Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reenqueue the job after shutdown hooks have run #153

Merged
merged 1 commit into from
Nov 17, 2021

Conversation

MatthewRBruce
Copy link
Contributor

@MatthewRBruce MatthewRBruce commented Nov 17, 2021

We discovered a race condition where a job can be re-enqueued a new run executed before the previous runs' shutdown hooks have been completed. This change ensures the shutdown hooks have run before the job is re-enqueued.

Fixes: https://github.com/Shopify/job-platform/issues/534

@MatthewRBruce MatthewRBruce force-pushed the mb-reenqueue-after-shutdown-hook branch from d857f2f to cbb0184 Compare November 17, 2021 17:19
@MatthewRBruce MatthewRBruce force-pushed the mb-reenqueue-after-shutdown-hook branch 3 times, most recently from 57d1930 to a38f62a Compare November 17, 2021 19:50
@MatthewRBruce MatthewRBruce marked this pull request as ready for review November 17, 2021 19:52
Copy link

@cardy31 cardy31 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job!

We discovered a race condition where a job can be re-enqueued a
new run executed before the previous runs' shutdown hooks have
been completed.  This change ensures the shutdown hooks have run
before the job is re-enqueued.

Add test

update CHANGELOG
@MatthewRBruce MatthewRBruce force-pushed the mb-reenqueue-after-shutdown-hook branch from 3952929 to 0fe8cfe Compare November 17, 2021 20:23
@MatthewRBruce MatthewRBruce merged commit f426057 into master Nov 17, 2021
@MatthewRBruce MatthewRBruce deleted the mb-reenqueue-after-shutdown-hook branch November 17, 2021 20:27
@shopify-shipit shopify-shipit bot temporarily deployed to rubygems November 17, 2021 20:31 Inactive
@etiennebarrie
Copy link
Member

We looked into something similar with #66. In addition to shutdown hooks, after_perform hooks are similarly affected. We had more compatibility issues with the other fix, but this is a good thing to have anyway. I just thought you'd be interested since it's related.

@shopify-shipit shopify-shipit bot temporarily deployed to rubygems January 18, 2022 12:25 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants