Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace threading.Lock with asyncio.Lock when batching to avoid deadlocks #1270

Open
wants to merge 24 commits into
base: main
Choose a base branch
from

Conversation

tsmith023
Copy link
Contributor

@tsmith023 tsmith023 commented Aug 29, 2024

  • Changes all threading.Lock in async def fns to asyncio.Lock to avoid blocking the event loop
  • Changes all time.sleep() in async def fns to asyncio.sleep() to avoid blocking the event loop
  • Improves logging when errors occur during the batching logic
  • Uses BatchObject instead of _BatchObject in ErrorObject and BatchReference instead of _BatchReference in ErrorReference for easier usage of errors

Closes: #1332

@tsmith023 tsmith023 requested a review from dirkkul August 29, 2024 10:13
@codecov-commenter
Copy link

codecov-commenter commented Aug 29, 2024

Codecov Report

Attention: Patch coverage is 87.23404% with 6 lines in your changes missing coverage. Please review.

Project coverage is 90.08%. Comparing base (a373ee6) to head (252d86f).
Report is 218 commits behind head on main.

Files with missing lines Patch % Lines
weaviate/collections/batch/base.py 88.88% 3 Missing ⚠️
weaviate/collections/classes/batch.py 82.35% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1270      +/-   ##
==========================================
- Coverage   93.98%   90.08%   -3.91%     
==========================================
  Files         221      175      -46     
  Lines       21332    15081    -6251     
==========================================
- Hits        20048    13585    -6463     
- Misses       1284     1496     +212     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@weaviate-git-bot
Copy link

Great to see you again! Thanks for the contribution.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Slack channel?

@tsmith023 tsmith023 marked this pull request as draft September 16, 2024 16:45
@tsmith023
Copy link
Contributor Author

Closing in favour of #1371

@tsmith023 tsmith023 marked this pull request as ready for review November 1, 2024 11:33
Copy link

@orca-security-eu orca-security-eu bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@tsmith023 tsmith023 requested a review from dirkkul January 8, 2025 11:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix event loop blocking and deadlocks in batching loop
4 participants