Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Looks like we're resuming the target during snapshots again #526

Open
cbiffle opened this issue Dec 10, 2024 · 1 comment
Open

Looks like we're resuming the target during snapshots again #526

cbiffle opened this issue Dec 10, 2024 · 1 comment

Comments

@cbiffle
Copy link
Contributor

cbiffle commented Dec 10, 2024

I was on Mastodon trumpeting the fact that Humility mostly works with my keyboard when I noticed...a party pooper.

Image

It's subtle, but: see how scanner has its timer set for T+10?

scanner always sets its timer for T+1. It sleeps for one millisecond at a time.

So why would Humility show it with a timer set for T+10? Because Humility captures the system time early in the process (see where it's printed up top) and uses that value to compare to task timers. The T+10 means that it's resumed the target for 9 ms during the task snapshot process, generating an inconsistent snapshot.

I can reproduce inconsistent snapshots here pretty reliably -- for instance, getting output where no tasks are running despite being runnable (the idle task, at least, should show as running!).

This suggests to me that the issue where we're not reliably halting during snapshots has regressed. 😢

@cbiffle
Copy link
Contributor Author

cbiffle commented Dec 16, 2024

This is causing me significant pain trying to debug the sidecar startup VPD read issue right now. It would be nice to see which tasks are blocking on which during the 11.6 second delay, but the dumps show random things blocked in impossible positions because they're not all captured at the same time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant