Could the task-list have a column for thread_id?

[dan-da]

Background:

We are using tokio multithreaded runtime and are presently chasing down a weird issue where our entire app hangs when two identical tasks begin operating concurrently. This has been happening on a VPS with only 2 cores, though it also happens when we bumped it to 4 cores. Other devices are unaffected so far.

So we fired up tokio-console and replicated the hang condition. We were hoping to see both thread_id and task_id, so that we could easily identify which tasks are sharing the same thread. Instead we see only the task IDs.

Proposal:

1. Add a column "Thread ID" to the task list that indicates the thread-id of the worker thread that is presently executing the task.
2. add a field "Thread ID" to the task details view.

It is unclear to me if this would be a simple addition, or is complicated to implement due to tracing spans, etc.

Afterword:

It turns out that tokio-console does not update at all after our app hangs. We guess because the tokio-console serving task is running on the same (blocked) thread as the stalled task. So a next test will be to run the app with 100 tokio worker threads to hopefully eliminate that situation.

[hds]

A quick suggestion: start the console subscriber on a dedicated tokio runtime. This will ensure that it keeps working even when you are blocking a worker.

Although this is what spawn does, so maybe that's another issue?

[dan-da]

thx. We are just using console_subscriber::init(). I'm not certain how to start it on a separate runtime, though maybe it's worth a try.

more info: we tried running with 100 tokio worker threads and we still get the same behavior. It seems as if the entire tokio runtime is hanging somehow, not just a particular thread. The hang occurs when a function verify() is called concurrently, ie at the second invokation... but not always. This verify() fn uses rayon's parallel iterator. There is some indication that the problem only occurs when an unrelated task is likewise using a rayon parallel iterator inside a spawn_blocking(). The code executing inside rayon is not async.

[hds]

Regarding the original request, tasks aren't bound to a single worker (hence not to a single thread). So any thread id world be more of a "last seen on thread" value.

[dan-da]

Regarding the original request, tasks aren't bound to a single worker (hence not to a single thread). So any thread id world be more of a "last seen on thread" value.

yeah I realize that. I'm not sure how often tasks hop between threads (I guess it will depend on workload) but I think that would be interesting to see. And I think it would be helpful to identify situations where two or more tasks are all stalled and they share a common thread.