How to prevent event drops

[faro-dr]

When analyzing an application with tokio-console I get a lot of event drops:

I found corresponding comments in the code, e.g. in tokio-console/console-api/src/generated/rs.tokio.console.tasks.rs:

    /// A count of how many task events (e.g. polls, spawns, etc) were not
    /// recorded because the application's event buffer was at capacity.
    ///
    /// If everything is working normally, this should be 0. If it is greater
    /// than 0, that may indicate that some data is missing from this update, and
    /// it may be necessary to increase the number of events buffered by the
    /// application to ensure that data loss is avoided.
    ///
    /// If the application's instrumentation ensures reliable delivery of events,
    /// this will always be 0.
    #[prost(uint64, tag = "4")]
    pub dropped_events: u64,

As I understand it, this means that event buffers in the instrumented application fill up, is this correct?

How can I prevent that? Is there a runtime parameter to tweak buffer size? Do I need to adapt console-subscriber?

Thank you!

[faro-dr]

I guess I should set environment varaible TOKIO_CONSOLE_BUFFER_CAPACITY

[faro-dr]

Increasing TOKIO_CONSOLE_BUFFER_CAPACITY and decreasing TOKIO_CONSOLE_PUBLISH_INTERVAL improved the situation a bit for me but I still get a lot of event drops after a couple of minutes. So if you have further ideas, I'd be happy to hear them :)

[Rustin170506]

Increasing TOKIO_CONSOLE_BUFFER_CAPACITY

I think this is the right approach to solve the problem. What was your TOKIO_CONSOLE_BUFFER_CAPACITY?

[faro-dr]

These settings improved the number of drops but didn't fix it entirely.

TOKIO_CONSOLE_BUFFER_CAPACITY=5242880
TOKIO_CONSOLE_PUBLISH_INTERVAL=200ms

After a couple of minutes, event drops started to occur again. I also tested an even larger buffer capacity namely TOKIO_CONSOLE_BUFFER_CAPACITY=10485760 but this created a huge latency and eventually ate up all the memory on my system.