Using an accurate benchmark tool

[waghanza]

We are currently using wrk. This tool is great, no doubt about that. However, the purpose of such a tool is to check how a framework could perform at its best (grab the maximum performance it could give).

This project aims to reflect how frameworks could perform in real world scenarios (for example, using constant throughput). For this reason, I'm considering switching this tool.

Candidates are :

• https://github.com/tsenart/vegeta
• https://github.com/tag1consulting/goose
• https://github.com/codesenberg/bombardier
• https://github.com/sharkdp/hyperfine

Goose seems accurate for me. Any suggestions @dominikzogg @jeremyevans @panesofglass @jodosha @timriley @ahopkins @appleboy @yusukebe @p8 @mcollina @fundon @ohler55 @sy-records @enghitalo @lukaszlenart ?

PS : The idea is to change this tool on Q2 2025

PSS : Sorry to ping some individuals, I do not know how to ping the whole team at once 😇

[waghanza]

Sorry : not hyperfine (more to benchmark cli tools)

[fundon]

https://github.com/hatoo/oha
It is also a good testing tool.

[waghanza]

Seems more like a PoC for me hatoo/oha#617

But to be honest, it's promising

[appleboy]

Yes, tsenart/vegeta is a versatile HTTP load testing tool that can handle real-world scenarios, including constant throughput, which is what you're looking for. Vegeta is designed to be both a command-line utility and a library, making it flexible for various use cases.

Key Features of Vegeta

1. Constant Throughput: Vegeta can maintain a constant request rate, which is useful for simulating real-world traffic patterns.
2. Ease of Use: It has a simple command-line interface and can be easily integrated into scripts and CI/CD pipelines.
3. Detailed Reporting: Vegeta provides detailed metrics and can generate reports in various formats, including JSON and HTML.
4. Flexibility: It supports multiple attack modes, including rate-limited and burst modes, and can handle complex scenarios with custom headers, bodies, and methods.

[waghanza]

I'm concerned about garbage collecting that can mess up results. Also concurrency model (async / await) seems more hard to learn, but more accurate in this context, I mean vs rust (goose)

[waghanza]

Also, how can vegeta handle constant throughput ? I mean, how can we make an equal sign between constant throughput and a real world scenario, there is no control about throughout

[waghanza]

Vegeta use 50rps per default, how can we define the rate to use facing the scenario we use ?

[waghanza]

I have tested vegeta (without rate limit), just to compare to wrk (since it has not rate limit) , and I can not explain result I've found in tsenart/vegeta#720

[mcollina]

I think wrk is ok. wrk2 is better ;).

[waghanza]

I do not understand how using constant throughput could be helpful.

I mean, with constant throughput approach, we must know the rate we should use. With wrk2, the rate used (by default) is 1000, but how can we know that 1000 is accurate for our scenario ? Or define any rate for it ?

[waghanza]

wrk2 is not yet compilable for arm

[yusukebe]

I prefer to bombardier, though I don't have any concrete reason. My favorite benchmark by @SaltyAom is using it:

https://github.com/SaltyAom/bun-http-framework-benchmark

[waghanza]

I there a way to enable any rate limit with it @yusukebe @SaltyAom ?

This feature is important, imho, to make more realistic results