Exit codes for cli

[ckocyigit]

https://docs.cypress.io/guides/guides/command-line#Exit-code documents that cypress basically has three exit behaviours.

Cypress succeeds means exit code 0

Cypress fails (complete run failure) exit code 1

Cypress: Specs are failing means exit code is amount of specs failed

But what if just one spec failed.

You could not differentiate if the whole run just failed or just one spec based on the exit codes.

[ckocyigit]

[flotwig]

We're following Mocha's behavior here: #3559

There's only so much data that can be communicated via an exit code, so there's not much we can do here. For detailed information about what passed/failed/crashed, you should use either the Junit Reporter or the Module API.

[spiltcoffee]

@flotwig I assume you meant this issue? mochajs/mocha#3559

Anyway, a related exit code issue due to this practice is that if you have a multiple of 256 failed tests, then Cypress inappropriately exits with an exit code of 0 in contexts where the exit code is only 8 bits (i.e. unix).

Mocha faced this issue previously (mochajs/mocha#2438) and resolved it by clamping to 255 if 255+ tests failed (mochajs/mocha#2445).