docs: explain panic mode

[zhaohuabing]

Explain the "panic mode" where failed endpoints exceeds 50% as users were asking why requests were sent to unhealth endpoints.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 66.71%. Comparing base (f71fa99) to head (283a63e).
Report is 18 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4990      +/-   ##
==========================================
- Coverage   66.77%   66.71%   -0.06%     
==========================================
  Files         209      209              
  Lines       32052    32055       +3     
==========================================
- Hits        21404    21387      -17     
- Misses       9374     9387      +13     
- Partials     1274     1281       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[arkodg]

// Note: Once the overall health of the backendRef drops below 50% (e.g. a backendRef having 10 endpoints
// with more than 5 unhealthy endpoints), the health check is ignored for the remaining endpoints i.e. they are not
// removed from the load balancing pool. This prevents cascading failures and retry storms in the distributed system

[zhaohuabing]

the health check is ignored for the remaining endpoints i.e. they are not
removed from the load balancing pool.

This description may not be accurate, I've verified the following behavior in my test culster:

When the percentage of the unhealth endpoints reach 50%, the healtch check will not be ignored - the unhealth endpoints will still be marked as unhealth, however, the load balancer will distribute requests across all the endpoints, including both the unhealth and the health endpoints.

I guess the reason is to prevent a lot of requests from being sent to a small number of remaining health endpoint, which may overwhelming these health endpoinds and tear town the wole culster.