support backoff between retries in tcp_proxy

[IssaAbuKalbein]

Commit Message: support backoff between retries in tcp_proxy
Additional Description: This feature also fixes a bug in the retry mechanism in tcp proxy in the following situation:

1. We are tunneling TCP over HTTP.
2. A new stream is created on existing upstream connection (multiplexing) and waiting for response headers.
3. The upstream connection is closed.
4. During the close process, all streams created on this connection will be reset.
5. The tcp_proxy receives a callback on the stream reset.
6. They retry to connect.
7. The closed connection is picked as it is still in the connection pool (we are still in the process of the close).
8. The new stream that is created on the second attempt will be reset immediately.
9. The same process will happen (step 5 - step 8) until we reach the max_connect_attepts.
10. This means that we are doing only one real attempt in this situation.

Risk Level: medium
Testing: unit tests, integration tests
Docs Changes: added
Release Notes: added

[repokitteh-read-only]

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @adisuissa
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #37709 was opened by IssaAbuKalbein.

see: more, trace.

[IssaAbuKalbein]

/retest

[IssaAbuKalbein]

/retest

[IssaAbuKalbein]

/retest

[adisuissa]

Thanks!
Left a couple of API comments.
I suggest breaking this down to 2 PRs, one for TCP-Proxy, and the other for UDP-Proxy.

[adisuissa]

Instead of deprecating the old max_connect_attempts field, I suggest adding the backoff_options directly to the parent's message.
In an ideal world, where all the servers and clients under the control of a single entity, it would probably be possible to refactor the API so fields that are coupled will live in the same sub-type. However, unfortunately in our case, the costs outweigh the benefits, and unless there's a good reason for this refactor, I don't think it adds any new feature.

[IssaAbuKalbein]

I did this refactor to align tcp_proxy config with udp_proxy, do you still think it will be better to add backoff_options directly to the parent's message?

[adisuissa]

/wait

[IssaAbuKalbein]

Thanks! Left a couple of API comments. I suggest breaking this down to 2 PRs, one for TCP-Proxy, and the other for UDP-Proxy.

Created a separate PR for udp_proxy: #37912