chore: add test for EG cert rotation

[guydc]

What type of PR is this?

What this PR does / why we need it:
In #4481, TLS config loading was changed to load the latest certs when a new client connection is established:

gateway/internal/crypto/cert_load.go

Line 54 in 2385672

return loadConfig()

This effectively made it possible to rotate certificate without restarting the xds runner.

This PR adds an e2e test that rotates Envoy Gateway/Envoy certificates and verifies that clients using rotated certs are able to connect to EG (which picks up the new cert). EG XDS is exposed with LB SVC for convenience.

rotation can take several seconds:

W1217 16:02:34.256378   10062 logging.go:55] [core] [Channel #4 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:18000", ServerName: "envoy-gateway.envoy-gateway-system.svc.cluster.local", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
    certificate_rotation.go:114: 2024-12-17T16:02:34.25645-06:00: failed to connect to Envoy Gateway with new tls credentials: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W1217 16:02:36.273081   10062 logging.go:55] [core] [Channel #6 SubChannel #7]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:18000", ServerName: "envoy-gateway.envoy-gateway-system.svc.cluster.local", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
    certificate_rotation.go:114: 2024-12-17T16:02:36.273226-06:00: failed to connect to Envoy Gateway with new tls credentials: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W1217 16:02:38.288275   10062 logging.go:55] [core] [Channel #8 SubChannel #9]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:18000", ServerName: "envoy-gateway.envoy-gateway-system.svc.cluster.local", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
    certificate_rotation.go:114: 2024-12-17T16:02:38.288346-06:00: failed to connect to Envoy Gateway with new tls credentials: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W1217 16:02:40.304879   10062 logging.go:55] [core] [Channel #10 SubChannel #11]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:18000", ServerName: "envoy-gateway.envoy-gateway-system.svc.cluster.local", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
    certificate_rotation.go:114: 2024-12-17T16:02:40.305028-06:00: failed to connect to Envoy Gateway with new tls credentials: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W1217 16:02:42.330178   10062 logging.go:55] [core] [Channel #12 SubChannel #13]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:18000", ServerName: "envoy-gateway.envoy-gateway-system.svc.cluster.local", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
    certificate_rotation.go:114: 2024-12-17T16:02:42.330257-06:00: failed to connect to Envoy Gateway with new tls credentials: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W1217 16:02:44.344827   10062 logging.go:55] [core] [Channel #14 SubChannel #15]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:18000", ServerName: "envoy-gateway.envoy-gateway-system.svc.cluster.local", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
    certificate_rotation.go:114: 2024-12-17T16:02:44.344926-06:00: failed to connect to Envoy Gateway with new tls credentials: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
W1217 16:02:46.366453   10062 logging.go:55] [core] [Channel #16 SubChannel #17]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:18000", ServerName: "envoy-gateway.envoy-gateway-system.svc.cluster.local", }. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
    certificate_rotation.go:114: 2024-12-17T16:02:46.366552-06:00: failed to connect to Envoy Gateway with new tls credentials: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
    certificate_rotation.go:121: 2024-12-17T16:02:48.395747-06:00: Connected to Envoy Gateway with new tls credentials

Which issue(s) this PR fixes:

Relates to #4891

Release Notes: No

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 11 lines in your changes missing coverage. Please review.

Project coverage is 66.75%. Comparing base (fcdbafa) to head (f948654).

Files with missing lines	Patch %	Lines
internal/xds/server/runner/runner.go	0.00%	11 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4944      +/-   ##
==========================================
- Coverage   66.77%   66.75%   -0.03%     
==========================================
  Files         209      209              
  Lines       32101    32108       +7     
==========================================
- Hits        21437    21433       -4     
- Misses       9381     9394      +13     
+ Partials     1283     1281       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.