JWT Authentication: Getting 404 'route_not_found' When Using WebSockets

[njegosrailic]

Description:

I'm using Envoy Gateway and adding an extra authentication layer based on JWT authentication with GitHub OIDC. It works perfectly with HTTP and gRPC traffic, but it doesn't seem to work when using WebSockets.

HTTP Route:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: test-app
  namespace: gateway
spec:
  hostnames:
  - test-app.[redacted]
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: gw-public
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: test-app
      namespace: test-app
      port: 80
      weight: 1
    matches:
    - headers:
      - name: x-claim-github-repository
        type: Exact
        value: myorg/myrepo <- [redacted]
      path:
        type: PathPrefix
        value: /
status:
  parents:
  - conditions:
    - lastTransitionTime: "2024-12-26T14:25:29Z"
      message: Route is accepted
      observedGeneration: 28
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-12-26T14:25:29Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 28
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs

Security policy:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: test-app-github-oidc
  namespace: gateway
spec:
  jwt:
    providers:
    - audiences:
      - gh-proxy
      claimToHeaders:
      - claim: repository
        header: x-claim-github-repository
      extractFrom:
        headers:
        - name: x-github-jwt
          valuePrefix: 'Bearer '
      issuer: https://token.actions.githubusercontent.com
      name: github-oidc
      recomputeRoute: true
      remoteJWKS:
        uri: https://token.actions.githubusercontent.com/.well-known/jwks
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: test-app

When testing this from GH Action:

  # Send WebSocket request with websocat
  websocat -v -v \
    -H="x-github-jwt: ***" \
    wss://test-app.[redacted]/hello-ws
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
[DEBUG websocat::lints] Substituting StdioClass with AsyncStdioClass at the left
[INFO  websocat::lints] Auto-inserting the line mode
[DEBUG websocat] Done third phase of interpreting options.
[DEBUG websocat] Done fourth phase of interpreting options.
[DEBUG websocat] Preparation done. Now actually starting.
[DEBUG websocat::sessionserve] Serving Line2Message(AsyncStdio) to Message2Line(WsClient("wss://test-app.[redacted]/hello-ws")) with Options { websocket_text_mode: true, websocket_protocol: None, websocket_reply_protocol: None, udp_oneshot_mode: false, udp_broadcast: false, udp_multicast_loop: false, udp_ttl: None, udp_join_multicast_addr: [], udp_join_multicast_iface_v4: [], udp_join_multicast_iface_v6: [], udp_reuseaddr: false, unidirectional: false, unidirectional_reverse: false, max_messages: None, max_messages_rev: None, exit_on_eof: false, oneshot: false, unlink_unix_socket: false, unix_socket_accept_from_fd: false, exec_args: [], ws_c_uri: "ws://0.0.0.0/", linemode_strip_newlines: false, linemode_strict: false, origin: None, custom_headers: [("x-github-jwt", [66, 101, 97, 114, 101, 114, 32, 101, 121, 74, 48, 101, 88, 65, 105, 79, 105, 74, 75, 86, 49, 81, 105, 76, 67, 74, 104, 98, 71, 99, 105, 79, 105, 74, 83, 85, 122, 73, 49, 78, 105, 73, 115, 73, 110, 103, 49, 100
[DEBUG websocat::stdio_peer] get_stdio_peer (async)
[DEBUG websocat::stdio_peer] Setting stdin to nonblocking mode
[DEBUG websocat::stdio_peer] Setting stdout to nonblocking mode
[DEBUG websocat::stdio_peer] Installing signal handler
[DEBUG websocat::sessionserve] Underlying connection established
[INFO  websocat::ws_client_peer] get_ws_client_peer
[DEBUG websocat::net_peer] Setting up a race between multiple TCP client sockets. Who connects the first?
[INFO  websocat::net_peer] Connected to TCP [redacted]:443
[DEBUG websocat::stdio_peer] restore_blocking_status
[DEBUG websocat::stdio_peer] Restoring blocking status for stdin
[DEBUG websocat::stdio_peer] Restoring blocking status for stdout
websocat: WebSocketError: WebSocketError: Received unexpected status code (404 Not Found)
[DEBUG websocat::stdio_peer] restore_blocking_status
[DEBUG websocat::stdio_peer] Restoring blocking status for stdin
[DEBUG websocat::stdio_peer] Restoring blocking status for stdout
websocat: error running
Error: Process completed with exit code 1.

I'm seeing the following in Envoy logs:

2024-12-26 14:35:33.571][66][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1826] removing hosts for TLS cluster token_actions_githubusercontent_com_443 removed 1
[2024-12-26 14:35:33.989][45][debug][conn_handler] [source/common/listener_manager/active_tcp_listener.cc:160] [Tags: "ConnectionId":"4583"] new connection from 20.43.229.22:65472
[2024-12-26 14:35:34.009][45][debug][http] [source/common/http/conn_manager_impl.cc:385] [Tags: "ConnectionId":"4583"] new stream
[2024-12-26 14:35:34.009][45][debug][http] [source/common/http/conn_manager_impl.cc:1135] [Tags: "ConnectionId":"4583","StreamId":"3806770929837629847"] request headers complete (end_stream=false):
':authority', 'test-app.[redacted]'
':path', '/hello-ws'
':method', 'GET'
'x-github-jwt', 'Bearer [redacted]'
'connection', 'Upgrade'
'upgrade', 'websocket'
'sec-websocket-version', '13'
'sec-websocket-key', 'PlEnaALt04CrzzILn7qtHQ=='

[2024-12-26 14:35:34.009][45][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"4583"] current connecting state: false
[2024-12-26 14:35:34.009][45][debug][router] [source/common/router/config_impl.cc:1974] route was resolved but final route list did not match incoming request
[2024-12-26 14:35:34.009][45][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:168] Called Filter : setDecoderFilterCallbacks
[2024-12-26 14:35:34.009][45][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:53] Called Filter : decodeHeaders
[2024-12-26 14:35:34.009][45][debug][jwt] [source/extensions/filters/http/jwt_authn/filter.cc:114] Jwt authentication completed with: OK
[2024-12-26 14:35:34.009][45][debug][router] [source/common/router/router.cc:457] [Tags: "ConnectionId":"4583","StreamId":"3806770929837629847"] no route match for URL '/hello-ws'
[2024-12-26 14:35:34.009][45][debug][http] [source/common/http/filter_manager.cc:1025] [Tags: "ConnectionId":"4583","StreamId":"3806770929837629847"] Preparing local reply with details route_not_found
[2024-12-26 14:35:34.009][45][debug][http] [source/common/http/filter_manager.cc:1067] [Tags: "ConnectionId":"4583","StreamId":"3806770929837629847"] Executing sending local reply.
[2024-12-26 14:35:34.009][45][debug][http] [source/common/http/conn_manager_impl.cc:1825] [Tags: "ConnectionId":"4583","StreamId":"3806770929837629847"] encoding headers via codec (end_stream=true):
':status', '404'
'date', 'Thu, 26 Dec 2024 14:35:33 GMT'
'connection', 'close'

What I don't understand is why it doesn't match the route, because during the handshake process, it uses the HTTP protocol, and we see in the logs that the JWT token is being sent. My question is: is JWT authentication supposed to work with WebSockets?

I'm assuming that there is no reason why JWT can't be validated. After the upgrade, it should receive a 101 status, the connection becomes a WebSocket, and everything should work because the connection will stay open. Anyway, even if this doesn't work, I'd like to understand why the JWT isn't being matched.

If I delete the security policy and header matching from the HTTPRoute, I can access the hello-ws endpoint without issues.

[zhaohuabing]

@njegosrailic Can you access the hello-ws endpoint if removing the x-claim-github-repository matching condition and the recomputeRoute: true in the SecurityPolicy ?

[njegosrailic]

@zhaohuabing I can access the endpoint if I remove the header matching in the HTTPRoute. It also works with recomputeRoute: true set in the SecurityPolicy.