add BackendSecurityPolicy for traffic (authN/authZ) from gateway to provider #43

aabchoo · 2024-12-12T18:24:40Z

add backend security policy for traffic from gateway to backend provider
introduces api key, oidc, and static key auth
add reference to backendSecurityPolicy on llmbackend
add provider type on llmbackend

api/v1alpha1/api.go

Signed-off-by: Aaron Choo <[email protected]> Signed-off-by: Aaron Choo <[email protected]>

fix yaml files Signed-off-by: Choo <[email protected]>

This commit is a follow up on #20. Basically, this makes LLMRoute a pure "addition" to the existing standardized HTTPRoute. This makes it possible to configure something like ``` kind: LLMRoute metadata: name: llm-route spec: inputSchema: OpenAI httpRouteRef: name: my-llm-route --- kind: HTTPRoute metadata: name: my-llm-route spec: matches: - headers: key: x-envoy-ai-gateway-llm-model value: llama3-70b backendRefs: - kserve: weight: 20 - aws-bedrock: weight: 80 ``` where LLMRoute is purely referencing HTTPRoute and users can configure whatever routing condition in a standardized way via HTTPRoute while leveraging the LLM specific information, in this case x-envoy-ai-gateway-llm-model header. In the implementation, though it's not merged yet, we have to do the routing calculation in the extproc by actually analyzing the referenced HTTPRoute, and emulate the behavior in order to do the transformation. The reason is that the routing decision is made at the very end of filter chain in general, and by the time we invoke extproc, we don't have that info. Furthermore, `x-envoy-ai-gateway-llm-model` is not available before extproc. As a bonus of this, we no longer need TargetRef at LLMRoute level since that's within the HTTPRoute resources. This will really simplify the PoC implementation. --------- Signed-off-by: Takeshi Yoneda <[email protected]>

This adds `backendRef` to LLMBackendSpec which specifies the "backend" either Service or Backend resource of Envoy Gateway. The choice of not embedding is intentional - A backend can be a target of routing in HTTPRoute and can be either of these two types. Hence this is not suitable for embedding. In addition, in the implementation, we won't directly use or reference them, but just simply attach the necessary logic by the names, so basically no benefit by doing so. --------- Signed-off-by: Takeshi Yoneda <[email protected]>

Signed-off-by: Aaron Choo <[email protected]>

api/v1alpha1/api.go

Signed-off-by: Aaron Choo <[email protected]>

api/v1alpha1/api.go

Signed-off-by: Aaron Choo <[email protected]>

api/v1alpha1/api.go

Signed-off-by: Aaron Choo <[email protected]>

Co-authored-by: Dan Sun <[email protected]> Signed-off-by: Aaron Choo <[email protected]>

Signed-off-by: Aaron Choo <[email protected]>

api/v1alpha1/api.go

yuzisun · 2025-01-04T00:09:05Z

@mathetake @arkodg @zhaohuabing can you take another look?

api/v1alpha1/api.go

mathetake

LGTM per nits above!

api/v1alpha1/api.go

mathetake · 2025-01-04T01:55:55Z

api/v1alpha1/api.go

+	// +optional
+	BackendSecurityPolicyRef *gwapiv1.LocalObjectReference `json:"backendSecurityPolicyRef,omitempty"`


can you add a CEL validation to ensure that both apiKey and cloudProviderCredentials won't be defined at the same time

I've set a MaxProperties validation to 2

This will allow for the combos:
(BackendSecurityPolicyType AND (BackendSecurityPolicyAPIKey OR BackendSecurityPolicyCloudProviderCredentials))

mathetake · 2025-01-04T01:58:03Z

api/v1alpha1/api.go

+	// CloudProviderCredentials is a mechanism to access a backend(s). Cloud provider specific logic will be applied.
+	//
+	// +optional
+	CloudProviderCredentials *AuthenticationCloudProviderCredentials `json:"cloudProviderCredentials,omitempty"`


so i wonder what kind of logic will make cloud providers special compared to others (currently only apiKey though). how do you envision this additional indirect layer will be used? in other words, what kind of logic will be shared among cloud providers

ApiKey is a simple authentication mechanism which distributes the token to user and user is responsible to rotate the keys. Providers like Anthropic, Mistral, Cohere only support API keys.

Cloud authentication is more sophisticated and support issuing temporary tokens like AWS STS, Google STS to access cloud resources. You do not have to distribute the token and major clouds also support OIDC federation to allow managing your user identity in external system.
see https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html#:~:text=You%20can%20use%20the%20AWS,has%20permissions%20to%20do%20so.
https://cloud.google.com/docs/authentication#credentials

yeah i get that it's complex, but my question is why this another layer is needed and what kind of logic will be shared among cloud providers? what becomes difficult to achieve when you have *AWSCredentials directly here without another layer of AuthenticationCloudProviderCredentials. How does having AuthenticationCloudProviderCredentials helps implement various cloud providers vs having them in here

This is mainly a yaml UX, the additional layer for grouping is there for readability, if we list the individual types then it is not intuitive for user to see the classification. The cloud provider security implementations are different but they share the same high level concepts.

api/v1alpha1/api.go

Co-authored-by: Takeshi Yoneda <[email protected]> Signed-off-by: Aaron Choo <[email protected]>

Signed-off-by: Aaron Choo <[email protected]>

Co-authored-by: Takeshi Yoneda <[email protected]> Signed-off-by: Aaron Choo <[email protected]>

Signed-off-by: Aaron Choo <[email protected]>

api/v1alpha1/api.go

mathetake · 2025-01-07T23:57:14Z

api/v1alpha1/api.go

+// Only one type of BackendSecurityPolicy can be defined.
+// +kubebuilder:validation:MaxProperties=2
+type BackendSecurityPolicySpec struct {
+	// Type specifies the auth mechanism used to access the provider. Currently, only "APIKey", AND "AWS_IAM" are supported.


well, shouldn't the type be "CloudProviders" instead of AWS_IAM? The "type" field is for union type assertion and this feels a weird because AWS_IAM is not for the union type assertion but for the higher layer. @yuzisun @arkodg

This is relevant to the thread to #43 (comment) I am still not convinced by the "Yaml UX" stuff there. I am fine with that but at least to be consistent could you make this CloudProviders and have a type inside it for a specific cloud provider? (which i think is a worse UX vs the flat one)

aabchoo requested review from mathetake, missBerg and wengyao04 as code owners December 12, 2024 18:24

mathetake reviewed Dec 12, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

aabchoo force-pushed the aaron/authorization-api branch 2 times, most recently from 52e389b to 89ee946 Compare December 12, 2024 19:35

aabchoo and others added 7 commits December 12, 2024 14:50

Set up API for AUTH api

08be1f7

Signed-off-by: Aaron Choo <[email protected]> Signed-off-by: Aaron Choo <[email protected]>

Signed-off-by: Aaron Choo <[email protected]>

2ac7698

fix yaml files Signed-off-by: Choo <[email protected]>

updating test-cel

e200702

Signed-off-by: Aaron Choo <[email protected]>

add llm provider policy as part of crd

43f77fc

Signed-off-by: Aaron Choo <[email protected]>

Merge remote-tracking branch 'origin/main' into aaron/authorization-api

ea69193

Signed-off-by: Aaron Choo <[email protected]>

aabchoo force-pushed the aaron/authorization-api branch from 9fb8a19 to ea69193 Compare December 12, 2024 19:54

typo in test-cel

7e9315c

Signed-off-by: Aaron Choo <[email protected]>

Krishanx92 reviewed Dec 13, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

Krishanx92 reviewed Dec 13, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

update llmproviderpolicy -> llmsecuritypolicy

57e0fa1

Signed-off-by: Aaron Choo <[email protected]>

aabchoo force-pushed the aaron/authorization-api branch from a829fe8 to 57e0fa1 Compare December 13, 2024 15:38

ref to llmsecuritypolicy in the backend

bc0dadd

Signed-off-by: Aaron Choo <[email protected]>

mathetake reviewed Dec 14, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

api/v1alpha1/api.go Outdated Show resolved Hide resolved

api/v1alpha1/api.go Outdated Show resolved Hide resolved

aabchoo added 2 commits December 16, 2024 10:55

update documentation and tests

b071160

Signed-off-by: Aaron Choo <[email protected]>

update crd name

d2f72ed

Signed-off-by: Aaron Choo <[email protected]>

aabchoo force-pushed the aaron/authorization-api branch from 19c1337 to d2f72ed Compare December 16, 2024 15:56

yuzisun reviewed Dec 16, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

yuzisun reviewed Dec 16, 2024

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

aabchoo added 4 commits December 16, 2024 17:54

update to generalize aws auth

52ef4f9

Signed-off-by: Aaron Choo <[email protected]>

update llmproviderpolicy type

7b811b7

Signed-off-by: Aaron Choo <[email protected]>

auto gen update

7dbb202

Signed-off-by: Aaron Choo <[email protected]>

add oidc as part of security policy spec

c04ec5e

Signed-off-by: Aaron Choo <[email protected]>

aabchoo and others added 5 commits January 3, 2025 09:24

Update api/v1alpha1/api.go

ef3158c

Co-authored-by: Dan Sun <[email protected]> Signed-off-by: Aaron Choo <[email protected]>

update prefix for auth mechanism

5ff45c2

Signed-off-by: Aaron Choo <[email protected]>

update oidcFederation -> oidcExchangeToken

1f5e736

Signed-off-by: Aaron Choo <[email protected]>

update manifest

658c1dd

Signed-off-by: Aaron Choo <[email protected]>

update comment

2bcda7a

Signed-off-by: Aaron Choo <[email protected]>

aabchoo requested review from arkodg and removed request for zhaohuabing January 3, 2025 20:37

yuzisun reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

api/v1alpha1/api.go Show resolved Hide resolved

mathetake reviewed Jan 4, 2025

View reviewed changes

zhaohuabing reviewed Jan 6, 2025

View reviewed changes

api/v1alpha1/api.go Outdated Show resolved Hide resolved

aabchoo and others added 9 commits January 6, 2025 14:54

Update api/v1alpha1/api.go

f7ee160

Co-authored-by: Takeshi Yoneda <[email protected]> Signed-off-by: Aaron Choo <[email protected]>

Update api/v1alpha1/api.go

c4f7baf

Co-authored-by: Takeshi Yoneda <[email protected]> Signed-off-by: Aaron Choo <[email protected]>

make secretRef required

6b8f698

Signed-off-by: Aaron Choo <[email protected]>

Update api/v1alpha1/api.go

1d71e7a

Co-authored-by: Takeshi Yoneda <[email protected]> Signed-off-by: Aaron Choo <[email protected]>

Update api/v1alpha1/api.go

70a4b29

Co-authored-by: Takeshi Yoneda <[email protected]> Signed-off-by: Aaron Choo <[email protected]>

Add validation and comments to BackendSecurityPolicy

99e8ae3

Signed-off-by: Aaron Choo <[email protected]>

credentialFile to ref secret

70aa411

Signed-off-by: Aaron Choo <[email protected]>

Merge branch 'main' into aaron/authorization-api

27df777

Signed-off-by: Aaron Choo <[email protected]>

update description of oidc audience

dbc37e2

Signed-off-by: Aaron Choo <[email protected]>

mathetake approved these changes Jan 7, 2025

View reviewed changes

api/v1alpha1/api.go Show resolved Hide resolved

mathetake reviewed Jan 7, 2025

View reviewed changes

mathetake mentioned this pull request Jan 8, 2025

Adds initial controller implementation #71

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add BackendSecurityPolicy for traffic (authN/authZ) from gateway to provider #43

add BackendSecurityPolicy for traffic (authN/authZ) from gateway to provider #43

aabchoo commented Dec 12, 2024 •

edited

Loading

yuzisun commented Jan 4, 2025

mathetake left a comment

mathetake Jan 4, 2025

aabchoo Jan 6, 2025

mathetake Jan 4, 2025

yuzisun Jan 4, 2025 •

edited

Loading

mathetake Jan 6, 2025

yuzisun Jan 7, 2025

mathetake Jan 7, 2025

mathetake Jan 7, 2025

		// +optional
		BackendSecurityPolicyRef *gwapiv1.LocalObjectReference `json:"backendSecurityPolicyRef,omitempty"`

add BackendSecurityPolicy for traffic (authN/authZ) from gateway to provider #43

Are you sure you want to change the base?

add BackendSecurityPolicy for traffic (authN/authZ) from gateway to provider #43

Conversation

aabchoo commented Dec 12, 2024 • edited Loading

yuzisun commented Jan 4, 2025

mathetake left a comment

Choose a reason for hiding this comment

mathetake Jan 4, 2025

Choose a reason for hiding this comment

aabchoo Jan 6, 2025

Choose a reason for hiding this comment

mathetake Jan 4, 2025

Choose a reason for hiding this comment

yuzisun Jan 4, 2025 • edited Loading

Choose a reason for hiding this comment

mathetake Jan 6, 2025

Choose a reason for hiding this comment

yuzisun Jan 7, 2025

Choose a reason for hiding this comment

mathetake Jan 7, 2025

Choose a reason for hiding this comment

mathetake Jan 7, 2025

Choose a reason for hiding this comment

aabchoo commented Dec 12, 2024 •

edited

Loading

yuzisun Jan 4, 2025 •

edited

Loading