Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Local Environment Variable Lookup for Variables & Connections in Task SDK #45435

Open
kaxil opened this issue Jan 6, 2025 · 0 comments

Comments

@kaxil
Copy link
Member

kaxil commented Jan 6, 2025

Currently, the Task Execution API server handles secrets backend logic, which is ideal for scenarios such as:

Deploying tasks in transient or ephemeral environments (e.g., GPU cloud services) where secrets need to be retrieved dynamically via the API.

However, there is a complementary use case where secrets or configurations are already provisioned locally on the worker. For simplicity and flexibility, we should enable the Task SDK to look up Variables and Connections from environment variables local to the worker.

This approach simplifies the logic by:

  • Supporting only environment variable lookups on the client-side (avoiding the complexity of implementing a full-fledged Secrets backend in both the client and server).
  • Keeping the Task Execution API server as the primary source for dynamic secrets retrieval, but allowing local fallback when needed.

This would delegate the responsibility to the worker provisioner for ensuring that environment variables are populated from external services (e.g., Vault, AWS Secrets Manager, etc.) before task execution for better security.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

2 participants