Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Serialization & De-serialization stricter with versioning with Task SDK #45428

Open
kaxil opened this issue Jan 6, 2025 · 0 comments
Open
Assignees

Comments

@kaxil
Copy link
Member

kaxil commented Jan 6, 2025

Currently the serialization and de-serialization logic lives in airflow/serialization in the Core. With Airflow 3 and the separation of Task SDK, we will need to make serialization and its versioning much stricter.

We should bump the current DAG serialization version to 2.

The serialization code should live closer to language-specific Task SDK as it knows best how to serialize objects in a language to a JSON-formatted string.

The Core/scheduler will contain the de-serialization code -- and it does need to be language specific as it contains only the info needed by the scheduler.

The contract between those two is the schema.json file that contains the serialization. Both the client and server could support multiple versions at a time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

2 participants