Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resharding: cold storage #580

Merged
merged 3 commits into from
Dec 13, 2024
Merged

resharding: cold storage #580

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
958c1da
cold storage
staffik Dec 13, 2024
5ad68f8
update
staffik Dec 13, 2024
9452035
fix
staffik Dec 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions neps/nep-0568.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,6 @@ Splitting a shard's Flat State is performed in multiple steps:

### State Storage - State

// TODO Describe integration with cold storage once design is ready

Each shard’s Trie is stored in the `State` column of the database, with keys prefixed by `ShardUId`, followed by a node's hash.
This structure uniquely identifies each shard’s data. To avoid copying all entries under a new `ShardUId` during resharding,
a mapping strategy allows child shards to access ancestor shard data without directly creating new entries.
Expand All @@ -155,11 +153,18 @@ This allows child shards to access and update state data under the ancestor shar
Initially, `ShardUIdMapping` is empty, as existing shards map to themselves. During resharding, a mapping entry is added to `ShardUIdMapping`,
pointing each child shard’s `ShardUId` to the appropriate ancestor. Mappings persist as long as any descendant shard references the ancestor’s data.
Once a node stops tracking all children and descendants of a shard, the entry for that shard can be removed, allowing its data to be garbage collected.
For archival nodes, mappings are retained indefinitely to maintain access to the full historical state.

This mapping strategy enables efficient shard management during resharding events,
supporting smooth transitions without altering storage structures directly.

#### Integration with cold storage (archival nodes)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: new line after


Cold storage uses the same mapping strategy to manage shard state during resharding:
* When state data is migrated from hot to cold storage, it retains the parent shard’s `ShardUId` prefix, ensuring consistency with the mapping strategy.
* While copying data for the last block of the epoch where resharding occured, the `DBCol::StateShardUIdMapping` column is copied into cold storage. This ensures that mappings are updated alongside the shard state data.
* These mappings are permanent in cold storage, aligning with its role in preserving historical state.

This approach minimizes complexity while maintaining consistency across hot and cold storage.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also mention that the mapping in cold storage is permanent?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In line 164 there is "These mappings are permanent in cold storage".


### Stateless Validation

Expand Down
Loading