storage controller: don't hold detached tenants in memory

[jcsp]

Problem

Typical deployments of neon have some tenants that stay in use continuously, and a background churning population of tenants that are created and then fall idle, and are configured to Detached state. Currently, this churn of short lived tenants results in an ever-increasing memory footprint.

Closes: #9712

Summary of changes

• At startup, filter to only load shards that don't have Detached policy
• In process_result, check if a tenant's shards are all Detached and observed=={}, and if so drop them from memory
• In tenant_location_conf and other tenant mutators, load the tenants' shards on-demand if they are not present

[github-actions]

7238 tests run: 6867 passed, 11 failed, 360 skipped (full report)

---

Failures on Postgres 17

• test_tenant_s3_restore: release-x86-64, release-x86-64, release-arm64, release-arm64, debug-x86-64

Failures on Postgres 16

• test_tenant_s3_restore: release-x86-64, release-arm64

Failures on Postgres 15

• test_tenant_s3_restore: release-x86-64, release-arm64

Failures on Postgres 14

• test_tenant_s3_restore: release-x86-64, release-arm64

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_tenant_s3_restore[release-pg14] or test_tenant_s3_restore[release-pg14] or test_tenant_s3_restore[release-pg15] or test_tenant_s3_restore[release-pg15] or test_tenant_s3_restore[release-pg16] or test_tenant_s3_restore[release-pg16] or test_tenant_s3_restore[release-pg17] or test_tenant_s3_restore[release-pg17] or test_tenant_s3_restore[release-pg17] or test_tenant_s3_restore[release-pg17] or test_tenant_s3_restore[debug-pg17]"

Flaky tests (3)

Postgres 17

• test_tenant_reattach[reset]: release-arm64
• test_tenants_normal_work: release-arm64

Postgres 14

• test_physical_replication_config_mismatch_too_many_known_xids: release-arm64

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
dd47681 at 2025-01-06T19:00:20.682Z :recycle:}

[jcsp]

Let's get this merged and give it a week in staging -- there's a lot of detach/attach churn there, so we should have an excellent chance of spotting any unforseen issues

[VladLazar]

Looks good!

---

I think attach hook handling is missing a maybe_load_tenant call around service.rs:1603 and that's causing the test failures.

[jcsp]

I think attach hook handling is missing a maybe_load_tenant call around service.rs:1603 and that's causing the test failures.

Yep, thanks for pointing that out.