pageserver: simplify block_deletions handling

[erikgrinaker]

Problem

Upload queue reordering from #10218 may allow us to simplify the handling of block_deletions (such as #9844). Quote from internal Slack thread:

I've been looking at the race with uploads and blocked deletions in the context of the new upload queue reordering, which prevents upload/delete races and head-of-line blocking. I think we can probably get rid of all the complexity around blocked_deletions and recently_deleted now by simply not allowing scheduling of deletions when block_deletions=true. They would be left in the queue, and any operations that have an ordering dependency on them (i.e. reference the same file) would wait behind them, while other operations would be allowed to jump the queue and proceed. To unblock deletions, we simply set block_deletions=false and the operations would be processed in order.

Summary of changes

Block deletions in old generations in UploadQueue::is_ready(), and rely on UploadOp::can_bypass() to allow writes in the current generation to bypass the queue.

[github-actions]

6450 tests run: 6181 passed, 10 failed, 259 skipped (full report)

---

Failures on Postgres 17

• test_issue_5878[same_generation]: release-x86-64, release-x86-64, release-arm64, release-arm64

Failures on Postgres 16

• test_issue_5878[same_generation]: release-x86-64, release-arm64

Failures on Postgres 15

• test_issue_5878[same_generation]: release-x86-64, release-arm64

Failures on Postgres 14

• test_issue_5878[same_generation]: release-x86-64, release-arm64

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_issue_5878[release-pg14-same_generation] or test_issue_5878[release-pg14-same_generation] or test_issue_5878[release-pg15-same_generation] or test_issue_5878[release-pg15-same_generation] or test_issue_5878[release-pg16-same_generation] or test_issue_5878[release-pg16-same_generation] or test_issue_5878[release-pg17-same_generation] or test_issue_5878[release-pg17-same_generation] or test_issue_5878[release-pg17-same_generation] or test_issue_5878[release-pg17-same_generation]"

Flaky tests (3)

Postgres 17

• test_physical_replication_config_mismatch_too_many_known_xids: release-arm64
• test_sharding_split_smoke: release-arm64
• test_timeline_archival_chaos: release-arm64

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
38148d3 at 2025-01-02T18:16:31.796Z :recycle:}