Troubleshooting Late runs

[sinacek]

prefect version

Version:             3.1.9
API version:         0.8.4
Python version:      3.11.10
Git commit:          e1fe7943
Built:               Fri, Dec 20, 2024 4:33 PM
OS/Arch:             darwin/arm64
Profile:             default
Server type:         server
Pydantic version:    2.10.4

helm server & worker version

2024.10.25010258

Queue details

Name: default
Type: Kubernetes
Concurrency Limit: None

prefect work-queue inspect default

WorkQueue(
    id='46472671-4d23-4e3a-8ca0-673a54791dde',
    created=DateTime(2024, 6, 19, 12, 30, 40, 739443, tzinfo=Timezone('UTC')),
    updated=DateTime(2024, 12, 25, 11, 8, 20, 602002, tzinfo=Timezone('UTC')),
    name='default',
    description="The work pool's default queue.",
    work_pool_name='k8s-worker',
    work_pool_id=UUID('1527dacf-df0e-4cd9-a54f-b2c9c2caaf34'),
    last_polled=DateTime(2024, 12, 25, 11, 8, 20, 600536, tzinfo=Timezone('UTC')),
    status=WorkQueueStatus.READY
)
WorkQueueStatusDetail(healthy=False, late_runs_count=68, last_polled=DateTime(2024, 12, 25, 11, 8, 20, 600536, tzinfo=Timezone('UTC')), health_check_policy=WorkQueueHealthPolicy())

I guess, that healthy=False means, thats containts too many late_runs.

I have self hosted prefect server and prefect worker (name: k8s-worker). I use just one queue (name: default). Time to time Prefect stucks and don't process any new jobs. For last two days have happened each night. When I restart worker and server, prefect slowly starts running jobs.

Do you have any idea, how to debug or get new insights?