prowler/api/docs/orphan-task-recovery.md

Orphan Celery task recovery

When a worker is terminated mid-task (a deploy, an OOM kill, a node eviction), the task it was running can be left non-terminal forever: the Scan stays EXECUTING, the TaskResult stays STARTED, and nothing re-runs it. This page describes the mechanisms that detect and recover allowlisted idempotent orphans so users never see a stuck scan and pending-task alerts do not fire.

How recovery works

1. Durable delivery. The broker is configured so a task message is acknowledged only after the task finishes (task_acks_late), one task is reserved at a time (worker_prefetch_multiplier = 1), and an abruptly-lost worker re-queues its task (task_reject_on_worker_lost). On SIGTERM the worker is given a soft-shutdown window (worker_soft_shutdown_timeout) to finish or re-queue in-flight work before it is force-killed.

2. Periodic watchdog. A Beat task, reconcile-orphan-tasks, runs every couple of minutes (a django_celery_beat periodic task created by migration). For each in-flight task result with an allowlisted idempotent task name, it pings the worker recorded on the task's TaskResult:

   • worker responds -> the task is still running, leave it alone;
   • worker is gone (and the scan started before a short grace window) -> it is a real orphan: the stale task is revoked and marked terminal (clearing the pending/started alert), and the scan is re-enqueued from scratch.

   The re-run is safe because only tasks with proven idempotency are allowlisted. Scan persistence, for example, clears the scan's prior findings and materialized summary/compliance rows before re-writing them. Jira sends are allowlisted too: each finding is reserved in a dispatch table before the external call, so a re-run skips already-ticketed findings (the worst case is one finding missed if a worker is hard-killed mid-send, never a duplicate issue). Other external side effects stay terminal: the S3 upload rebuilds from worker-local files that do not survive a crash, and report/Security Hub recovery is out of scope.

3. Recovery cap. Each automatic re-enqueue increments Scan.recovery_count. After --max-attempts recoveries (default 3) the scan is marked FAILED instead of re-enqueued, so a task that repeatedly kills its worker cannot loop forever.

A Postgres advisory lock ensures that, even with multiple API/worker replicas, only one reconciliation runs at a time; the others no-op.

On-demand command

The same logic is available as a management command, useful right after a deploy or for manual intervention:

python manage.py reconcile_orphan_tasks            # recover now
python manage.py reconcile_orphan_tasks --dry-run  # report orphans, change nothing
python manage.py reconcile_orphan_tasks --grace-minutes 5 --max-attempts 3

Configuration

All settings have safe defaults; override via environment variables.

Env var	Default	Purpose
DJANGO_CELERY_WORKER_PREFETCH_MULTIPLIER	1	Tasks reserved per worker process.
DJANGO_CELERY_WORKER_SOFT_SHUTDOWN_TIMEOUT	60	Seconds the worker drains/re-queues on SIGTERM before force-kill.
DJANGO_CELERY_TASK_TIME_LIMIT	21600 (6h)	Hard limit for most tasks; connection checks are capped at 120s.
DJANGO_CELERY_TASK_SOFT_TIME_LIMIT	hard - 600	Soft limit; raises SoftTimeLimitExceeded for cleanup.
DJANGO_CELERY_LONG_TASK_TIME_LIMIT	172800 (48h)	Hard limit for scans and provider/tenant deletions, which can legitimately run for more than a day.
DJANGO_CELERY_LONG_TASK_SOFT_TIME_LIMIT	long hard - 600	Soft limit for the long-running tasks above.

task_acks_late and task_reject_on_worker_lost are enabled in config/celery.py.

Deployment requirement

Two conditions must both hold for the soft shutdown to actually drain work:

1. The worker must receive SIGTERM. The container entrypoint execs the Celery process so it runs as PID 1; otherwise SIGTERM from docker stop/ECS hits the entrypoint shell, never reaches Celery, and the worker is hard-killed (SIGKILL) at the grace deadline without draining. Custom entrypoints must preserve the exec.
2. The orchestrator must give the worker enough time before force-killing it. Set the stop grace period to exceed DJANGO_CELERY_WORKER_SOFT_SHUTDOWN_TIMEOUT plus a margin:
   • docker-compose: stop_grace_period on the worker services (set to 120s).
   • AWS ECS: the worker container stopTimeout (configured in the deployment repository).

If either condition is missing, long tasks are still recovered by the watchdog, but they are cut mid-run on every deploy instead of draining.