ben.carrasco/prowler
1
0
Fork 0
mirror of https://github.com/prowler-cloud/prowler.git synced 2026-07-24 13:01:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into chore/remove-test-init-files

Browse Source
This commit is contained in:
Hugo P.Brito
2026-06-30 15:35:29 +01:00
parent 6a34ba051c 34e8e3ca61
commit d2fd272e0e
3451 changed files with 328059 additions and 103052 deletions
Download Patch File Download Diff File Expand all files Collapse all files
api/.env.example
+3
View File
@@ -24,6 +24,9 @@ DJANGO_THROTTLE_TOKEN_OBTAIN=50/minute
# Decide whether to allow Django manage database table partitions
DJANGO_MANAGE_DB_PARTITIONS=[True|False]
DJANGO_CELERY_DEADLOCK_ATTEMPTS=5
# Optional: bound Celery's prefork pool size. Unset → Celery uses os.cpu_count().
# Useful on Kubernetes nodes with many CPUs where unbounded prefork balloons memory.
# DJANGO_CELERY_WORKER_CONCURRENCY=4
DJANGO_BROKER_VISIBILITY_TIMEOUT=86400
DJANGO_SENTRY_DSN=
api/AGENTS.md
+12 -12
View File
@@ -10,7 +10,7 @@
> - [`jsonapi`](../skills/jsonapi/SKILL.md) - Strict JSON:API v1.1 spec compliance
> - [`pytest`](../skills/pytest/SKILL.md) - Generic pytest patterns
### Auto-invoke Skills
## Auto-invoke Skills
When performing these actions, ALWAYS invoke the corresponding skill FIRST:
@@ -81,7 +81,7 @@ When performing these actions, ALWAYS invoke the corresponding skill FIRST:
## DECISION TREES
### Serializer Selection
```
```text
Read → <Model>Serializer
Create → <Model>CreateSerializer
Update → <Model>UpdateSerializer
@@ -89,7 +89,7 @@ Nested read → <Model>IncludeSerializer
```
### Task vs View
```
```text
< 100ms → View
> 100ms or external API → Celery task
Needs retry → Celery task
@@ -105,7 +105,7 @@ Django 5.1.x | DRF 3.15.x | djangorestframework-jsonapi 7.x | Celery 5.4.x | Pos
## PROJECT STRUCTURE
```
```text
api/src/backend/
├── api/ # Main Django app
│ ├── v1/ # API version 1 (views, serializers, urls)
@@ -124,24 +124,24 @@ api/src/backend/
```bash
# Development
poetry run python src/backend/manage.py runserver
poetry run celery -A config.celery worker -l INFO
uv run python src/backend/manage.py runserver
uv run celery -A config.celery worker -l INFO
# Database
poetry run python src/backend/manage.py makemigrations
poetry run python src/backend/manage.py migrate
uv run python src/backend/manage.py makemigrations
uv run python src/backend/manage.py migrate
# Testing & Linting
poetry run pytest -x --tb=short
poetry run make lint
uv run pytest -x --tb=short
uv run make lint
```
---
## QA CHECKLIST
- [ ] `poetry run pytest` passes
- [ ] `poetry run make lint` passes
- [ ] `uv run pytest` passes
- [ ] `uv run make lint` passes
- [ ] Migrations created if models changed
- [ ] New endpoints have `@extend_schema` decorators
- [ ] RLS properly applied for tenant data
api/CHANGELOG.md
+312 -8
View File
@@ -2,22 +2,322 @@
All notable changes to the **Prowler API** are documented in this file.
## [1.24.0] (Prowler UNRELEASED)
### 🚀 Added
- Pin all unpinned dependencies to exact versions to prevent supply chain attacks and ensure reproducible builds [(#10469)](https://github.com/prowler-cloud/prowler/pull/10469)
- Filter RBAC role lookup by `tenant_id` to prevent cross-tenant privilege leak [(#10491)](https://github.com/prowler-cloud/prowler/pull/10491)
- `VALKEY_SCHEME`, `VALKEY_USERNAME`, and `VALKEY_PASSWORD` environment variables to configure Celery broker TLS/auth connection details for Valkey/ElastiCache [(#10420)](https://github.com/prowler-cloud/prowler/pull/10420)
- `Vercel` provider support [(#10190)](https://github.com/prowler-cloud/prowler/pull/10190)
## [1.33.0] (Prowler UNRELEASED)
### 🔄 Changed
- Attack Paths: AWS Neptune is now supported as a persistent sink database, selectable via `ATTACK_PATHS_SINK_DATABASE=neptune` (default `neo4j`), Cartography's (bumped to 0.138.1) per-scan ingest database stays on Neo4j [(#11524)](https://github.com/prowler-cloud/prowler/pull/11524)
---
## [1.32.2] (Prowler UNRELEASED)
### 🐞 Fixed
- `scan-perform` no longer reports an error when a provider is deleted during a running scan [(#11696)](https://github.com/prowler-cloud/prowler/pull/11696)
---
## [1.32.1] (Prowler v5.31.1)
### 🐞 Fixed
- API key auth no longer mutates `TenantAPIKey.objects` during admin DB lookups [(#11686)](https://github.com/prowler-cloud/prowler/pull/11686)
---
## [1.32.0] (Prowler v5.31.0)
### 🚀 Added
- Provider group filters for API endpoints that support cloud provider filtering, including exact and `__in` variants [(#11573)](https://github.com/prowler-cloud/prowler/pull/11573)
- Provider filters for `GET /api/v1/compliance-overviews`, `/metadata`, and `/requirements`, using latest completed scans per matching provider [(#11587)](https://github.com/prowler-cloud/prowler/pull/11587)
- Server-Sent Events (SSE) infrastructure for the API: a base viewset, a tenant-aware channel manager, and channel-name helpers backed by `django-eventstream` over Valkey Pub/Sub and served through the Gunicorn ASGI worker, so feature endpoints can stream events to clients over a single long-lived connection [(#11556)](https://github.com/prowler-cloud/prowler/pull/11556)
- `DJANGO_CELERY_WORKER_CONCURRENCY` to configure Celery workers concurrency. Unset for default behaviour [(#11075)](https://github.com/prowler-cloud/prowler/pull/11075)
### 🔄 Changed
- Gunicorn worker timeout raised from the 30s default to 120s, so long-running requests are no longer killed prematurely [(#11631)](https://github.com/prowler-cloud/prowler/pull/11631)
- Sentry now drops ASGI's `RequestAborted` errors from health-check probe disconnects on `/health/live` [(#11632)](https://github.com/prowler-cloud/prowler/pull/11632)
- Gunicorn keep-alive timeout now exceeds the load balancer idle timeout, stopping 502s from reused connections [(#11647)](https://github.com/prowler-cloud/prowler/pull/11647)
- API runs under the Uvicorn worker so keep-alive outlives the load balancer idle timeout, fixing Gunicorn's intermittent 502s [(#11663)](https://github.com/prowler-cloud/prowler/pull/11663)
- SAML logins no longer wipe a user's roles when the IdP does not send the `userType` attribute; existing roles are kept, and when `userType` names a role that does not exist it is now created with read-only access (visibility over all providers, no management permissions) instead of no permissions at all [(#11520)](https://github.com/prowler-cloud/prowler/pull/11520)
### 🐞 Fixed
- Database connections no longer leak under the ASGI worker, which previously exhausted the read replica's connection slots and caused 500s on read endpoints [(#11640)](https://github.com/prowler-cloud/prowler/pull/11640)
### 🔐 Security
- `aiohttp` to 3.14.0 and `idna` to 3.15, patching known CVEs [(#11596)](https://github.com/prowler-cloud/prowler/pull/11596)
- Container base image to `python:3.12.13-slim-bookworm` and `trivy` to 0.71.0, patching OS and Go module CVEs [(#11596)](https://github.com/prowler-cloud/prowler/pull/11596)
- `trivy` binary bumped to 0.71.0 patching embedded `golang.org/x/crypto`, `golang.org/x/net`, and Go `stdlib` CVEs [(#11592)](https://github.com/prowler-cloud/prowler/pull/11592)
---
## [1.31.3] (Prowler v5.30.3)
### 🔐 Security
- SAML logins now link to an existing account only when the asserted email domain matches the ACS endpoint and the user is already a member of that domain's tenant, fixing a cross-tenant account takeover [(GHSA-h8m9-jgf8-vwvp)](https://github.com/prowler-cloud/prowler/security/advisories/GHSA-h8m9-jgf8-vwvp)
---
## [1.31.2] (Prowler v5.30.2)
### 🔄 Changed
- `scan-compliance-overviews` task now streams the findings aggregation and the requirement-row writes so it runs faster and its peak memory no longer grows with the number of regions and frameworks [(#11591)](https://github.com/prowler-cloud/prowler/pull/11591)
---
## [1.31.1] (Prowler v5.30.1)
### 🐞 Fixed
- `compliance-overviews/attributes` now resolves the provider from the scan, so multi-provider universal frameworks (e.g. CSA CCM) return the check IDs of the scan's provider and Azure/GCP requirement details show their findings instead of appearing empty [(#11546)](https://github.com/prowler-cloud/prowler/pull/11546)
- Attack Paths: `drop_subgraph` now deletes relationships first and then nodes in batches, using less memory on Neo4j when clearing a dense provider graph [(#11557)](https://github.com/prowler-cloud/prowler/pull/11557)
- OCI scans now use API key credentials with the configured region instead of falling back to `/home/prowler/.oci/config` [(#11558)](https://github.com/prowler-cloud/prowler/pull/11558)
---
## [1.31.0] (Prowler v5.30.0)
### 🚀 Added
- Opt-in automatic recovery of allowlisted idempotent background tasks whose worker died during a deploy or crash: when enabled via `DJANGO_TASK_RECOVERY_ENABLED` (off by default), stuck summary and deletion tasks are detected and re-run instead of staying pending forever (scan and Jira tasks are excluded), with a `reconcile_orphan_tasks` management command for on-demand recovery [(#11416)](https://github.com/prowler-cloud/prowler/pull/11416)
- DORA compliance framework support [(#11131)](https://github.com/prowler-cloud/prowler/pull/11131)
- Label Postgres connections with `application_name="<component>:<alias>"` (component injected per process via `DJANGO_APP_COMPONENT`) so connections are attributable by component in `pg_stat_activity` [(#11494)](https://github.com/prowler-cloud/prowler/pull/11494)
- DISA Okta IDaaS STIG V1R2 compliance framework export support for the Okta provider [(#11428)](https://github.com/prowler-cloud/prowler/pull/11428)
### 🔄 Changed
- Allowlisted idempotent background tasks are no longer lost when a worker is stopped or crashes mid-task; tasks with external side effects are marked terminal instead of blindly re-running [(#11416)](https://github.com/prowler-cloud/prowler/pull/11416)
### 🐞 Fixed
- Workers now shut down gracefully on deploy or restart, finishing or re-queueing in-flight tasks instead of being force-killed and leaving them stuck [(#11416)](https://github.com/prowler-cloud/prowler/pull/11416)
- Resource `name` is now stored and refreshed on every scan, so resources no longer keep an empty name [(#11476)](https://github.com/prowler-cloud/prowler/pull/11476)
- Compliance catalog now warms in background during startup. `compliance-overviews/attributes` returns `503` while warming, so the first request after a deploy no longer trips the API timeout [(#11530)](https://github.com/prowler-cloud/prowler/pull/11530)
### 🔐 Security
- `dulwich` from 0.23.0 to 1.2.5 and `pyjwt` from 2.12.1 to 2.13.0, patching `GHSA-897w-fcg9-f6xj` (arbitrary file write) and `PYSEC-2026-179` (HMAC/JWK key confusion) [(#11499)](https://github.com/prowler-cloud/prowler/pull/11499)
---
## [1.30.3] (Prowler v5.29.3)
### 🐞 Fixed
- API startup no longer crashes when Neo4j is unreachable, as the Neo4j driver now connects lazily on first use rather than during app initialization [(#11491)](https://github.com/prowler-cloud/prowler/pull/11491)
---
## [1.30.1] (Prowler v5.29.1)
### 🐞 Fixed
- `GET /api/v1/findings` N+1 query loading `resources__tags` when listing findings [(#11420)](https://github.com/prowler-cloud/prowler/pull/11420)
- Clean up the scan tmp output directory when `scan-report` fails so partial files do not accumulate and fill the worker disk (`No space left on device`) [(#11421)](https://github.com/prowler-cloud/prowler/pull/11421)
---
## [1.30.0] (Prowler v5.29.0)
### 🔄 Changed
- Scan finding ingestion: bulk-resolve `Resource`/`ResourceTag` rows, replace per-mapping `SELECT FOR UPDATE` with deferred `ResourceTagMapping.bulk_create(ignore_conflicts=True)`, wrap each micro-batch in a single `rls_transaction`, and raise `SCAN_DB_BATCH_SIZE` to 1000 [(#11249)](https://github.com/prowler-cloud/prowler/pull/11249)
- Faster `GET /api/v1/finding-groups/latest` aggregation on tenants where one recent scan holds most findings [(#11380)](https://github.com/prowler-cloud/prowler/pull/11380)
---
## [1.29.1] (Prowler v5.28.1)
### 🐞 Fixed
- `finding-groups` slow response with finding-level filters such as `region`; check title and description are now read from the daily summaries, which drops sorting by `check_title` [(#11326)](https://github.com/prowler-cloud/prowler/pull/11326)
---
## [1.29.0] (Prowler v5.28.0)
### 🚀 Added
- `okta` provider support [(#11184)](https://github.com/prowler-cloud/prowler/pull/11184)
- `resource.metadata` attribute included in `/api/v1/findings?include=resources` [(#11187)](https://github.com/prowler-cloud/prowler/pull/11187)
---
## [1.28.0] (Prowler v5.27.0)
### 🚀 Added
- GIN index on `findings(categories, resource_services, resource_regions, resource_types)` to speed up `/api/v1/finding-groups` array filters [(#11001)](https://github.com/prowler-cloud/prowler/pull/11001)
- `GET /health/live` and `GET /health/ready` Kubernetes-style probe endpoints following the IETF Health Check Response Format (`application/health+json`). Readiness verifies PostgreSQL, Valkey and Neo4j connectivity and returns 503 with per-dependency detail when any is unreachable [(#11200)](https://github.com/prowler-cloud/prowler/pull/11200)
### 🔄 Changed
- Replace `poetry` with `uv` as package manager [(#10775)](https://github.com/prowler-cloud/prowler/pull/10775)
- Remove orphaned `gin_resources_search_idx` declaration from `Resource.Meta.indexes` (DB index dropped in `0072_drop_unused_indexes`) [(#11001)](https://github.com/prowler-cloud/prowler/pull/11001)
- PDF compliance reports cap detail tables at 100 failed findings per check (configurable via `DJANGO_PDF_MAX_FINDINGS_PER_CHECK`) to bound worker memory on large scans [(#11160)](https://github.com/prowler-cloud/prowler/pull/11160)
### 🐞 Fixed
- `perform_scan_task` and `perform_scheduled_scan_task` now short-circuit with a warning and `return None` when the target provider no longer exists, instead of letting `handle_provider_deletion` raise `ProviderDeletedException`. `perform_scheduled_scan_task` also removes any orphan `PeriodicTask` it finds so beat stops re-firing scans for deleted providers. Prevents queued messages for deleted providers from being recorded as `FAILURE` [(#11185)](https://github.com/prowler-cloud/prowler/pull/11185)
- Attack Paths: `BEDROCK-001` and `BEDROCK-002` now target roles trusting `bedrock-agentcore.amazonaws.com` instead of `bedrock.amazonaws.com`, eliminating false positives against regular Bedrock service roles (Agents, Knowledge Bases, model invocation) [(#11141)](https://github.com/prowler-cloud/prowler/pull/11141)
---
## [1.27.1] (Prowler v5.26.1)
### 🐞 Fixed
- `POST /api/v1/scans` was intermittently failing with `Scan matching query does not exist` in the `scan-perform` worker; the Celery task is now published via `transaction.on_commit` so the worker cannot read the Scan before the dispatch-wide transaction commits [(#11122)](https://github.com/prowler-cloud/prowler/pull/11122)
---
## [1.27.0] (Prowler v5.26.0)
### 🚀 Added
- `scan-reset-ephemeral-resources` post-scan task zeroes `failed_findings_count` for resources missing from the latest full-scope scan, keeping ephemeral resources from polluting the Resources page sort [(#10929)](https://github.com/prowler-cloud/prowler/pull/10929)
### 🔄 Changed
- ASD Essential Eight (AWS) compliance framework support [(#10982)](https://github.com/prowler-cloud/prowler/pull/10982)
### 🔐 Security
- `trivy` binary from 0.69.2 to 0.70.0 and `cryptography` from 46.0.6 to 46.0.7 (transitive via prowler SDK) in the API image for CVE-2026-33186 and CVE-2026-39892 [(#10978)](https://github.com/prowler-cloud/prowler/pull/10978)
---
## [1.26.1] (Prowler v5.25.1)
### 🐞 Fixed
- Attack Paths: AWS scans no longer fail when enabled regions cannot be retrieved, and scans stuck in `scheduled` state are now cleaned up after the stale threshold [(#10917)](https://github.com/prowler-cloud/prowler/pull/10917)
- Scan report and compliance downloads now redirect to a presigned S3 URL instead of streaming through the API worker, preventing gunicorn timeouts on large files [(#10927)](https://github.com/prowler-cloud/prowler/pull/10927)
---
## [1.26.0] (Prowler v5.25.0)
### 🚀 Added
- CIS Benchmark PDF report generation for scans, exposing the latest CIS version per provider via `GET /scans/{id}/cis/{name}/` [(#10650)](https://github.com/prowler-cloud/prowler/pull/10650)
- `/overviews/resource-groups` (resource inventory), `/overviews/categories` and `/overviews/attack-surfaces` now reflect newly-muted findings without waiting for the next scan. The post-mute `reaggregate-all-finding-group-summaries` task now also dispatches `aggregate_scan_resource_group_summaries_task`, `aggregate_scan_category_summaries_task` and `aggregate_attack_surface_task` per latest scan of every `(provider, day)` pair, rebuilding `ScanGroupSummary`, `ScanCategorySummary` and `AttackSurfaceOverview` alongside the tables already covered in #10827 [(#10843)](https://github.com/prowler-cloud/prowler/pull/10843)
- Install zizmor v1.24.1 in API Docker image for GitHub Actions workflow scanning [(#10607)](https://github.com/prowler-cloud/prowler/pull/10607)
### 🔄 Changed
- Allows tenant owners to expel users from their organizations [(#10787)](https://github.com/prowler-cloud/prowler/pull/10787)
- `aggregate_findings`, `aggregate_attack_surface`, `aggregate_scan_resource_group_summaries` and `aggregate_scan_category_summaries` now upsert via `bulk_create(update_conflicts=True, ...)` instead of the prior `ignore_conflicts=True` / plain INSERT / `already backfilled` short-circuit. Re-runs triggered by the post-mute reaggregation pipeline no longer trip the `unique_*_per_scan` constraints nor silently drop updates, and are race-safe under concurrent writers (e.g. scan completion overlapping with a fresh mute rule) [(#10843)](https://github.com/prowler-cloud/prowler/pull/10843)
- Rename the scan-category and scan-resource-group summary aggregators from `backfill_*` to `aggregate_*` [(#10843)](https://github.com/prowler-cloud/prowler/pull/10843)
### 🐞 Fixed
- `generate_outputs_task` crashing with `KeyError` for compliance frameworks listed by `get_compliance_frameworks` but not loadable by `Compliance.get_bulk` [(#10903)](https://github.com/prowler-cloud/prowler/pull/10903)
---
## [1.25.4] (Prowler v5.24.4)
### 🚀 Added
- `DJANGO_SENTRY_TRACES_SAMPLE_RATE` env var (default `0.02`) enables Sentry performance tracing for the API [(#10873)](https://github.com/prowler-cloud/prowler/pull/10873)
### 🔄 Changed
- Attack Paths: Neo4j driver `connection_acquisition_timeout` is now configurable via `NEO4J_CONN_ACQUISITION_TIMEOUT` (default lowered from 120 s to 15 s) [(#10873)](https://github.com/prowler-cloud/prowler/pull/10873)
### 🐞 Fixed
- `/tmp/prowler_api_output` saturation in compliance report workers: the final `rmtree` in `generate_compliance_reports` now only waits on frameworks actually generated for the provider (so unsupported frameworks no longer leave a placeholder `results` entry that blocks cleanup), output directories are created lazily per enabled framework, and both `generate_compliance_reports` and `generate_outputs_task` run an opportunistic stale cleanup at task start with a 48h age threshold, a per-host `fcntl` throttle, a 50-deletions-per-run cap, and guards that protect EXECUTING scans and scans whose `output_location` still points to a local path (metadata lookups routed through the admin DB so RLS does not hide those rows) [(#10874)](https://github.com/prowler-cloud/prowler/pull/10874)
---
## [1.25.3] (Prowler v5.24.3)
### 🚀 Added
- `/overviews/findings`, `/overviews/findings-severity` and `/overviews/services` now reflect newly-muted findings without waiting for the next scan. The post-mute `reaggregate-all-finding-group-summaries` task was extended to re-run the same per-scan pipeline that scan completion runs (`ScanSummary`, `DailySeveritySummary`, `FindingGroupDailySummary`) on the latest scan of every `(provider, day)` pair, keeping the pre-aggregated tables in sync with `Finding.muted` updates [(#10827)](https://github.com/prowler-cloud/prowler/pull/10827)
### 🐞 Fixed
- Finding groups aggregated `status` now treats muted findings as resolved: a group is `FAIL` only while at least one non-muted FAIL remains, otherwise it is `PASS` (including fully-muted groups). The `filter[status]` filter and the `sort=status` ordering share the same semantics, keeping `status` consistent with `fail_count` and the orthogonal `muted` flag [(#10825)](https://github.com/prowler-cloud/prowler/pull/10825)
- `aggregate_findings` is now idempotent: it deletes the scan's existing `ScanSummary` rows before `bulk_create`, so re-runs (such as the post-mute reaggregation pipeline) no longer violate the `unique_scan_summary` constraint and no longer abort the downstream `DailySeveritySummary` / `FindingGroupDailySummary` recomputation for the affected scan [(#10827)](https://github.com/prowler-cloud/prowler/pull/10827)
- Attack Paths: Findings on AWS were silently dropped during the Neo4j merge for resources whose Cartography node is keyed by a short identifier (e.g. EC2 instances) rather than the full ARN [(#10839)](https://github.com/prowler-cloud/prowler/pull/10839)
---
## [1.25.2] (Prowler v5.24.2)
### 🔄 Changed
- Finding groups `/resources` endpoints now materialize the filtered finding IDs into a Python list before filtering `ResourceFindingMapping`, so PostgreSQL switches from a Merge Semi Join that read hundreds of thousands of RFM index entries to a Nested Loop Index Scan over `finding_id`. The `has_mappings.exists()` pre-check is removed, and a request-scoped cache deduplicates the finding-id round-trip across the helpers that build different RFM querysets [(#10816)](https://github.com/prowler-cloud/prowler/pull/10816)
### 🐞 Fixed
- `/finding-groups/latest/<check_id>/resources` now selects the latest completed scan per provider by `-completed_at` (then `-inserted_at`) instead of `-inserted_at`, matching the `/finding-groups/latest` summary path and the daily-summary upsert so overlapping scans no longer produce diverging `delta`/`new_count` between the two endpoints [(#10802)](https://github.com/prowler-cloud/prowler/pull/10802)
## [1.25.1] (Prowler v5.24.1)
### 🔄 Changed
- Attack Paths: Restore `SYNC_BATCH_SIZE` and `FINDINGS_BATCH_SIZE` defaults to 1000, upgrade Cartography to 0.135.0, enable Celery queue priority for cleanup task, rewrite Finding insertion, remove AWS graph cleanup and add timing logs [(#10729)](https://github.com/prowler-cloud/prowler/pull/10729)
### 🐞 Fixed
- Finding group resources endpoints now include findings without associated resources (orphaned IaC findings) as simulated resource rows, and return one row per finding when multiple findings share a resource [(#10708)](https://github.com/prowler-cloud/prowler/pull/10708)
- Attack Paths: Missing `tenant_id` filter while getting related findings after scan completes [(#10722)](https://github.com/prowler-cloud/prowler/pull/10722)
- Finding group counters `pass_count`, `fail_count` and `manual_count` now exclude muted findings [(#10753)](https://github.com/prowler-cloud/prowler/pull/10753)
- Silent data loss in `ResourceFindingMapping` bulk insert that left findings orphaned when `INSERT ... ON CONFLICT DO NOTHING` dropped rows without raising; added explicit `unique_fields` [(#10724)](https://github.com/prowler-cloud/prowler/pull/10724)
- `DELETE /tenants/{tenant_pk}/memberships/{id}` now deletes the expelled user's account when the removed membership was their last one, and blacklists every outstanding refresh token for that user so their existing sessions can no longer mint new access tokens [(#10787)](https://github.com/prowler-cloud/prowler/pull/10787)
---
## [1.25.0] (Prowler v5.24.0)
### 🔄 Changed
- Bump Poetry to `2.3.4` in Dockerfile and pre-commit hooks. Regenerate `api/poetry.lock` [(#10681)](https://github.com/prowler-cloud/prowler/pull/10681)
- Attack Paths: Remove dead `cleanup_findings` no-op and its supporting `prowler_finding_lastupdated` index [(#10684)](https://github.com/prowler-cloud/prowler/pull/10684)
### 🐞 Fixed
- Worker-beat race condition on cold start: replaced `sleep 15` with API service healthcheck dependency (Docker Compose) and init containers (Helm), aligned Gunicorn default port to `8080` [(#10603)](https://github.com/prowler-cloud/prowler/pull/10603)
- API container startup crash on Linux due to root-owned bind-mount preventing JWT key generation [(#10646)](https://github.com/prowler-cloud/prowler/pull/10646)
### 🔐 Security
- `pytest` from 8.2.2 to 9.0.3 to fix CVE-2025-71176 [(#10678)](https://github.com/prowler-cloud/prowler/pull/10678)
---
## [1.24.0] (Prowler v5.23.0)
### 🚀 Added
- RBAC role lookup filtered by `tenant_id` to prevent cross-tenant privilege leak [(#10491)](https://github.com/prowler-cloud/prowler/pull/10491)
- `VALKEY_SCHEME`, `VALKEY_USERNAME`, and `VALKEY_PASSWORD` environment variables to configure Celery broker TLS/auth connection details for Valkey/ElastiCache [(#10420)](https://github.com/prowler-cloud/prowler/pull/10420)
- `Vercel` provider support [(#10190)](https://github.com/prowler-cloud/prowler/pull/10190)
- Finding groups list and latest endpoints support `sort=delta`, ordering by `new_count` then `changed_count` so groups with the most new findings rank highest [(#10606)](https://github.com/prowler-cloud/prowler/pull/10606)
- Finding group resources endpoints (`/finding-groups/{check_id}/resources` and `/finding-groups/latest/{check_id}/resources`) now expose `finding_id` per row, pointing to the most recent matching Finding for each resource. UUIDv7 ordering guarantees `Max(finding__id)` resolves to the latest snapshot [(#10630)](https://github.com/prowler-cloud/prowler/pull/10630)
- Handle CIS and CISA SCuBA compliance framework from google workspace [(#10629)](https://github.com/prowler-cloud/prowler/pull/10629)
- Sort support for all finding group counter fields: `pass_muted_count`, `fail_muted_count`, `manual_muted_count`, and all `new_*`/`changed_*` status-mute breakdown counters [(#10655)](https://github.com/prowler-cloud/prowler/pull/10655)
### 🔄 Changed
- Finding groups list/latest/resources now expose `status` ∈ `{FAIL, PASS, MANUAL}` and `muted: bool` as orthogonal fields. The aggregated `status` reflects the underlying check outcome regardless of mute state, and `muted=true` signals that every finding in the group/resource is muted. New `manual_count` is exposed alongside `pass_count`/`fail_count`, plus `pass_muted_count`/`fail_muted_count`/`manual_muted_count` siblings so clients can isolate the muted half of each status. The `new_*`/`changed_*` deltas are now broken down by status and mute state via 12 new counters (`new_fail_count`, `new_fail_muted_count`, `new_pass_count`, `new_pass_muted_count`, `new_manual_count`, `new_manual_muted_count` and the matching `changed_*` set). New `filter[muted]=true|false` and `sort=status` (FAIL > PASS > MANUAL) / `sort=muted` are supported. `filter[status]=MUTED` is no longer accepted [(#10630)](https://github.com/prowler-cloud/prowler/pull/10630)
- Attack Paths: Periodic cleanup of stale scans with dead-worker detection via Celery inspect, marking orphaned `EXECUTING` scans as `FAILED` and recovering `graph_data_ready` [(#10387)](https://github.com/prowler-cloud/prowler/pull/10387)
- Attack Paths: Replace `_provider_id` property with `_Provider_{uuid}` label for provider isolation, add regex-based label injection for custom queries [(#10402)](https://github.com/prowler-cloud/prowler/pull/10402)
### 🐞 Fixed
- `reaggregate_all_finding_group_summaries_task` now refreshes finding group daily summaries for every `(provider, day)` combination instead of only the latest scan per provider, matching the unbounded scope of `mute_historical_findings_task`. Mute rule operations no longer leave older daily summaries drifting from the underlying muted findings [(#10630)](https://github.com/prowler-cloud/prowler/pull/10630)
- Finding groups list/latest now apply computed status/severity filters and finding-level prefilters (delta, region, service, category, resource group, scan, resource type), plus `check_title` support for sort/filter consistency [(#10428)](https://github.com/prowler-cloud/prowler/pull/10428)
- Populate compliance data inside `check_metadata` for findings, which was always returned as `null` [(#10449)](https://github.com/prowler-cloud/prowler/pull/10449)
- 403 error for admin users listing tenants due to roles query not using the admin database connection [(#10460)](https://github.com/prowler-cloud/prowler/pull/10460)
@@ -28,10 +328,14 @@ All notable changes to the **Prowler API** are documented in this file.
- Membership `post_delete` signal using raw FK ids to avoid `DoesNotExist` during cascade deletions [(#10497)](https://github.com/prowler-cloud/prowler/pull/10497)
- Finding group resources endpoints returning false 404 when filters match no results, and `sort` parameter being ignored [(#10510)](https://github.com/prowler-cloud/prowler/pull/10510)
- Jira integration failing with `JiraInvalidIssueTypeError` on non-English Jira instances due to hardcoded `"Task"` issue type; now dynamically fetches available issue types per project [(#10534)](https://github.com/prowler-cloud/prowler/pull/10534)
- Finding group `first_seen_at` now reflects when a new finding appeared in the scan instead of the oldest carry-forward date across all unchanged findings [(#10595)](https://github.com/prowler-cloud/prowler/pull/10595)
- Attack Paths: Remove `clear_cache` call from read-only query endpoints; cache clearing belongs to the scan/ingestion flow, not API queries [(#10586)](https://github.com/prowler-cloud/prowler/pull/10586)
### 🔐 Security
- Pin all unpinned dependencies to exact versions to prevent supply chain attacks and ensure reproducible builds [(#10469)](https://github.com/prowler-cloud/prowler/pull/10469)
- `authlib` bumped from 1.6.6 to 1.6.9 to fix CVE-2026-28802 (JWT `alg: none` validation bypass) [(#10579)](https://github.com/prowler-cloud/prowler/pull/10579)
- `aiohttp` bumped from 3.13.3 to 3.13.5 to fix CVE-2026-34520 (the C parser accepted null bytes and control characters in response headers) [(#10538)](https://github.com/prowler-cloud/prowler/pull/10538)
---
api/Dockerfile
+48 -10
View File
@@ -1,16 +1,20 @@
FROM python:3.12.10-slim-bookworm@sha256:fd95fa221297a88e1cf49c55ec1828edd7c5a428187e67b5d1805692d11588db AS build
FROM python:3.12.13-slim-bookworm@sha256:76d4b7b6305788c6b4c6a19d6a22a3921bf802e9af4d5e1e5bd771208dba74bf AS build
LABEL maintainer="https://github.com/prowler-cloud/api"
ARG POWERSHELL_VERSION=7.5.0
ENV POWERSHELL_VERSION=${POWERSHELL_VERSION}
ARG TRIVY_VERSION=0.69.2
ARG TRIVY_VERSION=0.71.2
ENV TRIVY_VERSION=${TRIVY_VERSION}
ARG ZIZMOR_VERSION=1.24.1
ENV ZIZMOR_VERSION=${ZIZMOR_VERSION}
# hadolint ignore=DL3008
RUN apt-get update && apt-get install -y --no-install-recommends \
wget \
git \
libicu72 \
gcc \
g++ \
@@ -22,6 +26,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libtool \
libxslt1-dev \
python3-dev \
git \
&& rm -rf /var/lib/apt/lists/*
# Install PowerShell
@@ -57,6 +62,22 @@ RUN ARCH=$(uname -m) && \
mkdir -p /tmp/.cache/trivy && \
chmod 777 /tmp/.cache/trivy
# Install zizmor for GitHub Actions workflow scanning
RUN ARCH=$(uname -m) && \
if [ "$ARCH" = "x86_64" ]; then \
ZIZMOR_ARCH="x86_64-unknown-linux-gnu" ; \
elif [ "$ARCH" = "aarch64" ]; then \
ZIZMOR_ARCH="aarch64-unknown-linux-gnu" ; \
else \
echo "Unsupported architecture for zizmor: $ARCH" && exit 1 ; \
fi && \
wget --progress=dot:giga "https://github.com/zizmorcore/zizmor/releases/download/v${ZIZMOR_VERSION}/zizmor-${ZIZMOR_ARCH}.tar.gz" -O /tmp/zizmor.tar.gz && \
mkdir -p /tmp/zizmor-extract && \
tar zxf /tmp/zizmor.tar.gz -C /tmp/zizmor-extract && \
mv /tmp/zizmor-extract/zizmor /usr/local/bin/zizmor && \
chmod +x /usr/local/bin/zizmor && \
rm -rf /tmp/zizmor.tar.gz /tmp/zizmor-extract
# Add prowler user
RUN addgroup --gid 1000 prowler && \
adduser --uid 1000 --gid 1000 --disabled-password --gecos "" prowler
@@ -68,21 +89,38 @@ WORKDIR /home/prowler
# Ensure output directory exists
RUN mkdir -p /tmp/prowler_api_output
COPY pyproject.toml ./
COPY --chown=prowler:prowler pyproject.toml uv.lock ./
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir poetry
pip install --no-cache-dir uv==0.11.14
ENV PATH="/home/prowler/.local/bin:$PATH"
# Add `--no-root` to avoid installing the current project as a package
RUN poetry install --no-root && \
rm -rf ~/.cache/pip
# Add `--no-install-project` to avoid installing the current project as a package
RUN uv sync --locked --no-install-project && \
rm -rf ~/.cache/uv
RUN poetry run python "$(poetry env info --path)/src/prowler/prowler/providers/m365/lib/powershell/m365_powershell.py"
RUN .venv/bin/python .venv/lib/python3.12/site-packages/prowler/providers/m365/lib/powershell/m365_powershell.py
COPY src/backend/ ./backend/
COPY docker-entrypoint.sh ./docker-entrypoint.sh
USER root
# Remove build-only packages from the final image after Python dependencies are installed.
RUN apt-get purge -y --auto-remove \
gcc \
g++ \
make \
libxml2-dev \
libxmlsec1-dev \
pkg-config \
libtool \
libxslt1-dev \
python3-dev \
&& rm -rf /var/lib/apt/lists/*
USER prowler
COPY --chown=prowler:prowler src/backend/ ./backend/
COPY --chown=prowler:prowler docker-entrypoint.sh ./docker-entrypoint.sh
WORKDIR /home/prowler/backend
api/README.md
+77 -48
View File
@@ -2,7 +2,7 @@
This repository contains the JSON API and Task Runner components for Prowler, which facilitate a complete backend that interacts with the Prowler SDK and is used by the Prowler UI.
# Components
## Components
The Prowler API is composed of the following components:
- The JSON API, which is an API built with Django Rest Framework.
@@ -10,13 +10,13 @@ The Prowler API is composed of the following components:
- The PostgreSQL database, which is used to store the data.
- The Valkey database, which is an in-memory database which is used as a message broker for the Celery workers.
## Note about Valkey
### Note about Valkey
[Valkey](https://valkey.io/) is an open source (BSD) high performance key/value datastore.
Valkey exposes a Redis 7.2 compliant API. Any service that exposes the Redis API can be used with Prowler API.
# Modify environment variables
## Modify environment variables
Under the root path of the project, you can find a file called `.env`. This file shows all the environment variables that the project uses. You should review it and set the values for the variables you want to change.
@@ -24,23 +24,22 @@ If you don’t set `DJANGO_TOKEN_SIGNING_KEY` or `DJANGO_TOKEN_VERIFYING_KEY`, t
**Important note**: Every Prowler version (or repository branches and tags) could have different variables set in its `.env` file. Please use the `.env` file that corresponds with each version.
## Local deployment
Keep in mind if you export the `.env` file to use it with local deployment that you will have to do it within the context of the Poetry interpreter, not before. Otherwise, variables will not be loaded properly.
### Local deployment
Keep in mind if you export the `.env` file to use it with local deployment that you will have to do it within the context of the virtual environment, not before. Otherwise, variables will not be loaded properly.
To do this, you can run:
```console
poetry shell
set -a
source .env
```
# 🚀 Production deployment
## Docker deployment
## 🚀 Production deployment
### Docker deployment
This method requires `docker` and `docker compose`.
### Clone the repository
#### Clone the repository
```console
# HTTPS
@@ -51,13 +50,13 @@ git clone git@github.com:prowler-cloud/api.git
```
### Build the base image
#### Build the base image
```console
docker compose --profile prod build
```
### Run the production service
#### Run the production service
This command will start the Django production server and the Celery worker and also the Valkey and PostgreSQL databases.
@@ -69,7 +68,7 @@ You can access the server in `http://localhost:8080`.
> **NOTE:** notice how the port is different. When developing using docker, the port will be `8080` to prevent conflicts.
### View the Production Server Logs
#### View the Production Server Logs
To view the logs for any component (e.g., Django, Celery worker), you can use the following command with a wildcard. This command will follow logs for any container that matches the specified pattern:
@@ -78,7 +77,7 @@ docker logs -f $(docker ps --format "{{.Names}}" | grep 'api-')
## Local deployment
To use this method, you'll need to set up a Python virtual environment (version ">=3.11,<3.13") and keep dependencies updated. Additionally, ensure that `poetry` and `docker compose` are installed.
To use this method, you'll need to set up a Python virtual environment (version ">=3.11,<3.13") and keep dependencies updated. Additionally, ensure that `uv` and `docker compose` are installed.
### Clone the repository
@@ -90,11 +89,10 @@ git clone https://github.com/prowler-cloud/api.git
git clone git@github.com:prowler-cloud/api.git
```
### Install all dependencies with Poetry
### Install all dependencies with uv
```console
poetry install
poetry shell
uv sync
```
## Start the PostgreSQL Database and Valkey
@@ -135,13 +133,13 @@ gunicorn -c config/guniconf.py config.wsgi:application
> By default, the Gunicorn server will try to use as many workers as your machine can handle. You can manually change that in the `src/backend/config/guniconf.py` file.
# 🧪 Development guide
## 🧪 Development guide
## Local deployment
### Local deployment
To use this method, you'll need to set up a Python virtual environment (version ">=3.11,<3.13") and keep dependencies updated. Additionally, ensure that `poetry` and `docker compose` are installed.
To use this method, you'll need to set up a Python virtual environment (version ">=3.11,<3.13") and keep dependencies updated. Additionally, ensure that `uv` and `docker compose` are installed.
### Clone the repository
#### Clone the repository
```console
# HTTPS
@@ -152,7 +150,7 @@ git clone git@github.com:prowler-cloud/api.git
```
### Start the PostgreSQL Database and Valkey
#### Start the PostgreSQL Database and Valkey
The PostgreSQL database (version 16.3) and Valkey (version 7) are required for the development environment. To make development easier, we have provided a `docker-compose` file that will start these components for you.
@@ -163,16 +161,15 @@ The PostgreSQL database (version 16.3) and Valkey (version 7) are required for t
docker compose up postgres valkey -d
```
### Install the Python dependencies
#### Install the Python dependencies
> You must have Poetry installed
> You must have uv installed
```console
poetry install
poetry shell
uv sync
```
### Apply migrations
#### Apply migrations
For migrations, you need to force the `admin` database router. Assuming you have the correct environment variables and Python virtual environment, run:
@@ -181,7 +178,7 @@ cd src/backend
python manage.py migrate --database admin
```
### Run the Django development server
#### Run the Django development server
```console
cd src/backend
@@ -191,7 +188,7 @@ python manage.py runserver
You can access the server in `http://localhost:8000`.
All changes in the code will be automatically reloaded in the server.
### Run the Celery worker
#### Run the Celery worker
```console
python -m celery -A config.celery worker -l info -E
@@ -199,11 +196,47 @@ python -m celery -A config.celery worker -l info -E
The Celery worker does not detect and reload changes in the code, so you need to restart it manually when you make changes.
## Docker deployment
### Makefile-Assisted Local Deployment
This method is an additional local development workflow. It does not replace the manual local deployment or the Docker deployment described in this guide.
PostgreSQL, Valkey, and Neo4j run with Docker Compose, while Django and the Celery worker run natively through `uv`. Additionally, this workflow creates a `tmux` session with panes for the API, worker, and PostgreSQL logs.
Before using this method, ensure `docker compose`, `tmux`, and `uv` are installed.
This workflow is designed for macOS and should also work on Linux when Docker, `tmux`, and `uv` are available. Windows requires script changes before it can be supported.
From the repository root, run:
```console
make dev
```
The API will be available at:
```console
http://localhost:8080/api/v1
```
Use these commands to manage the local stack:
```console
make dev-setup # Bootstrap dependencies, migrations, and fixtures
make dev-attach # Attach to the tmux session
make dev-launch # Start the stack on fixed ports and attach
make dev-stop # Stop the tmux session and containers
make dev-clean # Remove stopped development containers
make dev-wipe # Stop everything and delete local development data
make dev-status # Show development container status
```
This workflow does not start the UI. Start it separately from the `ui/` directory when needed.
### Docker deployment
This method requires `docker` and `docker compose`.
### Clone the repository
#### Clone the repository
```console
# HTTPS
@@ -214,13 +247,13 @@ git clone git@github.com:prowler-cloud/api.git
```
### Build the base image
#### Build the base image
```console
docker compose --profile dev build
```
### Run the development service
#### Run the development service
This command will start the Django development server and the Celery worker and also the Valkey and PostgreSQL databases.
@@ -233,7 +266,7 @@ All changes in the code will be automatically reloaded in the server.
> **NOTE:** notice how the port is different. When developing using docker, the port will be `8080` to prevent conflicts.
### View the development server logs
#### View the development server logs
To view the logs for any component (e.g., Django, Celery worker), you can use the following command with a wildcard. This command will follow logs for any container that matches the specified pattern:
@@ -241,41 +274,38 @@ To view the logs for any component (e.g., Django, Celery worker), you can use th
docker logs -f $(docker ps --format "{{.Names}}" | grep 'api-')
```
## Applying migrations
### Applying migrations
For migrations, you need to force the `admin` database router. Assuming you have the correct environment variables and Python virtual environment, run:
```console
poetry shell
cd src/backend
python manage.py migrate --database admin
uv run python manage.py migrate --database admin
```
## Apply fixtures
### Apply fixtures
Fixtures are used to populate the database with initial development data.
```console
poetry shell
cd src/backend
python manage.py loaddata api/fixtures/0_dev_users.json --database admin
uv run python manage.py loaddata api/fixtures/0_dev_users.json --database admin
```
> The default credentials are `dev@prowler.com:Thisisapassword123@` or `dev2@prowler.com:Thisisapassword123@`
## Run tests
### Run tests
Note that the tests will fail if you use the same `.env` file as the development environment.
For best results, run in a new shell with no environment variables set.
```console
poetry shell
cd src/backend
pytest
uv run pytest
```
# Custom commands
## Custom commands
Django provides a way to create custom commands that can be run from the command line.
@@ -284,11 +314,10 @@ Django provides a way to create custom commands that can be run from the command
To run a custom command, you need to be in the `prowler/api/src/backend` directory and run:
```console
poetry shell
python manage.py <command_name>
uv run python manage.py <command_name>
```
## Generate dummy data
### Generate dummy data
```console
python manage.py findings --tenant
@@ -305,10 +334,10 @@ This command creates, for a given tenant, a provider, scan and a set of findings
>
> The last step is required to access the findings details, since the UI needs that to print all the information.
### Example
#### Example
```console
~/backend $ poetry run python manage.py findings --tenant
~/backend $ uv run python manage.py findings --tenant
fffb1893-3fc7-4623-a5d9-fae47da1c528 --findings 25000 --re
sources 1000 --batch 5000 --alias test-script
api/docker-entrypoint.sh
+24 -10
View File
@@ -5,9 +5,9 @@ apply_migrations() {
echo "Applying database migrations..."
# Fix Inconsistent migration history after adding sites app
poetry run python manage.py check_and_fix_socialaccount_sites_migration --database admin
uv run python manage.py check_and_fix_socialaccount_sites_migration --database admin
poetry run python manage.py migrate --database admin
uv run python manage.py migrate --database admin
}
apply_fixtures() {
@@ -15,19 +15,25 @@ apply_fixtures() {
for fixture in api/fixtures/dev/*.json; do
if [ -f "$fixture" ]; then
echo "Loading $fixture"
poetry run python manage.py loaddata "$fixture" --database admin
uv run python manage.py loaddata "$fixture" --database admin
fi
done
}
start_dev_server() {
echo "Starting the development server..."
poetry run python manage.py runserver 0.0.0.0:"${DJANGO_PORT:-8080}"
echo "Starting the development server (Gunicorn ASGI, debug + reload)..."
# Same server/worker as prod (config.asgi via the native `asgi` worker), so
# SSE streams run on the event loop exactly as they do in production. DEBUG is
# on so guniconf's `reload = DEBUG` hot-reloads edited code (and flips
# `preload_app` off so reload actually takes).
export DJANGO_DEBUG="${DJANGO_DEBUG:-True}"
export DJANGO_BIND_ADDRESS="${DJANGO_BIND_ADDRESS:-0.0.0.0}"
exec uv run gunicorn -c config/guniconf.py config.asgi:application
}
start_prod_server() {
echo "Starting the Gunicorn server..."
poetry run gunicorn -c config/guniconf.py config.wsgi:application
exec uv run gunicorn -c config/guniconf.py config.asgi:application
}
resolve_worker_hostname() {
@@ -47,7 +53,7 @@ resolve_worker_hostname() {
start_worker() {
echo "Starting the worker..."
poetry run python -m celery -A config.celery worker \
exec uv run python -m celery -A config.celery worker \
-n "$(resolve_worker_hostname)" \
-l "${DJANGO_LOGGING_LEVEL:-info}" \
-Q celery,scans,scan-reports,deletion,backfill,overview,integrations,compliance,attack-paths-scans \
@@ -56,8 +62,7 @@ start_worker() {
start_worker_beat() {
echo "Starting the worker-beat..."
sleep 15
poetry run python -m celery -A config.celery beat -l "${DJANGO_LOGGING_LEVEL:-info}" --scheduler django_celery_beat.schedulers:DatabaseScheduler
exec uv run python -m celery -A config.celery beat -l "${DJANGO_LOGGING_LEVEL:-info}" --scheduler django_celery_beat.schedulers:DatabaseScheduler
}
manage_db_partitions() {
@@ -65,10 +70,19 @@ manage_db_partitions() {
echo "Managing DB partitions..."
# For now we skip the deletion of partitions until we define the data retention policy
# --yes auto approves the operation without the need of an interactive terminal
poetry run python manage.py pgpartition --using admin --skip-delete --yes
uv run python manage.py pgpartition --using admin --skip-delete --yes
fi
}
# Identify this process to Postgres (application_name=<component>:<alias>) so
# connections are attributable by component in pg_stat_activity. Web tiers
# report "api"; everything else uses the launch subcommand.
case "$1" in
prod|dev) DJANGO_APP_COMPONENT="api" ;;
*) DJANGO_APP_COMPONENT="$1" ;;
esac
export DJANGO_APP_COMPONENT
case "$1" in
dev)
apply_migrations
api/docs/orphan-task-recovery.md
+106
View File
@@ -0,0 +1,106 @@
# Orphan Celery task recovery
When a worker is terminated mid-task (a deploy, an OOM kill, a node eviction), the
task it was running can be left non-terminal forever: the `TaskResult` stays
`STARTED` and nothing re-runs it. This page describes the mechanisms that detect and
recover allowlisted idempotent orphans so pending-task alerts do not fire. Scan tasks
are not auto-recovered (re-running a scan is not safe to do automatically); the
watchdog covers the summary/aggregation and deletion tasks.
## How recovery works
1. **Durable delivery.** The broker is configured so a task message is acknowledged
only after the task finishes (`task_acks_late`), one task is reserved at a time
(`worker_prefetch_multiplier = 1`), and an abruptly-lost worker re-queues its task
(`task_reject_on_worker_lost`). On `SIGTERM` the worker is given a soft-shutdown
window (`worker_soft_shutdown_timeout`) to finish or re-queue in-flight work
before it is force-killed. `scan-perform`, `scan-perform-scheduled` and
`integration-jira` opt out of redelivery with `acks_late=False`, so a crash drops
them rather than re-running and duplicating findings or Jira issues. Other
non-recovered side-effect tasks keep `acks_late=True`, so the broker can still
re-deliver them after a worker loss: the S3 upload rebuilds from worker-local files
that did not survive the crash and so no-ops, but Security Hub re-reads findings from
the DB and re-sends them to AWS.
2. **Periodic watchdog.** A Beat task, `reconcile-orphan-tasks`, runs every couple of
minutes (a `django_celery_beat` periodic task created by migration). For each
in-flight task result with an allowlisted idempotent task name, it pings the
worker recorded on the task's `TaskResult`:
- worker responds -> the task is still running, leave it alone;
- worker is gone (and the task started before a short grace window) -> it is a
real orphan: the stale task is revoked and marked terminal (clearing the
pending/started alert), and the task is re-enqueued from its stored name and
kwargs.
The re-run is safe because only tasks with proven idempotency are allowlisted: the
summary/aggregation tasks clear and re-write their own rows, and deletions are
idempotent. Scan tasks and external side effects are excluded: re-running a scan is
not safe to do automatically, Jira sends would create duplicate issues, the S3
upload rebuilds from worker-local files that do not survive a crash, and
report/Security Hub recovery is out of scope.
3. **Recovery cap.** A per-task Valkey counter limits how often the same task is
re-enqueued. After `--max-attempts` recoveries (default 3) the orphan is marked
terminal instead of re-enqueued, so a task that repeatedly kills its worker cannot
loop forever.
A Postgres advisory lock ensures that, even with multiple API/worker replicas, only
one reconciliation runs at a time; the others no-op.
## On-demand command
The same logic is available as a management command, useful right after a deploy or
for manual intervention:
```bash
python manage.py reconcile_orphan_tasks # recover now
python manage.py reconcile_orphan_tasks --dry-run # report orphans, change nothing
python manage.py reconcile_orphan_tasks --grace-minutes 5 --max-attempts 3
```
## Configuration
All settings have safe defaults; override via environment variables.
| Env var | Default | Purpose |
| --- | --- | --- |
| `DJANGO_CELERY_WORKER_PREFETCH_MULTIPLIER` | `1` | Tasks reserved per worker process. |
| `DJANGO_CELERY_WORKER_CONCURRENCY` | unset | Optional Celery prefork pool size. When unset, Celery uses its CPU-based default. Set this on worker containers to bound idle memory on hosts with many CPUs. |
| `DJANGO_CELERY_WORKER_SOFT_SHUTDOWN_TIMEOUT` | `60` | Seconds the worker drains/re-queues on `SIGTERM` before force-kill. |
| `DJANGO_CELERY_TASK_TIME_LIMIT` | `21600` (6h) | Hard limit for most tasks; connection checks are capped at 120s. |
| `DJANGO_CELERY_TASK_SOFT_TIME_LIMIT` | hard - 600 | Soft limit; raises `SoftTimeLimitExceeded` for cleanup. |
| `DJANGO_CELERY_LONG_TASK_TIME_LIMIT` | `172800` (48h) | Hard limit for scans and provider/tenant deletions, which can legitimately run for more than a day. |
| `DJANGO_CELERY_LONG_TASK_SOFT_TIME_LIMIT` | long hard - 600 | Soft limit for the long-running tasks above. |
| `DJANGO_TASK_RECOVERY_ENABLED` | `false` | Master switch for orphan-task recovery, disabled by default (opt-in); set to `true` to enable. When off, no orphan is detected, marked terminal, or re-enqueued (attack-paths stale cleanup still runs). |
| `DJANGO_TASK_RECOVERY_SUMMARIES_ENABLED` | `true` | Auto re-enqueue orphaned scan summary/aggregation tasks. |
| `DJANGO_TASK_RECOVERY_DELETIONS_ENABLED` | `true` | Auto re-enqueue orphaned provider/tenant deletion tasks. |
Recovery is opt-in: with the master flag off (the default) the sweep does nothing.
Once enabled, the per-group flags default to on, so every group recovers unless you
turn one off; a task whose group flag is off is marked terminal instead of
re-enqueued.
Turning recovery off only disables this watchdog sweep; it does not change Celery's
broker-level redelivery (`task_acks_late`/`task_reject_on_worker_lost`), which still
re-delivers tasks that keep `acks_late=True` on worker loss, independently of this flag.
`task_acks_late` and `task_reject_on_worker_lost` are enabled in `config/celery.py`.
## Deployment requirement
Two conditions must both hold for the soft shutdown to actually drain work:
1. **The worker must receive `SIGTERM`.** The container entrypoint `exec`s the
Celery process so it runs as PID 1; otherwise `SIGTERM` from `docker stop`/ECS
hits the entrypoint shell, never reaches Celery, and the worker is hard-killed
(SIGKILL) at the grace deadline without draining. Custom entrypoints must
preserve the `exec`.
2. **The orchestrator must give the worker enough time** before force-killing it.
Set the stop grace period to exceed `DJANGO_CELERY_WORKER_SOFT_SHUTDOWN_TIMEOUT`
plus a margin:
- **docker-compose:** `stop_grace_period` on the worker services (set to `120s`).
- **AWS ECS:** the worker container `stopTimeout` (configured in the deployment
repository).
If either condition is missing, long tasks are still recovered by the watchdog,
but they are cut mid-run on every deploy instead of draining.
api/poetry.lock
Generated
-9375
View File
File diff suppressed because it is too large Load Diff
api/pyproject.toml
+441 -31
View File
@@ -1,6 +1,24 @@
[build-system]
build-backend = "poetry.core.masonry.api"
requires = ["poetry-core"]
[dependency-groups]
dev = [
"bandit==1.7.9",
"coverage==7.5.4",
"django-silk==5.3.2",
"docker==7.1.0",
"filelock==3.20.3",
"freezegun==1.5.1",
"mypy==1.10.1",
"pylint==3.2.5",
"pytest==9.0.3",
"pytest-cov==5.0.0",
"pytest-django==4.8.0",
"pytest-env==1.1.3",
"pytest-randomly==3.15.0",
"pytest-xdist==3.6.1",
"ruff==0.15.11",
"tqdm==4.67.1",
"vulture==2.14",
"prek==0.3.9"
]
[project]
authors = [{name = "Prowler Engineering", email = "engineering@prowler.com"}]
@@ -23,26 +41,29 @@ dependencies = [
"drf-spectacular==0.27.2",
"drf-spectacular-jsonapi==0.5.1",
"defusedxml==0.7.1",
"gunicorn==23.0.0",
"lxml==5.3.2",
"django-eventstream==5.3.3",
"gunicorn==26.0.0",
"uvloop==0.22.1",
"lxml==6.1.0",
"prowler @ git+https://github.com/prowler-cloud/prowler.git@master",
"psycopg2-binary==2.9.9",
"pytest-celery[redis] (==1.3.0)",
"sentry-sdk[django] (==2.56.0)",
"uuid6==2024.7.10",
"openai (==1.109.1)",
"xmlsec==1.3.14",
"xmlsec==1.3.17",
"h2 (==4.3.0)",
"markdown (==3.10.2)",
"drf-simple-apikey (==2.2.1)",
"matplotlib (==3.10.8)",
"reportlab (==4.4.10)",
"neo4j (==6.1.0)",
"cartography (==0.132.0)",
"cartography (==0.138.1)",
"gevent (==25.9.1)",
"werkzeug (==3.1.7)",
"sqlparse (==0.5.5)",
"fonttools (==4.62.1)"
"fonttools (==4.62.1)",
"uvicorn-worker (==0.4.0)",
]
description = "Prowler's API (Django/DRF)"
license = "Apache-2.0"
@@ -50,28 +71,417 @@ name = "prowler-api"
package-mode = false
# Needed for the SDK compatibility
requires-python = ">=3.11,<3.13"
version = "1.24.0"
version = "1.33.0"
[project.scripts]
celery = "src.backend.config.settings.celery"
# Shared ruff baseline (kept in sync with mcp_server/pyproject.toml).
# target-version tracks this project's lowest supported Python.
[tool.ruff]
src = ["src"]
target-version = "py311"
[tool.poetry.group.dev.dependencies]
bandit = "1.7.9"
coverage = "7.5.4"
django-silk = "5.3.2"
docker = "7.1.0"
filelock = "3.20.3"
freezegun = "1.5.1"
marshmallow = "==3.26.2"
mypy = "1.10.1"
pylint = "3.2.5"
pytest = "8.2.2"
pytest-cov = "5.0.0"
pytest-django = "4.8.0"
pytest-env = "1.1.3"
pytest-randomly = "3.15.0"
pytest-xdist = "3.6.1"
ruff = "0.5.0"
safety = "3.7.0"
tqdm = "4.67.1"
vulture = "2.14"
[tool.ruff.lint]
# Defaults (E4/E7/E9, F) plus import sorting, modern-syntax upgrades, and
# comprehension lints — all mechanically auto-fixable. flake8-bugbear (B) is a
# good next step but needs manual cleanup (e.g. B904 raise-from), so it is left
# out of the shared baseline for now.
extend-select = [
"I", # isort — import ordering (prek's isort hook covers only the SDK)
"UP", # pyupgrade — modern syntax for the min supported Python
"C4" # flake8-comprehensions
]
[tool.uv]
# Transitive pins matching master to avoid silent drift; bump deliberately.
constraint-dependencies = [
"about-time==4.2.1",
"adal==1.2.7",
"aioboto3==15.5.0",
"aiobotocore==2.25.1",
"aiofiles==24.1.0",
"aiohappyeyeballs==2.6.1",
"aiohttp==3.14.0",
"aioitertools==0.13.0",
"aiosignal==1.4.0",
"alibabacloud-actiontrail20200706==2.4.1",
"alibabacloud-credentials==1.0.3",
"alibabacloud-credentials-api==1.0.0",
"alibabacloud-cs20151215==6.1.0",
"alibabacloud-darabonba-array==0.1.0",
"alibabacloud-darabonba-encode-util==0.0.2",
"alibabacloud-darabonba-map==0.0.1",
"alibabacloud-darabonba-signature-util==0.0.4",
"alibabacloud-darabonba-string==0.0.4",
"alibabacloud-darabonba-time==0.0.1",
"alibabacloud-ecs20140526==7.2.5",
"alibabacloud-endpoint-util==0.0.4",
"alibabacloud-gateway-oss==0.0.17",
"alibabacloud-gateway-oss-util==0.0.3",
"alibabacloud-gateway-sls==0.4.0",
"alibabacloud-gateway-sls-util==0.4.0",
"alibabacloud-gateway-spi==0.0.3",
"alibabacloud-openapi-util==0.2.4",
"alibabacloud-oss-util==0.0.6",
"alibabacloud-oss20190517==1.0.6",
"alibabacloud-ram20150501==1.2.0",
"alibabacloud-rds20140815==12.0.0",
"alibabacloud-sas20181203==6.1.0",
"alibabacloud-sls20201230==5.9.0",
"alibabacloud-sts20150401==1.1.6",
"alibabacloud-tea==0.4.3",
"alibabacloud-tea-openapi==0.4.4",
"alibabacloud-tea-util==0.3.14",
"alibabacloud-tea-xml==0.0.3",
"alibabacloud-vpc20160428==6.13.0",
"alive-progress==3.3.0",
"aliyun-log-fastpb==0.2.0",
"amqp==5.3.1",
"annotated-types==0.7.0",
"anyio==4.12.1",
"applicationinsights==0.11.10",
"apscheduler==3.11.2",
"argcomplete==3.5.3",
"asgiref==3.11.0",
"astroid==3.2.4",
"async-timeout==5.0.1",
"attrs==25.4.0",
"authlib==1.6.12",
"autopep8==2.3.2",
"azure-cli-core==2.83.0",
"azure-cli-telemetry==1.1.0",
"azure-common==1.1.28",
"azure-core==1.38.1",
"azure-identity==1.21.0",
"azure-keyvault-certificates==4.10.0",
"azure-keyvault-keys==4.10.0",
"azure-keyvault-secrets==4.10.0",
"azure-mgmt-apimanagement==5.0.0",
"azure-mgmt-applicationinsights==4.1.0",
"azure-mgmt-authorization==4.0.0",
"azure-mgmt-compute==34.0.0",
"azure-mgmt-containerinstance==10.1.0",
"azure-mgmt-containerregistry==12.0.0",
"azure-mgmt-containerservice==34.1.0",
"azure-mgmt-core==1.6.0",
"azure-mgmt-cosmosdb==9.7.0",
"azure-mgmt-databricks==2.0.0",
"azure-mgmt-datafactory==9.2.0",
"azure-mgmt-eventgrid==10.4.0",
"azure-mgmt-eventhub==11.2.0",
"azure-mgmt-keyvault==10.3.1",
"azure-mgmt-loganalytics==12.0.0",
"azure-mgmt-logic==10.0.0",
"azure-mgmt-monitor==6.0.2",
"azure-mgmt-network==28.1.0",
"azure-mgmt-postgresqlflexibleservers==1.1.0",
"azure-mgmt-rdbms==10.1.0",
"azure-mgmt-recoveryservices==3.1.0",
"azure-mgmt-recoveryservicesbackup==9.2.0",
"azure-mgmt-resource==24.0.0",
"azure-mgmt-search==9.1.0",
"azure-mgmt-security==7.0.0",
"azure-mgmt-sql==3.0.1",
"azure-mgmt-storage==22.1.1",
"azure-mgmt-subscription==3.1.1",
"azure-mgmt-synapse==2.0.0",
"azure-mgmt-web==8.0.0",
"azure-monitor-query==2.0.0",
"azure-storage-blob==12.24.1",
"azure-synapse-artifacts==0.21.0",
"backoff==2.2.1",
"bandit==1.7.9",
"billiard==4.2.4",
"blinker==1.9.0",
"boto3==1.40.61",
"botocore==1.40.61",
"cartography==0.138.1",
"celery==5.6.2",
"certifi==2026.1.4",
"cffi==2.0.0",
"charset-normalizer==3.4.4",
"circuitbreaker==2.1.3",
"click==8.3.1",
"click-didyoumean==0.3.1",
"click-plugins==1.1.1.2",
"click-repl==0.3.0",
"cloudflare==4.3.1",
"colorama==0.4.6",
"contextlib2==21.6.0",
"contourpy==1.3.3",
"coverage==7.5.4",
"cron-descriptor==1.4.5",
"crowdstrike-falconpy==1.6.0",
"cryptography==46.0.7",
"cycler==0.12.1",
"darabonba-core==1.0.5",
"dash==3.1.1",
"dash-bootstrap-components==2.0.3",
"debugpy==1.8.20",
"decorator==5.2.1",
"defusedxml==0.7.1",
"dill==0.4.1",
"distro==1.9.0",
"dj-rest-auth==7.0.1",
"django==5.1.15",
"django-allauth==65.15.0",
"django-celery-beat==2.9.0",
"django-celery-results==2.6.0",
"django-cors-headers==4.4.0",
"django-environ==0.11.2",
"django-eventstream==5.3.3",
"django-filter==24.3",
"django-guid==3.5.0",
"django-postgres-extra==2.0.9",
"django-silk==5.3.2",
"django-timezone-field==7.2.1",
"djangorestframework==3.15.2",
"djangorestframework-jsonapi==7.0.2",
"djangorestframework-simplejwt==5.5.1",
"dnspython==2.8.0",
"docker==7.1.0",
"dogpile-cache==1.5.0",
"dparse==0.6.4",
"drf-extensions==0.8.0",
"drf-nested-routers==0.95.0",
"drf-simple-apikey==2.2.1",
"drf-spectacular==0.27.2",
"drf-spectacular-jsonapi==0.5.1",
"dulwich==1.2.5",
"duo-client==5.5.0",
"durationpy==0.10",
"email-validator==2.2.0",
"execnet==2.1.2",
"filelock==3.20.3",
"flask==3.1.3",
"fonttools==4.62.1",
"freezegun==1.5.1",
"frozenlist==1.8.0",
"gevent==25.9.1",
"google-api-core==2.29.0",
"google-api-python-client==2.163.0",
"google-auth==2.48.0",
"google-auth-httplib2==0.2.0",
"google-cloud-access-context-manager==0.3.0",
"google-cloud-asset==4.2.0",
"google-cloud-org-policy==1.16.0",
"google-cloud-os-config==1.23.0",
"google-cloud-resource-manager==1.16.0",
"googleapis-common-protos==1.72.0",
"gprof2dot==2025.4.14",
"graphemeu==0.7.2",
"greenlet==3.3.1",
"grpc-google-iam-v1==0.14.3",
"grpcio==1.76.0",
"grpcio-status==1.76.0",
"gunicorn==26.0.0",
"h11==0.16.0",
"h2==4.3.0",
"hpack==4.1.0",
"httpcore==1.0.9",
"httplib2==0.31.2",
"httpx==0.28.1",
"humanfriendly==10.0",
"hyperframe==6.1.0",
"iamdata==0.1.202605131",
"idna==3.15",
"importlib-metadata==8.7.1",
"inflection==0.5.1",
"iniconfig==2.3.0",
"iso8601==2.1.0",
"isodate==0.7.2",
"isort==5.13.2",
"itsdangerous==2.2.0",
"jinja2==3.1.6",
"jiter==0.13.0",
"jmespath==1.1.0",
"joblib==1.5.3",
"jsonpatch==1.33",
"jsonpickle==4.1.1",
"jsonpointer==3.0.0",
"jsonschema==4.23.0",
"jsonschema-specifications==2025.9.1",
"keystoneauth1==5.13.0",
"kingfisher-bin==1.104.0",
"kiwisolver==1.4.9",
"knack==0.11.0",
"kombu==5.6.2",
"kubernetes==32.0.1",
"lxml==6.1.0",
"lz4==4.4.5",
"markdown==3.10.2",
"markdown-it-py==4.0.0",
"markupsafe==3.0.3",
"marshmallow==4.3.0",
"matplotlib==3.10.8",
"mccabe==0.7.0",
"mdurl==0.1.2",
"microsoft-kiota-abstractions==1.9.9",
"microsoft-kiota-authentication-azure==1.9.9",
"microsoft-kiota-http==1.9.9",
"microsoft-kiota-serialization-form==1.9.9",
"microsoft-kiota-serialization-json==1.9.9",
"microsoft-kiota-serialization-multipart==1.9.9",
"microsoft-kiota-serialization-text==1.9.9",
"microsoft-security-utilities-secret-masker==1.0.0b4",
"msal==1.35.0b1",
"msal-extensions==1.2.0",
"msgraph-core==1.3.8",
"msgraph-sdk==1.55.0",
"msrest==0.7.1",
"msrestazure==0.6.4.post1",
"multidict==6.7.1",
"mypy==1.10.1",
"mypy-extensions==1.1.0",
"narwhals==2.16.0",
"neo4j==6.1.0",
"nest-asyncio==1.6.0",
"nltk==3.9.4",
"numpy==2.2.6",
"oauthlib==3.3.1",
"oci==2.169.0",
"openai==1.109.1",
"openstacksdk==4.2.0",
"opentelemetry-api==1.39.1",
"opentelemetry-sdk==1.39.1",
"opentelemetry-semantic-conventions==0.60b1",
"os-service-types==1.8.2",
"packageurl-python==0.17.6",
"packaging==26.0",
"pagerduty==6.1.0",
"pandas==2.2.3",
"pbr==7.0.3",
"pillow==12.2.0",
"pkginfo==1.12.1.2",
"platformdirs==4.5.1",
"plotly==6.5.2",
"pluggy==1.6.0",
"policyuniverse==1.5.1.20231109",
"portalocker==2.10.1",
"prek==0.3.9",
"prompt-toolkit==3.0.52",
"propcache==0.4.1",
"proto-plus==1.27.0",
"protobuf==6.33.5",
"psutil==7.2.2",
"psycopg2-binary==2.9.9",
"py-deviceid==0.1.1",
"py-iam-expand==0.3.0",
"py-ocsf-models==0.8.1",
"pyasn1==0.6.3",
"pyasn1-modules==0.4.2",
"pycodestyle==2.14.0",
"pycparser==3.0",
"pydantic==2.12.5",
"pydantic-core==2.41.5",
"pygithub==2.8.0",
"pygments==2.20.0",
"pyjwt==2.13.0",
"pylint==3.2.5",
"pymsalruntime==0.18.1",
"pynacl==1.6.2",
"pyopenssl==26.0.0",
"pyparsing==3.3.2",
"pyreadline3==3.5.4",
"pysocks==1.7.1",
"pytest==9.0.3",
"pytest-celery==1.3.0",
"pytest-cov==5.0.0",
"pytest-django==4.8.0",
"pytest-docker-tools==3.1.9",
"pytest-env==1.1.3",
"pytest-randomly==3.15.0",
"pytest-xdist==3.6.1",
"python-crontab==3.3.0",
"python-dateutil==2.9.0.post0",
"python-digitalocean==1.17.0",
"python3-saml==1.16.0",
"pytz==2025.1",
"pywin32==311",
"pyyaml==6.0.3",
"redis==7.1.0",
"referencing==0.37.0",
"regex==2026.1.15",
"reportlab==4.4.10",
"requests==2.33.1",
"requests-file==3.0.1",
"requests-oauthlib==2.0.0",
"requestsexceptions==1.4.0",
"retrying==1.4.2",
"rich==14.3.2",
"rpds-py==0.30.0",
"rsa==4.9.1",
"ruamel-yaml==0.19.1",
"ruff==0.15.11",
"s3transfer==0.14.0",
"scaleway==2.10.3",
"scaleway-core==2.10.3",
"schema==0.7.5",
"sentry-sdk==2.56.0",
"setuptools==80.10.2",
"shellingham==1.5.4",
"shodan==1.31.0",
"six==1.17.0",
"slack-sdk==3.39.0",
"sniffio==1.3.1",
"sqlparse==0.5.5",
"statsd==4.0.1",
"std-uritemplate==2.0.8",
"stevedore==5.6.0",
"tabulate==0.9.0",
"tenacity==9.1.2",
"tldextract==5.3.1",
"tomlkit==0.14.0",
"tqdm==4.67.1",
"typer==0.21.1",
"types-aiobotocore-ecr==3.1.1",
"typing-extensions==4.15.0",
"typing-inspection==0.4.2",
"tzdata==2025.3",
"tzlocal==5.3.1",
"uritemplate==4.2.0",
"urllib3==2.7.0",
"uuid6==2024.7.10",
"uvicorn==0.49.0",
"uvloop==0.22.1",
"vine==5.1.0",
"vulture==2.14",
"wcwidth==0.5.3",
"websocket-client==1.9.0",
"werkzeug==3.1.7",
"workos==6.0.8",
"wrapt==1.17.3",
"xlsxwriter==3.2.9",
"xmlsec==1.3.17",
"xmltodict==1.0.2",
"yarl==1.22.0",
"zipp==3.23.0",
"zope-event==6.1",
"zope-interface==8.2",
"zstd==1.5.7.3"
]
# prowler@master needs okta==3.4.2, but cartography 0.138.1 requires okta<1.0.0.
# Attack Paths does not ingest Okta today, so override the Cartography
# dependency to the Prowler pin.
#
# prowler@master needs azure-mgmt-containerservice==34.1.0, but cartography
# 0.138.1 requires azure-mgmt-containerservice>=41.0.0. Attack Paths does not
# ingest Azure today, so override the Cartography dependency to the Prowler pin.
#
# prowler@master hard-pins microsoft-kiota-abstractions==1.9.2 in [project.dependencies].
# The microsoft-kiota-http security bump to 1.9.9 (GHSA-7j59-v9qr-6fq9) requires
# microsoft-kiota-abstractions>=1.9.9, which a constraint cannot satisfy against the
# SDK's hard pin; override it to the patched, kiota-aligned version.
#
# prowler@master hard-pins dulwich==0.23.0 and pyjwt==2.12.1 in [project.dependencies].
# dulwich 1.2.5 patches GHSA-897w-fcg9-f6xj (arbitrary file write) and pyjwt 2.13.0
# patches PYSEC-2026-179 (HMAC/JWK key-confusion); a constraint cannot satisfy these
# against the SDK's hard pins, so override them to the patched versions until the SDK
# bump propagates to the pinned master rev. pyjwt keeps the [crypto] extra because an
# override replaces the whole requirement; bare pyjwt would drop it from the consumers
# that request pyjwt[crypto] and leave cryptography (needed for RS256) only transitive.
override-dependencies = [
"okta==3.4.2",
"azure-mgmt-containerservice==34.1.0",
"microsoft-kiota-abstractions==1.9.9",
"dulwich==1.2.5",
"pyjwt[crypto]==2.13.0"
]
api/src/backend/api/adapters.py
+44 -3
View File
@@ -1,9 +1,15 @@
from allauth.socialaccount.adapter import DefaultSocialAccountAdapter
from django.db import transaction
from api.db_router import MainRouter
from api.db_utils import rls_transaction
from api.models import Membership, Role, Tenant, User, UserRoleRelationship
from api.models import (
Membership,
Role,
SAMLConfiguration,
Tenant,
User,
UserRoleRelationship,
)
from django.db import transaction
class ProwlerSocialAccountAdapter(DefaultSocialAccountAdapter):
@@ -18,7 +24,42 @@ class ProwlerSocialAccountAdapter(DefaultSocialAccountAdapter):
# Link existing accounts with the same email address
email = sociallogin.account.extra_data.get("email")
if sociallogin.provider.id == "saml":
# For SAML, the asserted NameID email cannot be trusted on its own:
# any tenant can claim any email domain in its SAML configuration. To
# prevent cross-tenant account takeover (GHSA-h8m9-jgf8-vwvp), only link
# the incoming SAML session to an existing account when (1) the email
# domain matches the tenant whose ACS endpoint is being used and (2) the
# existing user is already a member of that tenant.
email = sociallogin.user.email
if not email:
return
domain = email.rsplit("@", 1)[-1].lower()
resolver_match = getattr(request, "resolver_match", None)
organization_slug = (
(resolver_match.kwargs or {}).get("organization_slug", "")
if resolver_match
else ""
).lower()
# The ACS endpoint is scoped per email domain; reject mismatches so an
# attacker cannot replay an assertion through another tenant's endpoint.
if organization_slug != domain:
return
try:
saml_config = SAMLConfiguration.objects.using(MainRouter.admin_db).get(
email_domain=domain
)
except SAMLConfiguration.DoesNotExist:
return
existing_user = self.get_user_by_email(email)
if existing_user and existing_user.is_member_of_tenant(
str(saml_config.tenant_id)
):
sociallogin.connect(request, existing_user)
return
if email:
existing_user = self.get_user_by_email(email)
if existing_user:
api/src/backend/api/apps.py
+4 -35
View File
@@ -28,9 +28,10 @@ class ApiConfig(AppConfig):
name = "api"
def ready(self):
from api import schema_extensions # noqa: F401
from api import signals # noqa: F401
from api.attack_paths import database as graph_database
from api import (
schema_extensions, # noqa: F401
signals, # noqa: F401
)
# Generate required cryptographic keys if not present, but only if:
# `"manage.py" not in sys.argv[0]`: If an external server (e.g., Gunicorn) is running the app
@@ -41,38 +42,6 @@ class ApiConfig(AppConfig):
):
self._ensure_crypto_keys()
# Commands that don't need Neo4j
SKIP_NEO4J_DJANGO_COMMANDS = [
"makemigrations",
"migrate",
"pgpartition",
"check",
"help",
"showmigrations",
"check_and_fix_socialaccount_sites_migration",
]
# Skip Neo4j initialization during tests, some Django commands, and Celery
if getattr(settings, "TESTING", False) or (
len(sys.argv) > 1
and (
(
"manage.py" in sys.argv[0]
and sys.argv[1] in SKIP_NEO4J_DJANGO_COMMANDS
)
or "celery" in sys.argv[0]
)
):
logger.info(
"Skipping Neo4j initialization because tests, some Django commands or Celery"
)
else:
graph_database.init_driver()
# Neo4j driver is initialized at API startup (see api.attack_paths.database)
# It remains lazy for Celery workers and selected Django commands
def _ensure_crypto_keys(self):
"""
Orchestrator method that ensures all required cryptographic keys are present.
api/src/backend/api/attack_paths/__init__.py
-1
View File
@@ -5,7 +5,6 @@ from api.attack_paths.queries import (
get_query_by_id,
)
__all__ = [
"AttackPathsQueryDefinition",
"AttackPathsQueryParameterDefinition",
api/src/backend/api/attack_paths/cypher_sanitizer.py
+12 -16
View File
@@ -4,10 +4,10 @@ Cypher sanitizer for custom (user-supplied) Attack Paths queries.
Two responsibilities:
1. **Validation** - reject queries containing SSRF or dangerous procedure
patterns (defense-in-depth; the primary control is ``neo4j.READ_ACCESS``).
patterns (defense-in-depth; the primary control is `neo4j.READ_ACCESS`).
2. **Provider-scoped label injection** - inject a dynamic
``_Provider_{uuid}`` label into every node pattern so the database can
`_Provider_{uuid}` label into every node pattern so the database can
use its native label index for provider isolation.
Label-injection pipeline:
@@ -22,18 +22,16 @@ Label-injection pipeline:
import re
from rest_framework.exceptions import ValidationError
from tasks.jobs.attack_paths.config import get_provider_label
# Step 1 - String / comment protection
# Single combined regex: strings first, then line comments.
# Single combined regex: strings first, then line comments
# The regex engine finds the leftmost match, so a string like 'https://prowler.com'
# is consumed as a string before the // inside it can match as a comment.
# is consumed as a string before the // inside it can match as a comment
_PROTECTED_RE = re.compile(r"'(?:[^'\\]|\\.)*'|\"(?:[^\"\\]|\\.)*\"|//[^\n]*")
# Step 2 - Clause splitting
# OPTIONAL MATCH must come before MATCH to avoid partial matching.
# `OPTIONAL MATCH` must come before `MATCH` to avoid partial matching
_CLAUSE_RE = re.compile(
r"\b(OPTIONAL\s+MATCH|MATCH|WHERE|RETURN|WITH|ORDER\s+BY"
r"|SKIP|LIMIT|UNION|UNWIND|CALL)\b",
@@ -41,10 +39,10 @@ _CLAUSE_RE = re.compile(
)
# Pass A - Labeled node patterns (all segments)
# Matches node patterns that have at least one :Label.
# (?<!\w)\( - open paren NOT preceded by a word char (excludes function calls).
# Group 1: optional variable + one or more :Label
# Group 2: optional {properties} + closing paren
# Matches node patterns that have at least one `:Label`
# `(?<!\w)\(` - open paren NOT preceded by a word char, excludes function calls
# Group 1: optional variable + one or more `:Label`
# Group 2: optional `{`properties`}` + closing paren
_LABELED_NODE_RE = re.compile(
r"(?<!\w)\("
r"("
@@ -57,9 +55,9 @@ _LABELED_NODE_RE = re.compile(
r")"
)
# Pass B - Bare node patterns (MATCH segments only)
# Matches (identifier) or (identifier {properties}) without any :Label.
# Only applied in MATCH/OPTIONAL MATCH segments.
# Pass B - Bare node patterns (`MATCH` segments only)
# Matches (identifier) or (identifier {properties}) without any `:Label`
# Only applied in `MATCH` / `OPTIONAL MATCH` segments
_BARE_NODE_RE = re.compile(
r"(?<!\w)\(" r"(\s*[a-zA-Z_]\w*)" r"(\s*(?:\{[^}]*\})?)" r"\s*\)"
)
@@ -136,9 +134,7 @@ def inject_provider_label(cypher: str, provider_id: str) -> str:
return work
# ---------------------------------------------------------------------------
# Validation
# ---------------------------------------------------------------------------
# Patterns that indicate SSRF or dangerous procedure calls
# Defense-in-depth layer - the primary control is `neo4j.READ_ACCESS`
api/src/backend/api/attack_paths/database.py
+171 -223
View File
@@ -1,232 +1,32 @@
import atexit
import logging
import threading
from contextlib import contextmanager
from typing import Any, Iterator
"""Backwards-compatible facade over the ingest and sink modules.
Historically this module owned a single Neo4j driver used for both the
cartography temp database and the per-tenant sink database. The port to AWS
Neptune split those roles: the cartography ingest (temp) database is always
Neo4j and lives in `api.attack_paths.ingest`; the sink is configurable
(Neo4j or Neptune) and lives in `api.attack_paths.sink`. This shim preserves
the public API that `tasks/` and `api/v1/views.py` already depend on, and
dispatches to the right module by database-name prefix.
A database name starting with `db-tmp-scan-` is a cartography temp DB and
routes to ingest. Everything else routes to the configured sink.
"""
from contextlib import AbstractContextManager
from typing import Any
from uuid import UUID
import neo4j
import neo4j.exceptions
import neo4j # noqa: F401 - kept for tests that patch api.attack_paths.database.neo4j
from api.attack_paths import ingest
from api.attack_paths import sink as sink_module
from config.env import env
from django.conf import settings
from tasks.jobs.attack_paths.config import (
BATCH_SIZE,
PROVIDER_RESOURCE_LABEL,
get_provider_label,
from django.conf import (
settings, # noqa: F401 - kept for tests that patch ...database.settings
)
from api.attack_paths.retryable_session import RetryableSession
# Without this Celery goes crazy with Neo4j logging
logging.getLogger("neo4j").setLevel(logging.ERROR)
logging.getLogger("neo4j").propagate = False
SERVICE_UNAVAILABLE_MAX_RETRIES = env.int(
"ATTACK_PATHS_SERVICE_UNAVAILABLE_MAX_RETRIES", default=3
)
READ_QUERY_TIMEOUT_SECONDS = env.int(
"ATTACK_PATHS_READ_QUERY_TIMEOUT_SECONDS", default=30
)
MAX_CUSTOM_QUERY_NODES = env.int("ATTACK_PATHS_MAX_CUSTOM_QUERY_NODES", default=250)
READ_EXCEPTION_CODES = [
"Neo.ClientError.Statement.AccessMode",
"Neo.ClientError.Procedure.ProcedureNotFound",
]
CLIENT_STATEMENT_EXCEPTION_PREFIX = "Neo.ClientError.Statement."
# Module-level process-wide driver singleton
_driver: neo4j.Driver | None = None
_lock = threading.Lock()
# Base Neo4j functions
def get_uri() -> str:
host = settings.DATABASES["neo4j"]["HOST"]
port = settings.DATABASES["neo4j"]["PORT"]
return f"bolt://{host}:{port}"
def init_driver() -> neo4j.Driver:
global _driver
if _driver is not None:
return _driver
with _lock:
if _driver is None:
uri = get_uri()
config = settings.DATABASES["neo4j"]
_driver = neo4j.GraphDatabase.driver(
uri,
auth=(config["USER"], config["PASSWORD"]),
keep_alive=True,
max_connection_lifetime=7200,
connection_acquisition_timeout=120,
max_connection_pool_size=50,
)
_driver.verify_connectivity()
# Register cleanup handler (only runs once since we're inside the _driver is None block)
atexit.register(close_driver)
return _driver
def get_driver() -> neo4j.Driver:
return init_driver()
def close_driver() -> None: # TODO: Use it
global _driver
with _lock:
if _driver is not None:
try:
_driver.close()
finally:
_driver = None
@contextmanager
def get_session(
database: str | None = None, default_access_mode: str | None = None
) -> Iterator[RetryableSession]:
session_wrapper: RetryableSession | None = None
try:
session_wrapper = RetryableSession(
session_factory=lambda: get_driver().session(
database=database, default_access_mode=default_access_mode
),
max_retries=SERVICE_UNAVAILABLE_MAX_RETRIES,
)
yield session_wrapper
except neo4j.exceptions.Neo4jError as exc:
if (
default_access_mode == neo4j.READ_ACCESS
and exc.code
and exc.code in READ_EXCEPTION_CODES
):
message = "Read query not allowed"
code = READ_EXCEPTION_CODES[0]
raise WriteQueryNotAllowedException(message=message, code=code)
message = exc.message if exc.message is not None else str(exc)
if exc.code and exc.code.startswith(CLIENT_STATEMENT_EXCEPTION_PREFIX):
raise ClientStatementException(message=message, code=exc.code)
raise GraphDatabaseQueryException(message=message, code=exc.code)
finally:
if session_wrapper is not None:
session_wrapper.close()
def execute_read_query(
database: str,
cypher: str,
parameters: dict[str, Any] | None = None,
) -> neo4j.graph.Graph:
with get_session(database, default_access_mode=neo4j.READ_ACCESS) as session:
def _run(tx: neo4j.ManagedTransaction) -> neo4j.graph.Graph:
result = tx.run(
cypher, parameters or {}, timeout=READ_QUERY_TIMEOUT_SECONDS
)
return result.graph()
return session.execute_read(_run)
def create_database(database: str) -> None:
query = "CREATE DATABASE $database IF NOT EXISTS"
parameters = {"database": database}
with get_session() as session:
session.run(query, parameters)
def drop_database(database: str) -> None:
query = f"DROP DATABASE `{database}` IF EXISTS DESTROY DATA"
with get_session() as session:
session.run(query)
def drop_subgraph(database: str, provider_id: str) -> int:
"""
Delete all nodes for a provider from the tenant database.
Uses batched deletion to avoid memory issues with large graphs.
Silently returns 0 if the database doesn't exist.
"""
provider_label = get_provider_label(provider_id)
deleted_nodes = 0
try:
with get_session(database) as session:
deleted_count = 1
while deleted_count > 0:
result = session.run(
f"""
MATCH (n:{PROVIDER_RESOURCE_LABEL}:`{provider_label}`)
WITH n LIMIT $batch_size
DETACH DELETE n
RETURN COUNT(n) AS deleted_nodes_count
""",
{"batch_size": BATCH_SIZE},
)
deleted_count = result.single().get("deleted_nodes_count", 0)
deleted_nodes += deleted_count
except GraphDatabaseQueryException as exc:
if exc.code == "Neo.ClientError.Database.DatabaseNotFound":
return 0
raise
return deleted_nodes
def has_provider_data(database: str, provider_id: str) -> bool:
"""
Check if any ProviderResource node exists for this provider.
Returns `False` if the database doesn't exist.
"""
provider_label = get_provider_label(provider_id)
query = f"MATCH (n:{PROVIDER_RESOURCE_LABEL}:`{provider_label}`) RETURN 1 LIMIT 1"
try:
with get_session(database, default_access_mode=neo4j.READ_ACCESS) as session:
result = session.run(query)
return result.single() is not None
except GraphDatabaseQueryException as exc:
if exc.code == "Neo.ClientError.Database.DatabaseNotFound":
return False
raise
def clear_cache(database: str) -> None:
query = "CALL db.clearQueryCaches()"
try:
with get_session(database) as session:
session.run(query)
except GraphDatabaseQueryException as exc:
logging.warning(f"Failed to clear query cache for database `{database}`: {exc}")
# Neo4j functions related to Prowler + Cartography
def get_database_name(entity_id: str | UUID, temporary: bool = False) -> str:
prefix = "tmp-scan" if temporary else "tenant"
return f"db-{prefix}-{str(entity_id).lower()}"
TEMP_DB_PREFIX = "db-tmp-scan-"
# Exceptions
@@ -241,7 +41,6 @@ class GraphDatabaseQueryException(Exception):
def __str__(self) -> str:
if self.code:
return f"{self.code}: {self.message}"
return self.message
@@ -251,3 +50,152 @@ class WriteQueryNotAllowedException(GraphDatabaseQueryException):
class ClientStatementException(GraphDatabaseQueryException):
pass
# Routing
def _is_ingest_database(database: str | None) -> bool:
return bool(database) and database.startswith(TEMP_DB_PREFIX)
# Driver lifecycle
def init_driver() -> Any:
"""Initialize the configured sink backend.
The ingest driver (Neo4j for cartography temp DBs) stays lazy: it is
only initialized when a temp-DB operation actually runs, which never
happens on API pods.
"""
return sink_module.init()
def close_driver() -> None:
"""Close every driver held by this process."""
sink_module.close()
ingest.close_driver()
def get_driver() -> neo4j.Driver:
"""Return the sink backend's underlying driver.
Only meaningful for the Neo4j sink (where the backend has a single Neo4j
driver). On Neptune this returns the writer driver. Kept for tests and
legacy call-sites; prefer `get_session` for new code.
"""
backend = sink_module.get_backend()
# Neo4jSink exposes get_driver(); NeptuneSink exposes get_writer()
if hasattr(backend, "get_driver"):
return backend.get_driver()
if hasattr(backend, "get_writer"):
return backend.get_writer()
raise RuntimeError("Active sink backend does not expose a driver handle")
def verify_connectivity() -> None:
"""Raise if the configured graph database is unreachable on the API read path.
Backend-agnostic entry point for the readiness probe: Neo4j verifies its
driver, Neptune verifies the reader endpoint.
"""
sink_module.get_backend().verify_connectivity()
def get_uri() -> str:
"""Return the sink URI. Retained for backwards compatibility."""
if settings.ATTACK_PATHS_SINK_DATABASE == "neptune":
cfg = settings.DATABASES["neptune"]
return f"bolt+s://{cfg['WRITER_ENDPOINT']}:{cfg['PORT']}"
cfg = settings.DATABASES["neo4j"]
return f"bolt://{cfg['HOST']}:{cfg['PORT']}"
def get_ingest_uri() -> str:
"""Neo4j URI for the cartography temp (ingest) database, which is always
Neo4j regardless of the configured sink."""
return ingest.get_uri()
# Session API
def get_session(
database: str | None = None,
default_access_mode: str | None = None,
) -> AbstractContextManager:
"""Return a session against the right backend.
- `database` names starting with `db-tmp-scan-` always go to ingest.
- No database name → ingest (used for CREATE / DROP DATABASE admin ops).
- Any other name → sink.
"""
if _is_ingest_database(database) or database is None:
return ingest.get_session(
database=database, default_access_mode=default_access_mode
)
return sink_module.get_backend().get_session(
database=database, default_access_mode=default_access_mode
)
def execute_read_query(
database: str,
cypher: str,
parameters: dict[str, Any] | None = None,
) -> neo4j.graph.Graph:
"""Read-only query against the sink."""
return sink_module.get_backend().execute_read_query(database, cypher, parameters)
def create_database(database: str) -> None:
"""Create a database. Temp DBs always land on ingest (Neo4j).
On the Neo4j sink, tenant DBs also route to ingest because both drivers
connect to the same Neo4j cluster. On the Neptune sink, tenant DB creates
are no-ops.
"""
if _is_ingest_database(database):
ingest.create_database(database)
return
sink_module.get_backend().create_database(database)
def drop_database(database: str) -> None:
"""Drop a database. Mirrors `create_database` routing."""
if _is_ingest_database(database):
ingest.drop_database(database)
return
sink_module.get_backend().drop_database(database)
def drop_subgraph(database: str, provider_id: str) -> int:
return sink_module.get_backend().drop_subgraph(database, provider_id)
def has_provider_data(database: str, provider_id: str) -> bool:
return sink_module.get_backend().has_provider_data(database, provider_id)
def clear_cache(database: str) -> None:
if _is_ingest_database(database):
ingest.clear_cache(database)
return
sink_module.get_backend().clear_cache(database)
# Name helper
def get_database_name(entity_id: str | UUID, temporary: bool = False) -> str:
prefix = "tmp-scan" if temporary else "tenant"
return f"db-{prefix}-{str(entity_id).lower()}"
api/src/backend/api/attack_paths/ingest/__init__.py
+29
View File
@@ -0,0 +1,29 @@
"""Cartography ingest layer.
Public surface for the per-scan Neo4j temp database driver. Implementation
lives in `api.attack_paths.ingest.driver`.
"""
from api.attack_paths.ingest.driver import (
clear_cache,
close_driver,
create_database,
drop_database,
get_driver,
get_session,
get_uri,
init_driver,
run_cypher,
)
__all__ = [
"clear_cache",
"close_driver",
"create_database",
"drop_database",
"get_driver",
"get_session",
"get_uri",
"init_driver",
"run_cypher",
]
api/src/backend/api/attack_paths/ingest/driver.py
+187
View File
@@ -0,0 +1,187 @@
"""Cartography ingest driver: per-scan throw-away Neo4j database.
Cartography writes each scan's graph into a throw-away Neo4j database named
`db-tmp-scan-{scan_uuid}`. This is always Neo4j, regardless of the configured
sink: Neptune is single-database and cannot host per-scan throw-away
databases. This module owns the Neo4j driver used for those temp DBs and the
admin ops they need (CREATE / DROP DATABASE).
"""
import atexit
import logging
import threading
from collections.abc import Iterator
from contextlib import contextmanager
from typing import Any
import neo4j
import neo4j.exceptions
from api.attack_paths.retryable_session import RetryableSession
from config.env import env
from django.conf import settings
logging.getLogger("neo4j").setLevel(logging.ERROR)
logging.getLogger("neo4j").propagate = False
SERVICE_UNAVAILABLE_MAX_RETRIES = env.int(
"ATTACK_PATHS_SERVICE_UNAVAILABLE_MAX_RETRIES", default=3
)
CONN_ACQUISITION_TIMEOUT = env.int("NEO4J_CONN_ACQUISITION_TIMEOUT", default=15)
# TCP connect timeout, ordered below the acquisition timeout so an unreachable
# host can't pin a worker on a temp-DB op longer than this.
CONNECTION_TIMEOUT = env.int("NEO4J_CONNECTION_TIMEOUT", default=5)
MAX_CONNECTION_LIFETIME = env.int("NEO4J_MAX_CONNECTION_LIFETIME", default=7200)
MAX_CONNECTION_POOL_SIZE = env.int("NEO4J_MAX_CONNECTION_POOL_SIZE", default=50)
_driver: neo4j.Driver | None = None
_lock = threading.Lock()
def _neo4j_config() -> dict:
return settings.DATABASES["neo4j"]
def get_uri() -> str:
"""Bolt URI for the Neo4j temp (ingest) database. Always Neo4j."""
config = _neo4j_config()
host = config["HOST"]
port = config["PORT"]
if not host or not port:
raise RuntimeError(
"NEO4J_HOST / NEO4J_PORT must be set to use the attack-paths "
"temp database. Workers require Neo4j env even when the sink is Neptune."
)
return f"bolt://{host}:{port}"
def init_driver() -> neo4j.Driver:
"""Initialize the temp-database Neo4j driver. Idempotent."""
global _driver
if _driver is not None:
return _driver
with _lock:
if _driver is None:
config = _neo4j_config()
_driver = neo4j.GraphDatabase.driver(
get_uri(),
auth=(config["USER"], config["PASSWORD"]),
keep_alive=True,
max_connection_lifetime=MAX_CONNECTION_LIFETIME,
connection_timeout=CONNECTION_TIMEOUT,
connection_acquisition_timeout=CONN_ACQUISITION_TIMEOUT,
max_connection_pool_size=MAX_CONNECTION_POOL_SIZE,
)
# Best-effort connectivity check: a Neo4j that is down at boot must
# not crash the worker. The driver reconnects lazily on first use.
try:
_driver.verify_connectivity()
except Exception:
logging.warning(
"Neo4j temp-database unreachable at init; continuing with a "
"lazily-reconnecting driver",
exc_info=True,
)
atexit.register(close_driver)
return _driver
def get_driver() -> neo4j.Driver:
return init_driver()
def close_driver() -> None:
global _driver
with _lock:
if _driver is not None:
try:
_driver.close()
finally:
_driver = None
@contextmanager
def get_session(
database: str | None = None,
default_access_mode: str | None = None,
) -> Iterator[RetryableSession]:
"""Session against the Neo4j temp-database cluster. Used for temp DB sessions
and for admin operations (CREATE / DROP DATABASE) when `database` is None."""
from api.attack_paths.database import (
ClientStatementException,
GraphDatabaseQueryException,
WriteQueryNotAllowedException,
)
READ_EXCEPTION_CODES = [
"Neo.ClientError.Statement.AccessMode",
"Neo.ClientError.Procedure.ProcedureNotFound",
]
CLIENT_STATEMENT_EXCEPTION_PREFIX = "Neo.ClientError.Statement."
session_wrapper: RetryableSession | None = None
try:
session_wrapper = RetryableSession(
session_factory=lambda: get_driver().session(
database=database, default_access_mode=default_access_mode
),
max_retries=SERVICE_UNAVAILABLE_MAX_RETRIES,
)
yield session_wrapper
except neo4j.exceptions.Neo4jError as exc:
if (
default_access_mode == neo4j.READ_ACCESS
and exc.code
and exc.code in READ_EXCEPTION_CODES
):
raise WriteQueryNotAllowedException(
message="Read query not allowed", code=READ_EXCEPTION_CODES[0]
)
message = exc.message if exc.message is not None else str(exc)
if exc.code and exc.code.startswith(CLIENT_STATEMENT_EXCEPTION_PREFIX):
raise ClientStatementException(message=message, code=exc.code)
raise GraphDatabaseQueryException(message=message, code=exc.code)
finally:
if session_wrapper is not None:
session_wrapper.close()
def create_database(database: str) -> None:
"""Create a database on the Neo4j cluster. Used for temp scan DBs."""
with get_session() as session:
session.run("CREATE DATABASE $database IF NOT EXISTS", {"database": database})
def drop_database(database: str) -> None:
"""Drop a database on the Neo4j cluster. Used for temp scan DBs."""
with get_session() as session:
session.run(f"DROP DATABASE `{database}` IF EXISTS DESTROY DATA")
def clear_cache(database: str) -> None:
"""Best-effort cache clear for a Neo4j database."""
from api.attack_paths.database import GraphDatabaseQueryException
try:
with get_session(database) as session:
session.run("CALL db.clearQueryCaches()")
except GraphDatabaseQueryException as exc:
logging.warning(f"Failed to clear query cache for database `{database}`: {exc}")
def run_cypher(
database: str | None,
cypher: str,
parameters: dict[str, Any] | None = None,
) -> Any:
"""Execute Cypher directly without the context manager. Thin helper."""
with get_session(database) as session:
return session.run(cypher, parameters or {})
api/src/backend/api/attack_paths/queries/__init__.py
+4 -5
View File
@@ -1,12 +1,11 @@
from api.attack_paths.queries.types import (
AttackPathsQueryDefinition,
AttackPathsQueryParameterDefinition,
)
from api.attack_paths.queries.registry import (
get_queries_for_provider,
get_query_by_id,
)
from api.attack_paths.queries.types import (
AttackPathsQueryDefinition,
AttackPathsQueryParameterDefinition,
)
__all__ = [
"AttackPathsQueryDefinition",
api/src/backend/api/attack_paths/queries/aws.py
+1098 -1397
View File
File diff suppressed because it is too large Load Diff
api/src/backend/api/attack_paths/queries/aws_deprecated.py
+3819
View File
File diff suppressed because it is too large Load Diff
api/src/backend/api/attack_paths/queries/registry.py
+44 -9
View File
@@ -1,13 +1,14 @@
from api.attack_paths.queries.types import AttackPathsQueryDefinition
from api.attack_paths.queries.aws import AWS_QUERIES
# TODO: drop after Neptune cutover
from api.attack_paths.queries.aws_deprecated import AWS_DEPRECATED_QUERIES
from api.attack_paths.queries.types import AttackPathsQueryDefinition
# Query definitions organized by provider
# Query definitions for scans synced with the current schema.
_QUERY_DEFINITIONS: dict[str, list[AttackPathsQueryDefinition]] = {
"aws": AWS_QUERIES,
}
# Flat lookup by query ID for O(1) access
_QUERIES_BY_ID: dict[str, AttackPathsQueryDefinition] = {
definition.id: definition
for definitions in _QUERY_DEFINITIONS.values()
@@ -15,11 +16,45 @@ _QUERIES_BY_ID: dict[str, AttackPathsQueryDefinition] = {
}
def get_queries_for_provider(provider: str) -> list[AttackPathsQueryDefinition]:
"""Get all attack path queries for a specific provider."""
return _QUERY_DEFINITIONS.get(provider, [])
# TODO: drop after Neptune cutover
#
# Query definitions for pre-cutover scans (`AttackPathsScan.is_migrated=False`)
# whose graph data was written under the previous schema. Both maps expose the
# same query IDs so the API contract is identical regardless of which set is
# routed to.
_DEPRECATED_QUERY_DEFINITIONS: dict[str, list[AttackPathsQueryDefinition]] = {
"aws": AWS_DEPRECATED_QUERIES,
}
_DEPRECATED_QUERIES_BY_ID: dict[str, AttackPathsQueryDefinition] = {
definition.id: definition
for definitions in _DEPRECATED_QUERY_DEFINITIONS.values()
for definition in definitions
}
def get_query_by_id(query_id: str) -> AttackPathsQueryDefinition | None:
"""Get a specific attack path query by its ID."""
return _QUERIES_BY_ID.get(query_id)
def get_queries_for_provider(
provider: str,
is_migrated: bool = True,
) -> list[AttackPathsQueryDefinition]:
"""Get all attack path queries for a provider.
`is_migrated` selects the catalog: True for scans synced with the current
schema, False for pre-cutover scans still using the legacy graph shape.
# TODO: drop the `is_migrated` parameter after Neptune cutover
"""
catalog = _QUERY_DEFINITIONS if is_migrated else _DEPRECATED_QUERY_DEFINITIONS
return catalog.get(provider, [])
def get_query_by_id(
query_id: str,
is_migrated: bool = True,
) -> AttackPathsQueryDefinition | None:
"""Get a specific attack path query by ID.
`is_migrated` selects the catalog (see `get_queries_for_provider`).
# TODO: drop the `is_migrated` parameter after Neptune cutover
"""
by_id = _QUERIES_BY_ID if is_migrated else _DEPRECATED_QUERIES_BY_ID
return by_id.get(query_id)
api/src/backend/api/attack_paths/retryable_session.py
-1
View File
@@ -1,5 +1,4 @@
import logging
from collections.abc import Callable
from typing import Any
api/src/backend/api/attack_paths/sink/__init__.py
+28
View File
@@ -0,0 +1,28 @@
"""Attack-paths sink database layer.
The sink is the persistent store where attack-paths graphs live after a scan
finishes. Currently selectable between Neo4j (OSS / local dev default) and
AWS Neptune (hosted dev/staging/prod). Backend is picked by the
`ATTACK_PATHS_SINK_DATABASE` setting at process init.
This package exposes the public factory API; the implementation lives in
`api.attack_paths.sink.factory`.
"""
from api.attack_paths.sink.factory import (
SinkBackend,
close,
get_backend,
get_backend_for_name,
get_backend_for_scan,
init,
)
__all__ = [
"SinkBackend",
"close",
"get_backend",
"get_backend_for_name",
"get_backend_for_scan",
"init",
]
api/src/backend/api/attack_paths/sink/base.py
+92
View File
@@ -0,0 +1,92 @@
"""Protocol every sink backend must implement."""
from contextlib import AbstractContextManager
from typing import Any, Protocol
import neo4j
class SinkDatabase(Protocol):
"""Contract for the persistent attack-paths graph store.
The `database` argument is an opaque identifier passed through from the
legacy `database.py` API surface. On Neo4j it is the per-tenant database
name (e.g. `db-tenant-{uuid}`). On Neptune it is ignored (the cluster
has a single graph, and isolation is label-based).
"""
def init(self) -> None: ...
def close(self) -> None: ...
def verify_connectivity(self) -> None:
"""Raise if the backend the API read path uses is unreachable.
Neo4j verifies its single driver. Neptune verifies the reader
driver (the endpoint the API serves reads from); on single-endpoint
clusters the reader aliases the writer, so that path is covered too.
Used by the readiness probe; must not block longer than the caller's
probe budget.
"""
...
def get_session(
self,
database: str | None = None,
default_access_mode: str | None = None,
) -> AbstractContextManager: ...
def execute_read_query(
self,
database: str,
cypher: str,
parameters: dict[str, Any] | None = None,
) -> neo4j.graph.Graph: ...
def create_database(self, database: str) -> None: ...
def drop_database(self, database: str) -> None: ...
def drop_subgraph(self, database: str, provider_id: str) -> int: ...
def has_provider_data(self, database: str, provider_id: str) -> bool: ...
def clear_cache(self, database: str) -> None: ...
def ensure_sync_indexes(self, database: str) -> None:
"""Create any index needed for the sync write path.
Called once at the start of each provider sync; must be idempotent.
Neo4j creates a `_provider_element_id` index on `_ProviderResource`;
Neptune is a no-op (its `~id` lookup needs no index).
"""
...
def write_nodes(
self,
database: str,
labels: str,
rows: list[dict[str, Any]],
) -> None:
"""Upsert a batch of nodes into the sink.
`labels` is a pre-rendered Cypher label string ready to drop after
the node variable (e.g. `` `AWSUser`:`_ProviderResource`:`_Tenant_x` ``).
Each row carries `provider_element_id` and `props`.
"""
...
def write_relationships(
self,
database: str,
rel_type: str,
provider_id: str,
rows: list[dict[str, Any]],
) -> None:
"""Upsert a batch of relationships into the sink.
Each row carries `start_element_id`, `end_element_id`,
`provider_element_id` and `props`. `rel_type` is the relationship
type (already a valid Cypher identifier).
"""
...
api/src/backend/api/attack_paths/sink/factory.py
+134
View File
@@ -0,0 +1,134 @@
"""Sink backend factory and process-wide handle cache.
Picks the active backend from `settings.ATTACK_PATHS_SINK_DATABASE` at first
use, holds the active backend plus any secondary backends needed to serve
scans written under the previous configuration, and tears them all down on
process shutdown. Imported via `from api.attack_paths import sink as
sink_module`.
"""
import threading
from enum import StrEnum, auto
from api.attack_paths.sink.base import SinkDatabase
from api.models import AttackPathsScan
from django.conf import settings
# Backend names
class SinkBackend(StrEnum):
NEO4J = auto()
NEPTUNE = auto()
# Backend cache
_backend: SinkDatabase | None = None
_secondary_backends: dict[SinkBackend, SinkDatabase] = {}
_lock = threading.Lock()
def _resolve_setting() -> SinkBackend:
raw = settings.ATTACK_PATHS_SINK_DATABASE.lower()
try:
return SinkBackend(raw)
except ValueError:
valid = sorted(b.value for b in SinkBackend)
raise RuntimeError(
f"ATTACK_PATHS_SINK_DATABASE must be one of {valid}; got {raw!r}"
)
def _build_backend(name: SinkBackend) -> SinkDatabase:
if name is SinkBackend.NEO4J:
from api.attack_paths.sink.neo4j import Neo4jSink
return Neo4jSink()
if name is SinkBackend.NEPTUNE:
from api.attack_paths.sink.neptune import NeptuneSink
return NeptuneSink()
raise RuntimeError(f"Unknown sink backend {name!r}")
# Lifecycle
def init(name: SinkBackend | str | None = None) -> SinkDatabase:
"""Initialize the configured sink backend. Idempotent."""
global _backend
if _backend is not None:
return _backend
with _lock:
if _backend is None:
resolved = SinkBackend(name) if name else _resolve_setting()
backend = _build_backend(resolved)
backend.init()
_backend = backend
return _backend
def close() -> None:
"""Close the active backend and every cached secondary backend."""
global _backend
with _lock:
backends = [
b for b in (_backend, *_secondary_backends.values()) if b is not None
]
_backend = None
_secondary_backends.clear()
for backend in backends:
try:
backend.close()
except Exception: # pragma: no cover - best-effort
pass
def get_backend() -> SinkDatabase:
"""Return the active sink. Initializes on first call."""
return init()
# Per-scan routing
def get_backend_for_scan(scan: AttackPathsScan) -> SinkDatabase:
"""Route reads by the sink that stores this scan's graph."""
raw_backend = getattr(scan, "sink_backend", SinkBackend.NEO4J.value)
if not isinstance(raw_backend, str):
raw_backend = SinkBackend.NEO4J.value
return get_backend_for_name(raw_backend)
def get_backend_for_name(name: SinkBackend | str) -> SinkDatabase:
"""Return the backend named by persisted scan metadata."""
resolved = SinkBackend(name)
if resolved is _resolve_setting():
return get_backend()
return _build_backend_cached(resolved)
def _build_backend_cached(name: SinkBackend) -> SinkDatabase:
# TODO: drop after Neptune cutover
# Needed only during cutover to serve Neo4j-written scans from a Neptune-
# configured API pod (and vice versa). Once every scan is on Neptune,
# `get_backend_for_scan` becomes a one-liner returning `get_backend()`.
if name in _secondary_backends:
return _secondary_backends[name]
with _lock:
if name not in _secondary_backends:
backend = _build_backend(name)
backend.init()
_secondary_backends[name] = backend
return _secondary_backends[name]
api/src/backend/api/attack_paths/sink/neo4j.py
+454
View File
@@ -0,0 +1,454 @@
"""Neo4j sink implementation.
Owns a Neo4j driver independent from the staging driver. On OSS and local dev
this is the only sink; on hosted deployments it runs only as a legacy read
path while phase-1 drains tenant DBs.
"""
import atexit
import logging
import threading
import time
from collections.abc import Iterator
from contextlib import AbstractContextManager, contextmanager
from typing import Any
import neo4j
import neo4j.exceptions
from api.attack_paths.retryable_session import RetryableSession
from api.attack_paths.sink.base import SinkDatabase
from config.env import env
from django.conf import settings
logging.getLogger("neo4j").setLevel(logging.ERROR)
logging.getLogger("neo4j").propagate = False
logger = logging.getLogger(__name__)
SERVICE_UNAVAILABLE_MAX_RETRIES = env.int(
"ATTACK_PATHS_SERVICE_UNAVAILABLE_MAX_RETRIES", default=3
)
READ_QUERY_TIMEOUT_SECONDS = env.int(
"ATTACK_PATHS_READ_QUERY_TIMEOUT_SECONDS", default=30
)
CONN_ACQUISITION_TIMEOUT = env.int("NEO4J_CONN_ACQUISITION_TIMEOUT", default=15)
# TCP connect timeout, ordered below the acquisition timeout so an unreachable
# host can't pin a request or the readiness probe longer than this.
CONNECTION_TIMEOUT = env.int("NEO4J_CONNECTION_TIMEOUT", default=5)
MAX_CONNECTION_LIFETIME = env.int("NEO4J_MAX_CONNECTION_LIFETIME", default=7200)
MAX_CONNECTION_POOL_SIZE = env.int("NEO4J_MAX_CONNECTION_POOL_SIZE", default=50)
READ_EXCEPTION_CODES = [
"Neo.ClientError.Statement.AccessMode",
"Neo.ClientError.Procedure.ProcedureNotFound",
]
CLIENT_STATEMENT_EXCEPTION_PREFIX = "Neo.ClientError.Statement."
DATABASE_NOT_FOUND_CODE = "Neo.ClientError.Database.DatabaseNotFound"
class Neo4jSink(SinkDatabase):
"""Neo4j-backed sink. Multi-database cluster; tenant isolation is physical."""
def __init__(self) -> None:
self._driver: neo4j.Driver | None = None
self._lock = threading.Lock()
self._atexit_registered = False
# Driver
def _config(self) -> dict:
return settings.DATABASES["neo4j"]
def _uri(self) -> str:
cfg = self._config()
host = cfg["HOST"]
port = cfg["PORT"]
if not host or not port:
raise RuntimeError(
"NEO4J_HOST / NEO4J_PORT must be set when ATTACK_PATHS_SINK_DATABASE=neo4j"
)
return f"bolt://{host}:{port}"
def init(self) -> neo4j.Driver:
if self._driver is not None:
return self._driver
with self._lock:
if self._driver is None:
cfg = self._config()
self._driver = neo4j.GraphDatabase.driver(
self._uri(),
auth=(cfg["USER"], cfg["PASSWORD"]),
keep_alive=True,
max_connection_lifetime=MAX_CONNECTION_LIFETIME,
connection_timeout=CONNECTION_TIMEOUT,
connection_acquisition_timeout=CONN_ACQUISITION_TIMEOUT,
max_connection_pool_size=MAX_CONNECTION_POOL_SIZE,
)
# Eager connectivity check is best-effort:
# A Neo4j that is down at boot must not crash the process, same degradation model as Postgres
# The driver reconnects lazily on first use
# /health/ready surfaces the outage until it recovers
try:
self._driver.verify_connectivity()
except Exception:
logger.warning(
"Neo4j sink unreachable at init; continuing with a lazily-reconnecting driver",
exc_info=True,
)
if not self._atexit_registered:
atexit.register(self.close)
self._atexit_registered = True
return self._driver
def _get_driver(self) -> neo4j.Driver:
return self.init()
def verify_connectivity(self) -> None:
self._get_driver().verify_connectivity()
def close(self) -> None:
with self._lock:
if self._driver is not None:
try:
self._driver.close()
finally:
self._driver = None
# Sessions
@contextmanager
def get_session(
self,
database: str | None = None,
default_access_mode: str | None = None,
) -> Iterator[RetryableSession]:
from api.attack_paths.database import (
ClientStatementException,
GraphDatabaseQueryException,
WriteQueryNotAllowedException,
)
session_wrapper: RetryableSession | None = None
try:
session_wrapper = RetryableSession(
session_factory=lambda: self._get_driver().session(
database=database, default_access_mode=default_access_mode
),
max_retries=SERVICE_UNAVAILABLE_MAX_RETRIES,
)
yield session_wrapper
except neo4j.exceptions.Neo4jError as exc:
if (
default_access_mode == neo4j.READ_ACCESS
and exc.code
and exc.code in READ_EXCEPTION_CODES
):
raise WriteQueryNotAllowedException(
message="Read query not allowed", code=READ_EXCEPTION_CODES[0]
)
message = exc.message if exc.message is not None else str(exc)
if exc.code and exc.code.startswith(CLIENT_STATEMENT_EXCEPTION_PREFIX):
raise ClientStatementException(message=message, code=exc.code)
raise GraphDatabaseQueryException(message=message, code=exc.code)
finally:
if session_wrapper is not None:
session_wrapper.close()
# Operations
def execute_read_query(
self,
database: str,
cypher: str,
parameters: dict[str, Any] | None = None,
) -> neo4j.graph.Graph:
with self.get_session(
database, default_access_mode=neo4j.READ_ACCESS
) as session:
def _run(tx: neo4j.ManagedTransaction) -> neo4j.graph.Graph:
result = tx.run(
cypher, parameters or {}, timeout=READ_QUERY_TIMEOUT_SECONDS
)
return result.graph()
return session.execute_read(_run)
def create_database(self, database: str) -> None:
with self.get_session() as session:
session.run(
"CREATE DATABASE $database IF NOT EXISTS", {"database": database}
)
def drop_database(self, database: str) -> None:
with self.get_session() as session:
session.run(f"DROP DATABASE `{database}` IF EXISTS DESTROY DATA")
def drop_subgraph(self, database: str, provider_id: str) -> int:
"""Delete all nodes for a provider from a tenant database, batched.
Deletes relationships then nodes in batches (not `DETACH DELETE`) so a
dense provider's graph cannot exceed Neo4j's transaction memory limit.
Silently returns 0 if the database doesn't exist.
"""
from api.attack_paths.database import GraphDatabaseQueryException
from tasks.jobs.attack_paths.config import (
BATCH_SIZE,
PROVIDER_RESOURCE_LABEL,
get_provider_label,
)
provider_label = get_provider_label(provider_id)
deleted_nodes = 0
deleted_relationships = 0
relationship_batches = 0
node_batches = 0
drop_t0 = time.perf_counter()
logger.info(
"Dropping provider graph from Neo4j sink database %s "
"(provider=%s, provider_label=%s)",
database,
provider_id,
provider_label,
)
try:
logger.info(
"Opening Neo4j sink session for provider graph drop "
"(database=%s, provider=%s)",
database,
provider_id,
)
with self.get_session(database) as session:
logger.info(
"Opened Neo4j sink session for provider graph drop "
"(database=%s, provider=%s)",
database,
provider_id,
)
# Phase 1: delete relationships incident to provider nodes in
# batches. The undirected pattern matches an edge between two
# provider nodes from both ends, so `DISTINCT r` dedupes it to
# delete a full batch of unique relationships each round.
deleted_count = 1
while deleted_count > 0:
next_batch = relationship_batches + 1
logger.info(
"Deleting relationship batch from Neo4j sink database %s "
"(provider=%s, batch=%s, total_rels=%s, elapsed=%.3fs)",
database,
provider_id,
next_batch,
deleted_relationships,
time.perf_counter() - drop_t0,
)
result = session.run(
f"""
MATCH (:`{provider_label}`)-[r]-()
WITH DISTINCT r LIMIT $batch_size
DELETE r
RETURN COUNT(r) AS deleted_rels_count
""",
{"batch_size": BATCH_SIZE},
)
deleted_count = result.single().get("deleted_rels_count", 0)
if deleted_count > 0:
relationship_batches += 1
deleted_relationships += deleted_count
logger.info(
"Deleted relationship batch from Neo4j sink database %s "
"(provider=%s, batch=%s, deleted_rels=%s, "
"total_rels=%s, elapsed=%.3fs)",
database,
provider_id,
relationship_batches,
deleted_count,
deleted_relationships,
time.perf_counter() - drop_t0,
)
# Phase 2: delete the now relationship-free nodes in batches.
deleted_count = 1
while deleted_count > 0:
next_batch = node_batches + 1
logger.info(
"Deleting node batch from Neo4j sink database %s "
"(provider=%s, batch=%s, total_nodes=%s, elapsed=%.3fs)",
database,
provider_id,
next_batch,
deleted_nodes,
time.perf_counter() - drop_t0,
)
result = session.run(
f"""
MATCH (n:{PROVIDER_RESOURCE_LABEL}:`{provider_label}`)
WITH n LIMIT $batch_size
DELETE n
RETURN COUNT(n) AS deleted_nodes_count
""",
{"batch_size": BATCH_SIZE},
)
deleted_count = result.single().get("deleted_nodes_count", 0)
if deleted_count > 0:
node_batches += 1
deleted_nodes += deleted_count
logger.info(
"Deleted node batch from Neo4j sink database %s "
"(provider=%s, batch=%s, deleted_nodes=%s, "
"total_nodes=%s, elapsed=%.3fs)",
database,
provider_id,
node_batches,
deleted_count,
deleted_nodes,
time.perf_counter() - drop_t0,
)
except GraphDatabaseQueryException as exc:
if exc.code == DATABASE_NOT_FOUND_CODE:
logger.info(
"Skipped provider graph drop from Neo4j sink database %s "
"(provider=%s, reason=database_not_found, elapsed=%.3fs)",
database,
provider_id,
time.perf_counter() - drop_t0,
)
return 0
raise
logger.info(
"Finished dropping provider graph from Neo4j sink database %s "
"(provider=%s, relationship_batches=%s, deleted_rels=%s, "
"node_batches=%s, deleted_nodes=%s, elapsed=%.3fs)",
database,
provider_id,
relationship_batches,
deleted_relationships,
node_batches,
deleted_nodes,
time.perf_counter() - drop_t0,
)
return deleted_nodes
def has_provider_data(self, database: str, provider_id: str) -> bool:
from api.attack_paths.database import GraphDatabaseQueryException
from tasks.jobs.attack_paths.config import (
PROVIDER_RESOURCE_LABEL,
get_provider_label,
)
provider_label = get_provider_label(provider_id)
query = (
f"MATCH (n:{PROVIDER_RESOURCE_LABEL}:`{provider_label}`) RETURN 1 LIMIT 1"
)
try:
with self.get_session(
database, default_access_mode=neo4j.READ_ACCESS
) as session:
result = session.run(query)
return result.single() is not None
except GraphDatabaseQueryException as exc:
if exc.code == DATABASE_NOT_FOUND_CODE:
return False
raise
def clear_cache(self, database: str) -> None:
from api.attack_paths.database import GraphDatabaseQueryException
try:
with self.get_session(database) as session:
session.run("CALL db.clearQueryCaches()")
except GraphDatabaseQueryException as exc:
logger.warning(
f"Failed to clear query cache for database `{database}`: {exc}"
)
# Sync write path
def ensure_sync_indexes(self, database: str) -> None:
"""Create the `_provider_element_id` lookup index on `_ProviderResource`.
Every synced node carries the `_ProviderResource` label, so a single
index covers both node-upserts and relationship endpoint MATCHes.
Without this index the rel sync degrades to a label scan per row and
large provider syncs become unworkable.
"""
from tasks.jobs.attack_paths.config import (
PROVIDER_ELEMENT_ID_PROPERTY,
PROVIDER_RESOURCE_LABEL,
)
query = (
f"CREATE INDEX provider_element_id_idx IF NOT EXISTS "
f"FOR (n:`{PROVIDER_RESOURCE_LABEL}`) "
f"ON (n.`{PROVIDER_ELEMENT_ID_PROPERTY}`)"
)
with self.get_session(database) as session:
session.run(query).consume()
def write_nodes(
self,
database: str,
labels: str,
rows: list[dict[str, Any]],
) -> None:
if not rows:
return
from tasks.jobs.attack_paths.config import (
PROVIDER_ELEMENT_ID_PROPERTY,
PROVIDER_RESOURCE_LABEL,
)
query = f"""
UNWIND $rows AS row
MERGE (n:`{PROVIDER_RESOURCE_LABEL}` {{`{PROVIDER_ELEMENT_ID_PROPERTY}`: row.provider_element_id}})
SET n:{labels}
SET n += row.props
"""
with self.get_session(database) as session:
session.run(query, {"rows": rows}).consume()
def write_relationships(
self,
database: str,
rel_type: str,
provider_id: str,
rows: list[dict[str, Any]],
) -> None:
if not rows:
return
from tasks.jobs.attack_paths.config import (
PROVIDER_ELEMENT_ID_PROPERTY,
PROVIDER_RESOURCE_LABEL,
get_provider_label,
)
provider_label = get_provider_label(provider_id)
query = f"""
UNWIND $rows AS row
MATCH (s:`{PROVIDER_RESOURCE_LABEL}`:`{provider_label}` {{`{PROVIDER_ELEMENT_ID_PROPERTY}`: row.start_element_id}})
MATCH (t:`{PROVIDER_RESOURCE_LABEL}`:`{provider_label}` {{`{PROVIDER_ELEMENT_ID_PROPERTY}`: row.end_element_id}})
MERGE (s)-[r:`{rel_type}` {{`{PROVIDER_ELEMENT_ID_PROPERTY}`: row.provider_element_id}}]->(t)
SET r += row.props
"""
with self.get_session(database) as session:
session.run(query, {"rows": rows}).consume()
# For compatibility with test harnesses that patch the concrete driver
def get_driver(self) -> neo4j.Driver:
return self._get_driver()
# Helper for tests / external callers that want a writer session specifically
def get_read_session(
sink: Neo4jSink, database: str
) -> AbstractContextManager[RetryableSession]:
return sink.get_session(database, default_access_mode=neo4j.READ_ACCESS)
api/src/backend/api/attack_paths/sink/neptune.py
+524
View File
@@ -0,0 +1,524 @@
"""AWS Neptune sink implementation.
Dual Bolt drivers: one against the writer endpoint for workers, one against
the reader endpoint for the API read path. If `NEPTUNE_READER_ENDPOINT` is
unset the reader falls back to the writer driver so single-node clusters work.
Neptune is single-database. The `database` argument on the SinkDatabase
protocol is ignored; tenant / provider isolation is enforced by labels that
the sync step already writes on every node (see tasks/jobs/attack_paths/sync.py).
SigV4 auth lives at the bottom of this file as `neptune_auth_provider`. The
neo4j driver invokes the returned callable on each token refresh.
"""
import atexit
import datetime
import json
import logging
import threading
import time
from collections.abc import Callable, Iterator
from contextlib import contextmanager
from typing import Any
from urllib.parse import urlsplit
import neo4j
import neo4j.exceptions
from api.attack_paths.retryable_session import RetryableSession
from api.attack_paths.sink.base import SinkDatabase
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
from botocore.session import Session as BotoSession
from config.env import env
from django.conf import settings
from neo4j.auth_management import AuthManagers, ExpiringAuth
logging.getLogger("neo4j").setLevel(logging.ERROR)
logging.getLogger("neo4j").propagate = False
logger = logging.getLogger(__name__)
SERVICE_UNAVAILABLE_MAX_RETRIES = env.int(
"ATTACK_PATHS_SERVICE_UNAVAILABLE_MAX_RETRIES", default=3
)
READ_QUERY_TIMEOUT_SECONDS = env.int(
"ATTACK_PATHS_READ_QUERY_TIMEOUT_SECONDS", default=30
)
# Neptune serverless cold-start can be >30s; give the driver room
CONN_ACQUISITION_TIMEOUT = env.int("NEPTUNE_CONN_ACQUISITION_TIMEOUT", default=60)
# TCP connect timeout, ordered below the acquisition timeout so an unreachable
# endpoint can't pin a request or the readiness probe longer than this. Kept
# generous: cold-start delays query execution, not the socket connect.
CONNECTION_TIMEOUT = env.int("NEPTUNE_CONNECTION_TIMEOUT", default=10)
# Roll connections hourly so SigV4 rotations and cert refreshes don't strand long-lived pool entries
MAX_CONNECTION_LIFETIME = env.int("NEPTUNE_MAX_CONNECTION_LIFETIME", default=3600)
MAX_CONNECTION_POOL_SIZE = env.int("NEPTUNE_MAX_CONNECTION_POOL_SIZE", default=50)
READ_EXCEPTION_CODES = [
"Neo.ClientError.Statement.AccessMode",
"Neo.ClientError.Procedure.ProcedureNotFound",
]
CLIENT_STATEMENT_EXCEPTION_PREFIX = "Neo.ClientError.Statement."
# Refresh 60s before the 5-minute SigV4 window closes
SIGV4_TOKEN_LIFETIME_MINUTES = 4
class NeptuneSink(SinkDatabase):
"""Neptune-backed sink. Single database; isolation is label-based."""
def __init__(self) -> None:
self._writer: neo4j.Driver | None = None
self._reader: neo4j.Driver | None = None
self._lock = threading.Lock()
self._atexit_registered = False
# Config
def _config(self) -> dict:
return settings.DATABASES["neptune"]
def _bolt_uri(self, endpoint: str, port: str) -> str:
return f"bolt+s://{endpoint}:{port}"
def _https_url(self, endpoint: str, port: str) -> str:
return f"https://{endpoint}:{port}"
def _build_driver(self, endpoint: str) -> neo4j.Driver:
cfg = self._config()
port = cfg["PORT"]
region = cfg["REGION"]
if not endpoint or not region:
raise RuntimeError(
"NEPTUNE_WRITER_ENDPOINT and AWS_REGION must be set when "
"ATTACK_PATHS_SINK_DATABASE=neptune"
)
return neo4j.GraphDatabase.driver(
self._bolt_uri(endpoint, port),
auth=AuthManagers.bearer(
neptune_auth_provider(region, self._https_url(endpoint, port))
),
keep_alive=True,
max_connection_lifetime=MAX_CONNECTION_LIFETIME,
connection_timeout=CONNECTION_TIMEOUT,
connection_acquisition_timeout=CONN_ACQUISITION_TIMEOUT,
max_connection_pool_size=MAX_CONNECTION_POOL_SIZE,
max_transaction_retry_time=0,
)
# Lifecycle
def init(self) -> None:
if self._writer is not None:
return
with self._lock:
if self._writer is None:
cfg = self._config()
writer_endpoint = cfg["WRITER_ENDPOINT"]
reader_endpoint = cfg["READER_ENDPOINT"] or writer_endpoint
# Eager connectivity checks are best-effort
# A Neptune that is down at boot must not crash the process, same degradation model as Postgres
# Drivers reconnect lazily on first use
# /health/ready surfaces the outage until it recovers
self._writer = self._build_driver(writer_endpoint)
self._verify_best_effort(self._writer, "writer")
if reader_endpoint == writer_endpoint:
self._reader = self._writer
else:
self._reader = self._build_driver(reader_endpoint)
self._verify_best_effort(self._reader, "reader")
if not self._atexit_registered:
atexit.register(self.close)
self._atexit_registered = True
def close(self) -> None:
with self._lock:
# `Driver.close()` is idempotent, so closing the same driver twice
# (when reader aliases writer on single-endpoint configs) is safe
for driver in (self._reader, self._writer):
if driver is None:
continue
try:
driver.close()
except Exception: # pragma: no cover - best-effort
pass
self._writer = None
self._reader = None
# Sessions
def _get_writer(self) -> neo4j.Driver:
self.init()
assert self._writer is not None
return self._writer
def _get_reader(self) -> neo4j.Driver:
self.init()
assert self._reader is not None
return self._reader
@staticmethod
def _verify_best_effort(driver: neo4j.Driver, role: str) -> None:
try:
driver.verify_connectivity()
except Exception:
logger.warning(
"Neptune %s endpoint unreachable at init; continuing with a lazily-reconnecting driver",
role,
exc_info=True,
)
def verify_connectivity(self) -> None:
# The API read path uses the reader driver
# On single-endpoint clusters it aliases the writer, so this also covers the writer
# A writer-only outage is a workers' concern (no HTTP probe there) and deliberately does not fail API readiness
self._get_reader().verify_connectivity()
@contextmanager
def get_session(
self,
database: str | None = None, # noqa: ARG002 - ignored on Neptune
default_access_mode: str | None = None,
) -> Iterator[RetryableSession]:
from api.attack_paths.database import (
ClientStatementException,
GraphDatabaseQueryException,
WriteQueryNotAllowedException,
)
driver = (
self._get_reader()
if default_access_mode == neo4j.READ_ACCESS
else self._get_writer()
)
session_wrapper: RetryableSession | None = None
try:
session_wrapper = RetryableSession(
session_factory=lambda: driver.session(
default_access_mode=default_access_mode
),
max_retries=SERVICE_UNAVAILABLE_MAX_RETRIES,
)
yield session_wrapper
except neo4j.exceptions.Neo4jError as exc:
if (
default_access_mode == neo4j.READ_ACCESS
and exc.code
and exc.code in READ_EXCEPTION_CODES
):
raise WriteQueryNotAllowedException(
message="Read query not allowed", code=READ_EXCEPTION_CODES[0]
)
message = exc.message if exc.message is not None else str(exc)
if exc.code and exc.code.startswith(CLIENT_STATEMENT_EXCEPTION_PREFIX):
raise ClientStatementException(message=message, code=exc.code)
raise GraphDatabaseQueryException(message=message, code=exc.code)
finally:
if session_wrapper is not None:
session_wrapper.close()
# Operations
def execute_read_query(
self,
database: str, # noqa: ARG002 - ignored on Neptune
cypher: str,
parameters: dict[str, Any] | None = None,
) -> neo4j.graph.Graph:
with self.get_session(default_access_mode=neo4j.READ_ACCESS) as session:
def _run(tx: neo4j.ManagedTransaction) -> neo4j.graph.Graph:
result = tx.run(
cypher, parameters or {}, timeout=READ_QUERY_TIMEOUT_SECONDS
)
return result.graph()
return session.execute_read(_run)
def create_database(self, database: str) -> None: # noqa: ARG002
# Neptune clusters are single-database; there is nothing to create.
return None
def drop_database(self, database: str) -> None: # noqa: ARG002
# Neptune clusters are single-database; there is nothing to drop.
return None
def drop_subgraph(self, database: str, provider_id: str) -> int: # noqa: ARG002
"""Delete a provider's subgraph in two bounded phases.
Neptune write transactions are capped at ~2 minutes. A naive
`DETACH DELETE` on a label-scanned batch grows unbounded with graph
density (one node can drag thousands of relationships into the same
transaction). Instead:
1. Delete relationships incident to provider nodes, one fixed-size
batch per transaction.
2. Delete the now-orphaned nodes, one fixed-size batch per transaction.
Each transaction does work proportional to `batch_size`, never to the
graph's branching factor.
"""
from tasks.jobs.attack_paths.config import (
BATCH_SIZE,
PROVIDER_RESOURCE_LABEL,
get_provider_label,
)
provider_label = get_provider_label(provider_id)
deleted_relationships = 0
relationship_batches = 0
node_batches = 0
drop_t0 = time.perf_counter()
logger.info(
"Dropping provider graph from Neptune sink "
"(provider=%s, provider_label=%s)",
provider_id,
provider_label,
)
logger.info(
"Opening Neptune writer session for provider graph drop (provider=%s)",
provider_id,
)
with self.get_session() as session:
logger.info(
"Opened Neptune writer session for provider graph drop (provider=%s)",
provider_id,
)
while True:
next_batch = relationship_batches + 1
logger.info(
"Deleting relationship batch from Neptune sink "
"(provider=%s, batch=%s, total_rels=%s, elapsed=%.3fs)",
provider_id,
next_batch,
deleted_relationships,
time.perf_counter() - drop_t0,
)
result = session.run(
f"""
MATCH (:`{provider_label}`)-[r]-()
WITH DISTINCT r LIMIT $batch_size
DELETE r
RETURN COUNT(r) AS deleted_rels_count
""",
{"batch_size": BATCH_SIZE},
)
record = result.single()
deleted_rels = (record["deleted_rels_count"] if record else 0) or 0
if deleted_rels == 0:
break
relationship_batches += 1
deleted_relationships += deleted_rels
logger.info(
"Deleted relationship batch from Neptune sink "
"(provider=%s, batch=%s, deleted_rels=%s, total_rels=%s, "
"elapsed=%.3fs)",
provider_id,
relationship_batches,
deleted_rels,
deleted_relationships,
time.perf_counter() - drop_t0,
)
deleted_nodes = 0
while True:
next_batch = node_batches + 1
logger.info(
"Deleting node batch from Neptune sink "
"(provider=%s, batch=%s, total_nodes=%s, elapsed=%.3fs)",
provider_id,
next_batch,
deleted_nodes,
time.perf_counter() - drop_t0,
)
result = session.run(
f"""
MATCH (n:`{PROVIDER_RESOURCE_LABEL}`:`{provider_label}`)
WITH n LIMIT $batch_size
DELETE n
RETURN COUNT(n) AS deleted_nodes_count
""",
{"batch_size": BATCH_SIZE},
)
record = result.single()
deleted = (record["deleted_nodes_count"] if record else 0) or 0
if deleted == 0:
break
node_batches += 1
deleted_nodes += deleted
logger.info(
"Deleted node batch from Neptune sink "
"(provider=%s, batch=%s, deleted_nodes=%s, total_nodes=%s, "
"elapsed=%.3fs)",
provider_id,
node_batches,
deleted,
deleted_nodes,
time.perf_counter() - drop_t0,
)
logger.info(
"Finished dropping provider graph from Neptune sink "
"(provider=%s, relationship_batches=%s, deleted_rels=%s, "
"node_batches=%s, deleted_nodes=%s, elapsed=%.3fs)",
provider_id,
relationship_batches,
deleted_relationships,
node_batches,
deleted_nodes,
time.perf_counter() - drop_t0,
)
return deleted_nodes
def has_provider_data(self, database: str, provider_id: str) -> bool: # noqa: ARG002
from tasks.jobs.attack_paths.config import (
PROVIDER_RESOURCE_LABEL,
get_provider_label,
)
provider_label = get_provider_label(provider_id)
query = (
f"MATCH (n:{PROVIDER_RESOURCE_LABEL}:`{provider_label}`) RETURN 1 LIMIT 1"
)
with self.get_session(default_access_mode=neo4j.READ_ACCESS) as session:
result = session.run(query)
return result.single() is not None
def clear_cache(self, database: str) -> None: # noqa: ARG002
# Neptune has no user-facing cache-clear procedure; no-op.
return None
# Sync write path
def ensure_sync_indexes(self, database: str) -> None: # noqa: ARG002
# Neptune routes node and relationship lookups through `~id`, which is the cluster's primary key
# No additional index is needed or supported
return None
def write_nodes(
self,
database: str, # noqa: ARG002
labels: str,
rows: list[dict[str, Any]],
) -> None:
if not rows:
return
from tasks.jobs.attack_paths.config import (
PROVIDER_ELEMENT_ID_PROPERTY,
PROVIDER_RESOURCE_LABEL,
)
# MERGE on `~id` is the documented and engine-optimized idempotent
# upsert pattern for Neptune openCypher. The label inside the MERGE
# matters: Neptune assigns a default `vertex` label to any node
# created without an explicit one, so we pin `_ProviderResource`
# (which every synced node carries anyway) at MERGE-time. Additional
# labels are added after
#
# We also write `_provider_element_id` as a regular property so
# non-sync code (drop_subgraph, query helpers) keeps a stable contract
# that doesn't know about `~id`
query = f"""
UNWIND $rows AS row
MERGE (n:`{PROVIDER_RESOURCE_LABEL}` {{`~id`: row.provider_element_id}})
SET n:{labels}
SET n += row.props
SET n.`{PROVIDER_ELEMENT_ID_PROPERTY}` = row.provider_element_id
"""
with self.get_session() as session:
session.run(query, {"rows": rows}).consume()
def write_relationships(
self,
database: str, # noqa: ARG002
rel_type: str,
provider_id: str, # noqa: ARG002 - encoded in start/end `~id` already
rows: list[dict[str, Any]],
) -> None:
if not rows:
return
from tasks.jobs.attack_paths.config import PROVIDER_ELEMENT_ID_PROPERTY
# `id(n) = $value` is Neptune's parameterized fast path; both endpoint
# MATCHes resolve in O(1) via the system `~id`, so per-row work stays
# bounded regardless of batch size
query = f"""
UNWIND $rows AS row
MATCH (s) WHERE id(s) = row.start_element_id
MATCH (e) WHERE id(e) = row.end_element_id
MERGE (s)-[r:`{rel_type}` {{`{PROVIDER_ELEMENT_ID_PROPERTY}`: row.provider_element_id}}]->(e)
SET r += row.props
"""
with self.get_session() as session:
session.run(query, {"rows": rows}).consume()
# Test helpers
def get_writer(self) -> neo4j.Driver:
return self._get_writer()
def get_reader(self) -> neo4j.Driver:
return self._get_reader()
# SigV4 auth provider
class _NeptuneAuthToken(neo4j.Auth):
"""Neo4j Auth backed by a SigV4-signed GET to `/opencypher`."""
def __init__(self, region: str, url: str) -> None:
session = BotoSession()
credentials = session.get_credentials()
if credentials is None:
raise RuntimeError(
"No AWS credentials available for Neptune SigV4 signing. "
"Ensure the boto3 credential chain can resolve."
)
credentials = credentials.get_frozen_credentials()
request = AWSRequest(method="GET", url=url + "/opencypher")
# SigV4 canonical Host must carry the real `host:port`
# Neptune runs on a non-default port (8182), so `.hostname` would drop it and break signing
request.headers.add_header("Host", urlsplit(url).netloc)
SigV4Auth(credentials, "neptune-db", region).add_auth(request)
auth_obj = {
header: request.headers[header]
for header in (
"Authorization",
"X-Amz-Date",
"X-Amz-Security-Token",
"Host",
)
if header in request.headers
}
auth_obj["HttpMethod"] = "GET"
super().__init__("basic", "username", json.dumps(auth_obj))
def neptune_auth_provider(region: str, https_url: str) -> Callable[[], ExpiringAuth]:
"""Return a callable the neo4j driver can invoke to refresh credentials."""
def _provider() -> ExpiringAuth:
token = _NeptuneAuthToken(region, https_url)
expires_at = (
datetime.datetime.now(datetime.UTC)
+ datetime.timedelta(minutes=SIGV4_TOKEN_LIFETIME_MINUTES)
).timestamp()
return ExpiringAuth(auth=token, expires_at=expires_at)
return _provider
api/src/backend/api/attack_paths/views_helpers.py
+36 -20
View File
@@ -1,12 +1,11 @@
import logging
from typing import Any, Iterable
from collections.abc import Iterable
from typing import Any
import neo4j
from rest_framework.exceptions import APIException, PermissionDenied, ValidationError
from api.attack_paths import database as graph_database, AttackPathsQueryDefinition
from api.attack_paths import AttackPathsQueryDefinition
from api.attack_paths import database as graph_database
from api.attack_paths import sink as sink_module
from api.attack_paths.cypher_sanitizer import (
inject_provider_label,
validate_custom_query,
@@ -16,7 +15,10 @@ from api.attack_paths.queries.schema import (
RAW_SCHEMA_URL,
get_cartography_schema_query,
)
from api.models import AttackPathsScan
from config.custom_logging import BackendLogger
from config.env import env
from rest_framework.exceptions import APIException, PermissionDenied, ValidationError
from tasks.jobs.attack_paths.config import (
INTERNAL_LABELS,
INTERNAL_PROPERTIES,
@@ -27,6 +29,10 @@ from tasks.jobs.attack_paths.config import (
logger = logging.getLogger(BackendLogger.API)
def _custom_query_timeout_ms() -> int:
return env.int("ATTACK_PATHS_READ_QUERY_TIMEOUT_SECONDS", default=30) * 1000
# Predefined query helpers
@@ -103,13 +109,13 @@ def execute_query(
definition: AttackPathsQueryDefinition,
parameters: dict[str, Any],
provider_id: str,
scan: AttackPathsScan,
) -> dict[str, Any]:
try:
graph = graph_database.execute_read_query(
database=database_name,
cypher=definition.cypher,
parameters=parameters,
)
# TODO: drop after Neptune cutover
# Route reads by the scan row's recorded sink, not by current settings.
backend = sink_module.get_backend_for_scan(scan)
graph = backend.execute_read_query(database_name, definition.cypher, parameters)
return _serialize_graph(graph, provider_id)
except graph_database.WriteQueryNotAllowedException:
@@ -143,22 +149,31 @@ def execute_custom_query(
database_name: str,
cypher: str,
provider_id: str,
scan: AttackPathsScan,
) -> dict[str, Any]:
# Defense-in-depth for custom queries:
# 1. neo4j.READ_ACCESS — prevents mutations at the driver level
# 2. inject_provider_label() — regex-based label injection scopes node patterns
# 3. _serialize_graph() — post-query filter drops nodes without the provider label
# 1. `neo4j.READ_ACCESS` — prevents mutations at the driver level
# 2. `inject_provider_label()` — regex-based label injection scopes node patterns
# 3. `_serialize_graph()` — post-query filter drops nodes without the provider label
# 4. `USING QUERY:TIMEOUTMILLISECONDS` on Neptune — server-side runaway cutoff
#
# Layer 2 is best-effort (regex can't fully parse Cypher);
# layer 3 is the safety net that guarantees provider isolation.
validate_custom_query(cypher)
cypher = inject_provider_label(cypher, provider_id)
# TODO: drop after Neptune cutover
backend = sink_module.get_backend_for_scan(scan)
# Neptune enforces a cluster-level query timeout; prepending the hint
# makes the limit explicit and matches the client-side read timeout.
# Applies only when the scan's graph lives in Neptune.
if getattr(scan, "sink_backend", None) == "neptune":
timeout_ms = _custom_query_timeout_ms()
cypher = f"USING QUERY:TIMEOUTMILLISECONDS {timeout_ms}\n{cypher}"
try:
graph = graph_database.execute_read_query(
database=database_name,
cypher=cypher,
)
graph = backend.execute_read_query(database_name, cypher, None)
serialized = _serialize_graph(graph, provider_id)
return _truncate_graph(serialized)
@@ -181,10 +196,11 @@ def execute_custom_query(
def get_cartography_schema(
database_name: str, provider_id: str
database_name: str, provider_id: str, scan: AttackPathsScan
) -> dict[str, str] | None:
try:
with graph_database.get_session(
backend = sink_module.get_backend_for_scan(scan)
with backend.get_session(
database_name, default_access_mode=neo4j.READ_ACCESS
) as session:
result = session.run(get_cartography_schema_query(provider_id))
api/src/backend/api/authentication.py
+73 -14
View File
@@ -1,18 +1,19 @@
from typing import Optional, Tuple
from math import isfinite
from uuid import UUID
from api.db_router import MainRouter
from api.models import TenantAPIKey, TenantAPIKeyManager
from cryptography.fernet import InvalidToken
from django.core.exceptions import ObjectDoesNotExist
from django.utils import timezone
from drf_simple_apikey.backends import APIKeyAuthentication as BaseAPIKeyAuth
from drf_simple_apikey.crypto import get_crypto
from drf_simple_apikey.settings import package_settings
from rest_framework.authentication import BaseAuthentication
from rest_framework.exceptions import AuthenticationFailed
from rest_framework.request import Request
from rest_framework_simplejwt.authentication import JWTAuthentication
from api.db_router import MainRouter
from api.models import TenantAPIKey, TenantAPIKeyManager
class TenantAPIKeyAuthentication(BaseAPIKeyAuth):
model = TenantAPIKey
@@ -23,18 +24,49 @@ class TenantAPIKeyAuthentication(BaseAPIKeyAuth):
def _authenticate_credentials(self, request, key):
"""
Override to use admin connection, bypassing RLS during authentication.
Delegates to parent after temporarily routing model queries to admin DB.
"""
# Temporarily point the model's manager to admin database
original_objects = self.model.objects
self.model.objects = self.model.objects.using(MainRouter.admin_db)
try:
payload = self.key_crypto.decrypt(key)
except ValueError:
raise AuthenticationFailed("Invalid API Key.")
if not isinstance(payload, dict):
raise AuthenticationFailed("Invalid API Key.")
payload_pk = payload.get("_pk")
payload_exp = payload.get("_exp")
if (
not isinstance(payload_pk, str)
or isinstance(payload_exp, bool)
or not isinstance(payload_exp, (int, float))
or not isfinite(payload_exp)
):
raise AuthenticationFailed("Invalid API Key.")
try:
# Call parent method which will now use admin database
return super()._authenticate_credentials(request, key)
finally:
# Restore original manager
self.model.objects = original_objects
api_key_pk = UUID(payload_pk)
except ValueError:
raise AuthenticationFailed("Invalid API Key.")
if payload_exp < timezone.now().timestamp():
raise AuthenticationFailed("API Key has already expired.")
try:
api_key = self.model.objects.using(MainRouter.admin_db).get(id=api_key_pk)
except ObjectDoesNotExist:
raise AuthenticationFailed("No entity matching this api key.")
if api_key.revoked:
raise AuthenticationFailed("This API Key has been revoked.")
client_ip = request.META.get(package_settings.IP_ADDRESS_HEADER)
if api_key.blacklisted_ips and client_ip in api_key.blacklisted_ips:
raise AuthenticationFailed("Access denied from blacklisted IP.")
if api_key.whitelisted_ips and client_ip not in api_key.whitelisted_ips:
raise AuthenticationFailed("Access restricted to specific IP addresses.")
return api_key.entity, key
def authenticate(self, request: Request):
prefixed_key = self.get_key(request)
@@ -81,7 +113,7 @@ class CombinedJWTOrAPIKeyAuthentication(BaseAuthentication):
jwt_auth = JWTAuthentication()
api_key_auth = TenantAPIKeyAuthentication()
def authenticate(self, request: Request) -> Optional[Tuple[object, dict]]:
def authenticate(self, request: Request) -> tuple[object, dict] | None:
auth_header = request.headers.get("Authorization", "")
# Prioritize JWT authentication if both are present
@@ -93,3 +125,30 @@ class CombinedJWTOrAPIKeyAuthentication(BaseAuthentication):
# Default fallback
return self.jwt_auth.authenticate(request)
class SSEAuthentication(CombinedJWTOrAPIKeyAuthentication):
"""JWT/API-Key auth that also accepts `?access_token=<jwt>`.
Browser `EventSource` is the only widely available SSE client API
and it cannot set the `Authorization` header (its constructor takes
only a URL and `withCredentials`). To keep browser SSE clients on
the same auth stack as the rest of the API, SSE endpoints additionally
accept a JWT via the `?access_token=<jwt>` query parameter — the
standard parameter name defined in RFC 6750 Section 2.3 for bearer tokens.
"""
def authenticate(self, request: Request):
auth_header = request.headers.get("Authorization", "")
if auth_header:
return super().authenticate(request)
raw_token = request.query_params.get("access_token")
if not raw_token:
# No header and no query token — let the default path raise
# the canonical AuthenticationFailed via the parent class.
return super().authenticate(request)
validated_token = self.jwt_auth.get_validated_token(raw_token)
user = self.jwt_auth.get_user(validated_token)
return user, validated_token
api/src/backend/api/base_views.py
+6 -7
View File
@@ -1,3 +1,9 @@
from api.authentication import CombinedJWTOrAPIKeyAuthentication
from api.db_router import MainRouter, reset_read_db_alias, set_read_db_alias
from api.db_utils import POSTGRES_USER_VAR, rls_transaction
from api.filters import CustomDjangoFilterBackend
from api.models import Role, UserRoleRelationship
from api.rbac.permissions import HasPermissions
from django.conf import settings
from django.db import transaction
from rest_framework import permissions
@@ -8,13 +14,6 @@ from rest_framework.response import Response
from rest_framework_json_api import filters
from rest_framework_json_api.views import ModelViewSet
from api.authentication import CombinedJWTOrAPIKeyAuthentication
from api.db_router import MainRouter, reset_read_db_alias, set_read_db_alias
from api.db_utils import POSTGRES_USER_VAR, rls_transaction
from api.filters import CustomDjangoFilterBackend
from api.models import Role, UserRoleRelationship
from api.rbac.permissions import HasPermissions
class BaseViewSet(ModelViewSet):
authentication_classes = [CombinedJWTOrAPIKeyAuthentication]
api/src/backend/api/compliance.py
+111 -38
View File
@@ -1,12 +1,26 @@
import logging
import threading
from collections.abc import Iterable, Mapping
from api.models import Provider
from prowler.config.config import get_available_compliance_frameworks
from prowler.lib.check.compliance_models import Compliance
from prowler.lib.check.compliance_models import (
get_bulk_compliance_frameworks_universal,
)
from prowler.lib.check.models import CheckMetadata
logger = logging.getLogger(__name__)
AVAILABLE_COMPLIANCE_FRAMEWORKS = {}
# Per-process readiness flags for the background compliance warm-up.
# `STARTED` is set as soon as warming begins (only happens under Gunicorn via
# the post_fork hook); `WARMED` is set when it finishes. The attributes
# endpoint checks both: it returns 503 only while warming is in progress.
# Under `runserver` warming never runs, so `STARTED` stays clear and the
# endpoint keeps lazy-loading as before.
COMPLIANCE_WARMING_STARTED = threading.Event()
COMPLIANCE_WARMED = threading.Event()
class LazyComplianceTemplate(Mapping):
"""Lazy-load compliance templates per provider on first access."""
@@ -95,25 +109,22 @@ PROWLER_CHECKS = LazyChecksMapping()
def get_compliance_frameworks(provider_type: Provider.ProviderChoices) -> list[str]:
"""
Retrieve and cache the list of available compliance frameworks for a specific cloud provider.
"""List compliance framework identifiers available for `provider_type`.
This function lazily loads and caches the available compliance frameworks (e.g., CIS, MITRE, ISO)
for each provider type (AWS, Azure, GCP, etc.) on first access. Subsequent calls for the same
provider will return the cached result.
Includes both per-provider frameworks and universal top-level frameworks
(e.g. ``dora_2022_2554``, ``csa_ccm_4.0``).
Args:
provider_type (Provider.ProviderChoices): The cloud provider type for which to retrieve
available compliance frameworks (e.g., "aws", "azure", "gcp", "m365").
provider_type (Provider.ProviderChoices): The cloud provider type
(e.g., "aws", "azure", "gcp", "m365").
Returns:
list[str]: A list of framework identifiers (e.g., "cis_1.4_aws", "mitre_attack_azure") available
for the given provider.
list[str]: Framework identifiers (e.g., "cis_1.4_aws", "dora_2022_2554").
"""
global AVAILABLE_COMPLIANCE_FRAMEWORKS
if provider_type not in AVAILABLE_COMPLIANCE_FRAMEWORKS:
AVAILABLE_COMPLIANCE_FRAMEWORKS[provider_type] = (
get_available_compliance_frameworks(provider_type)
AVAILABLE_COMPLIANCE_FRAMEWORKS[provider_type] = list(
get_bulk_compliance_frameworks_universal(provider_type).keys()
)
return AVAILABLE_COMPLIANCE_FRAMEWORKS[provider_type]
@@ -140,18 +151,14 @@ def get_prowler_provider_compliance(provider_type: Provider.ProviderChoices) ->
"""
Retrieve the Prowler compliance data for a specified provider type.
This function fetches the compliance frameworks and their associated
requirements for the given cloud provider.
Args:
provider_type (Provider.ProviderChoices): The provider type
(e.g., 'aws', 'azure') for which to retrieve compliance data.
Returns:
dict: A dictionary mapping compliance framework names to their respective
Compliance objects for the specified provider.
dict: Mapping of framework name to `ComplianceFramework` for the provider.
"""
return Compliance.get_bulk(provider_type)
return get_bulk_compliance_frameworks_universal(provider_type)
def _load_provider_assets(provider_type: Provider.ProviderChoices) -> tuple[dict, dict]:
@@ -180,6 +187,56 @@ def _ensure_provider_loaded(provider_type: Provider.ProviderChoices) -> None:
PROWLER_CHECKS._cache[provider_type] = checks
def warm_compliance_caches(
provider_types: Iterable[str] | None = None,
) -> list[str]:
"""
Eagerly populate the per-process compliance caches at server startup.
Moves the cold-cache catalog load off the request thread so the first
request does not trip the Gunicorn worker timeout. Reads only on-disk
metadata (no database access). Each provider is warmed in isolation;
failures are logged and fall back to lazy loading.
Args:
provider_types (Iterable[str] | None): Subset to warm. Defaults to all.
Returns:
list[str]: Provider types that could not be warmed.
"""
if provider_types is None:
provider_types = Provider.ProviderChoices.values
provider_types = list(provider_types)
COMPLIANCE_WARMING_STARTED.set()
logger.info("Compliance cache warm-up started for providers: %s", provider_types)
failed = []
for provider_type in provider_types:
try:
get_compliance_frameworks(provider_type)
_ensure_provider_loaded(provider_type)
# Prowler check loading may sys.exit (SystemExit, not Exception).
except (Exception, SystemExit):
logger.warning(
"Failed to warm compliance caches for provider '%s'; "
"loading lazily on first request",
provider_type,
exc_info=True,
)
failed.append(provider_type)
# Mark as warmed even when some providers failed: a failed provider falls
# back to a single-provider lazy load, which stays under the worker timeout.
COMPLIANCE_WARMED.set()
logger.info(
"Compliance cache warm-up finished (providers warmed: %d, failed: %s)",
len(provider_types) - len(failed),
failed,
)
return failed
def load_prowler_checks(
prowler_compliance, provider_types: Iterable[str] | None = None
):
@@ -210,8 +267,8 @@ def load_prowler_checks(
for compliance_name, compliance_data in prowler_compliance.get(
provider_type, {}
).items():
for requirement in compliance_data.Requirements:
for check in requirement.Checks:
for requirement in compliance_data.requirements:
for check in requirement.checks.get(provider_type, []):
try:
checks[provider_type][check].add(compliance_name)
except KeyError:
@@ -291,24 +348,40 @@ def generate_compliance_overview_template(
requirements_status = {"passed": 0, "failed": 0, "manual": 0}
total_requirements = 0
for requirement in compliance_data.Requirements:
for requirement in compliance_data.requirements:
total_requirements += 1
total_checks = len(requirement.Checks)
checks_dict = {check: None for check in requirement.Checks}
provider_check_list = list(requirement.checks.get(provider_type, []))
total_checks = len(provider_check_list)
checks_dict = dict.fromkeys(provider_check_list)
req_status_val = "MANUAL" if total_checks == 0 else "PASS"
# MITRE attrs are wrapped under `_raw_attributes` by the
# universal adapter — unwrap so consumers see the flat list.
requirement_attributes = requirement.attributes
if (
isinstance(requirement_attributes, dict)
and "_raw_attributes" in requirement_attributes
):
attributes_payload = list(requirement_attributes["_raw_attributes"])
elif isinstance(requirement_attributes, dict):
attributes_payload = (
[dict(requirement_attributes)] if requirement_attributes else []
)
else:
attributes_payload = [
dict(attribute) for attribute in requirement_attributes
]
# Build requirement dictionary
requirement_dict = {
"name": requirement.Name or requirement.Id,
"description": requirement.Description,
"tactics": getattr(requirement, "Tactics", []),
"subtechniques": getattr(requirement, "SubTechniques", []),
"platforms": getattr(requirement, "Platforms", []),
"technique_url": getattr(requirement, "TechniqueURL", ""),
"attributes": [
dict(attribute) for attribute in requirement.Attributes
],
"name": requirement.name or requirement.id,
"description": requirement.description,
"tactics": requirement.tactics or [],
"subtechniques": requirement.sub_techniques or [],
"platforms": requirement.platforms or [],
"technique_url": requirement.technique_url or "",
"attributes": attributes_payload,
"checks": checks_dict,
"checks_status": {
"pass": 0,
@@ -326,15 +399,15 @@ def generate_compliance_overview_template(
requirements_status["passed"] += 1
# Add requirement to compliance requirements
compliance_requirements[requirement.Id] = requirement_dict
compliance_requirements[requirement.id] = requirement_dict
# Build compliance dictionary
compliance_dict = {
"framework": compliance_data.Framework,
"name": compliance_data.Name,
"version": compliance_data.Version,
"framework": compliance_data.framework,
"name": compliance_data.name,
"version": compliance_data.version,
"provider": provider_type,
"description": compliance_data.Description,
"description": compliance_data.description,
"requirements": compliance_requirements,
"requirements_status": requirements_status,
"total_requirements": total_requirements,
api/src/backend/api/db_utils.py
+10 -11
View File
@@ -3,8 +3,14 @@ import secrets
import time
import uuid
from contextlib import contextmanager
from datetime import datetime, timedelta, timezone
from datetime import UTC, datetime, timedelta
from api.db_router import (
READ_REPLICA_ALIAS,
get_read_db_alias,
reset_read_db_alias,
set_read_db_alias,
)
from celery.utils.log import get_task_logger
from config.env import env
from django.conf import settings
@@ -22,13 +28,6 @@ from psycopg2 import sql as psycopg2_sql
from psycopg2.extensions import AsIs, new_type, register_adapter, register_type
from rest_framework_json_api.serializers import ValidationError
from api.db_router import (
READ_REPLICA_ALIAS,
get_read_db_alias,
reset_read_db_alias,
set_read_db_alias,
)
logger = get_task_logger(__name__)
DB_USER = settings.DATABASES["default"]["USER"] if not settings.TESTING else "test"
@@ -170,7 +169,7 @@ def one_week_from_now():
"""
Return a datetime object with a date one week from now.
"""
return datetime.now(timezone.utc) + timedelta(days=7)
return datetime.now(UTC) + timedelta(days=7)
def generate_random_token(length: int = 14, symbols: str | None = None) -> str:
@@ -405,10 +404,10 @@ def _should_create_index_on_partition(
# Unknown month abbreviation, include it to be safe
return True
partition_date = datetime(year, month, 1, tzinfo=timezone.utc)
partition_date = datetime(year, month, 1, tzinfo=UTC)
# Get current month start
now = datetime.now(timezone.utc)
now = datetime.now(UTC)
current_month_start = now.replace(
day=1, hour=0, minute=0, second=0, microsecond=0
)
api/src/backend/api/decorators.py
+3 -4
View File
@@ -1,14 +1,13 @@
import uuid
from functools import wraps
from django.core.exceptions import ObjectDoesNotExist
from django.db import DatabaseError, connection, transaction
from rest_framework_json_api.serializers import ValidationError
from api.db_router import READ_REPLICA_ALIAS
from api.db_utils import POSTGRES_TENANT_VAR, SET_CONFIG_QUERY, rls_transaction
from api.exceptions import ProviderDeletedException
from api.models import Provider, Scan
from django.core.exceptions import ObjectDoesNotExist
from django.db import DatabaseError, connection, transaction
from rest_framework_json_api.serializers import ValidationError
def set_tenant(func=None, *, keep_tenant=False):
api/src/backend/api/exceptions.py
+26
View File
@@ -187,6 +187,32 @@ class UpstreamServiceUnavailableError(APIException):
)
class ComplianceWarmingError(APIException):
"""Compliance catalog is still warming (503 Service Unavailable).
Returned by the compliance attributes endpoint while the per-process
catalog warm-up is in progress, so the request thread never triggers the
slow cold load that would trip the Gunicorn worker timeout.
"""
status_code = status.HTTP_503_SERVICE_UNAVAILABLE
default_detail = (
"Compliance data is still loading. Please try again in a few seconds."
)
default_code = "compliance_warming"
def __init__(self, detail=None):
super().__init__(
detail=[
{
"detail": detail or self.default_detail,
"status": str(self.status_code),
"code": self.default_code,
}
]
)
class UpstreamInternalError(APIException):
"""Unexpected error communicating with provider (500 Internal Server Error).
api/src/backend/api/filters.py
+148 -34
View File
@@ -1,19 +1,4 @@
from datetime import date, datetime, timedelta, timezone
from dateutil.parser import parse
from django.conf import settings
from django.db.models import F, Q
from django_filters.rest_framework import (
BaseInFilter,
BooleanFilter,
CharFilter,
ChoiceFilter,
DateFilter,
FilterSet,
UUIDFilter,
)
from rest_framework_json_api.django_filters.backends import DjangoFilterBackend
from rest_framework_json_api.serializers import ValidationError
from datetime import UTC, date, datetime, timedelta
from api.constants import SEVERITY_ORDER
from api.db_utils import (
@@ -68,6 +53,20 @@ from api.uuid_utils import (
uuid7_start,
)
from api.v1.serializers import TaskBase
from dateutil.parser import parse
from django.conf import settings
from django.db.models import F, Q
from django_filters.rest_framework import (
BaseInFilter,
BooleanFilter,
CharFilter,
ChoiceFilter,
DateFilter,
FilterSet,
UUIDFilter,
)
from rest_framework_json_api.django_filters.backends import DjangoFilterBackend
from rest_framework_json_api.serializers import ValidationError
class CustomDjangoFilterBackend(DjangoFilterBackend):
@@ -102,7 +101,7 @@ class BaseProviderFilter(FilterSet):
"""
Abstract base filter for models with direct FK to Provider.
Provides standard provider_id and provider_type filters.
Provides standard provider_id, provider_type, and provider_groups filters.
Subclasses must define Meta.model.
"""
@@ -116,6 +115,16 @@ class BaseProviderFilter(FilterSet):
choices=Provider.ProviderChoices.choices,
lookup_expr="in",
)
provider_groups = UUIDFilter(
field_name="provider__provider_groups__id",
lookup_expr="exact",
distinct=True,
)
provider_groups__in = UUIDInFilter(
field_name="provider__provider_groups__id",
lookup_expr="in",
distinct=True,
)
class Meta:
abstract = True
@@ -126,7 +135,7 @@ class BaseScanProviderFilter(FilterSet):
"""
Abstract base filter for models with FK to Scan (and Scan has FK to Provider).
Provides standard provider_id and provider_type filters via scan relationship.
Provides standard provider_id, provider_type, and provider_groups filters via scan relationship.
Subclasses must define Meta.model.
"""
@@ -140,6 +149,16 @@ class BaseScanProviderFilter(FilterSet):
choices=Provider.ProviderChoices.choices,
lookup_expr="in",
)
provider_groups = UUIDFilter(
field_name="scan__provider__provider_groups__id",
lookup_expr="exact",
distinct=True,
)
provider_groups__in = UUIDInFilter(
field_name="scan__provider__provider_groups__id",
lookup_expr="in",
distinct=True,
)
class Meta:
abstract = True
@@ -160,6 +179,16 @@ class CommonFindingFilters(FilterSet):
provider_type__in = ChoiceInFilter(
choices=Provider.ProviderChoices.choices, field_name="scan__provider__provider"
)
provider_groups = UUIDFilter(
field_name="scan__provider__provider_groups__id",
lookup_expr="exact",
distinct=True,
)
provider_groups__in = UUIDInFilter(
field_name="scan__provider__provider_groups__id",
lookup_expr="in",
distinct=True,
)
provider_uid = CharFilter(field_name="scan__provider__uid", lookup_expr="exact")
provider_uid__in = CharInFilter(field_name="scan__provider__uid", lookup_expr="in")
provider_uid__icontains = CharFilter(
@@ -330,6 +359,7 @@ class MembershipFilter(FilterSet):
model = Membership
fields = {
"tenant": ["exact"],
"user": ["exact"],
"role": ["exact"],
"date_joined": ["date", "gte", "lte"],
}
@@ -369,6 +399,12 @@ class ProviderFilter(FilterSet):
choices=Provider.ProviderChoices.choices,
lookup_expr="in",
)
provider_groups = UUIDFilter(
field_name="provider_groups__id", lookup_expr="exact", distinct=True
)
provider_groups__in = UUIDInFilter(
field_name="provider_groups__id", lookup_expr="in", distinct=True
)
class Meta:
model = Provider
@@ -394,6 +430,16 @@ class ProviderRelationshipFilterSet(FilterSet):
provider_type__in = ChoiceInFilter(
choices=Provider.ProviderChoices.choices, field_name="provider__provider"
)
provider_groups = UUIDFilter(
field_name="provider__provider_groups__id",
lookup_expr="exact",
distinct=True,
)
provider_groups__in = UUIDInFilter(
field_name="provider__provider_groups__id",
lookup_expr="in",
distinct=True,
)
provider_uid = CharFilter(field_name="provider__uid", lookup_expr="exact")
provider_uid__in = CharInFilter(field_name="provider__uid", lookup_expr="in")
provider_uid__icontains = CharFilter(
@@ -551,12 +597,12 @@ class ResourceFilter(ProviderRelationshipFilterSet):
gte_date = (
parse(self.data.get("updated_at__gte")).date()
if self.data.get("updated_at__gte")
else datetime.now(timezone.utc).date()
else datetime.now(UTC).date()
)
lte_date = (
parse(self.data.get("updated_at__lte")).date()
if self.data.get("updated_at__lte")
else datetime.now(timezone.utc).date()
else datetime.now(UTC).date()
)
if abs(lte_date - gte_date) > timedelta(
@@ -701,9 +747,9 @@ class FindingFilter(CommonFindingFilters):
lte_date = cleaned.get("inserted_at__lte") or exact_date
if gte_date is None:
gte_date = datetime.now(timezone.utc).date()
gte_date = datetime.now(UTC).date()
if lte_date is None:
lte_date = datetime.now(timezone.utc).date()
lte_date = datetime.now(UTC).date()
if abs(lte_date - gte_date) > timedelta(
days=settings.FINDINGS_MAX_DAYS_IN_RANGE
@@ -797,7 +843,7 @@ class FindingFilter(CommonFindingFilters):
def maybe_date_to_datetime(value):
dt = value
if isinstance(value, date):
dt = datetime.combine(value, datetime.min.time(), tzinfo=timezone.utc)
dt = datetime.combine(value, datetime.min.time(), tzinfo=UTC)
return dt
@@ -886,9 +932,9 @@ class FindingGroupFilter(CommonFindingFilters):
lte_date = cleaned.get("inserted_at__lte") or exact_date
if gte_date is None:
gte_date = datetime.now(timezone.utc).date()
gte_date = datetime.now(UTC).date()
if lte_date is None:
lte_date = datetime.now(timezone.utc).date()
lte_date = datetime.now(UTC).date()
if abs(lte_date - gte_date) > timedelta(
days=settings.FINDINGS_MAX_DAYS_IN_RANGE
@@ -930,7 +976,7 @@ class FindingGroupFilter(CommonFindingFilters):
"""Convert date to datetime if needed."""
dt = value
if isinstance(value, date):
dt = datetime.combine(value, datetime.min.time(), tzinfo=timezone.utc)
dt = datetime.combine(value, datetime.min.time(), tzinfo=UTC)
return dt
@@ -1000,6 +1046,16 @@ class FindingGroupSummaryFilter(_CheckTitleToCheckIdMixin, FilterSet):
field_name="provider__provider", choices=Provider.ProviderChoices.choices
)
provider_type__in = CharInFilter(field_name="provider__provider", lookup_expr="in")
provider_groups = UUIDFilter(
field_name="provider__provider_groups__id",
lookup_expr="exact",
distinct=True,
)
provider_groups__in = UUIDInFilter(
field_name="provider__provider_groups__id",
lookup_expr="in",
distinct=True,
)
class Meta:
model = FindingGroupDailySummary
@@ -1034,9 +1090,9 @@ class FindingGroupSummaryFilter(_CheckTitleToCheckIdMixin, FilterSet):
lte_date = cleaned.get("inserted_at__lte") or exact_date
if gte_date is None:
gte_date = datetime.now(timezone.utc).date()
gte_date = datetime.now(UTC).date()
if lte_date is None:
lte_date = datetime.now(timezone.utc).date()
lte_date = datetime.now(UTC).date()
if abs(lte_date - gte_date) > timedelta(
days=settings.FINDINGS_MAX_DAYS_IN_RANGE
@@ -1075,7 +1131,7 @@ class FindingGroupSummaryFilter(_CheckTitleToCheckIdMixin, FilterSet):
def _maybe_date_to_datetime(value):
dt = value
if isinstance(value, date):
dt = datetime.combine(value, datetime.min.time(), tzinfo=timezone.utc)
dt = datetime.combine(value, datetime.min.time(), tzinfo=UTC)
return dt
@@ -1100,6 +1156,16 @@ class LatestFindingGroupSummaryFilter(_CheckTitleToCheckIdMixin, FilterSet):
field_name="provider__provider", choices=Provider.ProviderChoices.choices
)
provider_type__in = CharInFilter(field_name="provider__provider", lookup_expr="in")
provider_groups = UUIDFilter(
field_name="provider__provider_groups__id",
lookup_expr="exact",
distinct=True,
)
provider_groups__in = UUIDInFilter(
field_name="provider__provider_groups__id",
lookup_expr="in",
distinct=True,
)
class Meta:
model = FindingGroupDailySummary
@@ -1115,13 +1181,14 @@ class FindingGroupAggregatedComputedFilter(FilterSet):
STATUS_CHOICES = (
("FAIL", "Fail"),
("PASS", "Pass"),
("MUTED", "Muted"),
("MANUAL", "Manual"),
)
status = ChoiceFilter(method="filter_status", choices=STATUS_CHOICES)
status__in = CharInFilter(method="filter_status_in", lookup_expr="in")
severity = ChoiceFilter(method="filter_severity", choices=SeverityChoices)
severity__in = CharInFilter(method="filter_severity_in", lookup_expr="in")
muted = BooleanFilter(field_name="muted")
include_muted = BooleanFilter(method="filter_include_muted")
def filter_status(self, queryset, name, value):
@@ -1198,7 +1265,7 @@ class FindingGroupAggregatedComputedFilter(FilterSet):
if value is True:
return queryset
# include_muted=false: exclude fully-muted groups
return queryset.exclude(fail_count=0, pass_count=0, muted_count__gt=0)
return queryset.exclude(muted=True)
class ProviderSecretFilter(FilterSet):
@@ -1278,12 +1345,19 @@ class RoleFilter(FilterSet):
}
class ComplianceOverviewFilter(FilterSet):
class ComplianceOverviewFilter(BaseScanProviderFilter):
"""
Keep provider filters in the schema while runtime filtering resolves scans first.
Compliance overview provider filters are applied to the latest completed scans
in the viewset, then this filterset handles the remaining compliance fields.
"""
inserted_at = DateFilter(field_name="inserted_at", lookup_expr="date")
scan_id = UUIDFilter(field_name="scan_id", required=True)
scan_id = UUIDFilter(field_name="scan_id")
region = CharFilter(field_name="region")
class Meta:
class Meta(BaseScanProviderFilter.Meta):
model = ComplianceRequirementOverview
fields = {
"inserted_at": ["date", "gte", "lte"],
@@ -1304,6 +1378,16 @@ class ScanSummaryFilter(FilterSet):
provider_type__in = ChoiceInFilter(
field_name="scan__provider__provider", choices=Provider.ProviderChoices.choices
)
provider_groups = UUIDFilter(
field_name="scan__provider__provider_groups__id",
lookup_expr="exact",
distinct=True,
)
provider_groups__in = UUIDInFilter(
field_name="scan__provider__provider_groups__id",
lookup_expr="in",
distinct=True,
)
region = CharFilter(field_name="region")
class Meta:
@@ -1327,6 +1411,16 @@ class DailySeveritySummaryFilter(FilterSet):
provider_type__in = ChoiceInFilter(
field_name="provider__provider", choices=Provider.ProviderChoices.choices
)
provider_groups = UUIDFilter(
field_name="provider__provider_groups__id",
lookup_expr="exact",
distinct=True,
)
provider_groups__in = UUIDInFilter(
field_name="provider__provider_groups__id",
lookup_expr="in",
distinct=True,
)
date_from = DateFilter(method="filter_noop")
date_to = DateFilter(method="filter_noop")
@@ -1583,6 +1677,16 @@ class ThreatScoreSnapshotFilter(FilterSet):
choices=Provider.ProviderChoices.choices,
lookup_expr="in",
)
provider_groups = UUIDFilter(
field_name="provider__provider_groups__id",
lookup_expr="exact",
distinct=True,
)
provider_groups__in = UUIDInFilter(
field_name="provider__provider_groups__id",
lookup_expr="in",
distinct=True,
)
compliance_id = CharFilter(field_name="compliance_id", lookup_expr="exact")
compliance_id__in = CharInFilter(field_name="compliance_id", lookup_expr="in")
@@ -1626,6 +1730,16 @@ class ResourceGroupOverviewFilter(FilterSet):
choices=Provider.ProviderChoices.choices,
lookup_expr="in",
)
provider_groups = UUIDFilter(
field_name="scan__provider__provider_groups__id",
lookup_expr="exact",
distinct=True,
)
provider_groups__in = UUIDInFilter(
field_name="scan__provider__provider_groups__id",
lookup_expr="in",
distinct=True,
)
resource_group = CharFilter(field_name="resource_group", lookup_expr="exact")
resource_group__in = CharInFilter(field_name="resource_group", lookup_expr="in")
api/src/backend/api/health.py
+289
View File
@@ -0,0 +1,289 @@
"""Liveness and readiness endpoints following the IETF Health Check Response
Format (draft-inadarei-api-health-check-06).
Liveness reports only process status. Readiness verifies that PostgreSQL,
Valkey and the attack-paths graph store (Neo4j or Neptune, per
``ATTACK_PATHS_SINK_DATABASE``) are reachable and returns per-dependency
detail when any of them is unreachable.
"""
from __future__ import annotations
import logging
import threading
import time
from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import TimeoutError as FuturesTimeoutError
from contextlib import suppress
from datetime import UTC, datetime
from typing import Any
import redis
from config.version import API_VERSION, RELEASE_ID
from django.conf import settings
from django.db import connections
from drf_spectacular.utils import extend_schema
from rest_framework import status
from rest_framework.renderers import JSONRenderer
from rest_framework.response import Response
from rest_framework.throttling import ScopedRateThrottle
from rest_framework.views import APIView
logger = logging.getLogger(__name__)
SERVICE_ID = "prowler-api"
SERVICE_DESCRIPTION = "Prowler API"
# Status vocabulary from the IETF draft (section 3.1).
STATUS_PASS = "pass"
STATUS_FAIL = "fail"
STATUS_WARN = "warn"
# Short socket timeout so a stuck Valkey cannot stall the probe.
VALKEY_PROBE_TIMEOUT_SECONDS = 2
# Probe-scoped budget for the graph database.
# ``Driver.verify_connectivity()`` takes no timeout; its only bound is the
# driver-level ``connection_acquisition_timeout`` (60s on Neptune). The
# probe needs its own budget, independent of the workload driver, so a
# graph-database outage cannot pin a worker thread (and the readiness lock)
# for a minute.
GRAPH_DB_PROBE_TIMEOUT_SECONDS = 5
# Bounded pool that enforces ``GRAPH_DB_PROBE_TIMEOUT_SECONDS``. If the
# graph database is unreachable the probe call blocks until the driver's
# own acquisition timeout fires; we abandon the future after the budget and
# report ``fail``. Orphaned tasks are capped by ``max_workers`` plus the 3s
# readiness cache plus the per-IP throttle, so they cannot pile up: worst
# case during a graph-database outage is every readiness call failing fast
# in ``GRAPH_DB_PROBE_TIMEOUT_SECONDS`` with at most 2 background threads
# stuck for <= the driver acquisition timeout.
_graph_db_probe_executor = ThreadPoolExecutor(
max_workers=2, thread_name_prefix="health-graph-db-probe"
)
# Brief cache window so high-frequency probes (ALB target groups, scrapers)
# do not stampede the actual dependency checks.
CACHE_CONTROL_HEADER = "max-age=3, must-revalidate"
# In-process readiness cache. Caps real dependency hits to roughly
# (gunicorn workers / TTL) per second regardless of incoming RPS or the
# source-IP distribution. Kept in sync with the Cache-Control max-age.
# Access is guarded by a lock so concurrent readers do not race on the
# read-decide-write cycle of the double-checked locking pattern below.
READINESS_CACHE_TTL_SECONDS = 3.0
_readiness_cache: tuple[float, dict[str, Any], int] | None = None
_readiness_cache_lock = threading.Lock()
class HealthJSONRenderer(JSONRenderer):
"""Emits responses with the ``application/health+json`` content type."""
media_type = "application/health+json"
format = "health"
def _now_iso() -> str:
return datetime.now(UTC).isoformat(timespec="milliseconds").replace("+00:00", "Z")
def _measure(name: str, check_fn) -> tuple[dict[str, Any], float]:
"""Time ``check_fn`` and return ``(result, elapsed_ms)``.
``check_fn`` returns ``None`` on success or raises on failure. The full
exception is logged for operator diagnostics under ``name``; the
response payload intentionally omits the error detail to avoid leaking
infrastructure information (DNS names, ports, credentials, certificate
chains) to anonymous clients.
"""
started = time.perf_counter()
try:
check_fn()
except Exception:
elapsed_ms = (time.perf_counter() - started) * 1000
logger.warning("Health probe '%s' failed", name, exc_info=True)
return ({"status": STATUS_FAIL}, elapsed_ms)
elapsed_ms = (time.perf_counter() - started) * 1000
return ({"status": STATUS_PASS}, elapsed_ms)
def _probe_postgres() -> None:
with connections["default"].cursor() as cursor:
cursor.execute("SELECT 1")
cursor.fetchone()
def _probe_valkey() -> None:
client = redis.Redis.from_url(
settings.CELERY_BROKER_URL,
socket_connect_timeout=VALKEY_PROBE_TIMEOUT_SECONDS,
socket_timeout=VALKEY_PROBE_TIMEOUT_SECONDS,
)
try:
if not client.ping():
raise RuntimeError("PING did not return PONG")
finally:
# Best-effort cleanup: a failure releasing the socket (e.g. broken
# connection, half-closed by the server) must not mask the probe
# result. Narrowed to the exception types redis-py and the stdlib
# socket layer can raise on close.
with suppress(redis.RedisError, OSError):
client.close()
def _graph_db_component_id() -> str:
"""Return the active graph database name for the ``componentId`` field."""
return settings.ATTACK_PATHS_SINK_DATABASE.strip().lower()
def _probe_graph_db() -> None:
# Lazy import: avoids pulling attack_paths into the boot import graph
from api.attack_paths.database import verify_connectivity
future = _graph_db_probe_executor.submit(verify_connectivity)
try:
future.result(timeout=GRAPH_DB_PROBE_TIMEOUT_SECONDS)
except FuturesTimeoutError as exc:
# Do not wait for the abandoned task; it ends when the driver's own acquisition timeout fires
future.cancel()
raise TimeoutError(
f"graph-db probe exceeded {GRAPH_DB_PROBE_TIMEOUT_SECONDS}s"
) from exc
def _build_check_entry(
component_id: str,
component_type: str,
result: dict[str, Any],
elapsed_ms: float,
) -> dict[str, Any]:
entry: dict[str, Any] = {
"componentId": component_id,
"componentType": component_type,
"observedValue": round(elapsed_ms, 2),
"observedUnit": "ms",
"status": result["status"],
"time": _now_iso(),
}
if "output" in result:
entry["output"] = result["output"]
return entry
def _aggregate_status(check_entries: list[dict[str, Any]]) -> str:
statuses = {entry["status"] for entry in check_entries}
if STATUS_FAIL in statuses:
return STATUS_FAIL
if STATUS_WARN in statuses:
return STATUS_WARN
return STATUS_PASS
def _base_payload(overall_status: str) -> dict[str, Any]:
return {
"status": overall_status,
"version": API_VERSION,
"releaseId": RELEASE_ID,
"serviceId": SERVICE_ID,
"description": SERVICE_DESCRIPTION,
}
def _readiness_payload() -> tuple[dict[str, Any], int]:
global _readiness_cache
# Lock-free fast path: a stale snapshot still satisfies the freshness
# check correctly because we re-check after acquiring the lock below.
snapshot = _readiness_cache
if (
snapshot is not None
and time.monotonic() - snapshot[0] < READINESS_CACHE_TTL_SECONDS
):
return snapshot[1], snapshot[2]
with _readiness_cache_lock:
# Double-checked locking: another thread may have refreshed while
# we were waiting on the lock.
snapshot = _readiness_cache
if (
snapshot is not None
and time.monotonic() - snapshot[0] < READINESS_CACHE_TTL_SECONDS
):
return snapshot[1], snapshot[2]
graph_db_component_id = _graph_db_component_id()
postgres_result, postgres_ms = _measure("postgres", _probe_postgres)
valkey_result, valkey_ms = _measure("valkey", _probe_valkey)
graph_db_result, graph_db_ms = _measure(graph_db_component_id, _probe_graph_db)
entries = [
_build_check_entry("postgres", "datastore", postgres_result, postgres_ms),
_build_check_entry("valkey", "datastore", valkey_result, valkey_ms),
_build_check_entry(
graph_db_component_id, "datastore", graph_db_result, graph_db_ms
),
]
overall = _aggregate_status(entries)
payload = _base_payload(overall)
payload["checks"] = {
"postgres:responseTime": [entries[0]],
"valkey:responseTime": [entries[1]],
"graphdb:responseTime": [entries[2]],
}
http_status = (
status.HTTP_503_SERVICE_UNAVAILABLE
if overall == STATUS_FAIL
else status.HTTP_200_OK
)
_readiness_cache = (time.monotonic(), payload, http_status)
return payload, http_status
def _health_response(payload: dict[str, Any], http_status: int) -> Response:
response = Response(payload, status=http_status)
response["Cache-Control"] = CACHE_CONTROL_HEADER
return response
@extend_schema(exclude=True)
class LivenessView(APIView):
"""Liveness probe. Always 200 when the process can serve requests.
Dependencies are intentionally not consulted: a failing liveness probe
triggers a container restart, which must not happen for transient
dependency outages. Throttled per-IP so the endpoint cannot be used as
a cheap availability oracle for the process.
"""
authentication_classes: list = []
permission_classes: list = []
renderer_classes = [HealthJSONRenderer]
throttle_classes = [ScopedRateThrottle]
throttle_scope = "health-live"
def get(self, _request, *_args, **_kwargs):
return _health_response(_base_payload(STATUS_PASS), status.HTTP_200_OK)
@extend_schema(exclude=True)
class ReadinessView(APIView):
"""Readiness probe.
Returns 200 when PostgreSQL, Valkey and the attack-paths graph store
all respond, or 503 with per-dependency detail when any of them is
unreachable. Per-IP throttle plus the short in-process result cache cap
the real dependency hits regardless of inbound traffic shape.
"""
authentication_classes: list = []
permission_classes: list = []
renderer_classes = [HealthJSONRenderer]
throttle_classes = [ScopedRateThrottle]
throttle_scope = "health-ready"
def get(self, _request, *_args, **_kwargs):
payload, http_status = _readiness_payload()
return _health_response(payload, http_status)
api/src/backend/api/management/commands/findings.py
+6 -9
View File
@@ -1,11 +1,8 @@
import random
from datetime import datetime, timezone
from datetime import UTC, datetime
from math import ceil
from uuid import uuid4
from django.core.management.base import BaseCommand
from tqdm import tqdm
from api.db_utils import rls_transaction
from api.models import (
Finding,
@@ -16,7 +13,9 @@ from api.models import (
Scan,
StatusChoices,
)
from django.core.management.base import BaseCommand
from prowler.lib.check.models import CheckMetadata
from tqdm import tqdm
class Command(BaseCommand):
@@ -116,7 +115,7 @@ class Command(BaseCommand):
trigger="manual",
state="executing",
progress=0,
started_at=datetime.now(timezone.utc),
started_at=datetime.now(UTC),
)
scan_state = "completed"
@@ -272,10 +271,8 @@ class Command(BaseCommand):
self.stdout.write(self.style.ERROR(f"Failed to populate test data: {e}"))
scan_state = "failed"
finally:
scan.completed_at = datetime.now(timezone.utc)
scan.duration = int(
(datetime.now(timezone.utc) - scan.started_at).total_seconds()
)
scan.completed_at = datetime.now(UTC)
scan.duration = int((datetime.now(UTC) - scan.started_at).total_seconds())
scan.progress = 100
scan.state = scan_state
scan.unique_resource_count = num_resources
api/src/backend/api/management/commands/reconcile_orphan_tasks.py
+58
View File
@@ -0,0 +1,58 @@
from django.core.management.base import BaseCommand
from tasks.jobs.orphan_recovery import reconcile_orphans
class Command(BaseCommand):
help = (
"Recover orphaned allowlisted Celery tasks whose worker is gone and mark "
"other stale task results terminal. Single-flight via a Postgres advisory lock."
)
def add_arguments(self, parser):
parser.add_argument(
"--grace-minutes",
type=int,
default=2,
help="Skip tasks started within this window (worker may still register).",
)
parser.add_argument(
"--max-attempts",
type=int,
default=3,
help="Give up re-running a task after this many recovery attempts; it is then left terminal instead of re-enqueued.",
)
parser.add_argument(
"--dry-run",
action="store_true",
help="Detect and report orphans without revoking or re-enqueuing.",
)
def handle(self, *args, **options):
result = reconcile_orphans(
grace_minutes=options["grace_minutes"],
max_attempts=options["max_attempts"],
dry_run=options["dry_run"],
)
if not result.get("acquired"):
self.stdout.write("Reconcile skipped: another run holds the lock.")
return
if result.get("enabled") is False:
message = (
"Task recovery is disabled (DJANGO_TASK_RECOVERY_ENABLED is off); "
"no orphans were recovered."
)
if result.get("attack_paths") is not None:
message += " Attack-paths stale cleanup still ran."
self.stdout.write(message)
return
self.stdout.write(
self.style.SUCCESS(
"Orphan reconcile complete: "
f"recovered={len(result.get('recovered', []))} "
f"failed={len(result.get('failed', []))} "
f"skipped(in-flight)={len(result.get('skipped', []))}"
)
)
api/src/backend/api/middleware.py
+25
View File
@@ -2,6 +2,31 @@ import logging
import time
from config.custom_logging import BackendLogger
from django.core.handlers.asgi import ASGIRequest
from django.db import connections
class CloseDBConnectionsMiddleware:
"""
Close request-scoped DB connections at the end of each ASGI request.
Under the ASGI worker, connections opened by sync views are not released
by Django's normal request-boundary cleanup, so they accumulate idle until
Postgres runs out of slots. Only ASGI requests are handled; the sync WSGI
test client manages its own connections and must be left alone.
"""
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
try:
return self.get_response(request)
finally:
if isinstance(request, ASGIRequest):
for conn in connections.all(initialized_only=True):
if not conn.in_atomic_block:
conn.close_if_unusable_or_obsolete()
def extract_auth_info(request) -> dict:
api/src/backend/api/migrations/0001_initial.py
+13 -14
View File
@@ -1,26 +1,13 @@
import uuid
from functools import partial
import api.rls
import django.contrib.auth.models
import django.contrib.postgres.indexes
import django.contrib.postgres.search
import django.core.validators
import django.db.models.deletion
import django.utils.timezone
from django.conf import settings
from django.db import migrations, models
from psqlextra.backend.migrations.operations.add_default_partition import (
PostgresAddDefaultPartition,
)
from psqlextra.backend.migrations.operations.create_partitioned_model import (
PostgresCreatePartitionedModel,
)
from psqlextra.manager.manager import PostgresManager
from psqlextra.models.partitioned import PostgresPartitionedModel
from psqlextra.types import PostgresPartitioningMethod
from uuid6 import uuid7
import api.rls
from api.db_utils import (
DB_PROWLER_PASSWORD,
DB_PROWLER_USER,
@@ -53,6 +40,18 @@ from api.models import (
StateChoices,
StatusChoices,
)
from django.conf import settings
from django.db import migrations, models
from psqlextra.backend.migrations.operations.add_default_partition import (
PostgresAddDefaultPartition,
)
from psqlextra.backend.migrations.operations.create_partitioned_model import (
PostgresCreatePartitionedModel,
)
from psqlextra.manager.manager import PostgresManager
from psqlextra.models.partitioned import PostgresPartitionedModel
from psqlextra.types import PostgresPartitioningMethod
from uuid6 import uuid7
DB_NAME = settings.DATABASES["default"]["NAME"]
api/src/backend/api/migrations/0002_token_migrations.py
+1 -2
View File
@@ -1,8 +1,7 @@
from api.db_utils import DB_PROWLER_USER
from django.conf import settings
from django.db import migrations
from api.db_utils import DB_PROWLER_USER
DB_NAME = settings.DATABASES["default"]["NAME"]
api/src/backend/api/migrations/0004_rbac.py
+1 -2
View File
@@ -2,12 +2,11 @@
import uuid
import api.rls
import django.db.models.deletion
from django.conf import settings
from django.db import migrations, models
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0005_rbac_missing_admin_roles.py
+1 -2
View File
@@ -1,6 +1,5 @@
from django.db import migrations
from api.db_router import MainRouter
from django.db import migrations
def create_admin_role(apps, schema_editor):
api/src/backend/api/migrations/0008_daily_scheduled_tasks_update.py
+5 -6
View File
@@ -1,12 +1,11 @@
import json
from datetime import datetime, timedelta, timezone
from datetime import UTC, datetime, timedelta
import django.db.models.deletion
from django.db import migrations, models
from django_celery_beat.models import PeriodicTask
from api.db_utils import rls_transaction
from api.models import Scan, StateChoices
from django.db import migrations, models
from django_celery_beat.models import PeriodicTask
def migrate_daily_scheduled_scan_tasks(apps, schema_editor):
@@ -17,11 +16,11 @@ def migrate_daily_scheduled_scan_tasks(apps, schema_editor):
tenant_id = task_kwargs["tenant_id"]
provider_id = task_kwargs["provider_id"]
current_time = datetime.now(timezone.utc)
current_time = datetime.now(UTC)
scheduled_time_today = datetime.combine(
current_time.date(),
daily_scheduled_scan_task.start_time.time(),
tzinfo=timezone.utc,
tzinfo=UTC,
)
if current_time < scheduled_time_today:
api/src/backend/api/migrations/0013_integrations_enum.py
+1 -2
View File
@@ -2,10 +2,9 @@
from functools import partial
from django.db import migrations
from api.db_utils import IntegrationTypeEnum, PostgresEnumMigration, register_enum
from api.models import Integration
from django.db import migrations
IntegrationTypeEnumMigration = PostgresEnumMigration(
enum_name="integration_type",
api/src/backend/api/migrations/0014_integrations.py
+2 -3
View File
@@ -2,12 +2,11 @@
import uuid
import django.db.models.deletion
from django.db import migrations, models
import api.db_utils
import api.rls
import django.db.models.deletion
from api.rls import RowLevelSecurityConstraint
from django.db import migrations, models
class Migration(migrations.Migration):
api/src/backend/api/migrations/0015_finding_muted.py
+1 -2
View File
@@ -1,8 +1,7 @@
# Generated by Django 5.1.5 on 2025-03-25 11:29
from django.db import migrations, models
import api.db_utils
from django.db import migrations, models
class Migration(migrations.Migration):
api/src/backend/api/migrations/0017_m365_provider.py
+1 -2
View File
@@ -1,8 +1,7 @@
# Generated by Django 5.1.7 on 2025-04-16 08:47
from django.db import migrations
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0018_resource_scan_summaries.py
+1 -2
View File
@@ -2,12 +2,11 @@
import uuid
import api.rls
import django.db.models.deletion
import uuid6
from django.db import migrations, models
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0020_findings_new_performance_indexes_partitions.py
+1 -2
View File
@@ -1,8 +1,7 @@
from functools import partial
from django.db import migrations
from api.db_utils import create_index_on_partitions, drop_index_on_partitions
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0024_findings_uid_index_partitions.py
+1 -2
View File
@@ -1,8 +1,7 @@
from functools import partial
from django.db import migrations
from api.db_utils import create_index_on_partitions, drop_index_on_partitions
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0027_compliance_requirement_overviews.py
+2 -3
View File
@@ -2,12 +2,11 @@
import uuid
import django.db.models.deletion
from django.db import migrations, models
import api.db_utils
import api.rls
import django.db.models.deletion
from api.rls import RowLevelSecurityConstraint
from django.db import migrations, models
class Migration(migrations.Migration):
api/src/backend/api/migrations/0028_findings_check_index_partitions.py
+1 -2
View File
@@ -1,8 +1,7 @@
from functools import partial
from django.db import migrations
from api.db_utils import create_index_on_partitions, drop_index_on_partitions
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0030_lighthouseconfiguration.py
+1 -2
View File
@@ -2,12 +2,11 @@
import uuid
import api.rls
import django.core.validators
import django.db.models.deletion
from django.db import migrations, models
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0032_saml.py
+2 -3
View File
@@ -2,13 +2,12 @@
import uuid
import api.db_utils
import api.rls
import django.db.models.deletion
from django.conf import settings
from django.db import migrations, models
import api.db_utils
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0033_processors_enum.py
+1 -2
View File
@@ -2,10 +2,9 @@
from functools import partial
from django.db import migrations
from api.db_utils import PostgresEnumMigration, ProcessorTypeEnum, register_enum
from api.models import Processor
from django.db import migrations
ProcessorTypeEnumMigration = PostgresEnumMigration(
enum_name="processor_type",
api/src/backend/api/migrations/0034_processors.py
+2 -3
View File
@@ -2,12 +2,11 @@
import uuid
import django.db.models.deletion
from django.db import migrations, models
import api.db_utils
import api.rls
import django.db.models.deletion
from api.rls import RowLevelSecurityConstraint
from django.db import migrations, models
class Migration(migrations.Migration):
api/src/backend/api/migrations/0036_rfm_tenant_finding_index_partitions.py
+1 -2
View File
@@ -1,8 +1,7 @@
from functools import partial
from django.db import migrations
from api.db_utils import create_index_on_partitions, drop_index_on_partitions
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0040_rfm_tenant_resource_index_partitions.py
+1 -2
View File
@@ -1,8 +1,7 @@
from functools import partial
from django.db import migrations
from api.db_utils import create_index_on_partitions, drop_index_on_partitions
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0043_github_provider.py
+1 -2
View File
@@ -1,8 +1,7 @@
# Generated by Django 5.1.7 on 2025-07-09 14:44
from django.db import migrations
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0048_api_key.py
+2 -3
View File
@@ -2,15 +2,14 @@
import uuid
import api.db_utils
import api.rls
import django.core.validators
import django.db.models.deletion
import drf_simple_apikey.models
from django.conf import settings
from django.db import migrations, models
import api.db_utils
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0050_lighthouse_multi_llm.py
+2 -3
View File
@@ -4,15 +4,14 @@ import json
import logging
import uuid
import api.rls
import django.db.models.deletion
from api.db_router import MainRouter
from config.custom_logging import BackendLogger
from cryptography.fernet import Fernet
from django.conf import settings
from django.db import migrations, models
import api.rls
from api.db_router import MainRouter
logger = logging.getLogger(BackendLogger.API)
api/src/backend/api/migrations/0051_oraclecloud_provider.py
+1 -2
View File
@@ -1,8 +1,7 @@
# Generated by Django 5.1.7 on 2025-10-14 00:00
from django.db import migrations
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0052_mute_rules.py
+1 -2
View File
@@ -2,14 +2,13 @@
import uuid
import api.rls
import django.contrib.postgres.fields
import django.core.validators
import django.db.models.deletion
from django.conf import settings
from django.db import migrations, models
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0054_iac_provider.py
+1 -2
View File
@@ -1,8 +1,7 @@
# Generated by Django 5.1.10 on 2025-09-09 09:25
from django.db import migrations
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0055_mongodbatlas_provider.py
+1 -2
View File
@@ -1,8 +1,7 @@
# Generated by Django 5.1.13 on 2025-11-05 08:37
from django.db import migrations
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0057_threatscoresnapshot.py
+1 -2
View File
@@ -2,11 +2,10 @@
import uuid
import api.rls
import django.db.models.deletion
from django.db import migrations, models
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0059_compliance_overview_summary.py
+1 -2
View File
@@ -2,11 +2,10 @@
import uuid
import api.rls
import django.db.models.deletion
from django.db import migrations, models
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0060_attack_surface_overview.py
+1 -2
View File
@@ -2,11 +2,10 @@
import uuid
import api.rls
import django.db.models.deletion
from django.db import migrations, models
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0061_daily_severity_summary.py
+1 -2
View File
@@ -2,11 +2,10 @@
import uuid
import api.rls
import django.db.models.deletion
from django.db import migrations, models
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0062_backfill_daily_severity_summaries.py
+2 -3
View File
@@ -1,10 +1,9 @@
# Generated by Django 5.1.14 on 2025-12-10
from django.db import migrations
from tasks.tasks import backfill_daily_severity_summaries_task
from api.db_router import MainRouter
from api.rls import Tenant
from django.db import migrations
from tasks.tasks import backfill_daily_severity_summaries_task
def trigger_backfill_task(apps, schema_editor):
api/src/backend/api/migrations/0063_scan_category_summary.py
+2 -3
View File
@@ -1,10 +1,9 @@
import uuid
import django.db.models.deletion
from django.db import migrations, models
import api.db_utils
import api.rls
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
api/src/backend/api/migrations/0065_alibabacloud_provider.py
+1 -2
View File
@@ -1,8 +1,7 @@
# Generated by Django migration for Alibaba Cloud provider support
from django.db import migrations
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0066_provider_compliance_score.py
+2 -3
View File
@@ -1,10 +1,9 @@
import uuid
import django.db.models.deletion
from django.db import migrations, models
import api.db_utils
import api.rls
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
api/src/backend/api/migrations/0067_tenant_compliance_summary.py
+1 -2
View File
@@ -1,10 +1,9 @@
import uuid
import api.rls
import django.db.models.deletion
from django.db import migrations, models
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0068_finding_resource_group_scangroupsummary.py
+2 -3
View File
@@ -1,10 +1,9 @@
import uuid
import django.db.models.deletion
from django.db import migrations, models
import api.db_utils
import api.rls
import django.db.models.deletion
from django.db import migrations, models
class Migration(migrations.Migration):
api/src/backend/api/migrations/0070_attack_paths_scan.py
+1 -3
View File
@@ -1,12 +1,10 @@
# Generated by Django 5.1.13 on 2025-11-06 16:20
import api.rls
import django.db.models.deletion
from django.db import migrations, models
from uuid6 import uuid7
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0073_findings_fail_new_index_partitions.py
+1 -2
View File
@@ -1,8 +1,7 @@
from functools import partial
from django.db import migrations
from api.db_utils import create_index_on_partitions, drop_index_on_partitions
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0075_cloudflare_provider.py
+1 -2
View File
@@ -1,8 +1,7 @@
# Generated by Django migration for Cloudflare provider support
from django.db import migrations
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0076_openstack_provider.py
+1 -2
View File
@@ -1,8 +1,7 @@
# Generated by Django migration for OpenStack provider support
from django.db import migrations
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0080_backfill_attack_paths_graph_data_ready.py
+1 -2
View File
@@ -2,9 +2,8 @@
# on different database connections, causing a deadlock when combined with RunPython
# in the same migration.
from django.db import migrations
from api.db_router import MainRouter
from django.db import migrations
def backfill_graph_data_ready(apps, schema_editor):
api/src/backend/api/migrations/0081_finding_group_daily_summary.py
+1 -2
View File
@@ -2,14 +2,13 @@
import uuid
import api.rls
import django.db.models.deletion
from django.contrib.postgres.indexes import GinIndex, OpClass
from django.db import migrations, models
from django.db.models.functions import Upper
from django.utils import timezone
import api.rls
class Migration(migrations.Migration):
dependencies = [
api/src/backend/api/migrations/0082_backfill_finding_group_summaries.py
+2 -3
View File
@@ -1,10 +1,9 @@
# Generated by Django 5.1.14 on 2026-02-02
from django.db import migrations
from tasks.tasks import backfill_finding_group_summaries_task
from api.db_router import MainRouter
from api.rls import Tenant
from django.db import migrations
from tasks.tasks import backfill_finding_group_summaries_task
def trigger_backfill_task(apps, schema_editor):
api/src/backend/api/migrations/0083_image_provider.py
+1 -2
View File
@@ -1,6 +1,5 @@
from django.db import migrations
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0084_googleworkspace_provider.py
+1 -2
View File
@@ -1,6 +1,5 @@
from django.db import migrations
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0086_attack_paths_cleanup_periodic_task.py
-1
View File
@@ -1,6 +1,5 @@
from django.db import migrations
TASK_NAME = "attack-paths-cleanup-stale-scans"
INTERVAL_HOURS = 1
api/src/backend/api/migrations/0087_vercel_provider.py
+1 -2
View File
@@ -1,6 +1,5 @@
from django.db import migrations
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
api/src/backend/api/migrations/0088_finding_group_status_muted_fields.py
+95
View File
@@ -0,0 +1,95 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("api", "0087_vercel_provider"),
]
operations = [
migrations.AddField(
model_name="findinggroupdailysummary",
name="manual_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="pass_muted_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="fail_muted_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="manual_muted_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="muted",
field=models.BooleanField(default=False),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="new_fail_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="new_fail_muted_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="new_pass_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="new_pass_muted_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="new_manual_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="new_manual_muted_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="changed_fail_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="changed_fail_muted_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="changed_pass_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="changed_pass_muted_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="changed_manual_count",
field=models.IntegerField(default=0),
),
migrations.AddField(
model_name="findinggroupdailysummary",
name="changed_manual_muted_count",
field=models.IntegerField(default=0),
),
]
api/src/backend/api/migrations/0089_backfill_finding_group_status_muted.py
+30
View File
@@ -0,0 +1,30 @@
from api.db_router import MainRouter
from api.rls import Tenant
from django.db import migrations
from tasks.tasks import backfill_finding_group_summaries_task
def trigger_backfill_task(apps, schema_editor):
"""
Re-dispatch the finding-group backfill task for every tenant so the new
`manual_count` and `muted` columns added in 0088 get populated from the
last 10 days of completed scans.
The aggregator (`aggregate_finding_group_summaries`) recomputes every
column on each call, so it back-populates the new fields without touching
the existing ones beyond a normal upsert.
"""
tenant_ids = Tenant.objects.using(MainRouter.admin_db).values_list("id", flat=True)
for tenant_id in tenant_ids:
backfill_finding_group_summaries_task.delay(tenant_id=str(tenant_id), days=10)
class Migration(migrations.Migration):
dependencies = [
("api", "0088_finding_group_status_muted_fields"),
]
operations = [
migrations.RunPython(trigger_backfill_task, migrations.RunPython.noop),
]
api/src/backend/api/migrations/0090_attack_paths_cleanup_priority.py
+23
View File
@@ -0,0 +1,23 @@
from django.db import migrations
TASK_NAME = "attack-paths-cleanup-stale-scans"
def set_cleanup_priority(apps, schema_editor):
PeriodicTask = apps.get_model("django_celery_beat", "PeriodicTask")
PeriodicTask.objects.filter(name=TASK_NAME).update(priority=0)
def unset_cleanup_priority(apps, schema_editor):
PeriodicTask = apps.get_model("django_celery_beat", "PeriodicTask")
PeriodicTask.objects.filter(name=TASK_NAME).update(priority=None)
class Migration(migrations.Migration):
dependencies = [
("api", "0089_backfill_finding_group_status_muted"),
]
operations = [
migrations.RunPython(set_cleanup_priority, unset_cleanup_priority),
]
api/src/backend/api/migrations/0091_findings_arrays_gin_index_partitions.py
+30
View File
@@ -0,0 +1,30 @@
from functools import partial
from api.db_utils import create_index_on_partitions, drop_index_on_partitions
from django.db import migrations
class Migration(migrations.Migration):
atomic = False
dependencies = [
("api", "0090_attack_paths_cleanup_priority"),
]
operations = [
migrations.RunPython(
partial(
create_index_on_partitions,
parent_table="findings",
index_name="gin_find_arrays_idx",
columns="categories, resource_services, resource_regions, resource_types",
method="GIN",
all_partitions=True,
),
reverse_code=partial(
drop_index_on_partitions,
parent_table="findings",
index_name="gin_find_arrays_idx",
),
)
]
api/src/backend/api/migrations/0092_findings_arrays_gin_index_parent.py
+73
View File
@@ -0,0 +1,73 @@
import django.contrib.postgres.indexes
from django.db import migrations
INDEX_NAME = "gin_find_arrays_idx"
PARENT_TABLE = "findings"
def create_parent_and_attach(apps, schema_editor):
with schema_editor.connection.cursor() as cursor:
# Idempotent: the parent index may already exist if it was created
# manually on an environment before this migration ran.
cursor.execute(
f"CREATE INDEX IF NOT EXISTS {INDEX_NAME} ON ONLY {PARENT_TABLE} "
f"USING gin (categories, resource_services, resource_regions, resource_types)"
)
cursor.execute(
"SELECT inhrelid::regclass::text "
"FROM pg_inherits "
"WHERE inhparent = %s::regclass",
[PARENT_TABLE],
)
for (partition,) in cursor.fetchall():
child_idx = f"{partition.replace('.', '_')}_{INDEX_NAME}"
# ALTER INDEX ... ATTACH PARTITION has no IF NOT ATTACHED clause,
# so check pg_inherits first to keep the migration re-runnable.
cursor.execute(
"""
SELECT 1
FROM pg_inherits i
JOIN pg_class p ON p.oid = i.inhparent
JOIN pg_class c ON c.oid = i.inhrelid
WHERE p.relname = %s AND c.relname = %s
""",
[INDEX_NAME, child_idx],
)
if cursor.fetchone() is None:
cursor.execute(f"ALTER INDEX {INDEX_NAME} ATTACH PARTITION {child_idx}")
def drop_parent_index(apps, schema_editor):
with schema_editor.connection.cursor() as cursor:
cursor.execute(f"DROP INDEX IF EXISTS {INDEX_NAME}")
class Migration(migrations.Migration):
dependencies = [
("api", "0091_findings_arrays_gin_index_partitions"),
]
operations = [
migrations.SeparateDatabaseAndState(
state_operations=[
migrations.AddIndex(
model_name="finding",
index=django.contrib.postgres.indexes.GinIndex(
fields=[
"categories",
"resource_services",
"resource_regions",
"resource_types",
],
name=INDEX_NAME,
),
),
],
database_operations=[
migrations.RunPython(
create_parent_and_attach,
reverse_code=drop_parent_index,
),
],
),
]
api/src/backend/api/migrations/0093_okta_provider.py
+40
View File
@@ -0,0 +1,40 @@
import api.db_utils
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("api", "0092_findings_arrays_gin_index_parent"),
]
operations = [
migrations.AlterField(
model_name="provider",
name="provider",
field=api.db_utils.ProviderEnumField(
choices=[
("aws", "AWS"),
("azure", "Azure"),
("gcp", "GCP"),
("kubernetes", "Kubernetes"),
("m365", "M365"),
("github", "GitHub"),
("mongodbatlas", "MongoDB Atlas"),
("iac", "IaC"),
("oraclecloud", "Oracle Cloud Infrastructure"),
("alibabacloud", "Alibaba Cloud"),
("cloudflare", "Cloudflare"),
("openstack", "OpenStack"),
("image", "Image"),
("googleworkspace", "Google Workspace"),
("vercel", "Vercel"),
("okta", "Okta"),
],
default="aws",
),
),
migrations.RunSQL(
"ALTER TYPE provider ADD VALUE IF NOT EXISTS 'okta';",
reverse_sql=migrations.RunSQL.noop,
),
]
api/src/backend/api/migrations/0095_reconcile_orphan_tasks_periodic_task.py
+48
View File
@@ -0,0 +1,48 @@
from django.db import migrations
TASK_NAME = "reconcile-orphan-tasks"
INTERVAL_MINUTES = 2
def create_periodic_task(apps, schema_editor):
IntervalSchedule = apps.get_model("django_celery_beat", "IntervalSchedule")
PeriodicTask = apps.get_model("django_celery_beat", "PeriodicTask")
schedule, _ = IntervalSchedule.objects.get_or_create(
every=INTERVAL_MINUTES,
period="minutes",
)
PeriodicTask.objects.update_or_create(
name=TASK_NAME,
defaults={
"task": TASK_NAME,
"interval": schedule,
"enabled": True,
},
)
def delete_periodic_task(apps, schema_editor):
IntervalSchedule = apps.get_model("django_celery_beat", "IntervalSchedule")
PeriodicTask = apps.get_model("django_celery_beat", "PeriodicTask")
PeriodicTask.objects.filter(name=TASK_NAME).delete()
# Clean up the schedule if no other task references it
IntervalSchedule.objects.filter(
every=INTERVAL_MINUTES,
period="minutes",
periodictask__isnull=True,
).delete()
class Migration(migrations.Migration):
dependencies = [
("api", "0093_okta_provider"),
("django_celery_beat", "0019_alter_periodictasks_options"),
]
operations = [
migrations.RunPython(create_periodic_task, delete_periodic_task),
]
api/src/backend/api/migrations/0096_attack_paths_scan_is_migrated.py
+24
View File
@@ -0,0 +1,24 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("api", "0095_reconcile_orphan_tasks_periodic_task"),
]
operations = [
migrations.AddField(
model_name="attackpathsscan",
name="is_migrated",
field=models.BooleanField(default=False),
),
migrations.AddField(
model_name="attackpathsscan",
name="sink_backend",
field=models.CharField(
choices=[("neo4j", "Neo4j"), ("neptune", "Neptune")],
default="neo4j",
max_length=16,
),
),
]
api/src/backend/api/models.py
+162 -34
View File
@@ -1,37 +1,11 @@
import json
import logging
import re
from datetime import datetime, timedelta, timezone
from datetime import UTC, datetime, timedelta
from uuid import UUID, uuid4
import defusedxml
from allauth.socialaccount.models import SocialApp
from config.custom_logging import BackendLogger
from config.settings.social_login import SOCIALACCOUNT_PROVIDERS
from cryptography.fernet import Fernet, InvalidToken
from defusedxml import ElementTree as ET
from django.conf import settings
from django.contrib.auth.models import AbstractBaseUser
from django.contrib.postgres.fields import ArrayField
from django.contrib.postgres.indexes import GinIndex, OpClass
from django.contrib.postgres.search import SearchVector, SearchVectorField
from django.contrib.sites.models import Site
from django.core.exceptions import ValidationError
from django.core.validators import MinLengthValidator
from django.db import models
from django.db.models import Q
from django.db.models.functions import Upper
from django.utils import timezone as django_timezone
from django.utils.translation import gettext_lazy as _
from django_celery_beat.models import PeriodicTask
from django_celery_results.models import TaskResult
from drf_simple_apikey.crypto import get_crypto
from drf_simple_apikey.models import AbstractAPIKey, AbstractAPIKeyManager
from psqlextra.manager import PostgresManager
from psqlextra.models import PostgresPartitionedModel
from psqlextra.types import PostgresPartitioningMethod
from uuid6 import uuid7
from api.db_router import MainRouter
from api.db_utils import (
CustomUserManager,
@@ -58,7 +32,32 @@ from api.rls import (
RowLevelSecurityProtectedModel,
Tenant,
)
from config.custom_logging import BackendLogger
from config.settings.social_login import SOCIALACCOUNT_PROVIDERS
from cryptography.fernet import Fernet, InvalidToken
from defusedxml import ElementTree as ET
from django.conf import settings
from django.contrib.auth.models import AbstractBaseUser
from django.contrib.postgres.fields import ArrayField
from django.contrib.postgres.indexes import GinIndex, OpClass
from django.contrib.postgres.search import SearchVector, SearchVectorField
from django.contrib.sites.models import Site
from django.core.exceptions import ValidationError
from django.core.validators import MinLengthValidator
from django.db import models
from django.db.models import Q
from django.db.models.functions import Upper
from django.utils import timezone as django_timezone
from django.utils.translation import gettext_lazy as _
from django_celery_beat.models import PeriodicTask
from django_celery_results.models import TaskResult
from drf_simple_apikey.crypto import get_crypto
from drf_simple_apikey.models import AbstractAPIKey, AbstractAPIKeyManager
from prowler.lib.check.models import Severity
from psqlextra.manager import PostgresManager
from psqlextra.models import PostgresPartitionedModel
from psqlextra.types import PostgresPartitioningMethod
from uuid6 import uuid7
fernet = Fernet(settings.SECRETS_ENCRYPTION_KEY.encode())
@@ -296,6 +295,7 @@ class Provider(RowLevelSecurityProtectedModel):
IMAGE = "image", _("Image")
GOOGLEWORKSPACE = "googleworkspace", _("Google Workspace")
VERCEL = "vercel", _("Vercel")
OKTA = "okta", _("Okta")
@staticmethod
def validate_aws_uid(value):
@@ -354,6 +354,26 @@ class Provider(RowLevelSecurityProtectedModel):
pointer="/data/attributes/uid",
)
@staticmethod
def validate_okta_uid(value):
if not re.match(
r"^[a-z0-9][a-z0-9-]*\.("
r"okta\.com|oktapreview\.com|okta-emea\.com|"
r"okta-gov\.com|okta\.mil|okta-miltest\.com|trex-govcloud\.com"
r")$",
value,
):
raise ModelValidationError(
detail=(
"Okta provider ID must be a valid Okta-managed org domain "
"(e.g., acme.okta.com, also .oktapreview.com / .okta-emea.com "
"/ .okta-gov.com / .okta.mil / .okta-miltest.com / "
".trex-govcloud.com), without scheme or path."
),
code="okta-uid",
pointer="/data/attributes/uid",
)
@staticmethod
def validate_kubernetes_uid(value):
if not re.match(
@@ -480,6 +500,12 @@ class Provider(RowLevelSecurityProtectedModel):
def clean(self):
super().clean()
if self.provider == self.ProviderChoices.OKTA and self.uid:
# Mirror the SDK, which lowercases the org domain before connecting.
# Without this the API would reject Acme.okta.com even though the
# SDK would accept it, and stored uids could disagree with the
# authenticated org domain.
self.uid = self.uid.strip().lower()
getattr(self, f"validate_{self.provider}_uid")(self.uid)
def save(self, *args, **kwargs):
@@ -595,10 +621,40 @@ class Scan(RowLevelSecurityProtectedModel):
objects = ActiveProviderManager()
all_objects = models.Manager()
_SCOPING_SCANNER_ARG_KEYS_CACHE: tuple[str, ...] | None = None
@classmethod
def get_scoping_scanner_arg_keys(cls) -> tuple[str, ...]:
"""Return the scanner_args keys that mark a scan as scoped.
Derived from ``prowler.lib.scan.scan.Scan.__init__`` so the API stays
in sync with whatever the SDK actually accepts as filters. Cached at
class level — the signature is stable for the process lifetime.
"""
if cls._SCOPING_SCANNER_ARG_KEYS_CACHE is None:
import inspect
from prowler.lib.scan.scan import Scan as ProwlerScan
params = inspect.signature(ProwlerScan.__init__).parameters
cls._SCOPING_SCANNER_ARG_KEYS_CACHE = tuple(
name for name in params if name not in ("self", "provider")
)
return cls._SCOPING_SCANNER_ARG_KEYS_CACHE
class TriggerChoices(models.TextChoices):
SCHEDULED = "scheduled", _("Scheduled")
MANUAL = "manual", _("Manual")
# Trigger values for scans that ran the SDK end-to-end. Imported scans (or
# any future trigger) are intentionally NOT in this set — they may carry
# only a partial slice of resources, so post-scan logic that depends on a
# full-scope sweep (e.g. resetting ephemeral resource findings) must skip
# them by default.
LIVE_SCAN_TRIGGERS = frozenset(
(TriggerChoices.SCHEDULED.value, TriggerChoices.MANUAL.value)
)
id = models.UUIDField(primary_key=True, default=uuid7, editable=False)
name = models.CharField(
blank=True, null=True, max_length=100, validators=[MinLengthValidator(3)]
@@ -681,8 +737,30 @@ class Scan(RowLevelSecurityProtectedModel):
class JSONAPIMeta:
resource_name = "scans"
def is_full_scope(self) -> bool:
"""Return True if this scan ran with no scoping filters at all.
Used to gate post-scan operations (such as resetting the
failed_findings_count of resources missing from the scan) that are only
safe when the scan covered every check, service, and category. Imported
scans are NOT full-scope by definition — they may carry only a partial
slice of resources, so they're rejected via ``trigger`` even before the
scanner_args check.
"""
if self.trigger not in self.LIVE_SCAN_TRIGGERS:
return False
scanner_args = self.scanner_args or {}
for key in self.get_scoping_scanner_arg_keys():
if scanner_args.get(key):
return False
return True
class AttackPathsScan(RowLevelSecurityProtectedModel):
class SinkBackendChoices(models.TextChoices):
NEO4J = "neo4j", "Neo4j"
NEPTUNE = "neptune", "Neptune"
objects = ActiveProviderManager()
all_objects = models.Manager()
@@ -731,6 +809,18 @@ class AttackPathsScan(RowLevelSecurityProtectedModel):
)
ingestion_exceptions = models.JSONField(default=dict, null=True, blank=True)
# True when the scan was synced with the current schema (list-typed
# properties materialised as child item nodes). False for pre-cutover scans
# still using the previous graph shape. Query catalog selection uses this
# flag; physical read routing uses sink_backend below.
# TODO: drop after Neptune cutover
is_migrated = models.BooleanField(default=False)
sink_backend = models.CharField(
choices=SinkBackendChoices.choices,
default=SinkBackendChoices.NEO4J,
max_length=16,
)
class Meta(RowLevelSecurityProtectedModel.Meta):
db_table = "attack_paths_scans"
@@ -898,7 +988,6 @@ class Resource(RowLevelSecurityProtectedModel):
OpClass(Upper("name"), name="gin_trgm_ops"),
name="res_name_trgm_idx",
),
GinIndex(fields=["text_search"], name="gin_resources_search_idx"),
models.Index(fields=["tenant_id", "id"], name="resources_tenant_id_idx"),
models.Index(
fields=["tenant_id", "provider_id"],
@@ -1104,6 +1193,15 @@ class Finding(PostgresPartitionedModel, RowLevelSecurityProtectedModel):
fields=["tenant_id", "scan_id", "check_id"],
name="find_tenant_scan_check_idx",
),
GinIndex(
fields=[
"categories",
"resource_services",
"resource_regions",
"resource_types",
],
name="gin_find_arrays_idx",
),
]
class JSONAPIMeta:
@@ -1344,8 +1442,8 @@ class Role(RowLevelSecurityProtectedModel):
@classmethod
def filter_by_permission_state(cls, queryset, value):
q_all_true = Q(**{field: True for field in cls.PERMISSION_FIELDS})
q_all_false = Q(**{field: False for field in cls.PERMISSION_FIELDS})
q_all_true = Q(**dict.fromkeys(cls.PERMISSION_FIELDS, True))
q_all_false = Q(**dict.fromkeys(cls.PERMISSION_FIELDS, False))
if value == PermissionChoices.UNLIMITED:
return queryset.filter(q_all_true)
@@ -1748,15 +1846,45 @@ class FindingGroupDailySummary(RowLevelSecurityProtectedModel):
# Severity stored as integer for MAX aggregation (5=critical, 4=high, etc.)
severity_order = models.SmallIntegerField(default=1)
# Finding counts
# Finding counts (inclusive of muted findings; use the `muted` flag to
# tell whether the group has any actionable findings).
pass_count = models.IntegerField(default=0)
fail_count = models.IntegerField(default=0)
manual_count = models.IntegerField(default=0)
muted_count = models.IntegerField(default=0)
# Delta counts
# Status counts restricted to muted findings, so clients can isolate the
# muted half of each status (e.g. `pass_count - pass_muted_count` gives the
# actionable PASS findings).
pass_muted_count = models.IntegerField(default=0)
fail_muted_count = models.IntegerField(default=0)
manual_muted_count = models.IntegerField(default=0)
# Whether every finding for this (provider, check, day) is muted.
muted = models.BooleanField(default=False)
# Delta counts (non-muted, kept for convenience and as a "total" view).
new_count = models.IntegerField(default=0)
changed_count = models.IntegerField(default=0)
# Delta breakdown by (status, muted) so clients can answer questions like
# "how many new failing findings appeared in this scan?" without scanning
# the underlying findings table. Mirrors the existing pass/fail/manual
# naming, with `_muted_count` siblings tracking the muted half of each
# bucket explicitly.
new_fail_count = models.IntegerField(default=0)
new_fail_muted_count = models.IntegerField(default=0)
new_pass_count = models.IntegerField(default=0)
new_pass_muted_count = models.IntegerField(default=0)
new_manual_count = models.IntegerField(default=0)
new_manual_muted_count = models.IntegerField(default=0)
changed_fail_count = models.IntegerField(default=0)
changed_fail_muted_count = models.IntegerField(default=0)
changed_pass_count = models.IntegerField(default=0)
changed_pass_muted_count = models.IntegerField(default=0)
changed_manual_count = models.IntegerField(default=0)
changed_manual_muted_count = models.IntegerField(default=0)
# Resource counts
resources_fail = models.IntegerField(default=0)
resources_total = models.IntegerField(default=0)
@@ -1898,11 +2026,11 @@ class SAMLToken(models.Model):
def save(self, *args, **kwargs):
if not self.expires_at:
self.expires_at = datetime.now(timezone.utc) + timedelta(seconds=15)
self.expires_at = datetime.now(UTC) + timedelta(seconds=15)
super().save(*args, **kwargs)
def is_expired(self) -> bool:
return datetime.now(timezone.utc) >= self.expires_at
return datetime.now(UTC) >= self.expires_at
class SAMLDomainIndex(models.Model):
api/src/backend/api/partitions.py
+20 -23
View File
@@ -1,21 +1,20 @@
from datetime import datetime, timezone
from typing import Generator, Optional
from dateutil.relativedelta import relativedelta
from django.conf import settings
from psqlextra.partitioning import (
PostgresPartitioningManager,
PostgresRangePartition,
PostgresRangePartitioningStrategy,
PostgresTimePartitionSize,
PostgresPartitioningError,
)
from psqlextra.partitioning.config import PostgresPartitioningConfig
from uuid6 import UUID
from collections.abc import Generator
from datetime import UTC, datetime
from api.models import Finding, ResourceFindingMapping
from api.rls import RowLevelSecurityConstraint
from api.uuid_utils import datetime_to_uuid7
from dateutil.relativedelta import relativedelta
from django.conf import settings
from psqlextra.partitioning import (
PostgresPartitioningError,
PostgresPartitioningManager,
PostgresRangePartition,
PostgresRangePartitioningStrategy,
PostgresTimePartitionSize,
)
from psqlextra.partitioning.config import PostgresPartitioningConfig
from uuid6 import UUID
class PostgresUUIDv7RangePartition(PostgresRangePartition):
@@ -24,7 +23,7 @@ class PostgresUUIDv7RangePartition(PostgresRangePartition):
from_values: UUID,
to_values: UUID,
size: PostgresTimePartitionSize,
name_format: Optional[str] = None,
name_format: str | None = None,
**kwargs,
) -> None:
self.from_values = from_values
@@ -38,9 +37,7 @@ class PostgresUUIDv7RangePartition(PostgresRangePartition):
start_timestamp_ms = self.from_values.time
self.start_datetime = datetime.fromtimestamp(
start_timestamp_ms / 1000, timezone.utc
)
self.start_datetime = datetime.fromtimestamp(start_timestamp_ms / 1000, UTC)
def name(self) -> str:
if not self.name_format:
@@ -82,8 +79,8 @@ class PostgresUUIDv7PartitioningStrategy(PostgresRangePartitioningStrategy):
size: PostgresTimePartitionSize,
count: int,
start_date: datetime = None,
max_age: Optional[relativedelta] = None,
name_format: Optional[str] = None,
max_age: relativedelta | None = None,
name_format: str | None = None,
**kwargs,
) -> None:
self.start_date = start_date.replace(
@@ -151,7 +148,7 @@ class PostgresUUIDv7PartitioningStrategy(PostgresRangePartitioningStrategy):
Returns:
datetime: A `datetime` object representing the start of the current month in UTC.
"""
return datetime.now(timezone.utc).replace(
return datetime.now(UTC).replace(
day=1, hour=0, minute=0, second=0, microsecond=0
)
@@ -171,7 +168,7 @@ manager = PostgresPartitioningManager(
PostgresPartitioningConfig(
model=Finding,
strategy=PostgresUUIDv7PartitioningStrategy(
start_date=datetime.now(timezone.utc),
start_date=datetime.now(UTC),
size=PostgresTimePartitionSize(
months=settings.FINDINGS_TABLE_PARTITION_MONTHS
),
@@ -187,7 +184,7 @@ manager = PostgresPartitioningManager(
PostgresPartitioningConfig(
model=ResourceFindingMapping,
strategy=PostgresUUIDv7PartitioningStrategy(
start_date=datetime.now(timezone.utc),
start_date=datetime.now(UTC),
size=PostgresTimePartitionSize(
months=settings.FINDINGS_TABLE_PARTITION_MONTHS
),
api/src/backend/api/rbac/permissions.py
+3 -4
View File
@@ -1,11 +1,10 @@
from enum import Enum
from django.db.models import QuerySet
from rest_framework.exceptions import PermissionDenied
from rest_framework.permissions import BasePermission
from api.db_router import MainRouter
from api.models import Provider, Role, User
from django.db.models import QuerySet
from rest_framework.exceptions import PermissionDenied
from rest_framework.permissions import BasePermission
class Permissions(Enum):
api/src/backend/api/renderers.py
+1 -2
View File
@@ -1,10 +1,9 @@
from contextlib import nullcontext
from api.db_utils import rls_transaction
from rest_framework.renderers import BaseRenderer
from rest_framework_json_api.renderers import JSONRenderer
from api.db_utils import rls_transaction
class PlainTextRenderer(BaseRenderer):
media_type = "text/plain"
api/src/backend/api/rls.py
+1 -2
View File
@@ -1,12 +1,11 @@
from typing import Any
from uuid import uuid4
from api.db_utils import DB_USER, POSTGRES_TENANT_VAR
from django.core.exceptions import ValidationError
from django.db import DEFAULT_DB_ALIAS, models
from django.db.backends.ddl_references import Statement, Table
from api.db_utils import DB_USER, POSTGRES_TENANT_VAR
class Tenant(models.Model):
"""
api/src/backend/api/signals.py
+6 -7
View File
@@ -1,10 +1,3 @@
from celery import states
from celery.signals import before_task_publish
from config.celery import celery_app
from django.db.models.signals import post_delete, pre_delete
from django.dispatch import receiver
from django_celery_results.backends.database import DatabaseBackend
from api.db_utils import delete_related_daily_task
from api.models import (
LighthouseProviderConfiguration,
@@ -14,6 +7,12 @@ from api.models import (
TenantAPIKey,
User,
)
from celery import states
from celery.signals import before_task_publish
from config.celery import celery_app
from django.db.models.signals import post_delete, pre_delete
from django.dispatch import receiver
from django_celery_results.backends.database import DatabaseBackend
def create_task_result_on_publish(sender=None, headers=None, **kwargs): # noqa: F841

Some files were not shown because too many files have changed in this diff Show More