Compare commits

..

1 Commits

Author SHA1 Message Date
Prowler Bot 68eb946326 chore(api): Update prowler dependency to v5.25 for release 5.25.0 (#10906)
Co-authored-by: prowler-bot <179230569+prowler-bot@users.noreply.github.com>
2026-04-28 11:00:51 +02:00
990 changed files with 9108 additions and 46498 deletions
+1 -1
View File
@@ -145,7 +145,7 @@ SENTRY_RELEASE=local
NEXT_PUBLIC_SENTRY_ENVIRONMENT=${SENTRY_ENVIRONMENT}
#### Prowler release version ####
NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v5.27.0
NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v5.25.0
# Social login credentials
SOCIAL_GOOGLE_OAUTH_CALLBACK_URL="${AUTH_URL}/api/auth/callback/google"
-15
View File
@@ -1,15 +0,0 @@
# These are supported funding model platforms
github: [prowler-cloud]
# patreon: # Replace with a single Patreon username
# open_collective: # Replace with a single Open Collective username
# ko_fi: # Replace with a single Ko-fi username
# tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
# community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
# liberapay: # Replace with a single Liberapay username
# issuehunt: # Replace with a single IssueHunt username
# lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
# polar: # Replace with a single Polar username
# buy_me_a_coffee: # Replace with a single Buy Me a Coffee username
# thanks_dev: # Replace with a single thanks.dev username
# custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']
@@ -1,143 +0,0 @@
name: "🔎 New Check Request"
description: Request a new Prowler security check
title: "[New Check]: "
labels: ["feature-request", "status/needs-triage"]
body:
- type: checkboxes
id: search
attributes:
label: Existing check search
description: Confirm this check does not already exist before opening a new request.
options:
- label: I have searched existing issues, Prowler Hub, and the public roadmap, and this check does not already exist.
required: true
- type: markdown
attributes:
value: |
Use this form to describe the security condition that Prowler should evaluate.
The most useful inputs for [Prowler Studio](https://github.com/prowler-cloud/prowler-studio) are:
- What should be detected
- What PASS and FAIL mean
- Vendor docs, API references, SDK methods, CLI commands, or reference code
- type: dropdown
id: provider
attributes:
label: Provider
description: Cloud or platform this check targets.
options:
- AWS
- Azure
- GCP
- Kubernetes
- GitHub
- Microsoft 365
- OCI
- Alibaba Cloud
- Cloudflare
- MongoDB Atlas
- Google Workspace
- OpenStack
- Vercel
- NHN
- Other / New provider
validations:
required: true
- type: input
id: other_provider_name
attributes:
label: New provider name
description: Only fill this if you selected "Other / New provider" above.
placeholder: "NewProviderName"
validations:
required: false
- type: input
id: service_name
attributes:
label: Service or product area
description: Optional. Main service, product, or feature to audit.
placeholder: "s3, bedrock, entra, repository, apiserver"
validations:
required: false
- type: input
id: suggested_check_name
attributes:
label: Suggested check name
description: Optional. Use `snake_case` following `<service>_<resource>_<best_practice>`, with lowercase letters and underscores only.
placeholder: "bedrock_guardrail_sensitive_information_filter_enabled"
validations:
required: false
- type: textarea
id: context
attributes:
label: Context and goal
description: Describe the security problem, why it matters, and what this new check should help detect.
placeholder: |-
- Security condition to validate:
- Why it matters:
- Resource, feature, or configuration involved:
validations:
required: true
- type: textarea
id: expected_behavior
attributes:
label: Expected behavior
description: Explain what the check should evaluate and what PASS, FAIL, or MANUAL should mean.
placeholder: |-
- Resource or scope to evaluate:
- PASS when:
- FAIL when:
- MANUAL when (if applicable):
- Exclusions, thresholds, or edge cases:
validations:
required: true
- type: textarea
id: references
attributes:
label: References
description: Add vendor docs, API references, SDK methods, CLI commands, endpoint docs, sample payloads, or similar reference material.
placeholder: |-
- Product or service documentation:
- API or SDK reference:
- CLI command or endpoint documentation:
- Sample payload or response:
- Security advisory or benchmark:
validations:
required: true
- type: dropdown
id: severity
attributes:
label: Suggested severity
description: Your best estimate. Reviewers will confirm during triage.
options:
- Critical
- High
- Medium
- Low
- Informational
- Not sure
validations:
required: true
- type: textarea
id: implementation_notes
attributes:
label: Additional implementation notes
description: Optional. Add permissions, unsupported regions, config knobs, product limitations, or anything else that may affect implementation.
placeholder: |-
- Required permissions or scopes:
- Region, tenant, or subscription limitations:
- Configurable behavior or thresholds:
- Other constraints:
validations:
required: false
-7
View File
@@ -72,11 +72,6 @@ provider/vercel:
- any-glob-to-any-file: "prowler/providers/vercel/**"
- any-glob-to-any-file: "tests/providers/vercel/**"
provider/okta:
- changed-files:
- any-glob-to-any-file: "prowler/providers/okta/**"
- any-glob-to-any-file: "tests/providers/okta/**"
github_actions:
- changed-files:
- any-glob-to-any-file: ".github/workflows/*"
@@ -114,8 +109,6 @@ mutelist:
- any-glob-to-any-file: "tests/providers/googleworkspace/lib/mutelist/**"
- any-glob-to-any-file: "prowler/providers/vercel/lib/mutelist/**"
- any-glob-to-any-file: "tests/providers/vercel/lib/mutelist/**"
- any-glob-to-any-file: "prowler/providers/okta/lib/mutelist/**"
- any-glob-to-any-file: "tests/providers/okta/lib/mutelist/**"
integration/s3:
- changed-files:
-1
View File
@@ -36,7 +36,6 @@ Please add a detailed description of how to review this PR.
#### UI
- [ ] All issue/task requirements work as expected on the UI
- [ ] If this PR adds or updates npm dependencies, include package-health evidence (maintenance, popularity, known vulnerabilities, license, release age) and explain why existing/native alternatives are insufficient.
- [ ] Screenshots/Video of the functionality flow (if applicable) - Mobile (X < 640px)
- [ ] Screenshots/Video of the functionality flow (if applicable) - Table (640px > X < 1024px)
- [ ] Screenshots/Video of the functionality flow (if applicable) - Desktop (X > 1024px)
@@ -158,7 +158,7 @@ jobs:
tags: |
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-${{ matrix.arch }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=${{ github.event_name == 'pull_request' && 'min' || 'max' }},scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
# Create and push multi-architecture manifest
create-manifest:
+17 -13
View File
@@ -5,16 +5,10 @@ on:
branches:
- 'master'
- 'v5.*'
paths:
- 'api/**'
- '.github/workflows/api-container-checks.yml'
pull_request:
branches:
- 'master'
- 'v5.*'
paths:
- 'api/**'
- '.github/workflows/api-container-checks.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -63,7 +57,16 @@ jobs:
api-container-build-and-scan:
if: github.repository == 'prowler-cloud/prowler'
runs-on: ubuntu-latest
runs-on: ${{ matrix.runner }}
strategy:
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
arch: amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
arch: arm64
timeout-minutes: 30
permissions:
contents: read
@@ -116,22 +119,23 @@ jobs:
if: steps.check-changes.outputs.any_changed == 'true'
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Build container
- name: Build container for ${{ matrix.arch }}
if: steps.check-changes.outputs.any_changed == 'true'
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v7.0.0
with:
context: ${{ env.API_WORKING_DIR }}
push: false
load: true
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=${{ github.event_name == 'pull_request' && 'min' || 'max' }}
platforms: ${{ matrix.platform }}
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}-${{ matrix.arch }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
- name: Scan container with Trivy
- name: Scan container with Trivy for ${{ matrix.arch }}
if: steps.check-changes.outputs.any_changed == 'true'
uses: ./.github/actions/trivy-scan
with:
image-name: ${{ env.IMAGE_NAME }}
image-tag: ${{ github.sha }}
image-tag: ${{ github.sha }}-${{ matrix.arch }}
fail-on-critical: 'false'
severity: 'CRITICAL'
-10
View File
@@ -5,20 +5,10 @@ on:
branches:
- "master"
- "v5.*"
paths:
- 'api/**'
- '.github/workflows/api-tests.yml'
- '.github/workflows/api-security.yml'
- '.github/actions/setup-python-poetry/**'
pull_request:
branches:
- "master"
- "v5.*"
paths:
- 'api/**'
- '.github/workflows/api-tests.yml'
- '.github/workflows/api-security.yml'
- '.github/actions/setup-python-poetry/**'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -4,6 +4,8 @@ on:
pull_request:
branches:
- 'master'
- 'v3'
- 'v4.*'
- 'v5.*'
types:
- 'opened'
@@ -43,11 +43,14 @@ jobs:
echo "Processing release tag: $RELEASE_TAG"
# Remove 'v' prefix if present (e.g., v3.2.0 -> 3.2.0)
VERSION_ONLY="${RELEASE_TAG#v}"
# Check if it's a minor version (X.Y.0)
if [[ "$VERSION_ONLY" =~ ^([0-9]+)\.([0-9]+)\.0$ ]]; then
echo "Release $RELEASE_TAG (version $VERSION_ONLY) is a minor version. Proceeding to create backport label."
# Extract X.Y from X.Y.0 (e.g., 5.6 from 5.6.0)
MAJOR="${BASH_REMATCH[1]}"
MINOR="${BASH_REMATCH[2]}"
TWO_DIGIT_VERSION="${MAJOR}.${MINOR}"
@@ -59,6 +62,7 @@ jobs:
echo "Label name: $LABEL_NAME"
echo "Label description: $LABEL_DESC"
# Check if label already exists
if gh label list --repo ${{ github.repository }} --limit 1000 | grep -q "^${LABEL_NAME}[[:space:]]"; then
echo "Label '$LABEL_NAME' already exists."
else
+3 -6
View File
@@ -37,13 +37,10 @@ jobs:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
# PRs only need the diff range; push to master/release walks the new range from event.before.
# 50 is enough headroom for the longest realistic PR/push chain without paying for a full clone.
fetch-depth: 50
fetch-depth: 0
persist-credentials: false
- name: Scan diff for secrets with TruffleHog
# Action auto-injects --since-commit/--branch from event payload; passing them in extra_args produces duplicate flags.
- name: Scan for secrets with TruffleHog
uses: trufflesecurity/trufflehog@ef6e76c3c4023279497fab4721ffa071a722fd05 # v3.92.4
with:
extra_args: --results=verified,unknown
extra_args: '--results=verified,unknown'
+1 -1
View File
@@ -62,7 +62,7 @@ jobs:
"Alan-TheGentleman"
"alejandrobailo"
"amitsharm"
# "andoniaf"
"andoniaf"
"cesararroba"
"danibarranqueroo"
"HugoPBrito"
@@ -152,7 +152,7 @@ jobs:
org.opencontainers.image.created=${{ github.event_name == 'release' && github.event.release.published_at || github.event.head_commit.timestamp }}
${{ github.event_name == 'release' && format('org.opencontainers.image.version={0}', env.RELEASE_TAG) || '' }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=${{ github.event_name == 'pull_request' && 'min' || 'max' }},scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
# Create and push multi-architecture manifest
create-manifest:
+17 -13
View File
@@ -5,16 +5,10 @@ on:
branches:
- 'master'
- 'v5.*'
paths:
- 'mcp_server/**'
- '.github/workflows/mcp-container-checks.yml'
pull_request:
branches:
- 'master'
- 'v5.*'
paths:
- 'mcp_server/**'
- '.github/workflows/mcp-container-checks.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -62,7 +56,16 @@ jobs:
mcp-container-build-and-scan:
if: github.repository == 'prowler-cloud/prowler'
runs-on: ubuntu-latest
runs-on: ${{ matrix.runner }}
strategy:
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
arch: amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
arch: arm64
timeout-minutes: 30
permissions:
contents: read
@@ -109,22 +112,23 @@ jobs:
if: steps.check-changes.outputs.any_changed == 'true'
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Build MCP container
- name: Build MCP container for ${{ matrix.arch }}
if: steps.check-changes.outputs.any_changed == 'true'
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v7.0.0
with:
context: ${{ env.MCP_WORKING_DIR }}
push: false
load: true
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=${{ github.event_name == 'pull_request' && 'min' || 'max' }}
platforms: ${{ matrix.platform }}
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}-${{ matrix.arch }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
- name: Scan MCP container with Trivy
- name: Scan MCP container with Trivy for ${{ matrix.arch }}
if: steps.check-changes.outputs.any_changed == 'true'
uses: ./.github/actions/trivy-scan
with:
image-name: ${{ env.IMAGE_NAME }}
image-tag: ${{ github.sha }}
image-tag: ${{ github.sha }}-${{ matrix.arch }}
fail-on-critical: 'false'
severity: 'CRITICAL'
-21
View File
@@ -86,32 +86,11 @@ jobs:
with:
python-version: ${{ env.PYTHON_VERSION }}
# The MCP server version (mcp_server/pyproject.toml) is decoupled from the Prowler release
# version: it only changes when MCP code changes. mcp-bump-version.yml normally keeps it in
# sync with mcp_server/CHANGELOG.md, but this publish workflow still runs on every release.
# Pre-flight PyPI check covers the legitimate "no MCP changes for this release" case (and any
# workflow_dispatch re-runs) without failing with HTTP 400 (version exists).
- name: Check if prowler-mcp version already exists on PyPI
id: pypi-check
working-directory: ${{ env.WORKING_DIRECTORY }}
run: |
MCP_VERSION=$(grep '^version' pyproject.toml | head -1 | sed -E 's/^version[[:space:]]*=[[:space:]]*"([^"]+)".*/\1/')
echo "mcp_version=${MCP_VERSION}" >> "$GITHUB_OUTPUT"
if curl -fsS "https://pypi.org/pypi/prowler-mcp/${MCP_VERSION}/json" >/dev/null 2>&1; then
echo "skip=true" >> "$GITHUB_OUTPUT"
echo "::notice title=Skipping prowler-mcp publish::Version ${MCP_VERSION} already exists on PyPI; bump mcp_server/pyproject.toml to publish a new release."
else
echo "skip=false" >> "$GITHUB_OUTPUT"
echo "::notice title=Publishing prowler-mcp::Version ${MCP_VERSION} not on PyPI yet; proceeding."
fi
- name: Build prowler-mcp package
if: steps.pypi-check.outputs.skip != 'true'
working-directory: ${{ env.WORKING_DIRECTORY }}
run: uv build
- name: Publish prowler-mcp package to PyPI
if: steps.pypi-check.outputs.skip != 'true'
uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0
with:
packages-dir: ${{ env.WORKING_DIRECTORY }}/dist/
@@ -1,98 +0,0 @@
name: 'Nightly: ARM64 Container Builds'
# Mitigation for amd64-only PR container-checks: build amd64+arm64 nightly against
# master to keep arm-specific Dockerfile regressions caught quickly. Build only —
# no push, no Trivy (weekly checks already cover that).
on:
schedule:
- cron: '0 4 * * *'
workflow_dispatch: {}
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false
permissions: {}
jobs:
build-arm64:
if: github.repository == 'prowler-cloud/prowler'
runs-on: ubuntu-24.04-arm
timeout-minutes: 60
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
- component: sdk
context: .
dockerfile: ./Dockerfile
image_name: prowler
- component: api
context: ./api
dockerfile: ./api/Dockerfile
image_name: prowler-api
- component: ui
context: ./ui
dockerfile: ./ui/Dockerfile
image_name: prowler-ui
target: prod
build_args: |
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=pk_test_51LwpXXXX
- component: mcp
context: ./mcp_server
dockerfile: ./mcp_server/Dockerfile
image_name: prowler-mcp
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@fa2e9d605c4eeb9fcad4c99c224cee0c6c7f3594 # v2.16.0
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Build ${{ matrix.component }} container (linux/arm64)
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v7.0.0
with:
context: ${{ matrix.context }}
file: ${{ matrix.dockerfile }}
target: ${{ matrix.target }}
push: false
load: false
platforms: linux/arm64
tags: ${{ matrix.image_name }}:nightly-arm64
build-args: ${{ matrix.build_args }}
cache-from: type=gha,scope=arm64
cache-to: type=gha,mode=min,scope=arm64
notify-failure:
needs: build-arm64
if: failure() && github.event_name == 'schedule'
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@fa2e9d605c4eeb9fcad4c99c224cee0c6c7f3594 # v2.16.0
with:
egress-policy: audit
- name: Notify Slack on failure
uses: slackapi/slack-github-action@91efab103c0de0a537f72a35f6b8cda0ee76bf0a # v2.1.1
with:
method: chat.postMessage
token: ${{ secrets.SLACK_BOT_TOKEN }}
payload: |
channel: ${{ secrets.SLACK_PLATFORM_DEPLOYMENTS }}
text: ":rotating_light: Nightly arm64 container build failed for prowler — <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|view run>"
errors: true
+1 -6
View File
@@ -41,15 +41,10 @@ jobs:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 1
fetch-depth: 0
# zizmor: ignore[artipacked]
persist-credentials: true # Required by tj-actions/changed-files to fetch PR branch
- name: Fetch PR base ref for tj-actions/changed-files
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
run: git fetch --depth=1 origin "${BASE_REF}"
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@22103cc46bda19c2b464ffe86db46df6922fd323 # v47.0.5
@@ -45,15 +45,10 @@ jobs:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 1
fetch-depth: 0
# zizmor: ignore[artipacked]
persist-credentials: true # Required by tj-actions/changed-files to fetch PR branch
- name: Fetch PR base ref for tj-actions/changed-files
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
run: git fetch --depth=1 origin "${BASE_REF}"
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@22103cc46bda19c2b464ffe86db46df6922fd323 # v47.0.5
+2 -8
View File
@@ -36,14 +36,8 @@ jobs:
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 1
# zizmor: ignore[artipacked]
persist-credentials: true # Required by tj-actions/changed-files to fetch PR branch
- name: Fetch PR base ref for tj-actions/changed-files
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
run: git fetch --depth=1 origin "${BASE_REF}"
fetch-depth: 0
persist-credentials: false
- name: Get changed files
id: changed-files
@@ -5,9 +5,6 @@ on:
branches:
- 'master'
- 'v5.*'
paths:
- 'tests/providers/**/*_test.py'
- '.github/workflows/sdk-check-duplicate-test-names.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
+66 -8
View File
@@ -3,7 +3,9 @@ name: 'SDK: Container Build and Push'
on:
push:
branches:
- 'master'
- 'v3' # For v3-latest
- 'v4.6' # For v4-latest
- 'master' # For latest
paths-ignore:
- '.github/**'
- '!.github/workflows/sdk-container-build-push.yml'
@@ -54,6 +56,7 @@ jobs:
timeout-minutes: 5
outputs:
prowler_version: ${{ steps.get-prowler-version.outputs.prowler_version }}
prowler_version_major: ${{ steps.get-prowler-version.outputs.prowler_version_major }}
latest_tag: ${{ steps.get-prowler-version.outputs.latest_tag }}
stable_tag: ${{ steps.get-prowler-version.outputs.stable_tag }}
permissions:
@@ -89,13 +92,32 @@ jobs:
PROWLER_VERSION="$(poetry version -s 2>/dev/null)"
echo "prowler_version=${PROWLER_VERSION}" >> "${GITHUB_OUTPUT}"
# Extract major version
PROWLER_VERSION_MAJOR="${PROWLER_VERSION%%.*}"
if [[ "${PROWLER_VERSION_MAJOR}" != "5" ]]; then
echo "::error::Unsupported Prowler major version: ${PROWLER_VERSION_MAJOR}"
exit 1
fi
echo "latest_tag=latest" >> "${GITHUB_OUTPUT}"
echo "stable_tag=stable" >> "${GITHUB_OUTPUT}"
echo "prowler_version_major=${PROWLER_VERSION_MAJOR}" >> "${GITHUB_OUTPUT}"
# Set version-specific tags
case ${PROWLER_VERSION_MAJOR} in
3)
echo "latest_tag=v3-latest" >> "${GITHUB_OUTPUT}"
echo "stable_tag=v3-stable" >> "${GITHUB_OUTPUT}"
echo "✓ Prowler v3 detected - tags: v3-latest, v3-stable"
;;
4)
echo "latest_tag=v4-latest" >> "${GITHUB_OUTPUT}"
echo "stable_tag=v4-stable" >> "${GITHUB_OUTPUT}"
echo "✓ Prowler v4 detected - tags: v4-latest, v4-stable"
;;
5)
echo "latest_tag=latest" >> "${GITHUB_OUTPUT}"
echo "stable_tag=stable" >> "${GITHUB_OUTPUT}"
echo "✓ Prowler v5 detected - tags: latest, stable"
;;
*)
echo "::error::Unsupported Prowler major version: ${PROWLER_VERSION_MAJOR}"
exit 1
;;
esac
notify-release-started:
if: github.repository == 'prowler-cloud/prowler' && (github.event_name == 'release' || github.event_name == 'workflow_dispatch')
@@ -206,7 +228,7 @@ jobs:
tags: |
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.latest_tag }}-${{ matrix.arch }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=${{ github.event_name == 'pull_request' && 'min' || 'max' }},scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
# Create and push multi-architecture manifest
create-manifest:
@@ -364,3 +386,39 @@ jobs:
payload-file-path: "./.github/scripts/slack-messages/container-release-completed.json"
step-outcome: ${{ steps.outcome.outputs.outcome }}
update-ts: ${{ needs.notify-release-started.outputs.message-ts }}
dispatch-v3-deployment:
needs: [setup, container-build-push]
if: always() && needs.setup.outputs.prowler_version_major == '3' && needs.setup.result == 'success' && needs.container-build-push.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
steps:
- name: Harden the runner (Audit all outbound calls)
uses: step-security/harden-runner@fa2e9d605c4eeb9fcad4c99c224cee0c6c7f3594 # v2.16.0
with:
egress-policy: audit
- name: Calculate short SHA
id: short-sha
run: echo "short_sha=${GITHUB_SHA::7}" >> $GITHUB_OUTPUT
- name: Dispatch v3 deployment (latest)
if: github.event_name == 'push'
uses: peter-evans/repository-dispatch@28959ce8df70de7be546dd1250a005dd32156697 # v4.0.1
with:
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
repository: ${{ secrets.DISPATCH_OWNER }}/${{ secrets.DISPATCH_REPO }}
event-type: dispatch
client-payload: '{"version":"v3-latest","tag":"${{ steps.short-sha.outputs.short_sha }}"}'
- name: Dispatch v3 deployment (release)
if: github.event_name == 'release'
uses: peter-evans/repository-dispatch@28959ce8df70de7be546dd1250a005dd32156697 # v4.0.1
with:
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
repository: ${{ secrets.DISPATCH_OWNER }}/${{ secrets.DISPATCH_REPO }}
event-type: dispatch
client-payload: '{"version":"release","tag":"${{ needs.setup.outputs.prowler_version }}"}'
+17 -19
View File
@@ -5,22 +5,10 @@ on:
branches:
- 'master'
- 'v5.*'
paths:
- 'prowler/**'
- 'Dockerfile*'
- 'pyproject.toml'
- 'poetry.lock'
- '.github/workflows/sdk-container-checks.yml'
pull_request:
branches:
- 'master'
- 'v5.*'
paths:
- 'prowler/**'
- 'Dockerfile*'
- 'pyproject.toml'
- 'poetry.lock'
- '.github/workflows/sdk-container-checks.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -68,7 +56,16 @@ jobs:
sdk-container-build-and-scan:
if: github.repository == 'prowler-cloud/prowler'
runs-on: ubuntu-latest
runs-on: ${{ matrix.runner }}
strategy:
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
arch: amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
arch: arm64
timeout-minutes: 30
permissions:
contents: read
@@ -135,22 +132,23 @@ jobs:
if: steps.check-changes.outputs.any_changed == 'true'
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Build SDK container
- name: Build SDK container for ${{ matrix.arch }}
if: steps.check-changes.outputs.any_changed == 'true'
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v7.0.0
with:
context: .
push: false
load: true
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=${{ github.event_name == 'pull_request' && 'min' || 'max' }}
platforms: ${{ matrix.platform }}
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}-${{ matrix.arch }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
- name: Scan SDK container with Trivy
- name: Scan SDK container with Trivy for ${{ matrix.arch }}
if: steps.check-changes.outputs.any_changed == 'true'
uses: ./.github/actions/trivy-scan
with:
image-name: ${{ env.IMAGE_NAME }}
image-tag: ${{ github.sha }}
image-tag: ${{ github.sha }}-${{ matrix.arch }}
fail-on-critical: 'false'
severity: 'CRITICAL'
-16
View File
@@ -5,26 +5,10 @@ on:
branches:
- 'master'
- 'v5.*'
paths:
- 'prowler/**'
- 'tests/**'
- 'pyproject.toml'
- 'poetry.lock'
- '.github/workflows/sdk-tests.yml'
- '.github/workflows/sdk-security.yml'
- '.github/actions/setup-python-poetry/**'
pull_request:
branches:
- 'master'
- 'v5.*'
paths:
- 'prowler/**'
- 'tests/**'
- 'pyproject.toml'
- 'poetry.lock'
- '.github/workflows/sdk-tests.yml'
- '.github/workflows/sdk-security.yml'
- '.github/actions/setup-python-poetry/**'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
+2 -26
View File
@@ -209,11 +209,11 @@ jobs:
echo "AWS service_paths='${STEPS_AWS_SERVICES_OUTPUTS_SERVICE_PATHS}'"
if [ "${STEPS_AWS_SERVICES_OUTPUTS_RUN_ALL}" = "true" ]; then
poetry run pytest -n auto --cov=./prowler/providers/aws --cov-report=xml:aws_coverage.xml tests/providers/aws
poetry run pytest -p no:randomly -n auto --cov=./prowler/providers/aws --cov-report=xml:aws_coverage.xml tests/providers/aws
elif [ -z "${STEPS_AWS_SERVICES_OUTPUTS_SERVICE_PATHS}" ]; then
echo "No AWS service paths detected; skipping AWS tests."
else
poetry run pytest -n auto --cov=./prowler/providers/aws --cov-report=xml:aws_coverage.xml ${STEPS_AWS_SERVICES_OUTPUTS_SERVICE_PATHS}
poetry run pytest -p no:randomly -n auto --cov=./prowler/providers/aws --cov-report=xml:aws_coverage.xml ${STEPS_AWS_SERVICES_OUTPUTS_SERVICE_PATHS}
fi
env:
STEPS_AWS_SERVICES_OUTPUTS_RUN_ALL: ${{ steps.aws-services.outputs.run_all }}
@@ -324,30 +324,6 @@ jobs:
flags: prowler-py${{ matrix.python-version }}-github
files: ./github_coverage.xml
# Okta Provider
- name: Check if Okta files changed
if: steps.check-changes.outputs.any_changed == 'true'
id: changed-okta
uses: tj-actions/changed-files@22103cc46bda19c2b464ffe86db46df6922fd323 # v47.0.5
with:
files: |
./prowler/**/okta/**
./tests/**/okta/**
./poetry.lock
- name: Run Okta tests
if: steps.changed-okta.outputs.any_changed == 'true'
run: poetry run pytest -n auto --cov=./prowler/providers/okta --cov-report=xml:okta_coverage.xml tests/providers/okta
- name: Upload Okta coverage to Codecov
if: steps.changed-okta.outputs.any_changed == 'true'
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
with:
flags: prowler-py${{ matrix.python-version }}-okta
files: ./okta_coverage.xml
# NHN Provider
- name: Check if NHN files changed
if: steps.check-changes.outputs.any_changed == 'true'
@@ -151,7 +151,7 @@ jobs:
tags: |
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-${{ matrix.arch }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=${{ github.event_name == 'pull_request' && 'min' || 'max' }},scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
# Create and push multi-architecture manifest
create-manifest:
+17 -13
View File
@@ -5,16 +5,10 @@ on:
branches:
- 'master'
- 'v5.*'
paths:
- 'ui/**'
- '.github/workflows/ui-container-checks.yml'
pull_request:
branches:
- 'master'
- 'v5.*'
paths:
- 'ui/**'
- '.github/workflows/ui-container-checks.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -63,7 +57,16 @@ jobs:
ui-container-build-and-scan:
if: github.repository == 'prowler-cloud/prowler'
runs-on: ubuntu-latest
runs-on: ${{ matrix.runner }}
strategy:
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
arch: amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
arch: arm64
timeout-minutes: 30
permissions:
contents: read
@@ -111,7 +114,7 @@ jobs:
if: steps.check-changes.outputs.any_changed == 'true'
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Build UI container
- name: Build UI container for ${{ matrix.arch }}
if: steps.check-changes.outputs.any_changed == 'true'
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v7.0.0
with:
@@ -119,17 +122,18 @@ jobs:
target: prod
push: false
load: true
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=${{ github.event_name == 'pull_request' && 'min' || 'max' }}
platforms: ${{ matrix.platform }}
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}-${{ matrix.arch }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
build-args: |
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=pk_test_51LwpXXXX
- name: Scan UI container with Trivy
- name: Scan UI container with Trivy for ${{ matrix.arch }}
if: steps.check-changes.outputs.any_changed == 'true'
uses: ./.github/actions/trivy-scan
with:
image-name: ${{ env.IMAGE_NAME }}
image-tag: ${{ github.sha }}
image-tag: ${{ github.sha }}-${{ matrix.arch }}
fail-on-critical: 'false'
severity: 'CRITICAL'
+1 -5
View File
@@ -15,10 +15,6 @@ on:
- 'ui/**'
- 'api/**' # API changes can affect UI E2E
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions: {}
jobs:
@@ -270,7 +266,7 @@ jobs:
with:
name: playwright-report
path: ui/playwright-report/
retention-days: 7
retention-days: 30
- name: Cleanup services
if: always()
+9 -34
View File
@@ -1,14 +1,14 @@
name: "UI: Tests"
name: 'UI: Tests'
on:
push:
branches:
- "master"
- "v5.*"
- 'master'
- 'v5.*'
pull_request:
branches:
- "master"
- "v5.*"
- 'master'
- 'v5.*'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
@@ -16,7 +16,7 @@ concurrency:
env:
UI_WORKING_DIR: ./ui
NODE_VERSION: "24.13.0"
NODE_VERSION: '24.13.0'
permissions: {}
@@ -42,9 +42,6 @@ jobs:
fonts.gstatic.com:443
api.github.com:443
release-assets.githubusercontent.com:443
cdn.playwright.dev:443
objects.githubusercontent.com:443
playwright.download.prss.microsoft.com:443
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -132,15 +129,11 @@ jobs:
if: steps.check-changes.outputs.any_changed == 'true'
run: pnpm run healthcheck
- name: Run pnpm audit
if: steps.check-changes.outputs.any_changed == 'true'
run: pnpm run audit
- name: Run unit tests (all - critical paths changed)
if: steps.check-changes.outputs.any_changed == 'true' && steps.critical-changes.outputs.any_changed == 'true'
run: |
echo "Critical paths changed - running ALL unit tests"
pnpm run test:unit
pnpm run test:run
- name: Run unit tests (related to changes only)
if: steps.check-changes.outputs.any_changed == 'true' && steps.critical-changes.outputs.any_changed != 'true' && steps.changed-source.outputs.all_changed_files != ''
@@ -149,7 +142,7 @@ jobs:
echo "${STEPS_CHANGED_SOURCE_OUTPUTS_ALL_CHANGED_FILES}"
# Convert space-separated to vitest related format (remove ui/ prefix for relative paths)
CHANGED_FILES=$(echo "${STEPS_CHANGED_SOURCE_OUTPUTS_ALL_CHANGED_FILES}" | tr ' ' '\n' | sed 's|^ui/||' | tr '\n' ' ')
pnpm exec vitest related $CHANGED_FILES --run --project unit
pnpm exec vitest related $CHANGED_FILES --run
env:
STEPS_CHANGED_SOURCE_OUTPUTS_ALL_CHANGED_FILES: ${{ steps.changed-source.outputs.all_changed_files }}
@@ -157,25 +150,7 @@ jobs:
if: steps.check-changes.outputs.any_changed == 'true' && steps.critical-changes.outputs.any_changed != 'true' && steps.changed-source.outputs.all_changed_files == ''
run: |
echo "Only test files changed - running ALL unit tests"
pnpm run test:unit
- name: Cache Playwright browsers
if: steps.check-changes.outputs.any_changed == 'true'
id: playwright-cache
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5.0.4
with:
path: ~/.cache/ms-playwright
key: ${{ runner.os }}-playwright-chromium-${{ hashFiles('ui/pnpm-lock.yaml') }}
restore-keys: |
${{ runner.os }}-playwright-chromium-
- name: Install Playwright Chromium browser
if: steps.check-changes.outputs.any_changed == 'true' && steps.playwright-cache.outputs.cache-hit != 'true'
run: pnpm exec playwright install chromium
- name: Run browser tests
if: steps.check-changes.outputs.any_changed == 'true'
run: pnpm run test:browser
pnpm run test:run
- name: Build application
if: steps.check-changes.outputs.any_changed == 'true'
-1
View File
@@ -8,7 +8,6 @@ rules:
- docs-bump-version.yml
- issue-triage.lock.yml
- mcp-container-build-push.yml
- nightly-arm64-container-builds.yml
- pr-merged.yml
- prepare-release.yml
- sdk-bump-version.yml
+19 -42
View File
@@ -1,34 +1,17 @@
# Priority tiers (lower = runs first, same priority = concurrent):
# P0 — fast file fixers
# P10 — validators and guards
# P20 — auto-formatters
# P30 — linters
# P40 — security scanners
# P50 — dependency validation
default_install_hook_types: [pre-commit]
repos:
## GENERAL (prek built-in — no external repo needed)
- repo: builtin
hooks:
- id: check-merge-conflict
priority: 10
- id: check-yaml
args: ["--allow-multiple-documents"]
exclude: (prowler/config/llm_config.yaml|contrib/)
priority: 10
- id: check-json
priority: 10
- id: end-of-file-fixer
priority: 0
- id: trailing-whitespace
priority: 0
- id: no-commit-to-branch
priority: 10
- id: pretty-format-json
args: ["--autofix", --no-sort-keys, --no-ensure-ascii]
priority: 10
## TOML
- repo: https://github.com/macisamuele/language-formatters-pre-commit-hooks
@@ -37,17 +20,13 @@ repos:
- id: pretty-format-toml
args: [--autofix]
files: pyproject.toml
priority: 20
## GITHUB ACTIONS
- repo: https://github.com/zizmorcore/zizmor-pre-commit
rev: v1.24.1
hooks:
- id: zizmor
# zizmor only audits workflows, composite actions and dependabot
# config; broader paths trip exit 3 ("no audit was performed").
files: ^\.github/(workflows|actions)/.+\.ya?ml$|^\.github/dependabot\.ya?ml$
priority: 30
files: ^\.github/
## BASH
- repo: https://github.com/koalaman/shellcheck-precommit
@@ -55,7 +34,6 @@ repos:
hooks:
- id: shellcheck
exclude: contrib
priority: 30
## PYTHON — SDK (prowler/, tests/, dashboard/, util/, scripts/)
- repo: https://github.com/myint/autoflake
@@ -64,8 +42,12 @@ repos:
- id: autoflake
name: "SDK - autoflake"
files: { glob: ["{prowler,tests,dashboard,util,scripts}/**/*.py"] }
args: ["--in-place", "--remove-all-unused-imports", "--remove-unused-variable"]
priority: 20
args:
[
"--in-place",
"--remove-all-unused-imports",
"--remove-unused-variable",
]
- repo: https://github.com/pycqa/isort
rev: 8.0.1
@@ -74,7 +56,6 @@ repos:
name: "SDK - isort"
files: { glob: ["{prowler,tests,dashboard,util,scripts}/**/*.py"] }
args: ["--profile", "black"]
priority: 20
- repo: https://github.com/psf/black
rev: 26.3.1
@@ -82,7 +63,6 @@ repos:
- id: black
name: "SDK - black"
files: { glob: ["{prowler,tests,dashboard,util,scripts}/**/*.py"] }
priority: 20
- repo: https://github.com/pycqa/flake8
rev: 7.3.0
@@ -91,7 +71,6 @@ repos:
name: "SDK - flake8"
files: { glob: ["{prowler,tests,dashboard,util,scripts}/**/*.py"] }
args: ["--ignore=E266,W503,E203,E501,W605"]
priority: 30
## PYTHON — API + MCP Server (ruff)
- repo: https://github.com/astral-sh/ruff-pre-commit
@@ -101,11 +80,9 @@ repos:
name: "API + MCP - ruff check"
files: { glob: ["{api,mcp_server}/**/*.py"] }
args: ["--fix"]
priority: 30
- id: ruff-format
name: "API + MCP - ruff format"
files: { glob: ["{api,mcp_server}/**/*.py"] }
priority: 20
## PYTHON — Poetry
- repo: https://github.com/python-poetry/poetry
@@ -116,28 +93,24 @@ repos:
args: ["--directory=./api"]
files: { glob: ["api/{pyproject.toml,poetry.lock}"] }
pass_filenames: false
priority: 50
- id: poetry-lock
name: API - poetry-lock
args: ["--directory=./api"]
files: { glob: ["api/{pyproject.toml,poetry.lock}"] }
pass_filenames: false
priority: 50
- id: poetry-check
name: SDK - poetry-check
args: ["--directory=./"]
files: { glob: ["{pyproject.toml,poetry.lock}"] }
pass_filenames: false
priority: 50
- id: poetry-lock
name: SDK - poetry-lock
args: ["--directory=./"]
files: { glob: ["{pyproject.toml,poetry.lock}"] }
pass_filenames: false
priority: 50
## CONTAINERS
- repo: https://github.com/hadolint/hadolint
@@ -145,7 +118,6 @@ repos:
hooks:
- id: hadolint
args: ["--ignore=DL3013"]
priority: 30
## LOCAL HOOKS
- repo: local
@@ -156,7 +128,6 @@ repos:
language: system
types: [python]
files: { glob: ["{prowler,tests,dashboard,util,scripts}/**/*.py"] }
priority: 30
- id: trufflehog
name: TruffleHog
@@ -167,7 +138,6 @@ repos:
language: system
pass_filenames: false
stages: ["pre-commit", "pre-push"]
priority: 40
- id: bandit
name: bandit
@@ -176,8 +146,8 @@ repos:
language: system
types: [python]
files: '.*\.py'
exclude: { glob: ["{contrib,skills}/**", "**/.venv/**", "**/*_test.py"] }
priority: 40
exclude:
{ glob: ["{contrib,skills}/**", "**/.venv/**", "**/*_test.py"] }
- id: safety
name: safety
@@ -186,8 +156,16 @@ repos:
entry: safety check --policy-file .safety-policy.yml
language: system
pass_filenames: false
files: { glob: ["**/pyproject.toml", "**/poetry.lock", "**/requirements*.txt", ".safety-policy.yml"] }
priority: 40
files:
{
glob:
[
"**/pyproject.toml",
"**/poetry.lock",
"**/requirements*.txt",
".safety-policy.yml",
],
}
- id: vulture
name: vulture
@@ -196,4 +174,3 @@ repos:
language: system
types: [python]
files: '.*\.py'
priority: 40
+3 -11
View File
@@ -15,7 +15,7 @@ Use these skills for detailed patterns on-demand:
|-------|-------------|-----|
| `typescript` | Const types, flat interfaces, utility types | [SKILL.md](skills/typescript/SKILL.md) |
| `react-19` | No useMemo/useCallback, React Compiler | [SKILL.md](skills/react-19/SKILL.md) |
| `nextjs-16` | App Router, Server Actions, proxy.ts, streaming | [SKILL.md](skills/nextjs-16/SKILL.md) |
| `nextjs-15` | App Router, Server Actions, streaming | [SKILL.md](skills/nextjs-15/SKILL.md) |
| `tailwind-4` | cn() utility, no var() in className | [SKILL.md](skills/tailwind-4/SKILL.md) |
| `playwright` | Page Object Model, MCP workflow, selectors | [SKILL.md](skills/playwright/SKILL.md) |
| `pytest` | Fixtures, mocking, markers, parametrize | [SKILL.md](skills/pytest/SKILL.md) |
@@ -60,14 +60,11 @@ When performing these actions, ALWAYS invoke the corresponding skill FIRST:
|--------|-------|
| Add changelog entry for a PR or feature | `prowler-changelog` |
| Adding DRF pagination or permissions | `django-drf` |
| Adding a compliance output formatter (per-provider class + table dispatcher) | `prowler-compliance` |
| Adding indexes or constraints to database tables | `django-migration-psql` |
| Adding new providers | `prowler-provider` |
| Adding privilege escalation detection queries | `prowler-attack-paths-query` |
| Adding services to existing providers | `prowler-provider` |
| After creating/modifying a skill | `skill-sync` |
| App Router / Server Actions | `nextjs-16` |
| Auditing check-to-requirement mappings as a cloud auditor | `prowler-compliance` |
| App Router / Server Actions | `nextjs-15` |
| Building AI chat features | `ai-sdk-5` |
| Committing changes | `prowler-commit` |
| Configuring MCP servers in agentic workflows | `gh-aw` |
@@ -81,7 +78,6 @@ When performing these actions, ALWAYS invoke the corresponding skill FIRST:
| Creating a git commit | `prowler-commit` |
| Creating new checks | `prowler-sdk-check` |
| Creating new skills | `skill-creator` |
| Creating or reviewing Django migrations | `django-migration-psql` |
| Creating/modifying Prowler UI components | `prowler-ui` |
| Creating/modifying models, views, serializers | `prowler-api` |
| Creating/updating compliance frameworks | `prowler-compliance` |
@@ -89,7 +85,6 @@ When performing these actions, ALWAYS invoke the corresponding skill FIRST:
| Debugging gh-aw compilation errors | `gh-aw` |
| Fill .github/pull_request_template.md (Context/Description/Steps to review/Checklist) | `prowler-pr` |
| Fixing bug | `tdd` |
| Fixing compliance JSON bugs (duplicate IDs, empty Section, stale refs) | `prowler-compliance` |
| General Prowler development questions | `prowler` |
| Implementing JSON:API endpoints | `django-drf` |
| Implementing feature | `tdd` |
@@ -107,8 +102,6 @@ When performing these actions, ALWAYS invoke the corresponding skill FIRST:
| Review changelog format and conventions | `prowler-changelog` |
| Reviewing JSON:API compliance | `jsonapi` |
| Reviewing compliance framework PRs | `prowler-compliance-review` |
| Running makemigrations or pgmakemigrations | `django-migration-psql` |
| Syncing compliance framework with upstream catalog | `prowler-compliance` |
| Testing RLS tenant isolation | `prowler-test-api` |
| Testing hooks or utilities | `vitest` |
| Troubleshoot why a skill is missing from AGENTS.md auto-invoke | `skill-sync` |
@@ -136,7 +129,6 @@ When performing these actions, ALWAYS invoke the corresponding skill FIRST:
| Writing React components | `react-19` |
| Writing TypeScript types/interfaces | `typescript` |
| Writing Vitest tests | `vitest` |
| Writing data backfill or data migration | `django-migration-psql` |
| Writing documentation | `prowler-docs` |
| Writing unit tests for UI | `vitest` |
@@ -150,7 +142,7 @@ Prowler is an open-source cloud security assessment tool supporting AWS, Azure,
|-----------|----------|------------|
| SDK | `prowler/` | Python 3.10+, Poetry 2.3+ |
| API | `api/` | Django 5.1, DRF, Celery |
| UI | `ui/` | Next.js 16, React 19, Tailwind 4 |
| UI | `ui/` | Next.js 15, React 19, Tailwind 4 |
| MCP Server | `mcp_server/` | FastMCP, Python 3.12+ |
| Dashboard | `dashboard/` | Dash, Plotly |
+6 -29
View File
@@ -1,34 +1,11 @@
# Do you want to learn on how to...
- [Contribute with your code or fixes to Prowler](https://docs.prowler.com/developer-guide/introduction)
- [Create a new provider](https://docs.prowler.com/developer-guide/provider)
- [Create a new service](https://docs.prowler.com/developer-guide/services)
- [Create a new check for a provider](https://docs.prowler.com/developer-guide/checks)
- [Create a new security compliance framework](https://docs.prowler.com/developer-guide/security-compliance-framework)
- [Add a custom output format](https://docs.prowler.com/developer-guide/outputs)
- [Add a new integration](https://docs.prowler.com/developer-guide/integrations)
- [Contribute with documentation](https://docs.prowler.com/developer-guide/documentation)
- [Write unit tests](https://docs.prowler.com/developer-guide/unit-testing)
- [Write integration tests](https://docs.prowler.com/developer-guide/integration-testing)
- [Write end-to-end tests](https://docs.prowler.com/developer-guide/end2end-testing)
- [Debug Prowler](https://docs.prowler.com/developer-guide/debugging)
- [Configure checks](https://docs.prowler.com/developer-guide/configurable-checks)
- [Rename checks](https://docs.prowler.com/developer-guide/renaming-checks)
- [Follow the check metadata guidelines](https://docs.prowler.com/developer-guide/check-metadata-guidelines)
- [Extend the MCP server](https://docs.prowler.com/developer-guide/mcp-server)
- [Extend Lighthouse AI](https://docs.prowler.com/developer-guide/lighthouse-architecture)
- [Add AI skills](https://docs.prowler.com/developer-guide/ai-skills)
Provider-specific developer notes:
- [AWS](https://docs.prowler.com/developer-guide/aws-details)
- [Azure](https://docs.prowler.com/developer-guide/azure-details)
- [Google Cloud](https://docs.prowler.com/developer-guide/gcp-details)
- [Alibaba Cloud](https://docs.prowler.com/developer-guide/alibabacloud-details)
- [Kubernetes](https://docs.prowler.com/developer-guide/kubernetes-details)
- [Microsoft 365](https://docs.prowler.com/developer-guide/m365-details)
- [GitHub](https://docs.prowler.com/developer-guide/github-details)
- [LLM](https://docs.prowler.com/developer-guide/llm-details)
- Contribute with your code or fixes to Prowler
- Create a new check for a provider
- Create a new security compliance framework
- Add a custom output format
- Add a new integration
- Contribute with documentation
Want some swag as appreciation for your contribution?
+1 -1
View File
@@ -6,7 +6,7 @@ LABEL org.opencontainers.image.source="https://github.com/prowler-cloud/prowler"
ARG POWERSHELL_VERSION=7.5.0
ENV POWERSHELL_VERSION=${POWERSHELL_VERSION}
ARG TRIVY_VERSION=0.70.0
ARG TRIVY_VERSION=0.69.2
ENV TRIVY_VERSION=${TRIVY_VERSION}
ARG ZIZMOR_VERSION=1.24.1
+12 -13
View File
@@ -104,23 +104,22 @@ Every AWS provider scan will enqueue an Attack Paths ingestion job automatically
| Provider | Checks | Services | [Compliance Frameworks](https://docs.prowler.com/projects/prowler-open-source/en/latest/tutorials/compliance/) | [Categories](https://docs.prowler.com/projects/prowler-open-source/en/latest/tutorials/misc/#categories) | Support | Interface |
|---|---|---|---|---|---|---|
| AWS | 595 | 84 | 43 | 17 | Official | UI, API, CLI |
| Azure | 167 | 22 | 19 | 16 | Official | UI, API, CLI |
| GCP | 102 | 18 | 17 | 12 | Official | UI, API, CLI |
| Kubernetes | 83 | 7 | 7 | 11 | Official | UI, API, CLI |
| GitHub | 24 | 3 | 1 | 5 | Official | UI, API, CLI |
| M365 | 101 | 10 | 4 | 10 | Official | UI, API, CLI |
| OCI | 51 | 14 | 4 | 10 | Official | UI, API, CLI |
| Alibaba Cloud | 61 | 9 | 4 | 9 | Official | UI, API, CLI |
| Cloudflare | 29 | 3 | 0 | 5 | Official | UI, API, CLI |
| AWS | 572 | 83 | 41 | 17 | Official | UI, API, CLI |
| Azure | 165 | 20 | 18 | 13 | Official | UI, API, CLI |
| GCP | 100 | 13 | 15 | 11 | Official | UI, API, CLI |
| Kubernetes | 83 | 7 | 7 | 9 | Official | UI, API, CLI |
| GitHub | 21 | 2 | 1 | 2 | Official | UI, API, CLI |
| M365 | 89 | 9 | 4 | 5 | Official | UI, API, CLI |
| OCI | 48 | 13 | 3 | 10 | Official | UI, API, CLI |
| Alibaba Cloud | 61 | 9 | 3 | 9 | Official | UI, API, CLI |
| Cloudflare | 29 | 2 | 0 | 5 | Official | UI, API, CLI |
| IaC | [See `trivy` docs.](https://trivy.dev/latest/docs/coverage/iac/) | N/A | N/A | N/A | Official | UI, API, CLI |
| MongoDB Atlas | 10 | 3 | 0 | 8 | Official | UI, API, CLI |
| LLM | [See `promptfoo` docs.](https://www.promptfoo.dev/docs/red-team/plugins/) | N/A | N/A | N/A | Official | CLI |
| Image | N/A | N/A | N/A | N/A | Official | CLI, API |
| Google Workspace | 25 | 4 | 2 | 4 | Official | UI, API, CLI |
| OpenStack | 34 | 5 | 0 | 9 | Official | UI, API, CLI |
| Vercel | 26 | 6 | 0 | 5 | Official | UI, API, CLI |
| Okta | 1 | 1 | 0 | 1 | Official | CLI |
| Google Workspace | 1 | 1 | 0 | 1 | Official | CLI |
| OpenStack | 27 | 4 | 0 | 8 | Official | UI, API, CLI |
| Vercel | 30 | 6 | 0 | 5 | Official | CLI |
| NHN | 6 | 2 | 1 | 0 | Unofficial | CLI |
> [!Note]
+1 -54
View File
@@ -2,59 +2,6 @@
All notable changes to the **Prowler API** are documented in this file.
## [1.28.0] (Prowler UNRELEASED)
### 🚀 Added
- GIN index on `findings(categories, resource_services, resource_regions, resource_types)` to speed up `/api/v1/finding-groups` array filters [(#11001)](https://github.com/prowler-cloud/prowler/pull/11001)
### 🔄 Changed
- Remove orphaned `gin_resources_search_idx` declaration from `Resource.Meta.indexes` (DB index dropped in `0072_drop_unused_indexes`) [(#11001)](https://github.com/prowler-cloud/prowler/pull/11001)
---
## [1.27.2] (Prowler UNRELEASED)
### 🐞 Fixed
- Attack Paths: BEDROCK-001 and BEDROCK-002 now target roles trusting `bedrock-agentcore.amazonaws.com` instead of `bedrock.amazonaws.com`, eliminating false positives against regular Bedrock service roles (Agents, Knowledge Bases, model invocation) [(#11141)](https://github.com/prowler-cloud/prowler/pull/11141)
---
## [1.27.1] (Prowler v5.26.1)
### 🐞 Fixed
- `POST /api/v1/scans` was intermittently failing with `Scan matching query does not exist` in the `scan-perform` worker; the Celery task is now published via `transaction.on_commit` so the worker cannot read the Scan before the dispatch-wide transaction commits [(#11122)](https://github.com/prowler-cloud/prowler/pull/11122)
---
## [1.27.0] (Prowler v5.26.0)
### 🚀 Added
- `scan-reset-ephemeral-resources` post-scan task zeroes `failed_findings_count` for resources missing from the latest full-scope scan, keeping ephemeral resources from polluting the Resources page sort [(#10929)](https://github.com/prowler-cloud/prowler/pull/10929)
### 🔄 Changed
- ASD Essential Eight (AWS) compliance framework support [(#10982)](https://github.com/prowler-cloud/prowler/pull/10982)
### 🔐 Security
- `trivy` binary from 0.69.2 to 0.70.0 and `cryptography` from 46.0.6 to 46.0.7 (transitive via prowler SDK) in the API image for CVE-2026-33186 and CVE-2026-39892 [(#10978)](https://github.com/prowler-cloud/prowler/pull/10978)
---
## [1.26.1] (Prowler v5.25.1)
### 🐞 Fixed
- Attack Paths: AWS scans no longer fail when enabled regions cannot be retrieved, and scans stuck in `scheduled` state are now cleaned up after the stale threshold [(#10917)](https://github.com/prowler-cloud/prowler/pull/10917)
- Scan report and compliance downloads now redirect to a presigned S3 URL instead of streaming through the API worker, preventing gunicorn timeouts on large files [(#10927)](https://github.com/prowler-cloud/prowler/pull/10927)
---
## [1.26.0] (Prowler v5.25.0)
### 🚀 Added
@@ -65,7 +12,7 @@ All notable changes to the **Prowler API** are documented in this file.
### 🔄 Changed
- Allows tenant owners to expel users from their organizations [(#10787)](https://github.com/prowler-cloud/prowler/pull/10787)
- Allows tenant owners to expel users from their organizations [(#10787)](https://github.com/prowler-cloud/prowler/pull/10787)
- `aggregate_findings`, `aggregate_attack_surface`, `aggregate_scan_resource_group_summaries` and `aggregate_scan_category_summaries` now upsert via `bulk_create(update_conflicts=True, ...)` instead of the prior `ignore_conflicts=True` / plain INSERT / `already backfilled` short-circuit. Re-runs triggered by the post-mute reaggregation pipeline no longer trip the `unique_*_per_scan` constraints nor silently drop updates, and are race-safe under concurrent writers (e.g. scan completion overlapping with a fresh mute rule) [(#10843)](https://github.com/prowler-cloud/prowler/pull/10843)
- Rename the scan-category and scan-resource-group summary aggregators from `backfill_*` to `aggregate_*` [(#10843)](https://github.com/prowler-cloud/prowler/pull/10843)
+1 -1
View File
@@ -5,7 +5,7 @@ LABEL maintainer="https://github.com/prowler-cloud/api"
ARG POWERSHELL_VERSION=7.5.0
ENV POWERSHELL_VERSION=${POWERSHELL_VERSION}
ARG TRIVY_VERSION=0.70.0
ARG TRIVY_VERSION=0.69.2
ENV TRIVY_VERSION=${TRIVY_VERSION}
ARG ZIZMOR_VERSION=1.24.1
+63 -63
View File
@@ -2504,61 +2504,61 @@ dev = ["bandit", "coverage", "flake8", "pydocstyle", "pylint", "pytest", "pytest
[[package]]
name = "cryptography"
version = "46.0.7"
version = "46.0.6"
description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers."
optional = false
python-versions = "!=3.9.0,!=3.9.1,>=3.8"
groups = ["main", "dev"]
files = [
{file = "cryptography-46.0.7-cp311-abi3-macosx_10_9_universal2.whl", hash = "sha256:ea42cbe97209df307fdc3b155f1b6fa2577c0defa8f1f7d3be7d31d189108ad4"},
{file = "cryptography-46.0.7-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b36a4695e29fe69215d75960b22577197aca3f7a25b9cf9d165dcfe9d80bc325"},
{file = "cryptography-46.0.7-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5ad9ef796328c5e3c4ceed237a183f5d41d21150f972455a9d926593a1dcb308"},
{file = "cryptography-46.0.7-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:73510b83623e080a2c35c62c15298096e2a5dc8d51c3b4e1740211839d0dea77"},
{file = "cryptography-46.0.7-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:cbd5fb06b62bd0721e1170273d3f4d5a277044c47ca27ee257025146c34cbdd1"},
{file = "cryptography-46.0.7-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:420b1e4109cc95f0e5700eed79908cef9268265c773d3a66f7af1eef53d409ef"},
{file = "cryptography-46.0.7-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:24402210aa54baae71d99441d15bb5a1919c195398a87b563df84468160a65de"},
{file = "cryptography-46.0.7-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:8a469028a86f12eb7d2fe97162d0634026d92a21f3ae0ac87ed1c4a447886c83"},
{file = "cryptography-46.0.7-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:9694078c5d44c157ef3162e3bf3946510b857df5a3955458381d1c7cfc143ddb"},
{file = "cryptography-46.0.7-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:42a1e5f98abb6391717978baf9f90dc28a743b7d9be7f0751a6f56a75d14065b"},
{file = "cryptography-46.0.7-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:91bbcb08347344f810cbe49065914fe048949648f6bd5c2519f34619142bbe85"},
{file = "cryptography-46.0.7-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:5d1c02a14ceb9148cc7816249f64f623fbfee39e8c03b3650d842ad3f34d637e"},
{file = "cryptography-46.0.7-cp311-abi3-win32.whl", hash = "sha256:d23c8ca48e44ee015cd0a54aeccdf9f09004eba9fc96f38c911011d9ff1bd457"},
{file = "cryptography-46.0.7-cp311-abi3-win_amd64.whl", hash = "sha256:397655da831414d165029da9bc483bed2fe0e75dde6a1523ec2fe63f3c46046b"},
{file = "cryptography-46.0.7-cp314-cp314t-macosx_10_9_universal2.whl", hash = "sha256:d151173275e1728cf7839aaa80c34fe550c04ddb27b34f48c232193df8db5842"},
{file = "cryptography-46.0.7-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:db0f493b9181c7820c8134437eb8b0b4792085d37dbb24da050476ccb664e59c"},
{file = "cryptography-46.0.7-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ebd6daf519b9f189f85c479427bbd6e9c9037862cf8fe89ee35503bd209ed902"},
{file = "cryptography-46.0.7-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:b7b412817be92117ec5ed95f880defe9cf18a832e8cafacf0a22337dc1981b4d"},
{file = "cryptography-46.0.7-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:fbfd0e5f273877695cb93baf14b185f4878128b250cc9f8e617ea0c025dfb022"},
{file = "cryptography-46.0.7-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:ffca7aa1d00cf7d6469b988c581598f2259e46215e0140af408966a24cf086ce"},
{file = "cryptography-46.0.7-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:60627cf07e0d9274338521205899337c5d18249db56865f943cbe753aa96f40f"},
{file = "cryptography-46.0.7-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:80406c3065e2c55d7f49a9550fe0c49b3f12e5bfff5dedb727e319e1afb9bf99"},
{file = "cryptography-46.0.7-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:c5b1ccd1239f48b7151a65bc6dd54bcfcc15e028c8ac126d3fada09db0e07ef1"},
{file = "cryptography-46.0.7-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:d5f7520159cd9c2154eb61eb67548ca05c5774d39e9c2c4339fd793fe7d097b2"},
{file = "cryptography-46.0.7-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:fcd8eac50d9138c1d7fc53a653ba60a2bee81a505f9f8850b6b2888555a45d0e"},
{file = "cryptography-46.0.7-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:65814c60f8cc400c63131584e3e1fad01235edba2614b61fbfbfa954082db0ee"},
{file = "cryptography-46.0.7-cp314-cp314t-win32.whl", hash = "sha256:fdd1736fed309b4300346f88f74cd120c27c56852c3838cab416e7a166f67298"},
{file = "cryptography-46.0.7-cp314-cp314t-win_amd64.whl", hash = "sha256:e06acf3c99be55aa3b516397fe42f5855597f430add9c17fa46bf2e0fb34c9bb"},
{file = "cryptography-46.0.7-cp38-abi3-macosx_10_9_universal2.whl", hash = "sha256:462ad5cb1c148a22b2e3bcc5ad52504dff325d17daf5df8d88c17dda1f75f2a4"},
{file = "cryptography-46.0.7-cp38-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:84d4cced91f0f159a7ddacad249cc077e63195c36aac40b4150e7a57e84fffe7"},
{file = "cryptography-46.0.7-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:128c5edfe5e5938b86b03941e94fac9ee793a94452ad1365c9fc3f4f62216832"},
{file = "cryptography-46.0.7-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:5e51be372b26ef4ba3de3c167cd3d1022934bc838ae9eaad7e644986d2a3d163"},
{file = "cryptography-46.0.7-cp38-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:cdf1a610ef82abb396451862739e3fc93b071c844399e15b90726ef7470eeaf2"},
{file = "cryptography-46.0.7-cp38-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:1d25aee46d0c6f1a501adcddb2d2fee4b979381346a78558ed13e50aa8a59067"},
{file = "cryptography-46.0.7-cp38-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:cdfbe22376065ffcf8be74dc9a909f032df19bc58a699456a21712d6e5eabfd0"},
{file = "cryptography-46.0.7-cp38-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:abad9dac36cbf55de6eb49badd4016806b3165d396f64925bf2999bcb67837ba"},
{file = "cryptography-46.0.7-cp38-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:935ce7e3cfdb53e3536119a542b839bb94ec1ad081013e9ab9b7cfd478b05006"},
{file = "cryptography-46.0.7-cp38-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:35719dc79d4730d30f1c2b6474bd6acda36ae2dfae1e3c16f2051f215df33ce0"},
{file = "cryptography-46.0.7-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:7bbc6ccf49d05ac8f7d7b5e2e2c33830d4fe2061def88210a126d130d7f71a85"},
{file = "cryptography-46.0.7-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:a1529d614f44b863a7b480c6d000fe93b59acee9c82ffa027cfadc77521a9f5e"},
{file = "cryptography-46.0.7-cp38-abi3-win32.whl", hash = "sha256:f247c8c1a1fb45e12586afbb436ef21ff1e80670b2861a90353d9b025583d246"},
{file = "cryptography-46.0.7-cp38-abi3-win_amd64.whl", hash = "sha256:506c4ff91eff4f82bdac7633318a526b1d1309fc07ca76a3ad182cb5b686d6d3"},
{file = "cryptography-46.0.7-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:fc9ab8856ae6cf7c9358430e49b368f3108f050031442eaeb6b9d87e4dcf4e4f"},
{file = "cryptography-46.0.7-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:d3b99c535a9de0adced13d159c5a9cf65c325601aa30f4be08afd680643e9c15"},
{file = "cryptography-46.0.7-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:d02c738dacda7dc2a74d1b2b3177042009d5cab7c7079db74afc19e56ca1b455"},
{file = "cryptography-46.0.7-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:04959522f938493042d595a736e7dbdff6eb6cc2339c11465b3ff89343b65f65"},
{file = "cryptography-46.0.7-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:3986ac1dee6def53797289999eabe84798ad7817f3e97779b5061a95b0ee4968"},
{file = "cryptography-46.0.7-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:258514877e15963bd43b558917bc9f54cf7cf866c38aa576ebf47a77ddbc43a4"},
{file = "cryptography-46.0.7.tar.gz", hash = "sha256:e4cfd68c5f3e0bfdad0d38e023239b96a2fe84146481852dffbcca442c245aa5"},
{file = "cryptography-46.0.6-cp311-abi3-macosx_10_9_universal2.whl", hash = "sha256:64235194bad039a10bb6d2d930ab3323baaec67e2ce36215fd0952fad0930ca8"},
{file = "cryptography-46.0.6-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:26031f1e5ca62fcb9d1fcb34b2b60b390d1aacaa15dc8b895a9ed00968b97b30"},
{file = "cryptography-46.0.6-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9a693028b9cbe51b5a1136232ee8f2bc242e4e19d456ded3fa7c86e43c713b4a"},
{file = "cryptography-46.0.6-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:67177e8a9f421aa2d3a170c3e56eca4e0128883cf52a071a7cbf53297f18b175"},
{file = "cryptography-46.0.6-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:d9528b535a6c4f8ff37847144b8986a9a143585f0540fbcb1a98115b543aa463"},
{file = "cryptography-46.0.6-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:22259338084d6ae497a19bae5d4c66b7ca1387d3264d1c2c0e72d9e9b6a77b97"},
{file = "cryptography-46.0.6-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:760997a4b950ff00d418398ad73fbc91aa2894b5c1db7ccb45b4f68b42a63b3c"},
{file = "cryptography-46.0.6-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:3dfa6567f2e9e4c5dceb8ccb5a708158a2a871052fa75c8b78cb0977063f1507"},
{file = "cryptography-46.0.6-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:cdcd3edcbc5d55757e5f5f3d330dd00007ae463a7e7aa5bf132d1f22a4b62b19"},
{file = "cryptography-46.0.6-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:d4e4aadb7fc1f88687f47ca20bb7227981b03afaae69287029da08096853b738"},
{file = "cryptography-46.0.6-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:2b417edbe8877cda9022dde3a008e2deb50be9c407eef034aeeb3a8b11d9db3c"},
{file = "cryptography-46.0.6-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:380343e0653b1c9d7e1f55b52aaa2dbb2fdf2730088d48c43ca1c7c0abb7cc2f"},
{file = "cryptography-46.0.6-cp311-abi3-win32.whl", hash = "sha256:bcb87663e1f7b075e48c3be3ecb5f0b46c8fc50b50a97cf264e7f60242dca3f2"},
{file = "cryptography-46.0.6-cp311-abi3-win_amd64.whl", hash = "sha256:6739d56300662c468fddb0e5e291f9b4d084bead381667b9e654c7dd81705124"},
{file = "cryptography-46.0.6-cp314-cp314t-macosx_10_9_universal2.whl", hash = "sha256:2ef9e69886cbb137c2aef9772c2e7138dc581fad4fcbcf13cc181eb5a3ab6275"},
{file = "cryptography-46.0.6-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7f417f034f91dcec1cb6c5c35b07cdbb2ef262557f701b4ecd803ee8cefed4f4"},
{file = "cryptography-46.0.6-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d24c13369e856b94892a89ddf70b332e0b70ad4a5c43cf3e9cb71d6d7ffa1f7b"},
{file = "cryptography-46.0.6-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:aad75154a7ac9039936d50cf431719a2f8d4ed3d3c277ac03f3339ded1a5e707"},
{file = "cryptography-46.0.6-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:3c21d92ed15e9cfc6eb64c1f5a0326db22ca9c2566ca46d845119b45b4400361"},
{file = "cryptography-46.0.6-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:4668298aef7cddeaf5c6ecc244c2302a2b8e40f384255505c22875eebb47888b"},
{file = "cryptography-46.0.6-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:8ce35b77aaf02f3b59c90b2c8a05c73bac12cea5b4e8f3fbece1f5fddea5f0ca"},
{file = "cryptography-46.0.6-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:c89eb37fae9216985d8734c1afd172ba4927f5a05cfd9bf0e4863c6d5465b013"},
{file = "cryptography-46.0.6-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:ed418c37d095aeddf5336898a132fba01091f0ac5844e3e8018506f014b6d2c4"},
{file = "cryptography-46.0.6-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:69cf0056d6947edc6e6760e5f17afe4bea06b56a9ac8a06de9d2bd6b532d4f3a"},
{file = "cryptography-46.0.6-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8e7304c4f4e9490e11efe56af6713983460ee0780f16c63f219984dab3af9d2d"},
{file = "cryptography-46.0.6-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:b928a3ca837c77a10e81a814a693f2295200adb3352395fad024559b7be7a736"},
{file = "cryptography-46.0.6-cp314-cp314t-win32.whl", hash = "sha256:97c8115b27e19e592a05c45d0dd89c57f81f841cc9880e353e0d3bf25b2139ed"},
{file = "cryptography-46.0.6-cp314-cp314t-win_amd64.whl", hash = "sha256:c797e2517cb7880f8297e2c0f43bb910e91381339336f75d2c1c2cbf811b70b4"},
{file = "cryptography-46.0.6-cp38-abi3-macosx_10_9_universal2.whl", hash = "sha256:12cae594e9473bca1a7aceb90536060643128bb274fcea0fc459ab90f7d1ae7a"},
{file = "cryptography-46.0.6-cp38-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:639301950939d844a9e1c4464d7e07f902fe9a7f6b215bb0d4f28584729935d8"},
{file = "cryptography-46.0.6-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ed3775295fb91f70b4027aeba878d79b3e55c0b3e97eaa4de71f8f23a9f2eb77"},
{file = "cryptography-46.0.6-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:8927ccfbe967c7df312ade694f987e7e9e22b2425976ddbf28271d7e58845290"},
{file = "cryptography-46.0.6-cp38-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:b12c6b1e1651e42ab5de8b1e00dc3b6354fdfd778e7fa60541ddacc27cd21410"},
{file = "cryptography-46.0.6-cp38-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:063b67749f338ca9c5a0b7fe438a52c25f9526b851e24e6c9310e7195aad3b4d"},
{file = "cryptography-46.0.6-cp38-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:02fad249cb0e090b574e30b276a3da6a149e04ee2f049725b1f69e7b8351ec70"},
{file = "cryptography-46.0.6-cp38-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:7e6142674f2a9291463e5e150090b95a8519b2fb6e6aaec8917dd8d094ce750d"},
{file = "cryptography-46.0.6-cp38-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:456b3215172aeefb9284550b162801d62f5f264a081049a3e94307fe20792cfa"},
{file = "cryptography-46.0.6-cp38-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:341359d6c9e68834e204ceaf25936dffeafea3829ab80e9503860dcc4f4dac58"},
{file = "cryptography-46.0.6-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:9a9c42a2723999a710445bc0d974e345c32adfd8d2fac6d8a251fa829ad31cfb"},
{file = "cryptography-46.0.6-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:6617f67b1606dfd9fe4dbfa354a9508d4a6d37afe30306fe6c101b7ce3274b72"},
{file = "cryptography-46.0.6-cp38-abi3-win32.whl", hash = "sha256:7f6690b6c55e9c5332c0b59b9c8a3fb232ebf059094c17f9019a51e9827df91c"},
{file = "cryptography-46.0.6-cp38-abi3-win_amd64.whl", hash = "sha256:79e865c642cfc5c0b3eb12af83c35c5aeff4fa5c672dc28c43721c2c9fdd2f0f"},
{file = "cryptography-46.0.6-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:2ea0f37e9a9cf0df2952893ad145fd9627d326a59daec9b0802480fa3bcd2ead"},
{file = "cryptography-46.0.6-pp311-pypy311_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:a3e84d5ec9ba01f8fd03802b2147ba77f0c8f2617b2aff254cedd551844209c8"},
{file = "cryptography-46.0.6-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:12f0fa16cc247b13c43d56d7b35287ff1569b5b1f4c5e87e92cc4fcc00cd10c0"},
{file = "cryptography-46.0.6-pp311-pypy311_pp73-manylinux_2_34_aarch64.whl", hash = "sha256:50575a76e2951fe7dbd1f56d181f8c5ceeeb075e9ff88e7ad997d2f42af06e7b"},
{file = "cryptography-46.0.6-pp311-pypy311_pp73-manylinux_2_34_x86_64.whl", hash = "sha256:90e5f0a7b3be5f40c3a0a0eafb32c681d8d2c181fc2a1bdabe9b3f611d9f6b1a"},
{file = "cryptography-46.0.6-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:6728c49e3b2c180ef26f8e9f0a883a2c585638db64cf265b49c9ba10652d430e"},
{file = "cryptography-46.0.6.tar.gz", hash = "sha256:27550628a518c5c6c903d84f637fbecf287f6cb9ced3804838a1295dc1fd0759"},
]
[package.dependencies]
@@ -2571,7 +2571,7 @@ nox = ["nox[uv] (>=2024.4.15)"]
pep8test = ["check-sdist", "click (>=8.0.1)", "mypy (>=1.14)", "ruff (>=0.11.11)"]
sdist = ["build (>=1.0.0)"]
ssh = ["bcrypt (>=3.1.5)"]
test = ["certifi (>=2024)", "cryptography-vectors (==46.0.7)", "pretend (>=0.7)", "pytest (>=7.4.0)", "pytest-benchmark (>=4.0)", "pytest-cov (>=2.10.1)", "pytest-xdist (>=3.5.0)"]
test = ["certifi (>=2024)", "cryptography-vectors (==46.0.6)", "pretend (>=0.7)", "pytest (>=7.4.0)", "pytest-benchmark (>=4.0)", "pytest-cov (>=2.10.1)", "pytest-xdist (>=3.5.0)"]
test-randomorder = ["pytest-randomly"]
[[package]]
@@ -6665,7 +6665,7 @@ files = [
[[package]]
name = "prowler"
version = "5.26.0"
version = "5.25.0"
description = "Prowler is an Open Source security tool to perform AWS, GCP and Azure security best practices assessments, audits, incident response, continuous monitoring, hardening and forensics readiness. It contains hundreds of controls covering CIS, NIST 800, NIST CSF, CISA, RBI, FedRAMP, PCI-DSS, GDPR, HIPAA, FFIEC, SOC2, GXP, AWS Well-Architected Framework Security Pillar, AWS Foundational Technical Review (FTR), ENS (Spanish National Security Scheme) and your custom security frameworks."
optional = false
python-versions = ">=3.10,<3.13"
@@ -6720,7 +6720,7 @@ boto3 = "1.40.61"
botocore = "1.40.61"
cloudflare = "4.3.1"
colorama = "0.4.6"
cryptography = "46.0.7"
cryptography = "46.0.6"
dash = "3.1.1"
dash-bootstrap-components = "2.0.3"
defusedxml = "0.7.1"
@@ -6754,8 +6754,8 @@ uuid6 = "2024.7.10"
[package.source]
type = "git"
url = "https://github.com/prowler-cloud/prowler.git"
reference = "master"
resolved_reference = "16798e293da365965120961e6539e3a9756564f9"
reference = "v5.25"
resolved_reference = "e252058af491b41608dbaaba2975acd7c1728174"
[[package]]
name = "psutil"
@@ -7912,26 +7912,26 @@ shaping = ["uharfbuzz"]
[[package]]
name = "requests"
version = "2.33.1"
version = "2.32.5"
description = "Python HTTP for Humans."
optional = false
python-versions = ">=3.10"
python-versions = ">=3.9"
groups = ["main", "dev"]
files = [
{file = "requests-2.33.1-py3-none-any.whl", hash = "sha256:4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a"},
{file = "requests-2.33.1.tar.gz", hash = "sha256:18817f8c57c6263968bc123d237e3b8b08ac046f5456bd1e307ee8f4250d3517"},
{file = "requests-2.32.5-py3-none-any.whl", hash = "sha256:2462f94637a34fd532264295e186976db0f5d453d1cdd31473c85a6a161affb6"},
{file = "requests-2.32.5.tar.gz", hash = "sha256:dbba0bac56e100853db0ea71b82b4dfd5fe2bf6d3754a8893c3af500cec7d7cf"},
]
[package.dependencies]
certifi = ">=2023.5.7"
certifi = ">=2017.4.17"
charset_normalizer = ">=2,<4"
idna = ">=2.5,<4"
PySocks = {version = ">=1.5.6,<1.5.7 || >1.5.7", optional = true, markers = "extra == \"socks\""}
urllib3 = ">=1.26,<3"
urllib3 = ">=1.21.1,<3"
[package.extras]
socks = ["PySocks (>=1.5.6,!=1.5.7)"]
use-chardet-on-py3 = ["chardet (>=3.0.2,<8)"]
use-chardet-on-py3 = ["chardet (>=3.0.2,<6)"]
[[package]]
name = "requests-file"
@@ -9424,4 +9424,4 @@ files = [
[metadata]
lock-version = "2.1"
python-versions = ">=3.11,<3.13"
content-hash = "a3ab982d11a87d951ff15694d2ca7fd51f1f51a451abb0baa067ccf6966367a8"
content-hash = "7446e89a46709f976a572231862072de86e7bf01ed90a72bea526b9ab05a82b3"
+3 -3
View File
@@ -25,7 +25,7 @@ dependencies = [
"defusedxml==0.7.1",
"gunicorn==23.0.0",
"lxml==5.3.2",
"prowler @ git+https://github.com/prowler-cloud/prowler.git@master",
"prowler @ git+https://github.com/prowler-cloud/prowler.git@v5.25",
"psycopg2-binary==2.9.9",
"pytest-celery[redis] (==1.3.0)",
"sentry-sdk[django] (==2.56.0)",
@@ -50,7 +50,7 @@ name = "prowler-api"
package-mode = false
# Needed for the SDK compatibility
requires-python = ">=3.11,<3.13"
version = "1.28.0"
version = "1.26.0"
[project.scripts]
celery = "src.backend.config.settings.celery"
@@ -63,7 +63,6 @@ docker = "7.1.0"
filelock = "3.20.3"
freezegun = "1.5.1"
mypy = "1.10.1"
prek = "0.3.9"
pylint = "3.2.5"
pytest = "9.0.3"
pytest-cov = "5.0.0"
@@ -75,3 +74,4 @@ ruff = "0.5.0"
safety = "3.7.0"
tqdm = "4.67.1"
vulture = "2.14"
prek = "0.3.9"
+2 -2
View File
@@ -52,7 +52,7 @@ class ApiConfig(AppConfig):
"check_and_fix_socialaccount_sites_migration",
]
# Skip eager Neo4j init for tests, some Django commands, and Celery (prefork pool: driver must stay lazy, no post_fork hook)
# Skip Neo4j initialization during tests, some Django commands, and Celery
if getattr(settings, "TESTING", False) or (
len(sys.argv) > 1
and (
@@ -64,7 +64,7 @@ class ApiConfig(AppConfig):
)
):
logger.info(
"Skipping eager Neo4j init: tests, some Django commands, or Celery prefork pool (driver stays lazy)"
"Skipping Neo4j initialization because tests, some Django commands or Celery"
)
else:
@@ -484,8 +484,8 @@ AWS_BEDROCK_PRIVESC_PASSROLE_CODE_INTERPRETER = AttackPathsQueryDefinition(
OR action = '*'
)
// Find roles that trust the Bedrock AgentCore service (can be passed to a code interpreter)
MATCH path_target = (aws)--(target_role:AWSRole)-[:TRUSTS_AWS_PRINCIPAL]->(:AWSPrincipal {{arn: 'bedrock-agentcore.amazonaws.com'}})
// Find roles that trust Bedrock service (can be passed to Bedrock)
MATCH path_target = (aws)--(target_role:AWSRole)-[:TRUSTS_AWS_PRINCIPAL]->(:AWSPrincipal {{arn: 'bedrock.amazonaws.com'}})
WHERE any(resource IN stmt_passrole.resource WHERE
resource = '*'
OR target_role.arn CONTAINS resource
@@ -536,8 +536,8 @@ AWS_BEDROCK_PRIVESC_INVOKE_CODE_INTERPRETER = AttackPathsQueryDefinition(
OR action = '*'
)
// Find roles that trust the Bedrock AgentCore service (already attached to existing code interpreters)
MATCH path_target = (aws)--(target_role:AWSRole)-[:TRUSTS_AWS_PRINCIPAL]->(:AWSPrincipal {{arn: 'bedrock-agentcore.amazonaws.com'}})
// Find roles that trust Bedrock service (already attached to existing code interpreters)
MATCH path_target = (aws)--(target_role:AWSRole)-[:TRUSTS_AWS_PRINCIPAL]->(:AWSPrincipal {{arn: 'bedrock.amazonaws.com'}})
WITH collect(path_principal) + collect(path_target) AS paths
UNWIND paths AS p
@@ -1,31 +0,0 @@
from functools import partial
from django.db import migrations
from api.db_utils import create_index_on_partitions, drop_index_on_partitions
class Migration(migrations.Migration):
atomic = False
dependencies = [
("api", "0090_attack_paths_cleanup_priority"),
]
operations = [
migrations.RunPython(
partial(
create_index_on_partitions,
parent_table="findings",
index_name="gin_find_arrays_idx",
columns="categories, resource_services, resource_regions, resource_types",
method="GIN",
all_partitions=True,
),
reverse_code=partial(
drop_index_on_partitions,
parent_table="findings",
index_name="gin_find_arrays_idx",
),
)
]
@@ -1,73 +0,0 @@
import django.contrib.postgres.indexes
from django.db import migrations
INDEX_NAME = "gin_find_arrays_idx"
PARENT_TABLE = "findings"
def create_parent_and_attach(apps, schema_editor):
with schema_editor.connection.cursor() as cursor:
# Idempotent: the parent index may already exist if it was created
# manually on an environment before this migration ran.
cursor.execute(
f"CREATE INDEX IF NOT EXISTS {INDEX_NAME} ON ONLY {PARENT_TABLE} "
f"USING gin (categories, resource_services, resource_regions, resource_types)"
)
cursor.execute(
"SELECT inhrelid::regclass::text "
"FROM pg_inherits "
"WHERE inhparent = %s::regclass",
[PARENT_TABLE],
)
for (partition,) in cursor.fetchall():
child_idx = f"{partition.replace('.', '_')}_{INDEX_NAME}"
# ALTER INDEX ... ATTACH PARTITION has no IF NOT ATTACHED clause,
# so check pg_inherits first to keep the migration re-runnable.
cursor.execute(
"""
SELECT 1
FROM pg_inherits i
JOIN pg_class p ON p.oid = i.inhparent
JOIN pg_class c ON c.oid = i.inhrelid
WHERE p.relname = %s AND c.relname = %s
""",
[INDEX_NAME, child_idx],
)
if cursor.fetchone() is None:
cursor.execute(f"ALTER INDEX {INDEX_NAME} ATTACH PARTITION {child_idx}")
def drop_parent_index(apps, schema_editor):
with schema_editor.connection.cursor() as cursor:
cursor.execute(f"DROP INDEX IF EXISTS {INDEX_NAME}")
class Migration(migrations.Migration):
dependencies = [
("api", "0091_findings_arrays_gin_index_partitions"),
]
operations = [
migrations.SeparateDatabaseAndState(
state_operations=[
migrations.AddIndex(
model_name="finding",
index=django.contrib.postgres.indexes.GinIndex(
fields=[
"categories",
"resource_services",
"resource_regions",
"resource_types",
],
name=INDEX_NAME,
),
),
],
database_operations=[
migrations.RunPython(
create_parent_and_attach,
reverse_code=drop_parent_index,
),
],
),
]
+1 -57
View File
@@ -595,40 +595,10 @@ class Scan(RowLevelSecurityProtectedModel):
objects = ActiveProviderManager()
all_objects = models.Manager()
_SCOPING_SCANNER_ARG_KEYS_CACHE: tuple[str, ...] | None = None
@classmethod
def get_scoping_scanner_arg_keys(cls) -> tuple[str, ...]:
"""Return the scanner_args keys that mark a scan as scoped.
Derived from ``prowler.lib.scan.scan.Scan.__init__`` so the API stays
in sync with whatever the SDK actually accepts as filters. Cached at
class level — the signature is stable for the process lifetime.
"""
if cls._SCOPING_SCANNER_ARG_KEYS_CACHE is None:
import inspect
from prowler.lib.scan.scan import Scan as ProwlerScan
params = inspect.signature(ProwlerScan.__init__).parameters
cls._SCOPING_SCANNER_ARG_KEYS_CACHE = tuple(
name for name in params if name not in ("self", "provider")
)
return cls._SCOPING_SCANNER_ARG_KEYS_CACHE
class TriggerChoices(models.TextChoices):
SCHEDULED = "scheduled", _("Scheduled")
MANUAL = "manual", _("Manual")
# Trigger values for scans that ran the SDK end-to-end. Imported scans (or
# any future trigger) are intentionally NOT in this set — they may carry
# only a partial slice of resources, so post-scan logic that depends on a
# full-scope sweep (e.g. resetting ephemeral resource findings) must skip
# them by default.
LIVE_SCAN_TRIGGERS = frozenset(
(TriggerChoices.SCHEDULED.value, TriggerChoices.MANUAL.value)
)
id = models.UUIDField(primary_key=True, default=uuid7, editable=False)
name = models.CharField(
blank=True, null=True, max_length=100, validators=[MinLengthValidator(3)]
@@ -711,24 +681,6 @@ class Scan(RowLevelSecurityProtectedModel):
class JSONAPIMeta:
resource_name = "scans"
def is_full_scope(self) -> bool:
"""Return True if this scan ran with no scoping filters at all.
Used to gate post-scan operations (such as resetting the
failed_findings_count of resources missing from the scan) that are only
safe when the scan covered every check, service, and category. Imported
scans are NOT full-scope by definition — they may carry only a partial
slice of resources, so they're rejected via ``trigger`` even before the
scanner_args check.
"""
if self.trigger not in self.LIVE_SCAN_TRIGGERS:
return False
scanner_args = self.scanner_args or {}
for key in self.get_scoping_scanner_arg_keys():
if scanner_args.get(key):
return False
return True
class AttackPathsScan(RowLevelSecurityProtectedModel):
objects = ActiveProviderManager()
@@ -946,6 +898,7 @@ class Resource(RowLevelSecurityProtectedModel):
OpClass(Upper("name"), name="gin_trgm_ops"),
name="res_name_trgm_idx",
),
GinIndex(fields=["text_search"], name="gin_resources_search_idx"),
models.Index(fields=["tenant_id", "id"], name="resources_tenant_id_idx"),
models.Index(
fields=["tenant_id", "provider_id"],
@@ -1151,15 +1104,6 @@ class Finding(PostgresPartitionedModel, RowLevelSecurityProtectedModel):
fields=["tenant_id", "scan_id", "check_id"],
name="find_tenant_scan_check_idx",
),
GinIndex(
fields=[
"categories",
"resource_services",
"resource_regions",
"resource_types",
],
name="gin_find_arrays_idx",
),
]
class JSONAPIMeta:
File diff suppressed because it is too large Load Diff
+28 -59
View File
@@ -3841,14 +3841,9 @@ class TestScanViewSet:
"prowler-output-123_threatscore_report.pdf",
)
presigned_url = (
"https://test-bucket.s3.amazonaws.com/"
"tenant-id/scan-id/threatscore/prowler-output-123_threatscore_report.pdf"
"?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=300"
)
mock_s3_client = Mock()
mock_s3_client.list_objects_v2.return_value = {"Contents": [{"Key": pdf_key}]}
mock_s3_client.generate_presigned_url.return_value = presigned_url
mock_s3_client.get_object.return_value = {"Body": io.BytesIO(b"pdf-bytes")}
mock_env_str.return_value = bucket
mock_get_s3_client.return_value = mock_s3_client
@@ -3857,26 +3852,19 @@ class TestScanViewSet:
url = reverse("scan-threatscore", kwargs={"pk": scan.id})
response = authenticated_client.get(url)
assert response.status_code == status.HTTP_302_FOUND
assert response["Location"] == presigned_url
mock_s3_client.list_objects_v2.assert_called_once()
mock_s3_client.generate_presigned_url.assert_called_once_with(
"get_object",
Params={
"Bucket": bucket,
"Key": pdf_key,
"ResponseContentDisposition": (
'attachment; filename="prowler-output-123_threatscore_report.pdf"'
),
"ResponseContentType": "application/pdf",
},
ExpiresIn=300,
assert response.status_code == status.HTTP_200_OK
assert response["Content-Type"] == "application/pdf"
assert response["Content-Disposition"].endswith(
'"prowler-output-123_threatscore_report.pdf"'
)
assert response.content == b"pdf-bytes"
mock_s3_client.list_objects_v2.assert_called_once()
mock_s3_client.get_object.assert_called_once_with(Bucket=bucket, Key=pdf_key)
def test_report_s3_success(self, authenticated_client, scans_fixture, monkeypatch):
"""
When output_location is an S3 URL and the object exists,
the view should return a 302 redirect to a presigned S3 URL.
When output_location is an S3 URL and the S3 client returns the file successfully,
the view should return the ZIP file with HTTP 200 and proper headers.
"""
scan = scans_fixture[0]
bucket = "test-bucket"
@@ -3890,33 +3878,22 @@ class TestScanViewSet:
type("env", (), {"str": lambda self, *args, **kwargs: "test-bucket"})(),
)
presigned_url = (
"https://test-bucket.s3.amazonaws.com/report.zip"
"?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=300"
)
class FakeS3Client:
def head_object(self, Bucket, Key):
def get_object(self, Bucket, Key):
assert Bucket == bucket
assert Key == key
return {}
def generate_presigned_url(self, ClientMethod, Params, ExpiresIn):
assert ClientMethod == "get_object"
assert Params["Bucket"] == bucket
assert Params["Key"] == key
assert Params["ResponseContentDisposition"] == (
'attachment; filename="report.zip"'
)
assert ExpiresIn == 300
return presigned_url
return {"Body": io.BytesIO(b"s3 zip content")}
monkeypatch.setattr("api.v1.views.get_s3_client", lambda: FakeS3Client())
url = reverse("scan-report", kwargs={"pk": scan.id})
response = authenticated_client.get(url)
assert response.status_code == status.HTTP_302_FOUND
assert response["Location"] == presigned_url
assert response.status_code == 200
expected_filename = os.path.basename("report.zip")
content_disposition = response.get("Content-Disposition")
assert content_disposition.startswith('attachment; filename="')
assert f'filename="{expected_filename}"' in content_disposition
assert response.content == b"s3 zip content"
def test_report_s3_success_no_local_files(
self, authenticated_client, scans_fixture, monkeypatch
@@ -4055,31 +4032,23 @@ class TestScanViewSet:
)
match_key = "path/compliance/mitre_attack_aws.csv"
presigned_url = (
"https://test-bucket.s3.amazonaws.com/path/compliance/mitre_attack_aws.csv"
"?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=300"
)
class FakeS3Client:
def list_objects_v2(self, Bucket, Prefix):
return {"Contents": [{"Key": match_key}]}
def generate_presigned_url(self, ClientMethod, Params, ExpiresIn):
assert ClientMethod == "get_object"
assert Params["Key"] == match_key
assert Params["ResponseContentDisposition"] == (
'attachment; filename="mitre_attack_aws.csv"'
)
assert ExpiresIn == 300
return presigned_url
def get_object(self, Bucket, Key):
return {"Body": io.BytesIO(b"ignored")}
monkeypatch.setattr("api.v1.views.get_s3_client", lambda: FakeS3Client())
framework = match_key.split("/")[-1].split(".")[0]
url = reverse("scan-compliance", kwargs={"pk": scan.id, "name": framework})
resp = authenticated_client.get(url)
assert resp.status_code == status.HTTP_302_FOUND
assert resp["Location"] == presigned_url
assert resp.status_code == status.HTTP_200_OK
cd = resp["Content-Disposition"]
assert cd.startswith('attachment; filename="')
assert cd.endswith('filename="mitre_attack_aws.csv"')
def test_compliance_s3_not_found(
self, authenticated_client, scans_fixture, monkeypatch
@@ -4282,8 +4251,8 @@ class TestScanViewSet:
scan.save()
fake_client = MagicMock()
fake_client.head_object.side_effect = ClientError(
{"Error": {"Code": "NoSuchKey"}}, "HeadObject"
fake_client.get_object.side_effect = ClientError(
{"Error": {"Code": "NoSuchKey"}}, "GetObject"
)
mock_get_s3_client.return_value = fake_client
@@ -4306,8 +4275,8 @@ class TestScanViewSet:
scan.save()
fake_client = MagicMock()
fake_client.head_object.side_effect = ClientError(
{"Error": {"Code": "AccessDenied"}}, "HeadObject"
fake_client.get_object.side_effect = ClientError(
{"Error": {"Code": "AccessDenied"}}, "GetObject"
)
mock_get_s3_client.return_value = fake_client
+56 -149
View File
@@ -4,7 +4,6 @@ import json
import logging
import os
import time
import uuid
from collections import defaultdict
from copy import deepcopy
from datetime import datetime, timedelta, timezone
@@ -17,7 +16,7 @@ from allauth.socialaccount.providers.github.views import GitHubOAuth2Adapter
from allauth.socialaccount.providers.google.views import GoogleOAuth2Adapter
from allauth.socialaccount.providers.saml.views import FinishACSView, LoginView
from botocore.exceptions import ClientError, NoCredentialsError, ParamValidationError
from celery import chain, states
from celery import chain
from celery.result import AsyncResult
from config.custom_logging import BackendLogger
from config.env import env
@@ -54,14 +53,13 @@ from django.db.models import (
)
from django.db.models.fields.json import KeyTextTransform
from django.db.models.functions import Cast, Coalesce, RowNumber
from django.http import HttpResponse, HttpResponseBase, HttpResponseRedirect, QueryDict
from django.http import HttpResponse, QueryDict
from django.shortcuts import redirect
from django.urls import reverse
from django.utils.dateparse import parse_date
from django.utils.decorators import method_decorator
from django.views.decorators.cache import cache_control
from django_celery_beat.models import PeriodicTask
from django_celery_results.models import TaskResult
from drf_spectacular.settings import spectacular_settings
from drf_spectacular.types import OpenApiTypes
from drf_spectacular.utils import (
@@ -424,7 +422,7 @@ class SchemaView(SpectacularAPIView):
def get(self, request, *args, **kwargs):
spectacular_settings.TITLE = "Prowler API"
spectacular_settings.VERSION = "1.28.0"
spectacular_settings.VERSION = "1.26.0"
spectacular_settings.DESCRIPTION = (
"Prowler API specification.\n\nThis file is auto-generated."
)
@@ -2082,38 +2080,24 @@ class ScanViewSet(BaseRLSViewSet):
},
)
def _load_file(
self,
path_pattern,
s3=False,
bucket=None,
list_objects=False,
content_type=None,
):
def _load_file(self, path_pattern, s3=False, bucket=None, list_objects=False):
"""
Resolve a report file location and return the bytes (filesystem) or a redirect (S3).
Loads a binary file (e.g., ZIP or CSV) and returns its content and filename.
Depending on the input parameters, this method supports loading:
- From S3 using a direct key, returns a 302 to a short-lived presigned URL.
- From S3 by listing objects under a prefix and matching suffix, returns a 302 to a short-lived presigned URL.
- From the local filesystem using glob pattern matching, returns the file bytes.
The S3 branch never streams bytes through the worker; this prevents gunicorn
worker timeouts on large reports.
- From S3 using a direct key.
- From S3 by listing objects under a prefix and matching suffix.
- From the local filesystem using glob pattern matching.
Args:
path_pattern (str): The key or glob pattern representing the file location.
s3 (bool, optional): Whether the file is stored in S3. Defaults to False.
bucket (str, optional): The name of the S3 bucket, required if `s3=True`. Defaults to None.
list_objects (bool, optional): If True and `s3=True`, list objects by prefix to find the file. Defaults to False.
content_type (str, optional): On the S3 branch, forwarded as `ResponseContentType`
so the presigned download advertises the same Content-Type the API used to send.
Ignored on the filesystem branch.
Returns:
tuple[bytes, str]: For the filesystem branch, the file content and filename.
HttpResponseRedirect: For the S3 branch on success, a 302 redirect to a presigned `GetObject` URL.
Response: For any error path, a DRF `Response` with an appropriate status and detail.
tuple[bytes, str]: A tuple containing the file content as bytes and the filename if successful.
Response: A DRF `Response` object with an appropriate status and error detail if an error occurs.
"""
if s3:
try:
@@ -2160,45 +2144,25 @@ class ScanViewSet(BaseRLSViewSet):
# path_pattern here is prefix, but in compliance we build correct suffix check before
key = keys[0]
else:
# path_pattern is exact key; HEAD before presigning to preserve the 404 contract.
# path_pattern is exact key
key = path_pattern
try:
client.head_object(Bucket=bucket, Key=key)
except ClientError as e:
code = e.response.get("Error", {}).get("Code")
if code in ("NoSuchKey", "404"):
return Response(
{
"detail": "The scan has no reports, or the report generation task has not started yet."
},
status=status.HTTP_404_NOT_FOUND,
)
try:
s3_obj = client.get_object(Bucket=bucket, Key=key)
except ClientError as e:
code = e.response.get("Error", {}).get("Code")
if code == "NoSuchKey":
return Response(
{"detail": "There is a problem with credentials."},
status=status.HTTP_403_FORBIDDEN,
{
"detail": "The scan has no reports, or the report generation task has not started yet."
},
status=status.HTTP_404_NOT_FOUND,
)
return Response(
{"detail": "There is a problem with credentials."},
status=status.HTTP_403_FORBIDDEN,
)
content = s3_obj["Body"].read()
filename = os.path.basename(key)
# escape quotes and strip CR/LF so a malformed key cannot break out of the header
safe_filename = (
filename.replace("\\", "\\\\")
.replace('"', '\\"')
.replace("\r", "")
.replace("\n", "")
)
params = {
"Bucket": bucket,
"Key": key,
"ResponseContentDisposition": f'attachment; filename="{safe_filename}"',
}
if content_type:
params["ResponseContentType"] = content_type
url = client.generate_presigned_url(
"get_object",
Params=params,
ExpiresIn=300,
)
return HttpResponseRedirect(url)
else:
files = glob.glob(path_pattern)
if not files:
@@ -2241,16 +2205,12 @@ class ScanViewSet(BaseRLSViewSet):
bucket = env.str("DJANGO_OUTPUT_S3_AWS_OUTPUT_BUCKET", "")
key_prefix = scan.output_location.removeprefix(f"s3://{bucket}/")
loader = self._load_file(
key_prefix,
s3=True,
bucket=bucket,
list_objects=False,
content_type="application/x-zip-compressed",
key_prefix, s3=True, bucket=bucket, list_objects=False
)
else:
loader = self._load_file(scan.output_location, s3=False)
if isinstance(loader, HttpResponseBase):
if isinstance(loader, Response):
return loader
content, filename = loader
@@ -2288,19 +2248,13 @@ class ScanViewSet(BaseRLSViewSet):
prefix = os.path.join(
os.path.dirname(key_prefix), "compliance", f"{name}.csv"
)
loader = self._load_file(
prefix,
s3=True,
bucket=bucket,
list_objects=True,
content_type="text/csv",
)
loader = self._load_file(prefix, s3=True, bucket=bucket, list_objects=True)
else:
base = os.path.dirname(scan.output_location)
pattern = os.path.join(base, "compliance", f"*_{name}.csv")
loader = self._load_file(pattern, s3=False)
if isinstance(loader, HttpResponseBase):
if isinstance(loader, Response):
return loader
content, filename = loader
@@ -2333,19 +2287,13 @@ class ScanViewSet(BaseRLSViewSet):
"cis",
"*_cis_report.pdf",
)
loader = self._load_file(
prefix,
s3=True,
bucket=bucket,
list_objects=True,
content_type="application/pdf",
)
loader = self._load_file(prefix, s3=True, bucket=bucket, list_objects=True)
else:
base = os.path.dirname(scan.output_location)
pattern = os.path.join(base, "cis", "*_cis_report.pdf")
loader = self._load_file(pattern, s3=False)
if isinstance(loader, HttpResponseBase):
if isinstance(loader, Response):
return loader
content, filename = loader
@@ -2379,19 +2327,13 @@ class ScanViewSet(BaseRLSViewSet):
"threatscore",
"*_threatscore_report.pdf",
)
loader = self._load_file(
prefix,
s3=True,
bucket=bucket,
list_objects=True,
content_type="application/pdf",
)
loader = self._load_file(prefix, s3=True, bucket=bucket, list_objects=True)
else:
base = os.path.dirname(scan.output_location)
pattern = os.path.join(base, "threatscore", "*_threatscore_report.pdf")
loader = self._load_file(pattern, s3=False)
if isinstance(loader, HttpResponseBase):
if isinstance(loader, Response):
return loader
content, filename = loader
@@ -2425,19 +2367,13 @@ class ScanViewSet(BaseRLSViewSet):
"ens",
"*_ens_report.pdf",
)
loader = self._load_file(
prefix,
s3=True,
bucket=bucket,
list_objects=True,
content_type="application/pdf",
)
loader = self._load_file(prefix, s3=True, bucket=bucket, list_objects=True)
else:
base = os.path.dirname(scan.output_location)
pattern = os.path.join(base, "ens", "*_ens_report.pdf")
loader = self._load_file(pattern, s3=False)
if isinstance(loader, HttpResponseBase):
if isinstance(loader, Response):
return loader
content, filename = loader
@@ -2470,19 +2406,13 @@ class ScanViewSet(BaseRLSViewSet):
"nis2",
"*_nis2_report.pdf",
)
loader = self._load_file(
prefix,
s3=True,
bucket=bucket,
list_objects=True,
content_type="application/pdf",
)
loader = self._load_file(prefix, s3=True, bucket=bucket, list_objects=True)
else:
base = os.path.dirname(scan.output_location)
pattern = os.path.join(base, "nis2", "*_nis2_report.pdf")
loader = self._load_file(pattern, s3=False)
if isinstance(loader, HttpResponseBase):
if isinstance(loader, Response):
return loader
content, filename = loader
@@ -2515,19 +2445,13 @@ class ScanViewSet(BaseRLSViewSet):
"csa",
"*_csa_report.pdf",
)
loader = self._load_file(
prefix,
s3=True,
bucket=bucket,
list_objects=True,
content_type="application/pdf",
)
loader = self._load_file(prefix, s3=True, bucket=bucket, list_objects=True)
else:
base = os.path.dirname(scan.output_location)
pattern = os.path.join(base, "csa", "*_csa_report.pdf")
loader = self._load_file(pattern, s3=False)
if isinstance(loader, HttpResponseBase):
if isinstance(loader, Response):
return loader
content, filename = loader
@@ -2536,45 +2460,28 @@ class ScanViewSet(BaseRLSViewSet):
def create(self, request, *args, **kwargs):
input_serializer = self.get_serializer(data=request.data)
input_serializer.is_valid(raise_exception=True)
# Broker publish is deferred to on_commit so the worker cannot read
# Scan before BaseRLSViewSet's dispatch-wide atomic commits.
pre_task_id = str(uuid.uuid4())
with transaction.atomic():
scan = input_serializer.save()
scan.task_id = pre_task_id
scan.save(update_fields=["task_id"])
attack_paths_db_utils.create_attack_paths_scan(
tenant_id=self.request.tenant_id,
scan_id=str(scan.id),
provider_id=str(scan.provider_id),
with transaction.atomic():
task = perform_scan_task.apply_async(
kwargs={
"tenant_id": self.request.tenant_id,
"scan_id": str(scan.id),
"provider_id": str(scan.provider_id),
# Disabled for now
# checks_to_execute=scan.scanner_args.get("checks_to_execute")
},
)
task_result, _ = TaskResult.objects.get_or_create(
task_id=pre_task_id,
defaults={"status": states.PENDING, "task_name": "scan-perform"},
)
prowler_task, _ = Task.objects.update_or_create(
id=pre_task_id,
tenant_id=self.request.tenant_id,
defaults={"task_runner_task": task_result},
)
attack_paths_db_utils.create_attack_paths_scan(
tenant_id=self.request.tenant_id,
scan_id=str(scan.id),
provider_id=str(scan.provider_id),
)
scan_kwargs = {
"tenant_id": self.request.tenant_id,
"scan_id": str(scan.id),
"provider_id": str(scan.provider_id),
# Disabled for now
# checks_to_execute=scan.scanner_args.get("checks_to_execute")
}
transaction.on_commit(
lambda: perform_scan_task.apply_async(
kwargs=scan_kwargs, task_id=pre_task_id
)
)
prowler_task = Task.objects.get(id=task.id)
scan.task_id = task.id
scan.save(update_fields=["task_id"])
self.response_serializer_class = TaskSerializer
output_serializer = self.get_serializer(prowler_task)
+1 -43
View File
@@ -49,7 +49,7 @@ def start_aws_ingestion(
}
boto3_session = get_boto3_session(prowler_api_provider, prowler_sdk_provider)
regions: list[str] = resolve_aws_regions(prowler_api_provider, prowler_sdk_provider)
regions: list[str] = list(prowler_sdk_provider._enabled_regions)
requested_syncs = list(cartography_aws.RESOURCE_FUNCTIONS.keys())
sync_args = cartography_aws._build_aws_sync_kwargs(
@@ -226,48 +226,6 @@ def get_boto3_session(
return boto3_session
def resolve_aws_regions(
prowler_api_provider: ProwlerAPIProvider,
prowler_sdk_provider: ProwlerSDKProvider,
) -> list[str]:
"""Resolve the regions to scan, falling back when `_enabled_regions` is `None`.
The SDK silently sets `_enabled_regions` to `None` when `ec2:DescribeRegions`
fails (missing IAM permission, transient error). Without a fallback the
Cartography ingestion crashes with a non-actionable `TypeError`. Try the
user's `audited_regions` next, then the partition's static region list.
Excluded regions are honored on every branch.
"""
if prowler_sdk_provider._enabled_regions is not None:
regions = set(prowler_sdk_provider._enabled_regions)
elif prowler_sdk_provider.identity.audited_regions:
regions = set(prowler_sdk_provider.identity.audited_regions)
else:
partition = prowler_sdk_provider.identity.partition
try:
regions = prowler_sdk_provider.get_available_aws_service_regions(
"ec2", partition
)
except KeyError:
raise RuntimeError(
f"No region data available for partition {partition!r}; "
f"cannot determine regions to scan for "
f"{prowler_api_provider.uid}"
)
logger.warning(
f"Could not enumerate enabled regions for AWS account "
f"{prowler_api_provider.uid}; falling back to all regions in "
f"partition {partition!r}"
)
excluded = set(getattr(prowler_sdk_provider, "_excluded_regions", None) or ())
return sorted(regions - excluded)
def get_aioboto3_session(boto3_session: boto3.Session) -> aioboto3.Session:
return aioboto3.Session(botocore_session=boto3_session._session)
@@ -18,45 +18,28 @@ logger = get_task_logger(__name__)
def cleanup_stale_attack_paths_scans() -> dict:
"""
Mark stale `AttackPathsScan` rows as `FAILED`.
Find `EXECUTING` `AttackPathsScan` scans whose workers are dead or that have
exceeded the stale threshold, and mark them as `FAILED`.
Covers two stuck-state scenarios:
1. `EXECUTING` scans whose workers are dead, or that have exceeded the
stale threshold while alive.
2. `SCHEDULED` scans that never made it to a worker — parent scan
crashed before dispatch, broker lost the message, etc. Detected by
age plus the parent `Scan` no longer being in flight.
"""
threshold = timedelta(minutes=ATTACK_PATHS_SCAN_STALE_THRESHOLD_MINUTES)
now = datetime.now(tz=timezone.utc)
cutoff = now - threshold
cleaned_up: list[str] = []
cleaned_up.extend(_cleanup_stale_executing_scans(cutoff))
cleaned_up.extend(_cleanup_stale_scheduled_scans(cutoff))
logger.info(
f"Stale `AttackPathsScan` cleanup: {len(cleaned_up)} scan(s) cleaned up"
)
return {"cleaned_up_count": len(cleaned_up), "scan_ids": cleaned_up}
def _cleanup_stale_executing_scans(cutoff: datetime) -> list[str]:
"""
Two-pass detection for `EXECUTING` scans:
Two-pass detection:
1. If `TaskResult.worker` exists, ping the worker.
- Dead worker: cleanup immediately (any age).
- Alive + past threshold: revoke the task, then cleanup.
- Alive + within threshold: skip.
2. If no worker field: fall back to time-based heuristic only.
"""
executing_scans = list(
threshold = timedelta(minutes=ATTACK_PATHS_SCAN_STALE_THRESHOLD_MINUTES)
now = datetime.now(tz=timezone.utc)
cutoff = now - threshold
executing_scans = (
AttackPathsScan.all_objects.using(MainRouter.admin_db)
.filter(state=StateChoices.EXECUTING)
.select_related("task__task_runner_task")
)
# Cache worker liveness so each worker is pinged at most once
executing_scans = list(executing_scans)
workers = {
tr.worker
for scan in executing_scans
@@ -65,7 +48,7 @@ def _cleanup_stale_executing_scans(cutoff: datetime) -> list[str]:
}
worker_alive = {w: _is_worker_alive(w) for w in workers}
cleaned_up: list[str] = []
cleaned_up = []
for scan in executing_scans:
task_result = (
@@ -82,7 +65,9 @@ def _cleanup_stale_executing_scans(cutoff: datetime) -> list[str]:
# Alive but stale — revoke before cleanup
_revoke_task(task_result)
reason = "Scan exceeded stale threshold — cleaned up by periodic task"
reason = (
"Scan exceeded stale threshold — " "cleaned up by periodic task"
)
else:
reason = "Worker dead — cleaned up by periodic task"
else:
@@ -97,57 +82,10 @@ def _cleanup_stale_executing_scans(cutoff: datetime) -> list[str]:
if _cleanup_scan(scan, task_result, reason):
cleaned_up.append(str(scan.id))
return cleaned_up
def _cleanup_stale_scheduled_scans(cutoff: datetime) -> list[str]:
"""
Cleanup `SCHEDULED` scans that never reached a worker.
Detection:
- `state == SCHEDULED`
- `started_at < cutoff`
- parent `Scan` is no longer in flight (terminal state or missing). This
avoids cleaning up rows whose parent Prowler scan is legitimately still
running.
For each match: revoke the queued task (best-effort; harmless if already
consumed), atomically flip to `FAILED`, and mark the `TaskResult`. The
temp Neo4j database is never created while `SCHEDULED`, so no drop is
needed.
"""
scheduled_scans = list(
AttackPathsScan.all_objects.using(MainRouter.admin_db)
.filter(
state=StateChoices.SCHEDULED,
started_at__lt=cutoff,
)
.select_related("task__task_runner_task", "scan")
logger.info(
f"Stale `AttackPathsScan` cleanup: {len(cleaned_up)} scan(s) cleaned up"
)
cleaned_up: list[str] = []
parent_terminal = (
StateChoices.COMPLETED,
StateChoices.FAILED,
StateChoices.CANCELLED,
)
for scan in scheduled_scans:
parent_scan = scan.scan
if parent_scan is not None and parent_scan.state not in parent_terminal:
continue
task_result = (
getattr(scan.task, "task_runner_task", None) if scan.task else None
)
if task_result:
_revoke_task(task_result, terminate=False)
reason = "Scan never started — cleaned up by periodic task"
if _cleanup_scheduled_scan(scan, task_result, reason):
cleaned_up.append(str(scan.id))
return cleaned_up
return {"cleaned_up_count": len(cleaned_up), "scan_ids": cleaned_up}
def _is_worker_alive(worker: str) -> bool:
@@ -160,17 +98,12 @@ def _is_worker_alive(worker: str) -> bool:
return True
def _revoke_task(task_result, terminate: bool = True) -> None:
"""Revoke a Celery task. Non-fatal on failure.
`terminate=True` SIGTERMs the worker if the task is mid-execution; use
for EXECUTING cleanup. `terminate=False` only marks the task id revoked
across workers, so any worker pulling the queued message discards it;
use for SCHEDULED cleanup where the task hasn't run yet.
"""
def _revoke_task(task_result) -> None:
"""Send `SIGTERM` to a hung Celery task. Non-fatal on failure."""
try:
kwargs = {"terminate": True, "signal": "SIGTERM"} if terminate else {}
current_app.control.revoke(task_result.task_id, **kwargs)
current_app.control.revoke(
task_result.task_id, terminate=True, signal="SIGTERM"
)
logger.info(f"Revoked task {task_result.task_id}")
except Exception:
logger.exception(f"Failed to revoke task {task_result.task_id}")
@@ -192,64 +125,28 @@ def _cleanup_scan(scan, task_result, reason: str) -> bool:
except Exception:
logger.exception(f"Failed to drop temp database {tmp_db_name}")
fresh_scan = _finalize_failed_scan(scan, StateChoices.EXECUTING, reason)
if fresh_scan is None:
return False
# Mark `TaskResult` as `FAILURE` (not RLS-protected, outside lock)
if task_result:
task_result.status = states.FAILURE
task_result.date_done = datetime.now(tz=timezone.utc)
task_result.save(update_fields=["status", "date_done"])
recover_graph_data_ready(fresh_scan)
logger.info(f"Cleaned up stale scan {scan_id_str}: {reason}")
return True
def _cleanup_scheduled_scan(scan, task_result, reason: str) -> bool:
"""
Clean up a `SCHEDULED` scan that never reached a worker.
Skips the temp Neo4j drop — the database is only created once the worker
enters `EXECUTING`, so dropping it here just produces noisy log output.
Returns `True` if the scan was actually cleaned up, `False` if skipped.
"""
scan_id_str = str(scan.id)
fresh_scan = _finalize_failed_scan(scan, StateChoices.SCHEDULED, reason)
if fresh_scan is None:
return False
if task_result:
task_result.status = states.FAILURE
task_result.date_done = datetime.now(tz=timezone.utc)
task_result.save(update_fields=["status", "date_done"])
logger.info(f"Cleaned up scheduled scan {scan_id_str}: {reason}")
return True
def _finalize_failed_scan(scan, expected_state: str, reason: str):
"""
Atomically lock the row, verify it's still in `expected_state`, and
mark it `FAILED`. Returns the locked row on success, `None` if the
row is gone or has already moved on.
"""
scan_id_str = str(scan.id)
# 2. Lock row, verify still EXECUTING, mark FAILED — all atomic
with rls_transaction(str(scan.tenant_id)):
try:
fresh_scan = AttackPathsScan.objects.select_for_update().get(id=scan.id)
except AttackPathsScan.DoesNotExist:
logger.warning(f"Scan {scan_id_str} no longer exists, skipping")
return None
return False
if fresh_scan.state != expected_state:
if fresh_scan.state != StateChoices.EXECUTING:
logger.info(f"Scan {scan_id_str} is now {fresh_scan.state}, skipping")
return None
return False
_mark_scan_finished(fresh_scan, StateChoices.FAILED, {"global_error": reason})
return fresh_scan
# 3. Mark `TaskResult` as `FAILURE` (not RLS-protected, outside lock)
if task_result:
task_result.status = states.FAILURE
task_result.date_done = datetime.now(tz=timezone.utc)
task_result.save(update_fields=["status", "date_done"])
# 4. Recover graph_data_ready if provider data still exists
recover_graph_data_ready(fresh_scan)
logger.info(f"Cleaned up stale scan {scan_id_str}: {reason}")
return True
@@ -67,52 +67,25 @@ def retrieve_attack_paths_scan(
return None
def set_attack_paths_scan_task_id(
tenant_id: str,
scan_pk: str,
task_id: str,
) -> None:
"""Persist the Celery `task_id` on the `AttackPathsScan` row.
Called at dispatch time (when `apply_async` returns) so the row carries
the task id even while still `SCHEDULED`. This lets the periodic
cleanup revoke queued messages for scans that never reached a worker.
"""
with rls_transaction(tenant_id):
ProwlerAPIAttackPathsScan.objects.filter(id=scan_pk).update(task_id=task_id)
def starting_attack_paths_scan(
attack_paths_scan: ProwlerAPIAttackPathsScan,
task_id: str,
cartography_config: CartographyConfig,
) -> bool:
"""Flip the row from `SCHEDULED` to `EXECUTING` atomically.
Returns `False` if the row is gone or has already moved past
`SCHEDULED` (e.g., periodic cleanup raced ahead and marked it
`FAILED` while the worker message was still in flight).
"""
) -> None:
with rls_transaction(attack_paths_scan.tenant_id):
try:
locked = ProwlerAPIAttackPathsScan.objects.select_for_update().get(
id=attack_paths_scan.id
)
except ProwlerAPIAttackPathsScan.DoesNotExist:
return False
attack_paths_scan.task_id = task_id
attack_paths_scan.state = StateChoices.EXECUTING
attack_paths_scan.started_at = datetime.now(tz=timezone.utc)
attack_paths_scan.update_tag = cartography_config.update_tag
if locked.state != StateChoices.SCHEDULED:
return False
locked.state = StateChoices.EXECUTING
locked.started_at = datetime.now(tz=timezone.utc)
locked.update_tag = cartography_config.update_tag
locked.save(update_fields=["state", "started_at", "update_tag"])
# Keep the in-memory object the caller is holding in sync.
attack_paths_scan.state = locked.state
attack_paths_scan.started_at = locked.started_at
attack_paths_scan.update_tag = locked.update_tag
return True
attack_paths_scan.save(
update_fields=[
"task_id",
"state",
"started_at",
"update_tag",
]
)
def _mark_scan_finished(
@@ -97,19 +97,6 @@ def run(tenant_id: str, scan_id: str, task_id: str) -> dict[str, Any]:
)
attack_paths_scan = db_utils.retrieve_attack_paths_scan(tenant_id, scan_id)
# Idempotency guard: cleanup may have flipped this row to a terminal state
# while the message was still in flight. Bail out before touching state.
if attack_paths_scan and attack_paths_scan.state in (
StateChoices.FAILED,
StateChoices.COMPLETED,
StateChoices.CANCELLED,
):
logger.warning(
f"Attack Paths scan {attack_paths_scan.id} already in terminal "
f"state {attack_paths_scan.state}; skipping execution"
)
return {}
# Checks before starting the scan
if not cartography_ingestion_function:
ingestion_exceptions = {
@@ -127,17 +114,12 @@ def run(tenant_id: str, scan_id: str, task_id: str) -> dict[str, Any]:
else:
if not attack_paths_scan:
# Safety net for in-flight messages or direct task invocations; dispatcher normally pre-creates the row.
logger.warning(
f"No Attack Paths Scan found for scan {scan_id} and tenant {tenant_id}, let's create it then"
)
attack_paths_scan = db_utils.create_attack_paths_scan(
tenant_id, scan_id, prowler_api_provider.id
)
if attack_paths_scan and task_id:
db_utils.set_attack_paths_scan_task_id(
tenant_id, attack_paths_scan.id, task_id
)
tmp_database_name = graph_database.get_database_name(
attack_paths_scan.id, temporary=True
@@ -159,13 +141,9 @@ def run(tenant_id: str, scan_id: str, task_id: str) -> dict[str, Any]:
)
# Starting the Attack Paths scan
if not db_utils.starting_attack_paths_scan(
attack_paths_scan, tenant_cartography_config
):
logger.warning(
f"Attack Paths scan {attack_paths_scan.id} no longer in SCHEDULED state; cleanup likely raced ahead"
)
return {}
db_utils.starting_attack_paths_scan(
attack_paths_scan, task_id, tenant_cartography_config
)
scan_t0 = time.perf_counter()
logger.info(
-4
View File
@@ -47,9 +47,6 @@ from prowler.lib.outputs.compliance.csa.csa_oraclecloud import OracleCloudCSA
from prowler.lib.outputs.compliance.ens.ens_aws import AWSENS
from prowler.lib.outputs.compliance.ens.ens_azure import AzureENS
from prowler.lib.outputs.compliance.ens.ens_gcp import GCPENS
from prowler.lib.outputs.compliance.asd_essential_eight.asd_essential_eight_aws import (
ASDEssentialEightAWS,
)
from prowler.lib.outputs.compliance.iso27001.iso27001_aws import AWSISO27001
from prowler.lib.outputs.compliance.iso27001.iso27001_azure import AzureISO27001
from prowler.lib.outputs.compliance.iso27001.iso27001_gcp import GCPISO27001
@@ -103,7 +100,6 @@ COMPLIANCE_CLASS_MAP = {
(lambda name: name.startswith("ccc_"), CCC_AWS),
(lambda name: name.startswith("c5_"), AWSC5),
(lambda name: name.startswith("csa_"), AWSCSA),
(lambda name: name == "asd_essential_eight_aws", ASDEssentialEightAWS),
],
"azure": [
(lambda name: name.startswith("cis_"), AzureCIS),
+4 -184
View File
@@ -10,29 +10,16 @@ from typing import Any
import sentry_sdk
from celery.utils.log import get_task_logger
from config.django.base import DJANGO_FINDINGS_BATCH_SIZE
from config.env import env
from config.settings.celery import CELERY_DEADLOCK_ATTEMPTS
from django.db import IntegrityError, OperationalError
from django.db.models import (
Case,
Count,
Exists,
IntegerField,
Max,
Min,
OuterRef,
Prefetch,
Q,
Sum,
When,
)
from django.db.models import Case, Count, IntegerField, Max, Min, Prefetch, Q, Sum, When
from django.utils import timezone as django_timezone
from tasks.jobs.queries import (
COMPLIANCE_UPSERT_PROVIDER_SCORE_SQL,
COMPLIANCE_UPSERT_TENANT_SUMMARY_SQL,
)
from tasks.utils import CustomEncoder, batched
from tasks.utils import CustomEncoder
from api.compliance import PROWLER_COMPLIANCE_OVERVIEW_TEMPLATE
from api.constants import SEVERITY_ORDER
@@ -202,9 +189,8 @@ def _get_attack_surface_mapping_from_provider(provider_type: str) -> dict:
"iam_inline_policy_allows_privilege_escalation",
},
"ec2-imdsv1": {
"ec2_instance_imdsv2_enabled",
"ec2_instance_account_imdsv2_enabled",
}, # AWS only - instance-level IMDSv1 exposure and account IMDS defaults
"ec2_instance_imdsv2_enabled"
}, # AWS only - IMDSv1 enabled findings
}
for category_name, check_ids in attack_surface_check_mappings.items():
if check_ids is None:
@@ -2083,169 +2069,3 @@ def aggregate_finding_group_summaries(tenant_id: str, scan_id: str):
"created": created_count,
"updated": updated_count,
}
def reset_ephemeral_resource_findings_count(tenant_id: str, scan_id: str) -> dict:
"""Zero failed_findings_count for resources missing from a completed full-scope scan.
Resources that exist in the database for the scan's provider but were not
touched by this scan are treated as ephemeral. We keep their historical
findings, but reset the denormalized counter that drives the Resources page
sort so they stop ranking at the top.
Skipped (no-op) when:
- The scan is not in COMPLETED state.
- The scan ran with any scoping filter in scanner_args (partial scope).
Query design (must scale to 500k+ resources per provider):
Phase 1 collect ephemeral IDs with one anti-join read.
Outer filter ``(tenant_id, provider_id, failed_findings_count > 0)``
uses ``resources_tenant_provider_idx``. The correlated
``NOT EXISTS`` subquery hits the implicit unique index
``(tenant_id, scan_id, resource_id)`` on ``ResourceScanSummary``.
``NOT EXISTS`` (vs ``NOT IN``) is null-safe and lets the planner
choose between hash anti-join and indexed nested-loop anti-join.
``.iterator(chunk_size=...)`` skips the queryset cache so memory
stays bounded while streaming UUIDs.
Phase 2 UPDATE in fixed-size batches.
One large UPDATE would hold row-exclusive locks for seconds and
create a WAL spike. Batched UPDATEs by ``id__in`` (~1k rows each)
hit the primary key, keep each lock window ~50ms, bound WAL chunks,
and let other writers proceed between batches.
``failed_findings_count__gt=0`` in the UPDATE is idempotent under
concurrent scans and skips no-op rewrites.
Reads use the primary DB, not the replica: ``ResourceScanSummary`` rows
were written by the same scan task that triggered this one, so replica
lag could falsely classify scanned resources as ephemeral.
Scope detection (``Scan.is_full_scope()``) derives the set of scoping
scanner_args from ``prowler.lib.scan.scan.Scan.__init__`` via
introspection, so the API can never drift from the SDK's filter
contract. Imported scans are also rejected by trigger they may only
cover a partial slice of resources.
"""
with rls_transaction(tenant_id):
scan = Scan.objects.filter(tenant_id=tenant_id, id=scan_id).first()
if scan is None:
logger.warning(f"Scan {scan_id} not found")
return {"status": "skipped", "reason": "scan not found"}
if scan.state != StateChoices.COMPLETED:
logger.info(f"Scan {scan_id} not completed; skipping ephemeral reset")
return {"status": "skipped", "reason": "scan not completed"}
if not scan.is_full_scope():
logger.info(
f"Scan {scan_id} ran with scoping filters; skipping ephemeral reset"
)
return {"status": "skipped", "reason": "partial scan scope"}
# Race protection: if a newer completed full-scope scan exists for this
# provider, our ResourceScanSummary set is stale relative to the resources'
# current failed_findings_count values (which the newer scan already
# refreshed). Wiping based on the older scan would zero counts the newer
# scan just set. Skip and let the newer scan's reset task do the work; if
# this task was delayed in the queue, that's the correct outcome.
# `completed_at__isnull=False` is required: Postgres orders NULL first in
# DESC, so a sibling COMPLETED scan with a missing completed_at would sort
# as "newest" and incorrectly cause us to skip.
with rls_transaction(tenant_id):
latest_full_scope_scan_id = (
Scan.objects.filter(
tenant_id=tenant_id,
provider_id=scan.provider_id,
state=StateChoices.COMPLETED,
completed_at__isnull=False,
)
.order_by("-completed_at", "-inserted_at")
.values_list("id", flat=True)
.first()
)
if latest_full_scope_scan_id != scan.id:
logger.info(
f"Scan {scan_id} is not the latest completed scan for provider "
f"{scan.provider_id}; skipping ephemeral reset"
)
return {"status": "skipped", "reason": "newer scan exists"}
# Defensive gate: ResourceScanSummary rows are written by perform_prowler_scan
# via best-effort bulk_create. If those writes failed silently (or the scan
# genuinely produced resources but no summaries were persisted), the
# ~Exists(in_scan) anti-join below would classify EVERY resource for this
# provider as ephemeral and zero their counts. Bail loudly instead.
with rls_transaction(tenant_id):
summaries_present = ResourceScanSummary.objects.filter(
tenant_id=tenant_id, scan_id=scan_id
).exists()
if scan.unique_resource_count > 0 and not summaries_present:
logger.error(
f"Scan {scan_id} reports {scan.unique_resource_count} unique "
f"resources but no ResourceScanSummary rows are persisted; "
f"skipping ephemeral reset to avoid wiping valid counts"
)
return {"status": "skipped", "reason": "summaries missing"}
# Stays on the primary DB intentionally. ResourceScanSummary rows are
# written by perform_prowler_scan in the same chain that triggered this
# task, so replica lag could return an empty/partial summary set; a stale
# read here would classify every Resource as ephemeral and wipe valid
# failed_findings_count values on the primary. Same rationale as
# update_provider_compliance_scores below in this module.
# Materializing the ID list (rather than streaming the iterator into
# batched UPDATEs) is intentional: it lets the UPDATEs run in their own
# short rls_transactions instead of one long transaction holding row locks
# on every batch. At 500k UUIDs the peak memory is ~40 MB — acceptable for
# a Celery worker — and is the better trade-off versus a multi-second
# write-lock window blocking concurrent scans.
with rls_transaction(tenant_id):
in_scan = ResourceScanSummary.objects.filter(
tenant_id=tenant_id,
scan_id=scan_id,
resource_id=OuterRef("pk"),
)
ephemeral_ids = list(
Resource.objects.filter(
tenant_id=tenant_id,
provider_id=scan.provider_id,
failed_findings_count__gt=0,
)
.filter(~Exists(in_scan))
.values_list("id", flat=True)
.iterator(chunk_size=DJANGO_FINDINGS_BATCH_SIZE)
)
if not ephemeral_ids:
logger.info(f"No ephemeral resources for scan {scan_id}")
return {
"status": "completed",
"scan_id": str(scan_id),
"provider_id": str(scan.provider_id),
"reset": 0,
}
total_updated = 0
for batch, _ in batched(ephemeral_ids, DJANGO_FINDINGS_BATCH_SIZE):
# batched() always yields a final tuple, which is empty when the input
# length is an exact multiple of the batch size. Skip it so we don't
# issue a no-op UPDATE ... WHERE id IN ().
if not batch:
continue
with rls_transaction(tenant_id):
total_updated += Resource.objects.filter(
tenant_id=tenant_id,
id__in=batch,
failed_findings_count__gt=0,
).update(failed_findings_count=0)
logger.info(
f"Ephemeral resource reset for scan {scan_id}: "
f"{total_updated} resources zeroed for provider {scan.provider_id}"
)
return {
"status": "completed",
"scan_id": str(scan_id),
"provider_id": str(scan.provider_id),
"reset": total_updated,
}
+2 -53
View File
@@ -58,7 +58,6 @@ from tasks.jobs.scan import (
aggregate_findings,
create_compliance_requirements,
perform_prowler_scan,
reset_ephemeral_resource_findings_count,
update_provider_compliance_scores,
)
from tasks.utils import (
@@ -78,7 +77,6 @@ from prowler.lib.check.compliance_models import Compliance
from prowler.lib.outputs.compliance.generic.generic import GenericCompliance
from prowler.lib.outputs.finding import Finding as FindingOutput
logger = get_task_logger(__name__)
@@ -160,13 +158,6 @@ def _perform_scan_complete_tasks(tenant_id: str, scan_id: str, provider_id: str)
generate_outputs_task.si(
scan_id=scan_id, provider_id=provider_id, tenant_id=tenant_id
),
# post-scan task — runs in the parallel group so a
# failure cannot cascade into reports or integrations. Its only
# prerequisite is that perform_prowler_scan has committed
# ResourceScanSummary, which is true by the time this chain fires.
reset_ephemeral_resource_findings_count_task.si(
tenant_id=tenant_id, scan_id=scan_id
),
),
group(
# Use optimized task that generates both reports with shared queries
@@ -182,25 +173,10 @@ def _perform_scan_complete_tasks(tenant_id: str, scan_id: str, provider_id: str)
).apply_async()
if can_provider_run_attack_paths_scan(tenant_id, provider_id):
# Row is normally created upstream, so this is a safeguard so we can attach the task id below
attack_paths_scan = attack_paths_db_utils.retrieve_attack_paths_scan(
tenant_id, scan_id
)
if attack_paths_scan is None:
attack_paths_scan = attack_paths_db_utils.create_attack_paths_scan(
tenant_id, scan_id, provider_id
)
# Persist the Celery task id so the periodic cleanup can revoke scans stuck in SCHEDULED
result = perform_attack_paths_scan_task.apply_async(
perform_attack_paths_scan_task.apply_async(
kwargs={"tenant_id": tenant_id, "scan_id": scan_id}
)
if attack_paths_scan and result:
attack_paths_db_utils.set_attack_paths_scan_task_id(
tenant_id, attack_paths_scan.id, result.task_id
)
@shared_task(base=RLSTask, name="provider-connection-check")
@set_tenant
@@ -402,8 +378,7 @@ class AttackPathsScanRLSTask(RLSTask):
SDK initialization, or Neo4j configuration errors during setup).
"""
def on_failure(self, exc, task_id, args, kwargs, _einfo): # noqa: ARG002
del args # Required by Celery's Task.on_failure signature; not used.
def on_failure(self, exc, task_id, args, kwargs, _einfo):
tenant_id = kwargs.get("tenant_id")
scan_id = kwargs.get("scan_id")
@@ -800,32 +775,6 @@ def aggregate_daily_severity_task(tenant_id: str, scan_id: str):
return aggregate_daily_severity(tenant_id=tenant_id, scan_id=scan_id)
@shared_task(name="scan-reset-ephemeral-resources", queue="overview")
@handle_provider_deletion
def reset_ephemeral_resource_findings_count_task(tenant_id: str, scan_id: str):
"""Reset failed_findings_count for resources missing from a completed full-scope scan.
Failures are swallowed and returned as a status: this task lives inside the
post-scan group, and Celery propagates group-member exceptions into the next
chain step meaning a crash here would block compliance reports and
integrations. The reset is purely cosmetic (UI sort optimization), so a
bad run is logged and absorbed rather than allowed to cascade.
"""
try:
return reset_ephemeral_resource_findings_count(
tenant_id=tenant_id, scan_id=scan_id
)
except Exception as exc: # noqa: BLE001 — intentionally broad
logger.exception(
f"reset_ephemeral_resource_findings_count failed for scan {scan_id}: {exc}"
)
return {
"status": "failed",
"scan_id": str(scan_id),
"reason": str(exc),
}
@shared_task(base=RLSTask, name="scan-finding-group-summaries", queue="overview")
@set_tenant(keep_tenant=True)
@handle_provider_deletion
@@ -135,7 +135,7 @@ class TestAttackPathsRun:
assert result == ingestion_result
mock_retrieve_scan.assert_called_once_with(str(tenant.id), str(scan.id))
mock_starting.assert_called_once()
config = mock_starting.call_args[0][1]
config = mock_starting.call_args[0][2]
assert config.neo4j_database == "tenant-db"
mock_get_db_name.assert_has_calls(
[call(attack_paths_scan.id, temporary=True), call(provider.tenant_id)]
@@ -2732,143 +2732,3 @@ class TestCleanupStaleAttackPathsScans:
assert result["cleaned_up_count"] == 2
# Worker should be pinged exactly once — cache prevents second ping
mock_alive.assert_called_once_with("shared-worker@host")
# `SCHEDULED` state cleanup
def _create_scheduled_scan(
self,
tenant,
provider,
*,
age_minutes,
parent_state,
with_task=True,
):
"""Create a SCHEDULED AttackPathsScan with a parent Scan in `parent_state`.
`age_minutes` controls how far in the past `started_at` is set, so
callers can place rows safely past the cleanup cutoff.
"""
parent_scan = Scan.objects.create(
name="Parent Prowler scan",
provider=provider,
trigger=Scan.TriggerChoices.MANUAL,
state=parent_state,
tenant_id=tenant.id,
)
ap_scan = AttackPathsScan.objects.create(
tenant_id=tenant.id,
provider=provider,
scan=parent_scan,
state=StateChoices.SCHEDULED,
started_at=datetime.now(tz=timezone.utc) - timedelta(minutes=age_minutes),
)
task_result = None
if with_task:
task_result = TaskResult.objects.create(
task_id=str(ap_scan.id),
task_name="attack-paths-scan-perform",
status="PENDING",
)
task = Task.objects.create(
id=task_result.task_id,
task_runner_task=task_result,
tenant_id=tenant.id,
)
ap_scan.task = task
ap_scan.save(update_fields=["task_id"])
return ap_scan, task_result
@patch("tasks.jobs.attack_paths.cleanup.recover_graph_data_ready")
@patch("tasks.jobs.attack_paths.cleanup.graph_database.drop_database")
@patch(
"tasks.jobs.attack_paths.cleanup.rls_transaction",
new=lambda *args, **kwargs: nullcontext(),
)
@patch("tasks.jobs.attack_paths.cleanup._revoke_task")
def test_cleans_up_scheduled_scan_when_parent_is_terminal(
self,
mock_revoke,
mock_drop_db,
mock_recover,
tenants_fixture,
providers_fixture,
):
from tasks.jobs.attack_paths.cleanup import cleanup_stale_attack_paths_scans
tenant = tenants_fixture[0]
provider = providers_fixture[0]
provider.provider = Provider.ProviderChoices.AWS
provider.save()
ap_scan, task_result = self._create_scheduled_scan(
tenant,
provider,
age_minutes=24 * 60 * 3, # 3 days, safely past any threshold
parent_state=StateChoices.FAILED,
)
result = cleanup_stale_attack_paths_scans()
assert result["cleaned_up_count"] == 1
assert str(ap_scan.id) in result["scan_ids"]
ap_scan.refresh_from_db()
assert ap_scan.state == StateChoices.FAILED
assert ap_scan.progress == 100
assert ap_scan.completed_at is not None
assert ap_scan.ingestion_exceptions == {
"global_error": "Scan never started — cleaned up by periodic task"
}
# SCHEDULED revoke must NOT terminate a running worker
mock_revoke.assert_called_once()
assert mock_revoke.call_args.kwargs == {"terminate": False}
# Temp DB never created for SCHEDULED, so no drop attempted
mock_drop_db.assert_not_called()
# Tenant Neo4j data is untouched in this path
mock_recover.assert_not_called()
task_result.refresh_from_db()
assert task_result.status == "FAILURE"
assert task_result.date_done is not None
@patch("tasks.jobs.attack_paths.cleanup.recover_graph_data_ready")
@patch("tasks.jobs.attack_paths.cleanup.graph_database.drop_database")
@patch(
"tasks.jobs.attack_paths.cleanup.rls_transaction",
new=lambda *args, **kwargs: nullcontext(),
)
@patch("tasks.jobs.attack_paths.cleanup._revoke_task")
def test_skips_scheduled_scan_when_parent_still_in_flight(
self,
mock_revoke,
mock_drop_db,
mock_recover,
tenants_fixture,
providers_fixture,
):
from tasks.jobs.attack_paths.cleanup import cleanup_stale_attack_paths_scans
tenant = tenants_fixture[0]
provider = providers_fixture[0]
provider.provider = Provider.ProviderChoices.AWS
provider.save()
ap_scan, _ = self._create_scheduled_scan(
tenant,
provider,
age_minutes=24 * 60 * 3,
parent_state=StateChoices.EXECUTING,
)
result = cleanup_stale_attack_paths_scans()
assert result["cleaned_up_count"] == 0
ap_scan.refresh_from_db()
assert ap_scan.state == StateChoices.SCHEDULED
mock_revoke.assert_not_called()
-315
View File
@@ -24,7 +24,6 @@ from tasks.jobs.scan import (
aggregate_findings,
create_compliance_requirements,
perform_prowler_scan,
reset_ephemeral_resource_findings_count,
update_provider_compliance_scores,
)
from tasks.utils import CustomEncoder
@@ -36,7 +35,6 @@ from api.models import (
MuteRule,
Provider,
Resource,
ResourceScanSummary,
Scan,
ScanSummary,
StateChoices,
@@ -3853,7 +3851,6 @@ class TestAggregateAttackSurface:
in result["privilege-escalation"]
)
assert "ec2_instance_imdsv2_enabled" in result["ec2-imdsv1"]
assert "ec2_instance_account_imdsv2_enabled" in result["ec2-imdsv1"]
@patch("tasks.jobs.scan.AttackSurfaceOverview.objects.bulk_create")
@patch("tasks.jobs.scan.Finding.all_objects.filter")
@@ -4338,315 +4335,3 @@ class TestUpdateProviderComplianceScores:
assert any("provider_compliance_scores" in c for c in calls)
assert any("tenant_compliance_summaries" in c for c in calls)
assert any("pg_advisory_xact_lock" in c for c in calls)
class TestScanIsFullScope:
def _live_trigger(self):
return Scan.TriggerChoices.MANUAL
@pytest.mark.parametrize(
"scanner_args",
[
{},
{"unrelated": "value"},
{"checks": None},
{"services": []},
{"severities": ""},
],
)
def test_full_scope_when_no_filters_present(self, scanner_args):
scan = Scan(scanner_args=scanner_args, trigger=self._live_trigger())
assert scan.is_full_scope() is True
def test_full_scope_covers_every_sdk_kwarg(self):
# Lock the predicate to whatever ProwlerScan's __init__ exposes today.
# If the SDK adds a new filter, this test still passes via the
# introspection-driven derivation; if it adds a non-filter kwarg
# (e.g. provider-like), keep the exclusion list in sync in models.py.
from prowler.lib.scan.scan import Scan as ProwlerScan
import inspect
expected = tuple(
name
for name in inspect.signature(ProwlerScan.__init__).parameters
if name not in ("self", "provider")
)
assert Scan.get_scoping_scanner_arg_keys() == expected
# Spot-check a few well-known filters survive the introspection.
assert "checks" in expected
assert "services" in expected
assert "severities" in expected
def test_partial_scope_for_each_sdk_filter(self):
for key in Scan.get_scoping_scanner_arg_keys():
scan = Scan(scanner_args={key: ["x"]}, trigger=self._live_trigger())
assert scan.is_full_scope() is False, f"{key} should mark scan as partial"
def test_imported_scan_is_never_full_scope(self):
# Forward-defensive: any trigger outside LIVE_SCAN_TRIGGERS (e.g. a
# future "imported" trigger) must never qualify, even with empty args.
scan = Scan(scanner_args={}, trigger="imported")
assert scan.is_full_scope() is False
def test_handles_none_scanner_args(self):
scan = Scan(scanner_args=None, trigger=self._live_trigger())
assert scan.is_full_scope() is True
@pytest.mark.django_db
class TestResetEphemeralResourceFindingsCount:
def _make_scan_summary(self, tenant_id, scan_id, resource):
return ResourceScanSummary.objects.create(
tenant_id=tenant_id,
scan_id=scan_id,
resource_id=resource.id,
service=resource.service,
region=resource.region,
resource_type=resource.type,
)
def test_resets_only_resources_missing_from_full_scope_scan(
self, tenants_fixture, scans_fixture, providers_fixture, resources_fixture
):
tenant, *_ = tenants_fixture
scan1, scan2, *_ = scans_fixture
resource1, resource2, resource3 = resources_fixture
Resource.objects.filter(id=resource1.id).update(failed_findings_count=3)
Resource.objects.filter(id=resource2.id).update(failed_findings_count=5)
Resource.objects.filter(id=resource3.id).update(failed_findings_count=7)
# Only resource1 was scanned in scan1; resource2 is ephemeral.
self._make_scan_summary(tenant.id, scan1.id, resource1)
result = reset_ephemeral_resource_findings_count(
tenant_id=str(tenant.id), scan_id=str(scan1.id)
)
assert result["status"] == "completed"
assert result["reset"] == 1
resource1.refresh_from_db()
resource2.refresh_from_db()
resource3.refresh_from_db()
assert resource1.failed_findings_count == 3
assert resource2.failed_findings_count == 0
# Other provider's resource is never touched.
assert resource3.failed_findings_count == 7
def test_skips_when_scan_not_completed(
self, tenants_fixture, scans_fixture, resources_fixture
):
tenant, *_ = tenants_fixture
scan1, *_ = scans_fixture
resource1, resource2, _ = resources_fixture
Scan.objects.filter(id=scan1.id).update(state=StateChoices.EXECUTING)
Resource.objects.filter(id=resource2.id).update(failed_findings_count=5)
result = reset_ephemeral_resource_findings_count(
tenant_id=str(tenant.id), scan_id=str(scan1.id)
)
assert result["status"] == "skipped"
assert result["reason"] == "scan not completed"
resource2.refresh_from_db()
assert resource2.failed_findings_count == 5
def test_skips_when_scan_has_scoping_filters(
self, tenants_fixture, scans_fixture, resources_fixture
):
tenant, *_ = tenants_fixture
scan1, *_ = scans_fixture
_, resource2, _ = resources_fixture
Scan.objects.filter(id=scan1.id).update(scanner_args={"checks": ["check1"]})
Resource.objects.filter(id=resource2.id).update(failed_findings_count=5)
result = reset_ephemeral_resource_findings_count(
tenant_id=str(tenant.id), scan_id=str(scan1.id)
)
assert result["status"] == "skipped"
assert result["reason"] == "partial scan scope"
resource2.refresh_from_db()
assert resource2.failed_findings_count == 5
def test_skips_when_scan_not_found(self, tenants_fixture):
tenant, *_ = tenants_fixture
result = reset_ephemeral_resource_findings_count(
tenant_id=str(tenant.id), scan_id=str(uuid.uuid4())
)
assert result["status"] == "skipped"
assert result["reason"] == "scan not found"
def test_skips_when_newer_scan_completed_for_same_provider(
self, tenants_fixture, scans_fixture, providers_fixture, resources_fixture
):
# If a newer completed scan exists for the same provider, our
# ResourceScanSummary set is stale relative to the resources' current
# counts, and applying the diff would corrupt them.
from datetime import timedelta
tenant, *_ = tenants_fixture
scan1, *_ = scans_fixture
provider, *_ = providers_fixture
_, resource2, _ = resources_fixture
Resource.objects.filter(id=resource2.id).update(failed_findings_count=5)
# Create a newer COMPLETED scan for the same provider, with an
# explicit completed_at strictly after scan1's so ordering is
# deterministic regardless of clock resolution.
newer_completed_at = scan1.completed_at + timedelta(minutes=5)
Scan.objects.create(
name="Newer Scan",
provider=provider,
trigger=Scan.TriggerChoices.MANUAL,
state=StateChoices.COMPLETED,
tenant_id=tenant.id,
started_at=newer_completed_at,
completed_at=newer_completed_at,
)
result = reset_ephemeral_resource_findings_count(
tenant_id=str(tenant.id), scan_id=str(scan1.id)
)
assert result["status"] == "skipped"
assert result["reason"] == "newer scan exists"
resource2.refresh_from_db()
assert resource2.failed_findings_count == 5
def test_does_not_touch_other_providers_resources(
self, tenants_fixture, scans_fixture, providers_fixture, resources_fixture
):
tenant, *_ = tenants_fixture
scan1, *_ = scans_fixture
_, _, resource3 = resources_fixture
# resource3 belongs to provider2 with failed_findings_count > 0 and is
# not in scan1's summary. It MUST NOT be reset.
Resource.objects.filter(id=resource3.id).update(failed_findings_count=9)
result = reset_ephemeral_resource_findings_count(
tenant_id=str(tenant.id), scan_id=str(scan1.id)
)
assert result["status"] == "completed"
assert result["reset"] == 0
resource3.refresh_from_db()
assert resource3.failed_findings_count == 9
def test_resources_already_zero_are_not_rewritten(
self, tenants_fixture, scans_fixture, resources_fixture
):
tenant, *_ = tenants_fixture
scan1, *_ = scans_fixture
resource1, resource2, _ = resources_fixture
# Both resources already at 0, neither in summary -> nothing to update.
Resource.objects.filter(id=resource1.id).update(failed_findings_count=0)
Resource.objects.filter(id=resource2.id).update(failed_findings_count=0)
result = reset_ephemeral_resource_findings_count(
tenant_id=str(tenant.id), scan_id=str(scan1.id)
)
assert result["status"] == "completed"
assert result["reset"] == 0
def test_skips_when_summaries_missing_for_scan_with_resources(
self, tenants_fixture, scans_fixture, resources_fixture
):
# Catastrophic guard: if a scan reports unique_resource_count > 0 but
# no ResourceScanSummary rows are persisted (e.g. bulk_create silently
# failed), the anti-join would classify EVERY resource as ephemeral
# and zero their counts. The gate must skip and preserve the data.
tenant, *_ = tenants_fixture
scan1, *_ = scans_fixture
resource1, resource2, _ = resources_fixture
Scan.objects.filter(id=scan1.id).update(unique_resource_count=10)
Resource.objects.filter(id=resource1.id).update(failed_findings_count=3)
Resource.objects.filter(id=resource2.id).update(failed_findings_count=5)
result = reset_ephemeral_resource_findings_count(
tenant_id=str(tenant.id), scan_id=str(scan1.id)
)
assert result["status"] == "skipped"
assert result["reason"] == "summaries missing"
resource1.refresh_from_db()
resource2.refresh_from_db()
assert resource1.failed_findings_count == 3
assert resource2.failed_findings_count == 5
def test_ignores_sibling_scan_with_null_completed_at(
self, tenants_fixture, scans_fixture, providers_fixture, resources_fixture
):
# Postgres orders NULL first in DESC; a sibling COMPLETED scan with a
# missing completed_at must not be treated as the latest scan and
# cause us to incorrectly skip the reset.
tenant, *_ = tenants_fixture
scan1, *_ = scans_fixture
provider, *_ = providers_fixture
resource1, resource2, _ = resources_fixture
Resource.objects.filter(id=resource2.id).update(failed_findings_count=5)
self._make_scan_summary(tenant.id, scan1.id, resource1)
Scan.objects.create(
name="Ghost Scan",
provider=provider,
trigger=Scan.TriggerChoices.MANUAL,
state=StateChoices.COMPLETED,
tenant_id=tenant.id,
started_at=scan1.completed_at,
completed_at=None,
)
result = reset_ephemeral_resource_findings_count(
tenant_id=str(tenant.id), scan_id=str(scan1.id)
)
assert result["status"] == "completed"
assert result["reset"] == 1
resource2.refresh_from_db()
assert resource2.failed_findings_count == 0
def test_batches_updates_when_many_ephemeral_resources(
self, tenants_fixture, scans_fixture, resources_fixture
):
# Forces multiple batches to confirm the chunked UPDATE path executes
# cleanly and the count is the sum across batches.
tenant, *_ = tenants_fixture
scan1, *_ = scans_fixture
resource1, resource2, _ = resources_fixture
Resource.objects.filter(id=resource1.id).update(failed_findings_count=2)
Resource.objects.filter(id=resource2.id).update(failed_findings_count=4)
# No ResourceScanSummary -> both resource1 and resource2 are ephemeral.
# Force a 1-row batch via the shared findings batch size knob.
with patch("tasks.jobs.scan.DJANGO_FINDINGS_BATCH_SIZE", 1):
result = reset_ephemeral_resource_findings_count(
tenant_id=str(tenant.id), scan_id=str(scan1.id)
)
assert result["status"] == "completed"
assert result["reset"] == 2
resource1.refresh_from_db()
resource2.refresh_from_db()
assert resource1.failed_findings_count == 0
assert resource2.failed_findings_count == 0
-66
View File
@@ -842,72 +842,6 @@ class TestScanCompleteTasks:
# Attack Paths task should be skipped when provider cannot run it
mock_attack_paths_task.assert_not_called()
@pytest.mark.parametrize(
"row_pre_existing",
[True, False],
ids=["row-pre-existing", "row-missing-fallback"],
)
@patch("tasks.tasks.aggregate_attack_surface_task.apply_async")
@patch("tasks.tasks.chain")
@patch("tasks.tasks.create_compliance_requirements_task.si")
@patch("tasks.tasks.update_provider_compliance_scores_task.si")
@patch("tasks.tasks.perform_scan_summary_task.si")
@patch("tasks.tasks.generate_outputs_task.si")
@patch("tasks.tasks.generate_compliance_reports_task.si")
@patch("tasks.tasks.check_integrations_task.si")
@patch("tasks.tasks.attack_paths_db_utils.set_attack_paths_scan_task_id")
@patch("tasks.tasks.attack_paths_db_utils.create_attack_paths_scan")
@patch("tasks.tasks.attack_paths_db_utils.retrieve_attack_paths_scan")
@patch("tasks.tasks.perform_attack_paths_scan_task.apply_async")
@patch("tasks.tasks.can_provider_run_attack_paths_scan", return_value=True)
def test_scan_complete_dispatches_attack_paths_scan(
self,
_mock_can_run_attack_paths,
mock_attack_paths_task,
mock_retrieve,
mock_create,
mock_set_task_id,
mock_check_integrations_task,
mock_compliance_reports_task,
mock_outputs_task,
mock_scan_summary_task,
mock_update_compliance_scores_task,
mock_compliance_requirements_task,
mock_chain,
mock_attack_surface_task,
row_pre_existing,
):
"""When a provider can run Attack Paths, dispatch must:
1. Reuse the existing row or create one if missing.
2. Call apply_async on the Attack Paths task.
3. Persist the returned Celery task id on the row.
"""
existing_row = MagicMock(id="ap-scan-id")
if row_pre_existing:
mock_retrieve.return_value = existing_row
else:
mock_retrieve.return_value = None
mock_create.return_value = existing_row
async_result = MagicMock(task_id="celery-task-id")
mock_attack_paths_task.return_value = async_result
_perform_scan_complete_tasks("tenant-id", "scan-id", "provider-id")
mock_retrieve.assert_called_once_with("tenant-id", "scan-id")
if row_pre_existing:
mock_create.assert_not_called()
else:
mock_create.assert_called_once_with("tenant-id", "scan-id", "provider-id")
mock_attack_paths_task.assert_called_once_with(
kwargs={"tenant_id": "tenant-id", "scan_id": "scan-id"}
)
mock_set_task_id.assert_called_once_with(
"tenant-id", "ap-scan-id", "celery-task-id"
)
class TestAttackPathsTasks:
@staticmethod
-335
View File
@@ -1,335 +0,0 @@
# AWS Inventory Connectivity Graph
A community-contributed tool that generates interactive connectivity graphs from Prowler AWS scans, visualizing relationships between AWS resources with zero additional API calls.
## Overview
This tool extends Prowler by producing two artifacts after a scan completes:
- **`<output>.inventory.json`** Machine-readable graph (nodes + edges)
- **`<output>.inventory.html`** Interactive D3.js force-directed visualization
### Why?
Prowler's existing outputs (CSV, ASFF, OCSF, HTML) report individual check findings but provide no cross-service topology view. Security engineers need to understand **how** resources are connected—which Lambda functions sit inside which VPC, which IAM roles can be assumed by which services, which event sources trigger which functions—before they can reason about attack paths, blast-radius, or lateral-movement risk.
This tool fills that gap by building a connectivity graph from the service clients that are already loaded during a Prowler scan.
## Features
### Supported AWS Services
The tool currently extracts connectivity information from:
- **Lambda** Functions, VPC/subnet/SG edges, event source mappings, layers, DLQ, KMS
- **EC2** Instances, security groups, subnet/VPC edges
- **VPC** VPCs, subnets, peering connections
- **RDS** DB instances, VPC/SG/cluster/KMS edges
- **ELBv2** ALB/NLB load balancers, SG and VPC edges
- **S3** Buckets, replication targets, logging buckets, KMS keys
- **IAM** Roles, trust-relationship edges (who can assume what)
### Edge Semantic Types
Edges are typed for downstream filtering and attack-path analysis:
- `network` Resources share a network path (VPC/subnet/SG)
- `iam` IAM trust or permission relationship
- `triggers` One resource can invoke another (event source → Lambda)
- `data_flow` Data is written/read (Lambda → SQS dead-letter queue)
- `depends_on` Soft dependency (Lambda layer, subnet belongs to VPC)
- `routes_to` Traffic routing (LB → target)
- `replicates_to` S3 replication
- `encrypts` KMS key encrypts the resource
- `logs_to` Logging relationship
### Interactive HTML Graph Features
- Force-directed layout with drag-and-drop node pinning
- Zoom / pan (mouse wheel + click-drag on background)
- Per-service color-coded nodes with a legend
- Hover tooltips showing ARN + all metadata properties
- Service filter dropdown (show only Lambda, EC2, RDS, etc.)
- Adjustable link-distance and charge-strength physics sliders
- Edge labels on every arrow
## Installation
### Prerequisites
- Python 3.9.1 or higher
- Prowler installed and configured (see [Prowler documentation](https://docs.prowler.com/))
### Setup
1. Clone or download this directory to your local machine
2. Ensure Prowler is installed and working
3. No additional dependencies required beyond Prowler's existing requirements
## Usage
### Basic Usage
Run Prowler with your desired checks, then use the inventory graph script:
```bash
# Run Prowler scan (example)
prowler aws --output-formats csv
# Generate inventory graph from the scan
python contrib/inventory-graph/inventory_graph.py --output-directory ./output
```
### Command-Line Options
```bash
python contrib/inventory-graph/inventory_graph.py [OPTIONS]
Options:
--output-directory DIR Directory to save output files (default: ./output)
--output-filename NAME Base filename without extension (default: prowler-inventory-<timestamp>)
--help Show this help message and exit
```
### Example Workflow
```bash
# 1. Run a Prowler scan on your AWS account
prowler aws --profile my-aws-profile --output-formats csv html
# 2. Generate the inventory graph
python contrib/inventory-graph/inventory_graph.py \
--output-directory ./output \
--output-filename my-aws-inventory
# 3. Open the HTML file in your browser
open output/my-aws-inventory.inventory.html
```
### Integration with Prowler Scans
The tool reads from already-loaded AWS service clients in memory (via `sys.modules`). This means:
- **Zero extra AWS API calls** Uses data already collected during the Prowler scan
- **Graceful degradation** Services not scanned are silently skipped
- **Flexible** Works with any subset of Prowler checks
## Output Files
### JSON Output (`*.inventory.json`)
Machine-readable graph structure:
```json
{
"generated_at": "2026-03-19T12:34:56Z",
"nodes": [
{
"id": "arn:aws:lambda:us-east-1:123456789012:function:my-function",
"type": "lambda_function",
"name": "my-function",
"service": "lambda",
"region": "us-east-1",
"account_id": "123456789012",
"properties": {
"runtime": "python3.9",
"vpc_id": "vpc-abc123"
}
}
],
"edges": [
{
"source_id": "arn:aws:lambda:...",
"target_id": "arn:aws:ec2:...:vpc/vpc-abc123",
"edge_type": "network",
"label": "in-vpc"
}
],
"stats": {
"node_count": 42,
"edge_count": 87
}
}
```
### HTML Output (`*.inventory.html`)
Self-contained interactive visualization that opens in any modern browser. No server or build step required.
## Architecture
### Design Decisions
| Decision | Rationale |
|----------|-----------|
| **Read from sys.modules** | Zero extra AWS API calls; services not scanned are silently skipped |
| **Self-contained HTML** | D3.js v7 via CDN; no server, no build step; opens in any browser |
| **One extractor per service** | Each extractor is independently testable; adding a new service = one new file + one line in the registry |
| **Typed edges** | Semantic types allow downstream consumers (attack-path tools, Neo4j import) to filter by relationship class |
### Project Structure
```
contrib/inventory-graph/
├── README.md # This file
├── inventory_graph.py # Main entry point script
├── lib/
│ ├── __init__.py
│ ├── models.py # ResourceNode, ResourceEdge, ConnectivityGraph dataclasses
│ ├── graph_builder.py # Reads loaded service clients from sys.modules
│ ├── inventory_output.py # write_json(), write_html()
│ └── extractors/
│ ├── __init__.py
│ ├── lambda_extractor.py # Lambda functions → VPC/subnet/SG/event-sources/layers/DLQ/KMS
│ ├── ec2_extractor.py # EC2 instances + security groups → subnet/VPC
│ ├── vpc_extractor.py # VPCs, subnets, peering connections
│ ├── rds_extractor.py # RDS instances → VPC/SG/cluster/KMS
│ ├── elbv2_extractor.py # ALB/NLB load balancers → SG/VPC
│ ├── s3_extractor.py # S3 buckets → replication targets/logging buckets/KMS keys
│ └── iam_extractor.py # IAM roles + trust-relationship edges
└── examples/
└── sample_output.html # Example output (optional)
```
## Testing
### Smoke Test (No AWS Credentials Needed)
```python
import sys
from unittest.mock import MagicMock
# Wire a fake Lambda client
mock_module = MagicMock()
mock_fn = MagicMock()
mock_fn.arn = "arn:aws:lambda:us-east-1:123:function:test"
mock_fn.name = "test"
mock_fn.region = "us-east-1"
mock_fn.vpc_id = "vpc-abc"
mock_fn.security_groups = ["sg-111"]
mock_fn.subnet_ids = {"subnet-aaa"}
mock_fn.environment = None
mock_fn.kms_key_arn = None
mock_fn.layers = []
mock_fn.dead_letter_config = None
mock_fn.event_source_mappings = []
mock_module.awslambda_client.functions = {mock_fn.arn: mock_fn}
mock_module.awslambda_client.audited_account = "123"
sys.modules["prowler.providers.aws.services.awslambda.awslambda_client"] = mock_module
from contrib.inventory_graph.lib.graph_builder import build_graph
from contrib.inventory_graph.lib.inventory_output import write_json, write_html
graph = build_graph()
write_json(graph, "/tmp/test.inventory.json")
write_html(graph, "/tmp/test.inventory.html")
# Open /tmp/test.inventory.html in a browser
```
## Extending
### Adding a New Service
1. Create a new extractor file in `lib/extractors/` (e.g., `dynamodb_extractor.py`)
2. Implement the `extract(client)` function that returns `(nodes, edges)`
3. Register it in `lib/graph_builder.py` in the `_SERVICE_REGISTRY` tuple
Example extractor template:
```python
from typing import List, Tuple
from prowler.lib.outputs.inventory.models import ResourceNode, ResourceEdge
def extract(client) -> Tuple[List[ResourceNode], List[ResourceEdge]]:
"""Extract DynamoDB tables and their relationships."""
nodes = []
edges = []
for table in client.tables:
nodes.append(
ResourceNode(
id=table.arn,
type="dynamodb_table",
name=table.name,
service="dynamodb",
region=table.region,
account_id=client.audited_account,
properties={"billing_mode": table.billing_mode}
)
)
# Add edges for KMS encryption, streams, etc.
if table.kms_key_arn:
edges.append(
ResourceEdge(
source_id=table.kms_key_arn,
target_id=table.arn,
edge_type="encrypts",
label="encrypts"
)
)
return nodes, edges
```
## Troubleshooting
### No nodes discovered
**Problem:** The tool reports "no nodes discovered" after running.
**Solution:** Ensure you've run a Prowler scan first. The tool reads from in-memory service clients loaded during the scan. If no services were scanned, no nodes will be discovered.
### Missing services in the graph
**Problem:** Some AWS services are not appearing in the graph.
**Solution:** The tool only includes services that have been scanned by Prowler. Run Prowler with the services you want to include, or run without service filters to scan all available services.
### HTML file doesn't display properly
**Problem:** The HTML visualization doesn't load or shows errors.
**Solution:**
- Ensure you're opening the file in a modern browser (Chrome, Firefox, Safari, Edge)
- Check your browser's console for JavaScript errors
- Verify the file was generated completely (check file size > 0)
- The HTML requires internet access to load D3.js from CDN
## Roadmap
Potential future enhancements:
- [ ] Support for additional AWS services (DynamoDB, SQS, SNS, etc.)
- [ ] Export to Neo4j / Cartography format
- [ ] Attack path analysis integration
- [ ] Multi-account/multi-region aggregation
- [ ] Custom edge type filtering in HTML UI
- [ ] Graph diff between two scans
## Contributing
This is a community contribution. If you'd like to enhance it:
1. Fork the Prowler repository
2. Make your changes in `contrib/inventory-graph/`
3. Test thoroughly
4. Submit a pull request with a clear description
## License
This tool is part of the Prowler project and is licensed under the Apache License 2.0.
## Credits
- **Author:** [@sandiyochristan](https://github.com/sandiyochristan)
- **Related PR:** [#10382](https://github.com/prowler-cloud/prowler/pull/10382)
- **Prowler Project:** [prowler-cloud/prowler](https://github.com/prowler-cloud/prowler)
## Support
For issues or questions:
- Open an issue in the [Prowler repository](https://github.com/prowler-cloud/prowler/issues)
- Join the [Prowler Community Slack](https://goto.prowler.com/slack)
- Tag your issue with `contrib:inventory-graph`
@@ -1,181 +0,0 @@
#!/usr/bin/env python3
"""
Example: Generate AWS Inventory Graph with Mock Data
This example demonstrates how to use the inventory graph tool with mock AWS data.
No AWS credentials required.
"""
import sys
from pathlib import Path
from unittest.mock import MagicMock
# Add parent directory to path
sys.path.insert(0, str(Path(__file__).parent.parent))
from lib.graph_builder import build_graph
from lib.inventory_output import write_json, write_html
def create_mock_lambda_client():
"""Create a mock Lambda client with sample data."""
mock_module = MagicMock()
# Create a mock Lambda function
mock_fn = MagicMock()
mock_fn.arn = "arn:aws:lambda:us-east-1:123456789012:function:my-test-function"
mock_fn.name = "my-test-function"
mock_fn.region = "us-east-1"
mock_fn.vpc_id = "vpc-abc123"
mock_fn.security_groups = ["sg-111222"]
mock_fn.subnet_ids = {"subnet-aaa111", "subnet-bbb222"}
mock_fn.environment = {"Variables": {"ENV": "production"}}
mock_fn.kms_key_arn = (
"arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
)
mock_fn.layers = []
mock_fn.dead_letter_config = None
mock_fn.event_source_mappings = []
mock_module.awslambda_client.functions = {mock_fn.arn: mock_fn}
mock_module.awslambda_client.audited_account = "123456789012"
return mock_module
def create_mock_ec2_client():
"""Create a mock EC2 client with sample data."""
mock_module = MagicMock()
# Create a mock EC2 instance
mock_instance = MagicMock()
mock_instance.arn = (
"arn:aws:ec2:us-east-1:123456789012:instance/i-1234567890abcdef0"
)
mock_instance.id = "i-1234567890abcdef0"
mock_instance.region = "us-east-1"
mock_instance.vpc_id = "vpc-abc123"
mock_instance.subnet_id = "subnet-aaa111"
mock_instance.security_groups = [MagicMock(id="sg-111222")]
mock_instance.state = "running"
mock_instance.type = "t3.micro"
mock_instance.tags = [{"Key": "Name", "Value": "test-instance"}]
# Create a mock security group
mock_sg = MagicMock()
mock_sg.arn = "arn:aws:ec2:us-east-1:123456789012:security-group/sg-111222"
mock_sg.id = "sg-111222"
mock_sg.name = "test-security-group"
mock_sg.region = "us-east-1"
mock_sg.vpc_id = "vpc-abc123"
mock_module.ec2_client.instances = [mock_instance]
mock_module.ec2_client.security_groups = [mock_sg]
mock_module.ec2_client.audited_account = "123456789012"
return mock_module
def create_mock_vpc_client():
"""Create a mock VPC client with sample data."""
mock_module = MagicMock()
# Create a mock VPC
mock_vpc = MagicMock()
mock_vpc.arn = "arn:aws:ec2:us-east-1:123456789012:vpc/vpc-abc123"
mock_vpc.id = "vpc-abc123"
mock_vpc.region = "us-east-1"
mock_vpc.cidr_block = "10.0.0.0/16"
mock_vpc.tags = [{"Key": "Name", "Value": "test-vpc"}]
# Create mock subnets
mock_subnet1 = MagicMock()
mock_subnet1.arn = "arn:aws:ec2:us-east-1:123456789012:subnet/subnet-aaa111"
mock_subnet1.id = "subnet-aaa111"
mock_subnet1.region = "us-east-1"
mock_subnet1.vpc_id = "vpc-abc123"
mock_subnet1.cidr_block = "10.0.1.0/24"
mock_subnet1.availability_zone = "us-east-1a"
mock_subnet2 = MagicMock()
mock_subnet2.arn = "arn:aws:ec2:us-east-1:123456789012:subnet/subnet-bbb222"
mock_subnet2.id = "subnet-bbb222"
mock_subnet2.region = "us-east-1"
mock_subnet2.vpc_id = "vpc-abc123"
mock_subnet2.cidr_block = "10.0.2.0/24"
mock_subnet2.availability_zone = "us-east-1b"
mock_module.vpc_client.vpcs = [mock_vpc]
mock_module.vpc_client.subnets = [mock_subnet1, mock_subnet2]
mock_module.vpc_client.vpc_peering_connections = []
mock_module.vpc_client.audited_account = "123456789012"
return mock_module
def main():
"""Main function to demonstrate the inventory graph generation."""
print("=" * 70)
print("AWS Inventory Graph - Mock Data Example")
print("=" * 70)
print()
# Create mock clients and inject them into sys.modules
print("Creating mock AWS service clients...")
sys.modules["prowler.providers.aws.services.awslambda.awslambda_client"] = (
create_mock_lambda_client()
)
sys.modules["prowler.providers.aws.services.ec2.ec2_client"] = (
create_mock_ec2_client()
)
sys.modules["prowler.providers.aws.services.vpc.vpc_client"] = (
create_mock_vpc_client()
)
print("✓ Mock clients created")
print()
# Build the graph
print("Building connectivity graph...")
graph = build_graph()
print(f"✓ Graph built: {len(graph.nodes)} nodes, {len(graph.edges)} edges")
print()
# Display discovered nodes
print("Discovered nodes:")
for node in graph.nodes:
print(f" - {node.type}: {node.name} ({node.region})")
print()
# Display discovered edges
print("Discovered edges:")
for edge in graph.edges:
source_node = next((n for n in graph.nodes if n.id == edge.source_id), None)
target_node = next((n for n in graph.nodes if n.id == edge.target_id), None)
source_name = source_node.name if source_node else edge.source_id
target_name = target_node.name if target_node else edge.target_id
print(f" - {source_name} --[{edge.edge_type}]--> {target_name}")
print()
# Write outputs
output_dir = Path(__file__).parent
json_path = output_dir / "example_output.inventory.json"
html_path = output_dir / "example_output.inventory.html"
print("Writing output files...")
write_json(graph, str(json_path))
write_html(graph, str(html_path))
print(f"✓ JSON written to: {json_path}")
print(f"✓ HTML written to: {html_path}")
print()
print("=" * 70)
print("✓ Example complete!")
print("=" * 70)
print()
print(f"Open the HTML file to view the interactive graph:")
print(f" open {html_path}")
print()
if __name__ == "__main__":
main()
-158
View File
@@ -1,158 +0,0 @@
#!/usr/bin/env python3
"""
AWS Inventory Connectivity Graph Generator
A standalone tool that generates interactive connectivity graphs from Prowler AWS scans.
This tool reads from already-loaded AWS service clients in memory and produces:
- JSON graph (nodes + edges)
- Interactive HTML visualization
Usage:
python inventory_graph.py --output-directory ./output --output-filename my-inventory
For more information, see README.md
"""
import argparse
import os
import sys
from datetime import datetime
from pathlib import Path
# Add the contrib directory to the path so we can import the lib modules
CONTRIB_DIR = Path(__file__).parent
sys.path.insert(0, str(CONTRIB_DIR))
from lib.graph_builder import build_graph
from lib.inventory_output import write_json, write_html
def parse_arguments():
"""Parse command-line arguments."""
parser = argparse.ArgumentParser(
description="Generate AWS inventory connectivity graph from Prowler scan data",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
# Generate graph with default settings
python inventory_graph.py
# Specify custom output directory and filename
python inventory_graph.py --output-directory ./my-output --output-filename aws-inventory
# After running a Prowler scan
prowler aws --profile my-profile
python inventory_graph.py --output-directory ./output
For more information, see README.md
""",
)
parser.add_argument(
"--output-directory",
"-o",
default="./output",
help="Directory to save output files (default: ./output)",
)
parser.add_argument(
"--output-filename",
"-f",
default=None,
help="Base filename without extension (default: prowler-inventory-<timestamp>)",
)
parser.add_argument(
"--verbose",
"-v",
action="store_true",
help="Enable verbose output",
)
return parser.parse_args()
def main():
"""Main entry point for the inventory graph generator."""
args = parse_arguments()
# Set up output paths
output_dir = Path(args.output_directory)
output_dir.mkdir(parents=True, exist_ok=True)
# Generate filename with timestamp if not provided
if args.output_filename:
base_filename = args.output_filename
else:
timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
base_filename = f"prowler-inventory-{timestamp}"
json_path = output_dir / f"{base_filename}.inventory.json"
html_path = output_dir / f"{base_filename}.inventory.html"
print("=" * 70)
print("AWS Inventory Connectivity Graph Generator")
print("=" * 70)
print()
# Build the graph from loaded service clients
if args.verbose:
print("Building connectivity graph from loaded AWS service clients...")
graph = build_graph()
# Check if any nodes were discovered
if not graph.nodes:
print("⚠️ WARNING: No nodes discovered!")
print()
print("This usually means:")
print(" 1. No Prowler scan has been run yet in this Python session")
print(" 2. No AWS service clients are loaded in memory")
print()
print("To fix this:")
print(" 1. Run a Prowler scan first: prowler aws --output-formats csv")
print(" 2. Then run this script in the same session")
print()
print(
"Alternatively, integrate this tool directly into Prowler's output pipeline."
)
sys.exit(1)
print(f"✓ Discovered {len(graph.nodes)} nodes and {len(graph.edges)} edges")
print()
# Write outputs
if args.verbose:
print(f"Writing JSON output to: {json_path}")
write_json(graph, str(json_path))
if args.verbose:
print(f"Writing HTML output to: {html_path}")
write_html(graph, str(html_path))
print()
print("=" * 70)
print("✓ Graph generation complete!")
print("=" * 70)
print()
print(f"📄 JSON: {json_path}")
print(f"🌐 HTML: {html_path}")
print()
print(f"Open the HTML file in your browser to explore the interactive graph:")
print(f" open {html_path}")
print()
if __name__ == "__main__":
try:
main()
except KeyboardInterrupt:
print("\n\nInterrupted by user. Exiting...")
sys.exit(130)
except Exception as e:
print(f"\n❌ Error: {e}", file=sys.stderr)
if "--verbose" in sys.argv or "-v" in sys.argv:
import traceback
traceback.print_exc()
sys.exit(1)
@@ -1,94 +0,0 @@
from typing import List, Tuple
from lib.models import ResourceEdge, ResourceNode
def extract(client) -> Tuple[List[ResourceNode], List[ResourceEdge]]:
"""
Extract EC2 instance and security-group nodes with their edges.
Edges produced:
- instance security-group [network]
- instance subnet [network]
- security-group VPC [network]
"""
nodes: List[ResourceNode] = []
edges: List[ResourceEdge] = []
# EC2 Instances
for instance in client.instances:
name = instance.id
for tag in instance.tags or []:
if tag.get("Key") == "Name":
name = tag["Value"]
break
props = {
"instance_type": getattr(instance, "type", None),
"state": getattr(instance, "state", None),
"vpc_id": getattr(instance, "vpc_id", None),
"subnet_id": getattr(instance, "subnet_id", None),
"public_ip": getattr(instance, "public_ip_address", None),
"private_ip": getattr(instance, "private_ip_address", None),
}
nodes.append(
ResourceNode(
id=instance.arn,
type="ec2_instance",
name=name,
service="ec2",
region=instance.region,
account_id=client.audited_account,
properties={k: v for k, v in props.items() if v is not None},
)
)
for sg_id in instance.security_groups or []:
edges.append(
ResourceEdge(
source_id=instance.arn,
target_id=sg_id,
edge_type="network",
label="sg",
)
)
if instance.subnet_id:
edges.append(
ResourceEdge(
source_id=instance.arn,
target_id=instance.subnet_id,
edge_type="network",
label="subnet",
)
)
# Security Groups
for sg in client.security_groups.values():
name = (
sg.name if hasattr(sg, "name") else sg.id if hasattr(sg, "id") else sg.arn
)
nodes.append(
ResourceNode(
id=sg.arn,
type="security_group",
name=name,
service="ec2",
region=sg.region,
account_id=client.audited_account,
properties={"vpc_id": sg.vpc_id},
)
)
if sg.vpc_id:
edges.append(
ResourceEdge(
source_id=sg.arn,
target_id=sg.vpc_id,
edge_type="network",
label="in-vpc",
)
)
return nodes, edges
@@ -1,60 +0,0 @@
from typing import List, Tuple
from lib.models import ResourceEdge, ResourceNode
def extract(client) -> Tuple[List[ResourceNode], List[ResourceEdge]]:
"""
Extract ELBv2 (ALB/NLB) load balancer nodes and their edges.
Edges produced:
- load_balancer security-group [network]
- load_balancer VPC [network]
"""
nodes: List[ResourceNode] = []
edges: List[ResourceEdge] = []
for lb in client.loadbalancersv2.values():
props = {
"type": getattr(lb, "type", None),
"scheme": getattr(lb, "scheme", None),
"dns_name": getattr(lb, "dns", None),
"vpc_id": getattr(lb, "vpc_id", None),
}
name = getattr(lb, "name", lb.arn.split("/")[-2] if "/" in lb.arn else lb.arn)
nodes.append(
ResourceNode(
id=lb.arn,
type="load_balancer",
name=name,
service="elbv2",
region=lb.region,
account_id=client.audited_account,
properties={k: v for k, v in props.items() if v is not None},
)
)
for sg_id in lb.security_groups or []:
edges.append(
ResourceEdge(
source_id=lb.arn,
target_id=sg_id,
edge_type="network",
label="sg",
)
)
vpc_id = getattr(lb, "vpc_id", None)
if vpc_id:
edges.append(
ResourceEdge(
source_id=lb.arn,
target_id=vpc_id,
edge_type="network",
label="in-vpc",
)
)
return nodes, edges
@@ -1,84 +0,0 @@
import json
from typing import Any, Dict, List, Tuple
from prowler.lib.logger import logger
from lib.models import ResourceEdge, ResourceNode
def _parse_trust_principals(assume_role_policy: Any) -> List[str]:
"""
Return a flat list of principal strings from an IAM assume-role policy document.
The policy may be a dict already or a JSON string.
"""
if not assume_role_policy:
return []
if isinstance(assume_role_policy, str):
try:
assume_role_policy = json.loads(assume_role_policy)
except (json.JSONDecodeError, ValueError):
return []
principals = []
for statement in assume_role_policy.get("Statement", []):
principal = statement.get("Principal", {})
if isinstance(principal, str):
principals.append(principal)
elif isinstance(principal, dict):
for v in principal.values():
if isinstance(v, list):
principals.extend(v)
else:
principals.append(v)
elif isinstance(principal, list):
principals.extend(principal)
return principals
def extract(client) -> Tuple[List[ResourceNode], List[ResourceEdge]]:
"""
Extract IAM role nodes and their trust-relationship edges.
Edges produced:
- trusted-principal role [iam] (who can assume this role)
"""
nodes: List[ResourceNode] = []
edges: List[ResourceEdge] = []
for role in client.roles:
props: Dict[str, Any] = {
"path": getattr(role, "path", None),
"create_date": str(getattr(role, "create_date", "") or ""),
}
nodes.append(
ResourceNode(
id=role.arn,
type="iam_role",
name=role.name,
service="iam",
region="global",
account_id=client.audited_account,
properties={k: v for k, v in props.items() if v},
)
)
# Trust-relationship edges: principal → role (principal CAN assume role)
try:
for principal in _parse_trust_principals(role.assume_role_policy):
if principal and principal != "*":
edges.append(
ResourceEdge(
source_id=principal,
target_id=role.arn,
edge_type="iam",
label="can-assume",
)
)
except Exception as e:
logger.debug(
f"inventory iam_extractor: could not parse trust policy for {role.arn}: {e}"
)
return nodes, edges
@@ -1,118 +0,0 @@
from typing import List, Tuple
from lib.models import ResourceEdge, ResourceNode
def extract(client) -> Tuple[List[ResourceNode], List[ResourceEdge]]:
"""
Extract Lambda function nodes and their edges from an awslambda_client.
Edges produced:
- lambda VPC [network]
- lambda subnet [network]
- lambda sg [network]
- lambda event-source[triggers] (from EventSourceMapping)
- lambda layer ARN [depends_on]
- lambda DLQ target [data_flow]
- lambda KMS key [encrypts]
"""
nodes: List[ResourceNode] = []
edges: List[ResourceEdge] = []
for fn in client.functions.values():
props = {
"runtime": fn.runtime,
"vpc_id": fn.vpc_id,
}
if fn.environment:
props["has_env_vars"] = True
if fn.kms_key_arn:
props["kms_key_arn"] = fn.kms_key_arn
nodes.append(
ResourceNode(
id=fn.arn,
type="lambda_function",
name=fn.name,
service="lambda",
region=fn.region,
account_id=client.audited_account,
properties=props,
)
)
# Network edges → VPC, subnets, security groups
if fn.vpc_id:
edges.append(
ResourceEdge(
source_id=fn.arn,
target_id=fn.vpc_id,
edge_type="network",
label="in-vpc",
)
)
for sg_id in fn.security_groups or []:
edges.append(
ResourceEdge(
source_id=fn.arn,
target_id=sg_id,
edge_type="network",
label="sg",
)
)
for subnet_id in fn.subnet_ids or set():
edges.append(
ResourceEdge(
source_id=fn.arn,
target_id=subnet_id,
edge_type="network",
label="subnet",
)
)
# Trigger edges from event source mappings
for esm in getattr(fn, "event_source_mappings", []):
edges.append(
ResourceEdge(
source_id=esm.event_source_arn,
target_id=fn.arn,
edge_type="triggers",
label=f"esm:{esm.state}",
)
)
# Layer dependency edges
for layer in getattr(fn, "layers", []):
edges.append(
ResourceEdge(
source_id=fn.arn,
target_id=layer.arn,
edge_type="depends_on",
label="layer",
)
)
# Dead-letter queue data-flow edge
dlq = getattr(fn, "dead_letter_config", None)
if dlq and dlq.target_arn:
edges.append(
ResourceEdge(
source_id=fn.arn,
target_id=dlq.target_arn,
edge_type="data_flow",
label="dlq",
)
)
# KMS encryption edge
if fn.kms_key_arn:
edges.append(
ResourceEdge(
source_id=fn.kms_key_arn,
target_id=fn.arn,
edge_type="encrypts",
label="kms",
)
)
return nodes, edges
@@ -1,86 +0,0 @@
from typing import List, Tuple
from lib.models import ResourceEdge, ResourceNode
def extract(client) -> Tuple[List[ResourceNode], List[ResourceEdge]]:
"""
Extract RDS DB instance nodes and their edges.
Edges produced:
- db_instance security-group [network]
- db_instance VPC [network]
- db_instance cluster [depends_on]
- db_instance KMS key [encrypts]
"""
nodes: List[ResourceNode] = []
edges: List[ResourceEdge] = []
for db in client.db_instances.values():
props = {
"engine": getattr(db, "engine", None),
"engine_version": getattr(db, "engine_version", None),
"instance_class": getattr(db, "db_instance_class", None),
"vpc_id": getattr(db, "vpc_id", None),
"multi_az": getattr(db, "multi_az", None),
"publicly_accessible": getattr(db, "publicly_accessible", None),
"storage_encrypted": getattr(db, "storage_encrypted", None),
}
nodes.append(
ResourceNode(
id=db.arn,
type="rds_instance",
name=db.id,
service="rds",
region=db.region,
account_id=client.audited_account,
properties={k: v for k, v in props.items() if v is not None},
)
)
for sg in getattr(db, "security_groups", []):
sg_id = sg if isinstance(sg, str) else getattr(sg, "id", str(sg))
edges.append(
ResourceEdge(
source_id=db.arn,
target_id=sg_id,
edge_type="network",
label="sg",
)
)
vpc_id = getattr(db, "vpc_id", None)
if vpc_id:
edges.append(
ResourceEdge(
source_id=db.arn,
target_id=vpc_id,
edge_type="network",
label="in-vpc",
)
)
cluster_arn = getattr(db, "cluster_arn", None)
if cluster_arn:
edges.append(
ResourceEdge(
source_id=db.arn,
target_id=cluster_arn,
edge_type="depends_on",
label="cluster-member",
)
)
kms_key_id = getattr(db, "kms_key_id", None)
if kms_key_id:
edges.append(
ResourceEdge(
source_id=kms_key_id,
target_id=db.arn,
edge_type="encrypts",
label="kms",
)
)
return nodes, edges
@@ -1,92 +0,0 @@
from typing import List, Tuple
from lib.models import ResourceEdge, ResourceNode
def extract(client) -> Tuple[List[ResourceNode], List[ResourceEdge]]:
"""
Extract S3 bucket nodes and their edges.
Edges produced:
- bucket replication-target bucket [replicates_to]
- bucket KMS key [encrypts]
- bucket logging bucket [logs_to]
"""
nodes: List[ResourceNode] = []
edges: List[ResourceEdge] = []
for bucket in client.buckets.values():
encryption = getattr(bucket, "encryption", None)
versioning = getattr(bucket, "versioning_enabled", None)
logging = getattr(bucket, "logging", None)
public = getattr(bucket, "public_access_block", None)
props = {}
if versioning is not None:
props["versioning"] = versioning
if encryption:
enc_type = getattr(encryption, "type", str(encryption))
props["encryption"] = enc_type
nodes.append(
ResourceNode(
id=bucket.arn,
type="s3_bucket",
name=bucket.name,
service="s3",
region=bucket.region,
account_id=client.audited_account,
properties=props,
)
)
# Replication edges
for rule in getattr(bucket, "replication_rules", None) or []:
dest_bucket = getattr(rule, "destination_bucket", None)
if dest_bucket:
dest_arn = (
dest_bucket
if dest_bucket.startswith("arn:")
else f"arn:aws:s3:::{dest_bucket}"
)
edges.append(
ResourceEdge(
source_id=bucket.arn,
target_id=dest_arn,
edge_type="replicates_to",
label="s3-replication",
)
)
# Logging edges
if logging:
target_bucket = getattr(logging, "target_bucket", None)
if target_bucket:
target_arn = (
target_bucket
if target_bucket.startswith("arn:")
else f"arn:aws:s3:::{target_bucket}"
)
edges.append(
ResourceEdge(
source_id=bucket.arn,
target_id=target_arn,
edge_type="logs_to",
label="access-logs",
)
)
# KMS encryption edges
if encryption:
kms_arn = getattr(encryption, "kms_master_key_id", None)
if kms_arn:
edges.append(
ResourceEdge(
source_id=kms_arn,
target_id=bucket.arn,
edge_type="encrypts",
label="kms",
)
)
return nodes, edges
@@ -1,92 +0,0 @@
from typing import List, Tuple
from lib.models import ResourceEdge, ResourceNode
def extract(client) -> Tuple[List[ResourceNode], List[ResourceEdge]]:
"""
Extract VPC and subnet nodes with their edges.
Edges produced:
- subnet VPC [depends_on]
- peering connection between VPCs [network]
"""
nodes: List[ResourceNode] = []
edges: List[ResourceEdge] = []
# VPCs
for vpc in client.vpcs.values():
name = vpc.id if hasattr(vpc, "id") else vpc.arn
for tag in vpc.tags or []:
if isinstance(tag, dict) and tag.get("Key") == "Name":
name = tag["Value"]
break
nodes.append(
ResourceNode(
id=vpc.arn,
type="vpc",
name=name,
service="vpc",
region=vpc.region,
account_id=client.audited_account,
properties={
"cidr_block": getattr(vpc, "cidr_block", None),
"is_default": getattr(vpc, "is_default", None),
},
)
)
# VPC Subnets
for subnet in client.vpc_subnets.values():
name = subnet.id if hasattr(subnet, "id") else subnet.arn
for tag in getattr(subnet, "tags", None) or []:
if isinstance(tag, dict) and tag.get("Key") == "Name":
name = tag["Value"]
break
nodes.append(
ResourceNode(
id=subnet.arn,
type="subnet",
name=name,
service="vpc",
region=subnet.region,
account_id=client.audited_account,
properties={
"vpc_id": getattr(subnet, "vpc_id", None),
"cidr_block": getattr(subnet, "cidr_block", None),
"availability_zone": getattr(subnet, "availability_zone", None),
"public": getattr(subnet, "public", None),
},
)
)
vpc_id = getattr(subnet, "vpc_id", None)
if vpc_id:
# Find the VPC ARN for this vpc_id
vpc_arn = next(
(v.arn for v in client.vpcs.values() if v.id == vpc_id),
vpc_id,
)
edges.append(
ResourceEdge(
source_id=subnet.arn,
target_id=vpc_arn,
edge_type="depends_on",
label="subnet-of",
)
)
# VPC Peering Connections
for peering in getattr(client, "vpc_peering_connections", {}).values():
edges.append(
ResourceEdge(
source_id=peering.arn,
target_id=getattr(peering, "accepter_vpc_id", peering.arn),
edge_type="network",
label="vpc-peer",
)
)
return nodes, edges
@@ -1,106 +0,0 @@
"""
graph_builder.py
----------------
Builds a ConnectivityGraph by reading already-loaded AWS service clients from
sys.modules. Only services that were actually scanned (i.e. whose client
module is already imported) contribute nodes and edges. Unknown / unloaded
services are silently skipped, so the output degrades gracefully when only a
subset of checks has been run.
"""
import sys
from typing import Tuple
from prowler.lib.logger import logger
from lib.models import ConnectivityGraph
# Registry: (sys.modules key, attribute name inside that module, extractor module path)
_SERVICE_REGISTRY: Tuple[Tuple[str, str, str], ...] = (
(
"prowler.providers.aws.services.awslambda.awslambda_client",
"awslambda_client",
"lib.extractors.lambda_extractor",
),
(
"prowler.providers.aws.services.ec2.ec2_client",
"ec2_client",
"lib.extractors.ec2_extractor",
),
(
"prowler.providers.aws.services.vpc.vpc_client",
"vpc_client",
"lib.extractors.vpc_extractor",
),
(
"prowler.providers.aws.services.rds.rds_client",
"rds_client",
"lib.extractors.rds_extractor",
),
(
"prowler.providers.aws.services.elbv2.elbv2_client",
"elbv2_client",
"lib.extractors.elbv2_extractor",
),
(
"prowler.providers.aws.services.s3.s3_client",
"s3_client",
"lib.extractors.s3_extractor",
),
(
"prowler.providers.aws.services.iam.iam_client",
"iam_client",
"lib.extractors.iam_extractor",
),
)
def build_graph() -> ConnectivityGraph:
"""
Iterate over every registered service, check whether its client module is
already loaded, and call the corresponding extractor.
Returns a ConnectivityGraph with all discovered nodes and edges.
Duplicate node IDs are silently deduplicated (first occurrence wins).
"""
graph = ConnectivityGraph()
seen_node_ids: set = set()
for client_module_key, client_attr, extractor_module_key in _SERVICE_REGISTRY:
client_module = sys.modules.get(client_module_key)
if client_module is None:
continue
service_client = getattr(client_module, client_attr, None)
if service_client is None:
continue
extractor_module = sys.modules.get(extractor_module_key)
if extractor_module is None:
try:
import importlib
extractor_module = importlib.import_module(extractor_module_key)
except ImportError as e:
logger.debug(
f"inventory graph_builder: cannot import extractor {extractor_module_key}: {e}"
)
continue
try:
nodes, edges = extractor_module.extract(service_client)
except Exception as e:
logger.error(
f"inventory graph_builder: extractor {extractor_module_key} failed: "
f"{e.__class__.__name__}[{e.__traceback__.tb_lineno}]: {e}"
)
continue
for node in nodes:
if node.id not in seen_node_ids:
graph.add_node(node)
seen_node_ids.add(node.id)
for edge in edges:
graph.add_edge(edge)
return graph
@@ -1,502 +0,0 @@
"""
inventory_output.py
-------------------
Writes the ConnectivityGraph produced by graph_builder to two files:
<output_path>.inventory.json machine-readable graph (nodes + edges)
<output_path>.inventory.html interactive D3.js force-directed graph
"""
import json
import os
from dataclasses import asdict
from datetime import datetime
from typing import Optional
from prowler.lib.logger import logger
from lib.models import ConnectivityGraph
# ---------------------------------------------------------------------------
# JSON output
# ---------------------------------------------------------------------------
def write_json(graph: ConnectivityGraph, file_path: str) -> None:
"""Serialise the graph to a JSON file."""
try:
os.makedirs(os.path.dirname(file_path), exist_ok=True)
data = {
"generated_at": datetime.utcnow().isoformat() + "Z",
"nodes": [asdict(n) for n in graph.nodes],
"edges": [asdict(e) for e in graph.edges],
"stats": {
"node_count": len(graph.nodes),
"edge_count": len(graph.edges),
},
}
with open(file_path, "w", encoding="utf-8") as fh:
json.dump(data, fh, indent=2, default=str)
logger.info(f"Inventory graph JSON written to {file_path}")
except Exception as e:
logger.error(
f"inventory_output.write_json: {e.__class__.__name__}[{e.__traceback__.tb_lineno}]: {e}"
)
# ---------------------------------------------------------------------------
# HTML output (self-contained, D3.js CDN)
# ---------------------------------------------------------------------------
# Colour palette per node type
_NODE_COLOURS = {
"lambda_function": "#f59e0b",
"ec2_instance": "#3b82f6",
"security_group": "#6366f1",
"vpc": "#10b981",
"subnet": "#34d399",
"rds_instance": "#ef4444",
"load_balancer": "#8b5cf6",
"s3_bucket": "#06b6d4",
"iam_role": "#f97316",
"default": "#94a3b8",
}
# Edge stroke colours per edge type
_EDGE_COLOURS = {
"network": "#64748b",
"iam": "#f97316",
"triggers": "#a855f7",
"data_flow": "#0ea5e9",
"depends_on": "#94a3b8",
"routes_to": "#22c55e",
"replicates_to": "#ec4899",
"encrypts": "#eab308",
"logs_to": "#78716c",
}
_HTML_TEMPLATE = """\
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<title>Prowler AWS Connectivity Graph</title>
<script src="https://d3js.org/d3.v7.min.js"></script>
<style>
*, *::before, *::after {{ box-sizing: border-box; }}
body {{
margin: 0;
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
background: #0f172a;
color: #e2e8f0;
}}
#header {{
padding: 12px 20px;
background: #1e293b;
border-bottom: 1px solid #334155;
display: flex;
align-items: center;
gap: 16px;
}}
#header h1 {{ margin: 0; font-size: 18px; font-weight: 700; }}
#header .stats {{ font-size: 13px; color: #94a3b8; }}
#controls {{
padding: 8px 20px;
background: #1e293b;
border-bottom: 1px solid #334155;
display: flex;
gap: 12px;
align-items: center;
flex-wrap: wrap;
}}
#controls label {{ font-size: 12px; color: #94a3b8; }}
#controls select, #controls input[type=range] {{
background: #0f172a;
color: #e2e8f0;
border: 1px solid #334155;
border-radius: 4px;
padding: 3px 6px;
font-size: 12px;
}}
#graph-container {{ width: 100%; height: calc(100vh - 100px); position: relative; }}
svg {{ width: 100%; height: 100%; }}
.node circle {{
stroke: #1e293b;
stroke-width: 1.5px;
cursor: pointer;
transition: r 0.15s;
}}
.node circle:hover {{ stroke-width: 3px; }}
.node text {{
font-size: 10px;
fill: #e2e8f0;
pointer-events: none;
text-shadow: 0 0 4px #0f172a;
}}
.link {{
stroke-opacity: 0.6;
stroke-width: 1.5px;
}}
.link-label {{
font-size: 8px;
fill: #94a3b8;
pointer-events: none;
}}
#tooltip {{
position: fixed;
background: #1e293b;
border: 1px solid #334155;
border-radius: 6px;
padding: 10px 14px;
font-size: 12px;
pointer-events: none;
max-width: 320px;
word-break: break-all;
z-index: 9999;
display: none;
}}
#tooltip strong {{ color: #f8fafc; }}
#tooltip .prop {{ color: #94a3b8; margin-top: 4px; }}
#legend {{
position: absolute;
top: 10px;
right: 10px;
background: rgba(30,41,59,0.9);
border: 1px solid #334155;
border-radius: 6px;
padding: 10px 14px;
font-size: 11px;
}}
#legend h3 {{ margin: 0 0 6px; font-size: 12px; }}
.legend-row {{ display: flex; align-items: center; gap: 6px; margin: 3px 0; }}
.legend-dot {{ width: 12px; height: 12px; border-radius: 50%; flex-shrink: 0; }}
.legend-line {{ width: 20px; height: 2px; flex-shrink: 0; }}
</style>
</head>
<body>
<div id="header">
<h1>🔗 AWS Connectivity Graph</h1>
<span class="stats" id="stat-label">Generated: {generated_at}</span>
</div>
<div id="controls">
<label>Filter service:
<select id="filter-service">
<option value="">All services</option>
</select>
</label>
<label>Link distance:
<input type="range" id="link-distance" min="40" max="300" value="120"/>
</label>
<label>Charge strength:
<input type="range" id="charge-strength" min="-800" max="-20" value="-250"/>
</label>
<span class="stats" id="visible-count"></span>
</div>
<div id="graph-container">
<svg id="graph-svg"></svg>
<div id="tooltip"></div>
<div id="legend">
<h3>Node types</h3>
{legend_nodes_html}
<h3 style="margin-top:8px">Edge types</h3>
{legend_edges_html}
</div>
</div>
<script>
const RAW_NODES = {nodes_json};
const RAW_EDGES = {edges_json};
const NODE_COLOURS = {node_colours_json};
const EDGE_COLOURS = {edge_colours_json};
// helpers
function nodeColour(d) {{
return NODE_COLOURS[d.type] || NODE_COLOURS["default"];
}}
function edgeColour(d) {{
return EDGE_COLOURS[d.edge_type] || "#94a3b8";
}}
function nodeRadius(d) {{
const base = {{
lambda_function: 9, ec2_instance: 10, vpc: 14, subnet: 8,
security_group: 7, rds_instance: 11, load_balancer: 12,
s3_bucket: 9, iam_role: 9
}};
return base[d.type] || 8;
}}
// filter controls
const services = [...new Set(RAW_NODES.map(n => n.service))].sort();
const sel = document.getElementById("filter-service");
services.forEach(s => {{
const o = document.createElement("option");
o.value = s; o.textContent = s;
sel.appendChild(o);
}});
// D3 setup
const svg = d3.select("#graph-svg");
const container = svg.append("g");
// zoom
svg.call(
d3.zoom().scaleExtent([0.05, 8])
.on("zoom", e => container.attr("transform", e.transform))
);
// arrowhead marker
const defs = svg.append("defs");
defs.append("marker")
.attr("id", "arrow")
.attr("viewBox", "0 -5 10 10")
.attr("refX", 20).attr("refY", 0)
.attr("markerWidth", 6).attr("markerHeight", 6)
.attr("orient", "auto")
.append("path")
.attr("d", "M0,-5L10,0L0,5")
.attr("fill", "#94a3b8");
// tooltip
const tooltip = document.getElementById("tooltip");
// simulation
let simulation, linkSel, nodeSel, labelSel;
function buildGraph(nodeFilter) {{
// Determine which nodes to show
const visibleNodes = nodeFilter
? RAW_NODES.filter(n => n.service === nodeFilter)
: RAW_NODES;
const visibleIds = new Set(visibleNodes.map(n => n.id));
// Only show edges where BOTH endpoints are visible
const visibleEdges = RAW_EDGES.filter(
e => visibleIds.has(e.source_id) && visibleIds.has(e.target_id)
);
document.getElementById("visible-count").textContent =
`Showing ${{visibleNodes.length}} nodes · ${{visibleEdges.length}} edges`;
container.selectAll("*").remove();
if (simulation) simulation.stop();
const nodes = visibleNodes.map(n => ({{ ...n }}));
const nodeIndex = Object.fromEntries(nodes.map(n => [n.id, n]));
const links = visibleEdges.map(e => ({{
...e,
source: nodeIndex[e.source_id] || e.source_id,
target: nodeIndex[e.target_id] || e.target_id,
}}));
const dist = +document.getElementById("link-distance").value;
const charge = +document.getElementById("charge-strength").value;
simulation = d3.forceSimulation(nodes)
.force("link", d3.forceLink(links).id(d => d.id).distance(dist))
.force("charge", d3.forceManyBody().strength(charge))
.force("center", d3.forceCenter(
document.getElementById("graph-container").clientWidth / 2,
document.getElementById("graph-container").clientHeight / 2
))
.force("collision", d3.forceCollide().radius(d => nodeRadius(d) + 6));
// Edges
linkSel = container.append("g").attr("class", "links")
.selectAll("line")
.data(links)
.join("line")
.attr("class", "link")
.attr("stroke", edgeColour)
.attr("marker-end", "url(#arrow)");
// Edge labels
labelSel = container.append("g").attr("class", "link-labels")
.selectAll("text")
.data(links)
.join("text")
.attr("class", "link-label")
.text(d => d.label || "");
// Nodes
nodeSel = container.append("g").attr("class", "nodes")
.selectAll("g")
.data(nodes)
.join("g")
.attr("class", "node")
.call(
d3.drag()
.on("start", (event, d) => {{
if (!event.active) simulation.alphaTarget(0.3).restart();
d.fx = d.x; d.fy = d.y;
}})
.on("drag", (event, d) => {{ d.fx = event.x; d.fy = event.y; }})
.on("end", (event, d) => {{
if (!event.active) simulation.alphaTarget(0);
d.fx = null; d.fy = null;
}})
)
.on("mouseover", (event, d) => {{
const props = Object.entries(d.properties || {{}})
.map(([k, v]) => `<div class="prop"><b>${{k}}</b>: ${{v}}</div>`)
.join("");
tooltip.innerHTML = `
<strong>${{d.name}}</strong>
<div class="prop"><b>type</b>: ${{d.type}}</div>
<div class="prop"><b>service</b>: ${{d.service}}</div>
<div class="prop"><b>region</b>: ${{d.region}}</div>
<div class="prop"><b>account</b>: ${{d.account_id}}</div>
<div class="prop" style="word-break:break-all"><b>arn</b>: ${{d.id}}</div>
${{props}}
`;
tooltip.style.display = "block";
tooltip.style.left = (event.clientX + 12) + "px";
tooltip.style.top = (event.clientY - 10) + "px";
}})
.on("mousemove", event => {{
tooltip.style.left = (event.clientX + 12) + "px";
tooltip.style.top = (event.clientY - 10) + "px";
}})
.on("mouseout", () => {{ tooltip.style.display = "none"; }});
nodeSel.append("circle")
.attr("r", nodeRadius)
.attr("fill", nodeColour);
nodeSel.append("text")
.attr("dx", d => nodeRadius(d) + 3)
.attr("dy", "0.35em")
.text(d => d.name.length > 24 ? d.name.slice(0, 22) + "" : d.name);
simulation.on("tick", () => {{
linkSel
.attr("x1", d => d.source.x)
.attr("y1", d => d.source.y)
.attr("x2", d => d.target.x)
.attr("y2", d => d.target.y);
labelSel
.attr("x", d => (d.source.x + d.target.x) / 2)
.attr("y", d => (d.source.y + d.target.y) / 2);
nodeSel.attr("transform", d => `translate(${{d.x}},${{d.y}})`);
}});
}}
// Initial render
buildGraph(null);
// Filter change
sel.addEventListener("change", () => buildGraph(sel.value || null));
// Simulation control sliders restart on change
document.getElementById("link-distance").addEventListener("input", () => buildGraph(sel.value || null));
document.getElementById("charge-strength").addEventListener("input", () => buildGraph(sel.value || null));
</script>
</body>
</html>
"""
def _build_legend_html(colours: dict, shape: str) -> str:
rows = []
for key, colour in sorted(colours.items()):
if shape == "dot":
rows.append(
f'<div class="legend-row">'
f'<div class="legend-dot" style="background:{colour}"></div>'
f"<span>{key}</span></div>"
)
else:
rows.append(
f'<div class="legend-row">'
f'<div class="legend-line" style="background:{colour}"></div>'
f"<span>{key}</span></div>"
)
return "\n".join(rows)
def write_html(graph: ConnectivityGraph, file_path: str) -> None:
"""Render the graph as a self-contained interactive HTML page."""
try:
os.makedirs(os.path.dirname(file_path), exist_ok=True)
nodes_json = json.dumps(
[
{
"id": n.id,
"type": n.type,
"name": n.name,
"service": n.service,
"region": n.region,
"account_id": n.account_id,
"properties": n.properties,
}
for n in graph.nodes
],
indent=None,
default=str,
)
edges_json = json.dumps(
[
{
"source_id": e.source_id,
"target_id": e.target_id,
"edge_type": e.edge_type,
"label": e.label or "",
}
for e in graph.edges
],
indent=None,
default=str,
)
html = _HTML_TEMPLATE.format(
generated_at=datetime.utcnow().strftime("%Y-%m-%d %H:%M UTC"),
nodes_json=nodes_json,
edges_json=edges_json,
node_colours_json=json.dumps(_NODE_COLOURS),
edge_colours_json=json.dumps(_EDGE_COLOURS),
legend_nodes_html=_build_legend_html(_NODE_COLOURS, "dot"),
legend_edges_html=_build_legend_html(_EDGE_COLOURS, "line"),
)
with open(file_path, "w", encoding="utf-8") as fh:
fh.write(html)
logger.info(f"Inventory graph HTML written to {file_path}")
except Exception as e:
logger.error(
f"inventory_output.write_html: {e.__class__.__name__}[{e.__traceback__.tb_lineno}]: {e}"
)
# ---------------------------------------------------------------------------
# Convenience entry-point called from __main__.py
# ---------------------------------------------------------------------------
def generate_inventory_outputs(output_path: str) -> None:
"""
Build the connectivity graph from currently-loaded service clients and write
both JSON and HTML outputs.
Args:
output_path: base file path WITHOUT extension, e.g.
"output/prowler-output-20240101120000".
The function appends .inventory.json and .inventory.html.
"""
from lib.graph_builder import build_graph
graph = build_graph()
if not graph.nodes:
logger.warning(
"Inventory graph: no nodes discovered. "
"Make sure at least one AWS service was scanned before generating the inventory."
)
write_json(graph, f"{output_path}.inventory.json")
write_html(graph, f"{output_path}.inventory.html")
-71
View File
@@ -1,71 +0,0 @@
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
@dataclass
class ResourceNode:
"""
Represents a single AWS resource as a node in the connectivity graph.
id : globally unique identifier always the resource ARN
type : coarse resource type used for grouping/colour, e.g. "lambda_function"
name : human-readable label shown on the graph
service : AWS service name, e.g. "lambda", "ec2", "rds"
region : AWS region the resource lives in
account_id: AWS account ID
properties: additional resource-specific metadata (runtime, vpc_id, etc.)
"""
id: str
type: str
name: str
service: str
region: str
account_id: str
properties: Dict[str, Any] = field(default_factory=dict)
@dataclass
class ResourceEdge:
"""
Represents a directional relationship between two resource nodes.
source_id : ARN of the source node
target_id : ARN of the target node
edge_type : semantic type of the relationship, e.g.:
"network" resources share a network path (VPC/subnet/SG)
"iam" IAM trust or permission relationship
"triggers" one resource can invoke another (event source Lambda)
"data_flow" data is written/read (Lambda SQS dead-letter queue)
"depends_on" soft dependency (Lambda layer, subnet belongs to VPC)
"routes_to" traffic routing (LB target)
"encrypts" KMS key encrypts the resource
label : optional short label rendered on the edge in the HTML graph
"""
source_id: str
target_id: str
edge_type: str
label: Optional[str] = None
@dataclass
class ConnectivityGraph:
"""
Container for the full inventory connectivity graph.
nodes: all discovered resource nodes
edges: all discovered edges between nodes
"""
nodes: List[ResourceNode] = field(default_factory=list)
edges: List[ResourceEdge] = field(default_factory=list)
def add_node(self, node: ResourceNode) -> None:
self.nodes.append(node)
def add_edge(self, edge: ResourceEdge) -> None:
self.edges.append(edge)
def node_ids(self) -> set:
return {n.id for n in self.nodes}
+2 -2
View File
@@ -10,10 +10,10 @@ This repository contains the Prowler Open Source documentation powered by [Mintl
## Local Development
Install a reviewed version of the [Mintlify CLI](https://www.npmjs.com/package/mint) to preview documentation changes locally:
Install the [Mintlify CLI](https://www.npmjs.com/package/mint) to preview documentation changes locally:
```bash
npm install --global mint@4.2.560
npm i -g mint
```
Run the following command at the root of your documentation (where `mint.json` is located):
-52
View File
@@ -73,58 +73,6 @@ The best reference to understand how to implement a new service is following the
- AWS API calls are wrapped in try/except blocks, with specific handling for `ClientError` and generic exceptions, always logging errors.
- If ARN is not present for some resource, it can be constructed using string interpolation, always including partition, service, region, account, and resource ID.
- Tags and additional attributes that cannot be retrieved from the default call, should be collected and stored for each resource using dedicated methods and threading using the resource object list as iterator.
- When accessing dictionary values from AWS API responses, always use `.get()` with a default value instead of direct dictionary access (e.g., `response.get("Policies", {})` instead of `response["Policies"]`). AWS API responses may not always include all keys, and direct access can cause `KeyError` exceptions that break the entire scan for that service.
### Extending an Existing Service with New Attributes
When adding a new check that requires data not yet collected by an existing service, you need to extend the service by adding new attributes to its resource models and updating the data collection methods. This is a common contributor task that follows a consistent pattern:
1. **Identify the missing data**: Determine which AWS API call provides the data you need and whether it's already being called by the service.
2. **Add new attributes to the resource model**: Extend the Pydantic `BaseModel` class for the resource with the new fields. Use `Optional` types with `None` as the default value to maintain backward compatibility with existing checks.
3. **Update the data collection method**: Modify the existing method that fetches resource details to also extract and store the new attributes. If no existing method fetches the data, add a new method and call it in the constructor using `self.__threading_call__` if possible.
4. **Use safe dictionary access**: When extracting values from API responses, always use `.get()` with appropriate defaults to prevent `KeyError` exceptions when the API doesn't return certain fields.
#### Example: Adding DKIM Status to SES Identities
```python
# Step 1 & 2: Add new fields to the resource model
class Identity(BaseModel):
name: str
arn: str
region: str
type: Optional[str]
policy: Optional[dict] = None
tags: Optional[list] = []
# New attributes for DKIM check
dkim_status: Optional[str] = None
dkim_signing_attributes_origin: Optional[str] = None
# Step 3: Update the data collection method
def _get_email_identities(self, identity):
try:
regional_client = self.regional_clients[identity.region]
identity_attributes = regional_client.get_email_identity(
EmailIdentity=identity.name
)
# Step 4: Use .get() for safe dictionary access
for content_key, content_value in identity_attributes.get("Policies", {}).items():
identity.policy = loads(content)
identity.tags = identity_attributes.get("Tags", [])
# Extract new DKIM attributes
identity.dkim_status = identity_attributes.get("DkimStatus")
identity.dkim_signing_attributes_origin = (
identity_attributes.get("DkimSigningAttributesOrigin")
)
except Exception as error:
logger.error(
f"{regional_client.region} -- {error.__class__.__name__}[{error.__traceback__.tb_lineno}]: {error}"
)
```
5. **Update the service tests**: Add the new attributes to the test mock data and assertions to verify correct data extraction.
## Specific Patterns in AWS Checks
@@ -215,6 +215,3 @@ Also is important to keep all code examples as short as possible, including the
| e5 | M365 and Azure Entra checks enabled by or dependent on an E5 license (e.g., advanced threat protection, audit, DLP, and eDiscovery) |
| privilege-escalation | Detects IAM policies or permissions that allow identities to elevate their privileges beyond their intended scope, potentially gaining administrator or higher-level access through specific action combinations |
| ec2-imdsv1 | Identifies EC2 instances using Instance Metadata Service version 1 (IMDSv1), which is vulnerable to SSRF attacks and should be replaced with IMDSv2 for enhanced security |
| vercel-hobby-plan | Vercel checks whose audited feature is available on the Hobby plan (and therefore also on Pro and Enterprise plans) |
| vercel-pro-plan | Vercel checks whose audited feature requires a Pro plan or higher, including features also available on Enterprise or via supported paid add-ons for Pro plans |
| vercel-enterprise-plan | Vercel checks whose audited feature requires the Enterprise plan |
+6 -20
View File
@@ -27,28 +27,14 @@ The most common high level steps to create a new check are:
### Naming Format for Checks
If you already know the check name when creating a request or implementing a check, use a descriptive identifier with lowercase letters and underscores only.
Recommended patterns:
- `<service>_<resource>_<best_practice>`
Checks must be named following the format: `service_subservice_resource_action`.
The name components are:
- `service` The main service or product area being audited (e.g., ec2, entra, iam, bedrock).
- `resource` The resource, feature, or configuration being evaluated. It can be a single word or a compound phrase joined with underscores (e.g., instance, policy, guardrail, sensitive_information_filter).
- `best_practice` The expected secure state or best practice being checked (e.g., enabled, encrypted, restricted, configured, not_publicly_accessible).
Additional guidance:
- Use underscores only. Do not use hyphens.
- Keep the name specific enough to describe the behavior of the check.
- The first segment should match the service or product area whenever possible.
Examples:
- `s3_bucket_versioning_enabled`
- `bedrock_guardrail_sensitive_information_filter_enabled`
- `service` The main service being audited (e.g., ec2, entra, iam, etc.)
- `subservice` An individual component or subset of functionality within the service that is being audited. This may correspond to a shortened version of the class attribute accessed within the check. If there is no subservice, just omit.
- `resource` The specific resource type being evaluated (e.g., instance, policy, role, etc.)
- `action` The security aspect or configuration being checked (e.g., public, encrypted, enabled, etc.)
### File Creation
@@ -401,7 +387,7 @@ Provides both code examples and best practice recommendations for addressing the
#### Categories
One or more functional groupings used for execution filtering (e.g., `internet-exposed`). Categories must match the predefined values enforced by `CheckMetadata`; adding a new category requires updating the validator and the metadata documentation.
One or more functional groupings used for execution filtering (e.g., `internet-exposed`). You can define new categories just by adding to this field.
For the complete list of available categories, see [Categories Guidelines](/developer-guide/check-metadata-guidelines#categories-guidelines).
+1 -1
View File
@@ -28,7 +28,7 @@ This includes the [AGENTS.md](https://github.com/prowler-cloud/prowler/blob/mast
<Steps>
<Step title="Install Mintlify CLI">
```bash
npm install --global mint@4.2.560
npm i -g mint
```
For detailed instructions, check the [Mintlify documentation](https://www.mintlify.com/docs/installation).
</Step>
-16
View File
@@ -134,22 +134,6 @@ prek installed at `.git/hooks/pre-commit`
If pre-commit hooks were previously installed, run `prek install --overwrite` to replace the existing hook. Otherwise, both tools will run on each commit.
</Warning>
#### Enable TruffleHog as a Pre-Push Hook
By default, only `pre-commit` hooks are installed. To enable [`TruffleHog`](https://github.com/trufflesecurity/trufflehog) secret scanning on every push, install the `pre-push` hook type explicitly:
```shell
prek install --hook-type pre-push
```
Successful installation should produce the following output:
```shell
prek installed at `.git/hooks/pre-push`
```
Once installed, TruffleHog runs before each push and blocks the operation when verified secrets are detected.
### Code Quality and Security Checks
Before merging pull requests, several automated checks and utilities ensure code security and updated dependencies:
+1 -1
View File
@@ -1003,7 +1003,7 @@ class ProwlerArgumentParser:
formatter_class=RawTextHelpFormatter,
usage="prowler [-h] [--version] {aws,azure,gcp,kubernetes,m365,github,nhn,dashboard,iac,your_provider} ...",
epilog="""
Available Providers:
Available Cloud Providers:
{aws,azure,gcp,kubernetes,m365,github,iac,nhn,your_provider}
aws AWS Provider
azure Azure Provider
-131
View File
@@ -1,131 +0,0 @@
---
title: 'Prowler Studio'
---
**Prowler Studio is an AI workflow that ensures Claude Code follows Prowler's skills, guardrails, and best practices when creating new security checks.** What lands in the resulting pull request is consistent, tested, and ready for human review — not half-correct boilerplate that needs to be rewritten.
<Info>
**Contributor Tool**: Prowler Studio is a workflow for advanced contributors adding new Prowler security checks. It is not part of Prowler Cloud, Prowler App, or Prowler CLI.
</Info>
<Warning>
**Preview Feature**: Prowler Studio is under active development and breaking changes are expected. Please report issues or share feedback on [GitHub](https://github.com/prowler-cloud/prowler-studio/issues) or in the [Slack community](https://goto.prowler.com/slack).
</Warning>
<Card title="Prowler Studio Repository" icon="github" href="https://github.com/prowler-cloud/prowler-studio" horizontal>
Clone the source code, install Prowler Studio, and explore the agent workflow in detail.
</Card>
## The Problem
Adding a new check to [Prowler](https://github.com/prowler-cloud/prowler) is more than writing detection logic. A correct check has to:
- Match Prowler's exact service and check folder structure and naming conventions
- Wire up metadata, severity, remediation, tests, and compliance mappings
- Mirror the patterns used by the hundreds of existing checks in the same provider
- Actually load when Prowler scans for available checks — silent structural mistakes are easy to make
Asking a general-purpose AI assistant to do this usually means guessing. It misses conventions, skips tests, or invents structure that looks right but does not load. The result is a half-correct PR that needs to be reviewed line by line or rewritten.
## The Solution
Prowler Studio enforces the workflow end-to-end. Describe the check once — a markdown ticket, a Jira issue, or a GitHub issue — and the workflow:
1. **Loads Prowler-specific skills into every agent.** Every step starts with the same context an experienced Prowler engineer would have in mind. See [AI Skills System](/developer-guide/ai-skills) for how skills are structured.
2. **Runs specialized agents in sequence.** Implementation → testing → compliance mapping → review → PR creation. Each agent has one job and a tight scope.
3. **Verifies as it goes.** The check must load in Prowler. Tests must pass. If something fails, the agent fixes it and re-runs (up to a bounded number of attempts) before moving on.
4. **Produces a complete pull request.** Branch, passing check, tests, compliance mappings, and a pull request waiting for human review.
The result is a consistent starting point, every time, on every supported provider.
## Quick Start
### Install
Prowler Studio requires [`uv`](https://docs.astral.sh/uv/getting-started/installation/) — see the official [installation guide](https://docs.astral.sh/uv/getting-started/installation/).
```bash
git clone https://github.com/prowler-cloud/prowler-studio
cd prowler-studio
uv sync
source .venv/bin/activate
```
### Describe the Check
A ticket is a structured markdown description of the check to create. It is the only input the workflow needs; every agent (implementation, testing, compliance mapping, review, PR creation) uses it as the source of truth, so the more concrete it is, the closer the first PR will land to the desired outcome.
The ticket can be supplied in three ways:
- **Local markdown file** → `--ticket path/to/ticket.md`
- **Jira issue** → `--jira-url https://...` (uses the issue body)
- **GitHub issue** → `--github-url https://...` (uses the issue body)
The content should follow the **New Check Request** template:
- The local copy at [`check_ticket_template.md`](https://github.com/prowler-cloud/prowler-studio/blob/main/check_ticket_template.md) covers `--ticket` and Jira tickets.
- A prefilled GitHub form is also available: [Create a New Check Request issue](https://github.com/prowler-cloud/prowler/issues/new?template=new-check-request.yml).
Sections marked *Optional* can be skipped; everything else helps the agents make the right decisions.
### Run the Workflow
From a local markdown ticket:
```bash
prowler-studio --ticket check_ticket.md
```
From a Jira ticket:
```bash
prowler-studio --jira-url https://mycompany.atlassian.net/browse/PROJ-123
```
From a GitHub issue:
```bash
prowler-studio --github-url https://github.com/owner/repo/issues/123
```
<Note>
Provide exactly one of `--ticket`, `--jira-url`, or `--github-url`.
</Note>
Keep changes local (no push, no pull request):
```bash
prowler-studio -b feat/my-check --ticket check_ticket.md --local
```
### What You Get
After a successful run the working environment contains:
- A new branch on a clean Prowler worktree containing the check, metadata, tests, and compliance mappings
- A pull request opened against Prowler (skipped with `--local`)
- A timestamped log file under `logs/` capturing every step the agents took
## CLI Options
| Option | Short | Description |
|--------|-------|-------------|
| `--branch` | `-b` | Branch name (default: `feat/<ticket>-<check_name>` or `feat/<check_name>`) |
| `--ticket` | `-t` | Path to a markdown check ticket file |
| `--jira-url` | `-j` | Jira ticket URL (e.g., `https://mycompany.atlassian.net/browse/PROJ-123`) |
| `--github-url` | `-g` | GitHub issue URL (e.g., `https://github.com/owner/repo/issues/123`) |
| `--working-dir` | `-w` | Working directory for the Prowler clone (default: `./working`) |
| `--no-worktree` | | Legacy mode — work directly on the main clone instead of using worktrees |
| `--cleanup-worktree` | | Remove the worktree after a successful pull request is created |
| `--local` | | Keep changes local — skip push and pull request creation |
## Configuration
Set these environment variables depending on the input source:
| Variable | When Needed | Purpose |
|----------|-------------|---------|
| `GITHUB_TOKEN` | `--github-url` (recommended) | Higher GitHub API rate limits and access to private issues |
| `JIRA_SITE_URL` | `--jira-url` | Jira site, e.g. `https://mycompany.atlassian.net` |
| `JIRA_EMAIL` | `--jira-url` | Email of the Jira account used to fetch the ticket |
| `JIRA_API_TOKEN` | `--jira-url` | API token for the Jira account |
@@ -2,378 +2,47 @@
title: 'Creating a New Security Compliance Framework in Prowler'
---
This guide explains how to add a new security compliance framework to Prowler, end to end. It covers directory layout, the JSON schema, check mapping conventions, the Pydantic models that validate each framework, the CSV output formatter, local validation, testing, and the pull request process.
## Introduction
A compliance framework in Prowler maps a public or custom control catalog (for example CIS, NIST 800-53, PCI DSS, HIPAA, ENS, CCC) to the security checks that Prowler already runs. Each requirement links to zero, one or more Prowler checks. When a scan executes, findings are aggregated per requirement to produce the compliance report rendered by Prowler CLI and Prowler Cloud.
To create or contribute a custom security framework for Prowler—or to integrate a public framework—you must ensure the necessary checks are available. If they are missing, they must be implemented before proceeding.
Prowler ships with 85+ compliance frameworks across All Providers. The catalog lives under `prowler/compliance/<provider>/` (or `prowler/compliance/` for universal compliance frameworks)
Each framework is defined in a compliance file per provider. The file should follow the structure used in `prowler/compliance/<provider>/` and be named `<framework>_<version>_<provider>.json`. Follow the format below to create your own.
<Warning>
A compliance framework must represent the **complete state** of the source catalog. Every requirement defined by the framework has to be present in the JSON file, even when none of the existing Prowler checks can automate it. In that case, leave `Checks` as an empty array, but do not omit the requirement.
## Compliance Framework
Requirement coverage feeds the compliance percentage calculations and the metadata surfaces (dashboards, widgets, exports). Missing requirements skew those metrics and break the report as a faithful snapshot of the framework.
</Warning>
### Compliance Framework Structure
### Prerequisites
Each compliance framework file consists of structured metadata that identifies the framework and maps security checks to requirements or controls. Please note that a single requirement can be linked to multiple Prowler checks:
Before adding a new framework, complete the following checks:
- **Verify the framework is not already supported.** Inspect `prowler/compliance/<provider>/` for an existing JSON file matching the name and version.
- **Confirm the required checks exist.** Every requirement that can be automated must point to one or more existing Prowler checks. For each missing check, implement it first by following the [Prowler Checks](/developer-guide/checks) guide.
- **Review a reference framework.** Use an existing framework with a similar structure as your template. `cis_2.0_aws.json` is the canonical reference for CIS-style frameworks. `ccc_aws.json`, `ens_rd2022_aws.json`, and `nist_800_53_revision_5_aws.json` illustrate other attribute shapes.
## Four-Layer Architecture
A compliance framework spans four layers. A complete contribution must touch each layer that applies.
- **Layer 1 Schema validation:** The Pydantic models in `prowler/lib/check/compliance_models.py` define the canonical schema for each attribute shape (CIS, ENS, Mitre, CCC, C5, CSA CCM, ISO 27001, KISA ISMS-P, AWS Well-Architected, Prowler ThreatScore, and a generic fallback).
- **Layer 2 JSON catalog:** The framework JSON file in `prowler/compliance/<provider>/` lists every requirement and maps it to checks.
- **Layer 3 Output formatter:** The Python module in `prowler/lib/outputs/compliance/<framework>/` builds the CSV row model, the per-provider transformer, and the CLI summary table.
- **Layer 4 Output dispatchers:** The dispatchers in `prowler/lib/outputs/compliance/compliance.py` and `prowler/lib/outputs/compliance/compliance_output.py` route findings to the right formatter based on the framework identifier.
The rest of this guide walks each layer in order.
## Directory Structure and File Naming
Compliance frameworks live at:
- `Framework`: string The distinguished name of the framework (e.g., CIS).
- `Provider`: string The cloud provider where the framework applies (AWS, Azure, OCI).
- `Version`: string The framework version (e.g., 1.4 for CIS).
- `Requirements`: array of objects. Defines security requirements and their mapping to Prowler checks. All requirements or controls are to be included with the mapping to Prowler.
- `Requirements_Id`: string A unique identifier for each requirement within the framework
- `Requirements_Description`: string The requirement description as specified in the framework.
- `Requirements_Attributes`: array of objects. Contains relevant metadata such as security levels, sections, and any additional data needed for reporting with the result of the findings. Attributes should be derived directly from the frameworks own terminology, ensuring consistency with its established definitions.
- `Requirements_Checks`: array. The Prowler checks that are needed to prove this requirement. It can be one or multiple checks. In case automation is not feasible, this can be empty.
```
prowler/compliance/<provider>/<framework>_<version>_<provider>.json
```
The filename conventions are:
- All lowercase, words separated with underscores.
- `<provider>` is a supported provider identifier: `aws`, `azure`, `gcp`, `kubernetes`, `m365`, `github`, `googleworkspace`, `alibabacloud`, `oraclecloud`, `cloudflare`, `mongodbatlas`, `nhn`, `openstack`, `iac`, `llm`.
- `<version>` is optional. Omit it when the framework has no versioning, as in `ccc_aws.json`.
- The file basename (without `.json`) is the framework key that Prowler CLI accepts via `--compliance`.
Examples:
- `prowler/compliance/aws/cis_2.0_aws.json`
- `prowler/compliance/aws/nist_800_53_revision_5_aws.json`
- `prowler/compliance/azure/ens_rd2022_azure.json`
- `prowler/compliance/kubernetes/cis_1.10_kubernetes.json`
- `prowler/compliance/aws/ccc_aws.json`
The output formatter directory mirrors the framework name:
```
prowler/lib/outputs/compliance/<framework>/
├── <framework>.py # CLI summary-table dispatcher
├── <framework>_<provider>.py # Per-provider transformer class
├── models.py # Pydantic CSV row model
└── __init__.py
```
## JSON Schema Reference
Every compliance file is a JSON document with the following top-level keys.
| Field | Type | Required | Description |
|---|---|---|---|
| `Framework` | string | Yes | Canonical framework identifier, for example `CIS`, `NIST-800-53-Revision-5`, `ENS`, `CCC`. |
| `Name` | string | Yes | Human-readable framework name displayed by Prowler App. |
| `Version` | string | Yes | Framework version, for example `2.0`. Use an empty string only for frameworks without versioning. See [Version Handling](#version-handling). |
| `Provider` | string | Yes | Upper-cased provider identifier: `AWS`, `AZURE`, `GCP`, `KUBERNETES`, `M365`, `GITHUB`, `GOOGLEWORKSPACE`, and so on. |
| `Description` | string | Yes | Short description of the framework's scope and purpose. |
| `Requirements` | array | Yes | List of [requirement objects](#requirement-object). |
### Requirement Object
Each entry in `Requirements` describes one control or requirement.
| Field | Type | Required | Description |
|---|---|---|---|
| `Id` | string | Yes | Unique identifier within the framework, for example `1.10` or `CCC.Core.CN01.AR01`. |
| `Name` | string | No | Optional human-readable name used by frameworks that distinguish control name from description, such as NIST. |
| `Description` | string | Yes | Verbatim description from the source framework. |
| `Attributes` | array | Yes | List of [attribute objects](#attribute-objects). The shape depends on the framework. |
| `Checks` | array of strings | Yes | Prowler check identifiers that automate the requirement. Leave the list empty when the control cannot be automated. |
### Attribute Objects
Attributes carry the metadata that Prowler App and the CSV output display for each requirement. The object shape is framework-specific and is validated by a dedicated Pydantic model in `prowler/lib/check/compliance_models.py`. The most common shapes are summarized below.
#### CIS_Requirement_Attribute
Used by every CIS benchmark.
| Field | Type | Required | Notes |
|---|---|---|---|
| `Section` | string | Yes | Top-level section, for example `1 Identity and Access Management`. |
| `SubSection` | string | No | Optional second-level grouping. |
| `Profile` | enum | Yes | One of `Level 1`, `Level 2`, `E3 Level 1`, `E3 Level 2`, `E5 Level 1`, `E5 Level 2`. |
| `AssessmentStatus` | enum | Yes | `Manual` or `Automated`. |
| `Description` | string | Yes | Control description. |
| `RationaleStatement` | string | Yes | Reason the control exists. |
| `ImpactStatement` | string | Yes | Impact of non-compliance. |
| `RemediationProcedure` | string | Yes | Remediation steps. |
| `AuditProcedure` | string | Yes | Audit steps. |
| `AdditionalInformation` | string | Yes | Free-form notes. |
| `DefaultValue` | string | No | Default configuration value, when relevant. |
| `References` | string | Yes | Colon-separated list of reference URLs. |
#### ENS_Requirement_Attribute
Used by the Spanish ENS (Esquema Nacional de Seguridad) frameworks.
| Field | Type | Required | Notes |
|---|---|---|---|
| `IdGrupoControl` | string | Yes | Control group identifier. |
| `Marco` | string | Yes | Framework block (`operacional`, `organizativo`, `proteccion`). |
| `Categoria` | string | Yes | Control category. |
| `DescripcionControl` | string | Yes | Control description in Spanish. |
| `Tipo` | enum | Yes | `refuerzo`, `requisito`, `recomendacion`, `medida`. |
| `Nivel` | enum | Yes | `opcional`, `bajo`, `medio`, `alto`. |
| `Dimensiones` | array of enum | Yes | Subset of `confidencialidad`, `integridad`, `trazabilidad`, `autenticidad`, `disponibilidad`. |
| `ModoEjecucion` | string | Yes | Execution mode (`manual`, `automático`, `híbrido`). |
| `Dependencias` | array of strings | Yes | Ids of prerequisite controls. Empty list when none. |
#### CCC_Requirement_Attribute
Used by the Common Cloud Controls Catalog.
| Field | Type | Required | Notes |
|---|---|---|---|
| `FamilyName` | string | Yes | Control family, for example `Data`. |
| `FamilyDescription` | string | Yes | Description of the family. |
| `Section` | string | Yes | Section title. |
| `SubSection` | string | Yes | Subsection title, or empty string. |
| `SubSectionObjective` | string | Yes | Stated objective for the subsection. |
| `Applicability` | array of strings | Yes | Applicability tags such as `tlp-green`, `tlp-amber`, `tlp-red`. |
| `Recommendation` | string | Yes | Implementation recommendation. |
| `SectionThreatMappings` | array of objects | Yes | Each entry has `ReferenceId` and `Identifiers`. |
| `SectionGuidelineMappings` | array of objects | Yes | Each entry has `ReferenceId` and `Identifiers`. |
#### Generic_Compliance_Requirement_Attribute
The fallback attribute model used when no framework-specific schema applies (for example NIST 800-53, PCI DSS, GDPR, HIPAA).
| Field | Type | Required | Notes |
|---|---|---|---|
| `ItemId` | string | No | Item identifier. |
| `Section` | string | No | Section name. |
| `SubSection` | string | No | Subsection name. |
| `SubGroup` | string | No | Subgroup name. |
| `Service` | string | No | Affected service, for example `aws`, `iam`. |
| `Type` | string | No | Control type. |
| `Comment` | string | No | Free-form comment. |
Additional per-framework attribute models exist for `AWS_Well_Architected_Requirement_Attribute`, `ISO27001_2013_Requirement_Attribute`, `Mitre_Requirement_Attribute_<Provider>`, `KISA_ISMSP_Requirement_Attribute`, `Prowler_ThreatScore_Requirement_Attribute`, `C5Germany_Requirement_Attribute`, and `CSA_CCM_Requirement_Attribute`. Consult `prowler/lib/check/compliance_models.py` for their full field sets.
<Note>
The `Attributes` field is a Pydantic `Union`. The generic attribute model must remain the last element of that Union, otherwise Pydantic v1 silently coerces every framework into the generic shape and your specialized fields are dropped.
</Note>
## Minimal Working Example
The following snippet is a complete, valid framework file named `my_framework_1.0_aws.json`, saved at `prowler/compliance/aws/my_framework_1.0_aws.json`. It uses the generic attribute shape for simplicity.
```json title="prowler/compliance/aws/my_framework_1.0_aws.json"
{
"Framework": "My-Framework",
"Name": "My Framework 1.0 for AWS",
"Version": "1.0",
"Provider": "AWS",
"Description": "Internal baseline for AWS accounts.",
"Framework": "<framework>-<provider>",
"Version": "<version>",
"Requirements": [
{
"Id": "MF-1.1",
"Description": "Root account must have multi-factor authentication enabled.",
"Id": "<unique-id>",
"Description": "Full description of the requirement",
"Checks": [
"Here is the prowler check or checks that will be executed"
],
"Attributes": [
{
"ItemId": "MF-1.1",
"Section": "Identity and Access Management",
"SubSection": "Root Account",
"Service": "iam"
<Add here your custom attributes.>
}
],
"Checks": [
"iam_root_mfa_enabled",
"iam_root_hardware_mfa_enabled"
]
},
{
"Id": "MF-2.1",
"Description": "S3 buckets must block public access at the account level.",
"Attributes": [
{
"ItemId": "MF-2.1",
"Section": "Data Protection",
"Service": "s3"
}
],
"Checks": [
"s3_account_level_public_access_blocks"
]
}
...
]
}
```
## Mapping Checks to Requirements
Each requirement links to the Prowler checks that, together, produce a PASS or FAIL verdict for that control.
- **Include every requirement from the source catalog.** The framework file must mirror the full control list, one-to-one. Compliance percentages, dashboards, and exported metadata are computed against the total requirement count, so omitting an unmappable control inflates coverage and misrepresents the framework.
- List every check by its canonical identifier, the value of `CheckID` inside the check's `.metadata.json` file.
- One requirement can reference multiple checks. The requirement is evaluated as FAIL when any referenced check produces a FAIL finding for a resource in scope.
- Leave `Checks` as an empty array when the requirement cannot be automated. The requirement still appears in the report, contributes to the total, and resolves to `MANUAL`. An empty mapping is valid; a missing requirement is not.
- Reuse checks across requirements when the same control applies in multiple places. Do not duplicate check logic to match framework structure.
- Avoid referencing checks from a different provider. A compliance file is bound to one provider, and cross-provider checks will never match findings in the scan.
To discover available checks, run:
```bash
poetry run python prowler-cli.py <provider> --list-checks
```
## Supporting Multiple Providers
Each compliance file targets a single provider. To cover several providers with the same framework (for example CIS across AWS, Azure, and GCP), ship one JSON file per provider:
```
prowler/compliance/aws/cis_2.0_aws.json
prowler/compliance/azure/cis_2.0_azure.json
prowler/compliance/gcp/cis_2.0_gcp.json
```
Keep the `Framework` and `Version` values identical across the files so the dispatcher matches them, and change only the `Provider`, `Checks`, and provider-specific metadata.
The CIS output formatter already supports every provider listed above. For a brand-new framework that spans several providers, add one transformer per provider in `prowler/lib/outputs/compliance/<framework>/` and extend the summary-table dispatcher accordingly. See [Output Formatter](#output-formatter).
## Output Formatter
Prowler renders every compliance framework in two forms: a detailed CSV report written to disk, and a summary table printed in the CLI. Both are produced by the output formatter package for the framework.
For a new framework named `my_framework`, create:
```
prowler/lib/outputs/compliance/my_framework/
├── __init__.py
├── my_framework.py # CLI summary table dispatcher
├── my_framework_aws.py # Per-provider transformer
└── models.py # CSV row Pydantic model
```
### Step 1 Define the CSV Row Model
In `models.py`, declare a Pydantic v1 model with one field per CSV column. Use existing models such as `AWSCISModel` in `prowler/lib/outputs/compliance/cis/models.py` as the reference. Fields typically include `Provider`, `Description`, `AccountId`, `Region`, `AssessmentDate`, `Requirements_Id`, `Requirements_Description`, one `Requirements_Attributes_*` field per attribute key, plus the finding fields `Status`, `StatusExtended`, `ResourceId`, `ResourceName`, `CheckId`, `Muted`, `Framework`, `Name`.
### Step 2 Implement the Transformer Class
In `my_framework_aws.py`, subclass `ComplianceOutput` from `prowler.lib.outputs.compliance.compliance_output` and implement `transform(findings, compliance, compliance_name)`. Iterate over `findings`, match each finding to the requirements it satisfies through `finding.compliance.get(compliance_name, [])`, and append one row per attribute to `self._data`.
### Step 3 Add the Summary-Table Dispatcher
In `my_framework.py`, implement `get_my_framework_table(findings, bulk_checks_metadata, compliance_framework, output_filename, output_directory, compliance_overview)` following the pattern in `prowler/lib/outputs/compliance/cis/cis.py`.
### Step 4 Register the Framework in the Dispatchers
- Add the dispatcher call in `prowler/lib/outputs/compliance/compliance.py`, inside `display_compliance_table`, with a branch such as `elif "my_framework" in compliance_framework:`.
- Register the CSV model and transformer in `prowler/lib/outputs/compliance/compliance_output.py` so the CSV file is emitted during the scan.
<Note>
For NIST-style catalogs that use `Generic_Compliance_Requirement_Attribute`, no custom formatter is needed. The generic formatter in `prowler/lib/outputs/compliance/generic/` handles them automatically, provided the JSON validates against the generic attribute schema.
</Note>
## Version Handling
Prowler matches frameworks by concatenating `Framework` and `Version`. A missing or empty `Version` collapses several frameworks to the same key and breaks CLI filtering with `--compliance`.
- Always set `Version` to a non-empty string, even for frameworks that rename editions rather than version them. Use the edition identifier (for example `RD2022`, `v2025.10`, `4.0`).
- When the source catalog has no version, use the first year of adoption or the release date.
- Make sure the version substring embedded in the filename matches `Version`, because the CLI dispatcher reads `compliance_framework.split("_")[1]` to select the correct version.
## Validating the Framework Locally
Follow the steps below before opening a pull request.
### 1. Run the Compliance Model Validator
```bash
poetry run python prowler-cli.py <provider> --list-compliance
```
The framework must appear in the output. A validation error indicates a schema mismatch between the JSON file and the attribute model.
### 2. Run a Scan Filtered by the Framework
```bash
poetry run python prowler-cli.py <provider> \
--compliance <framework>_<version>_<provider> \
--log-level ERROR
```
Verify that:
- Prowler produces a CSV file under `output/compliance/` with the expected name.
- The CLI summary table lists every section in the framework.
- Findings roll up under the expected requirements.
### 3. Inspect the CSV Output
Open the generated CSV and confirm:
- All columns defined in `models.py` appear.
- Every requirement has at least one row per scanned resource.
- Values such as `Requirements_Attributes_Section` reflect the JSON content.
### 4. Verify the Framework in Prowler App
Launch Prowler App locally (`docker compose up` from the repository root) and run a scan with the new compliance framework. Confirm the compliance page renders the requirements, sections, and status widgets correctly.
## Testing
Compliance contributions require two layers of tests.
- **Schema tests** exercise the Pydantic models. Extend `tests/lib/check/universal_compliance_models_test.py` with a case that loads the new JSON file and asserts the attribute type matches the expected model.
- **Output tests** exercise the transformer. Mirror the structure under `tests/lib/outputs/compliance/<framework>/` with fixtures that feed synthetic findings through the transformer and assert the resulting CSV rows.
Run the suite with:
```bash
poetry run pytest -n auto tests/lib/check/universal_compliance_models_test.py \
tests/lib/outputs/compliance/
```
For guidance on writing Prowler SDK tests, refer to [Unit Testing](/developer-guide/unit-testing).
## Submitting the Pull Request
Before opening the pull request:
1. Run the complete QA pipeline:
```bash
poetry run pre-commit run --all-files
poetry run pytest -n auto
```
2. Add a changelog entry under the `### 🚀 Added` section of `prowler/CHANGELOG.md`, describing the new framework and the providers it covers.
3. Follow the [Pull Request Template](https://github.com/prowler-cloud/prowler/blob/master/.github/pull_request_template.md) and set the PR title using Conventional Commits, for example `feat(compliance): add My Framework 1.0 for AWS`.
4. Request review from the compliance codeowners listed in `.github/CODEOWNERS`.
## Troubleshooting
The following issues are the most common when contributing a compliance framework.
- **`ValidationError: field required` during scan.** The JSON is missing a required attribute field. Re-check the matching Pydantic model in `prowler/lib/check/compliance_models.py`.
- **All attributes collapse to `Generic_Compliance_Requirement_Attribute` values.** The Pydantic `Union` is ordered incorrectly, or the JSON matches only the generic shape. Move the generic model to the last Union position and ensure every required field is present in the JSON.
- **`--compliance` filter does not find the framework.** The filename does not match the expected pattern `<framework>_<version>_<provider>.json`, the version is empty, or the file lives outside `prowler/compliance/<provider>/`.
- **CLI summary table is empty but the CSV is populated.** The dispatcher branch in `prowler/lib/outputs/compliance/compliance.py` is missing or its substring match does not catch the framework key.
- **CSV file is missing after the scan.** The transformer class is not registered in `prowler/lib/outputs/compliance/compliance_output.py`, or `transform()` raises silently. Run the scan with `--log-level DEBUG`.
- **Findings do not roll up under a requirement.** A check listed in `Checks` either does not exist for that provider or is spelled incorrectly. Run `--list-checks | grep <check_name>` to confirm.
## Reference Examples
Use the following files as templates when modeling a new contribution.
- `prowler/compliance/aws/cis_2.0_aws.json` CIS attribute shape.
- `prowler/compliance/aws/nist_800_53_revision_5_aws.json` Generic attribute shape.
- `prowler/compliance/aws/ccc_aws.json` CCC attribute shape.
- `prowler/compliance/azure/ens_rd2022_azure.json` ENS attribute shape.
- `prowler/lib/check/compliance_models.py` Canonical Pydantic schemas.
- `prowler/lib/outputs/compliance/cis/` Reference implementation of a multi-provider output formatter.
- `prowler/lib/outputs/compliance/generic/` Reference implementation of a generic output formatter.
Finally, to have a proper output file for your reports, your framework data model has to be created in `prowler/lib/outputs/models.py` and also the CLI table output in `prowler/lib/outputs/compliance.py`. Also, you need to add a new conditional in `prowler/lib/outputs/file_descriptors.py` if creating a new CSV model.
+2 -15
View File
@@ -119,7 +119,6 @@
"user-guide/tutorials/prowler-app-multi-tenant",
"user-guide/tutorials/prowler-app-api-keys",
"user-guide/tutorials/prowler-app-import-findings",
"user-guide/tutorials/prowler-app-alerts",
{
"group": "Mutelist",
"expanded": true,
@@ -177,6 +176,7 @@
"pages": [
"user-guide/cli/tutorials/misc",
"user-guide/cli/tutorials/reporting",
"user-guide/cli/tutorials/compliance",
"user-guide/cli/tutorials/dashboard",
"user-guide/cli/tutorials/configuration_file",
"user-guide/cli/tutorials/logging",
@@ -332,20 +332,12 @@
"user-guide/providers/vercel/getting-started-vercel",
"user-guide/providers/vercel/authentication"
]
},
{
"group": "Okta",
"pages": [
"user-guide/providers/okta/getting-started-okta",
"user-guide/providers/okta/authentication"
]
}
]
},
{
"group": "Compliance",
"pages": [
"user-guide/compliance/tutorials/compliance",
"user-guide/compliance/tutorials/threatscore"
]
},
@@ -373,8 +365,7 @@
"developer-guide/security-compliance-framework",
"developer-guide/lighthouse-architecture",
"developer-guide/mcp-server",
"developer-guide/ai-skills",
"developer-guide/prowler-studio"
"developer-guide/ai-skills"
]
},
{
@@ -511,10 +502,6 @@
}
},
"redirects": [
{
"source": "/user-guide/cli/tutorials/compliance",
"destination": "/user-guide/compliance/tutorials/compliance"
},
{
"source": "/projects/prowler-open-source/en/latest/tutorials/prowler-app-lighthouse",
"destination": "/user-guide/tutorials/prowler-app-lighthouse"
@@ -32,11 +32,11 @@ Access Prowler App by logging in with **email and password**.
<img src="/images/log-in.png" alt="Log In" width="285" />
## Add Provider
## Add Cloud Provider
Configure a provider for scanning:
Configure a cloud provider for scanning:
1. Navigate to `Settings > Providers` and click `Add Provider`.
1. Navigate to `Settings > Cloud Providers` and click `Add Account`.
2. Select the cloud provider.
3. Enter the provider's identifier (Optional: Add an alias):
- **AWS**: Account ID
@@ -44,21 +44,13 @@ Choose the configuration based on your deployment:
<Tab title="Generic without Native HTTP Support">
**Configuration:**
<Warning>
Avoid configuring MCP clients to run `npx mcp-remote` directly. `npx` can download and execute a new package version on each run. Install a reviewed version of `mcp-remote` in a dedicated local workspace, then point the MCP client to the installed binary.
</Warning>
```bash
mkdir -p ~/.local/share/prowler-mcp-bridge
cd ~/.local/share/prowler-mcp-bridge
npm init -y
npm install --save-exact mcp-remote@0.1.38
```
```json
{
"mcpServers": {
"prowler": {
"command": "/absolute/path/to/.local/share/prowler-mcp-bridge/node_modules/.bin/mcp-remote",
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.prowler.com/mcp", // or your self-hosted Prowler MCP Server URL
"--header",
"Authorization: Bearer ${PROWLER_APP_API_KEY}"
@@ -80,20 +72,14 @@ Choose the configuration based on your deployment:
2. Go to "Developer" tab
3. Click in "Edit Config" button
4. Edit the `claude_desktop_config.json` file with your favorite editor
5. Install a reviewed version of `mcp-remote` in a dedicated local workspace:
```bash
mkdir -p ~/.local/share/prowler-mcp-bridge
cd ~/.local/share/prowler-mcp-bridge
npm init -y
npm install --save-exact mcp-remote@0.1.38
```
6. Add the following configuration:
5. Add the following configuration:
```json
{
"mcpServers": {
"prowler": {
"command": "/absolute/path/to/.local/share/prowler-mcp-bridge/node_modules/.bin/mcp-remote",
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.prowler.com/mcp",
"--header",
"Authorization: Bearer ${PROWLER_APP_API_KEY}"
@@ -38,7 +38,7 @@ Refer to the [Prowler App Tutorial](/user-guide/tutorials/prowler-app) for detai
- `git` installed.
- `poetry` installed: [poetry installation](https://python-poetry.org/docs/#installation).
- `pnpm` installed through [Corepack](https://pnpm.io/installation#using-corepack) or the standalone [pnpm installation](https://pnpm.io/installation).
- `npm` installed: [npm installation](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).
- `Docker Compose` installed: https://docs.docker.com/compose/install/.
<Warning>
@@ -97,11 +97,9 @@ Refer to the [Prowler App Tutorial](/user-guide/tutorials/prowler-app) for detai
```bash
git clone https://github.com/prowler-cloud/prowler \
cd prowler/ui \
corepack enable \
corepack install \
pnpm install --frozen-lockfile \
pnpm run build \
pnpm start
npm install \
npm run build \
npm start
```
> Enjoy Prowler App at http://localhost:3000 by signing up with your email and password.
@@ -123,8 +121,8 @@ To update the environment file:
Edit the `.env` file and change version values:
```env
PROWLER_UI_VERSION="5.26.1"
PROWLER_API_VERSION="5.26.1"
PROWLER_UI_VERSION="5.24.0"
PROWLER_API_VERSION="5.24.0"
```
<Note>
Binary file not shown.

Before

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 534 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 659 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 759 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 62 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 534 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 257 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 399 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 425 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 222 KiB

+2 -3
View File
@@ -47,12 +47,11 @@ Prowler supports a wide range of providers organized by category:
| Provider | Support | Audit Scope/Entities | Interface |
| ----------------------------------------------------------------------------------------- | -------- | ---------------------------- | ------------ |
| [GitHub](/user-guide/providers/github/getting-started-github) | Official | Organizations / Repositories | UI, API, CLI |
| [Google Workspace](/user-guide/providers/googleworkspace/getting-started-googleworkspace) | Official | Domains | UI, API, CLI |
| [Google Workspace](/user-guide/providers/googleworkspace/getting-started-googleworkspace) | Official | Domains | CLI |
| [LLM](/user-guide/providers/llm/getting-started-llm) | Official | Models | CLI |
| [M365](/user-guide/providers/microsoft365/getting-started-m365) | Official | Tenants | UI, API, CLI |
| [MongoDB Atlas](/user-guide/providers/mongodbatlas/getting-started-mongodbatlas) | Official | Organizations | UI, API, CLI |
| [Okta](/user-guide/providers/okta/getting-started-okta) | Official | Organizations | CLI |
| [Vercel](/user-guide/providers/vercel/getting-started-vercel) | Official | Teams / Projects | UI, API, CLI |
| [Vercel](/user-guide/providers/vercel/getting-started-vercel) | Official | Teams / Projects | CLI |
### Kubernetes
+8 -13
View File
@@ -1,17 +1,12 @@
export const VersionBadge = ({ version }) => {
return (
<a
href={`https://github.com/prowler-cloud/prowler/releases/tag/${version}`}
target="_blank"
rel="noopener noreferrer"
className="version-badge-link"
>
<span className="version-badge-container">
<span className="version-badge">
<span className="version-badge-label">Added in:</span>&nbsp;
<span className="version-badge-version">{version}</span>
</span>
</span>
</a>
<code className="version-badge-container">
<p className="version-badge">
<span className="version-badge-label">Added in:</span>&nbsp;
<code className="version-badge-version">{version}</code>
</p>
</code>
);
};
-17
View File
@@ -1,21 +1,4 @@
/* Version Badge Styling */
.version-badge-link,
.version-badge-link:hover,
.version-badge-link:focus,
.version-badge-link:active,
.version-badge-link:visited {
display: inline-block;
text-decoration: none !important;
background-image: none !important;
border-bottom: none !important;
color: inherit;
transition: opacity 0.15s ease-in-out;
}
.version-badge-link:hover {
opacity: 0.85;
}
.version-badge-container {
display: inline-block;
margin: 0 0 1rem 0;
-34
View File
@@ -159,40 +159,6 @@ When these environment variables are set, the API will use them directly instead
A fix addressing this permission issue is being evaluated in [PR #9953](https://github.com/prowler-cloud/prowler/pull/9953).
</Note>
### Scan Stuck in Executing State After Worker Crash
When running Prowler App via Docker Compose, a scan may remain indefinitely in the `executing` state if the worker process crashes (for example, due to an Out of Memory condition) before it can update the scan status. Since it is not currently possible to cancel a scan in `executing` state through the UI, the workaround is to manually update the scan record in the database.
**Root Cause:**
The Celery worker process terminates unexpectedly (OOM, node failure, etc.) before transitioning the scan state to `completed` or `failed`. The scan record remains in `executing` with no active process to advance it.
**Solution:**
Connect to the database using the `prowler_admin` user. Due to Row-Level Security (RLS), the default database user cannot see scan records — you must use `prowler_admin`:
```bash
psql -U prowler_admin -d prowler_db
```
Identify the stuck scan by filtering for scans in `executing` state:
```sql
SELECT id, name, state, started_at FROM scans WHERE state = 'executing';
```
Update the scan state to `failed` using the scan ID:
```sql
UPDATE scans SET state = 'failed' WHERE id = '<scan-id>';
```
After this change, the scan will appear as failed in the UI and you can launch a new scan.
<Note>
A feature to cancel executing scans directly from the UI is being tracked in [GitHub Issue #6893](https://github.com/prowler-cloud/prowler/issues/6893).
</Note>
### SAML/OAuth ACS URL Incorrect When Running Behind a Proxy or Load Balancer
See [GitHub Issue #9724](https://github.com/prowler-cloud/prowler/issues/9724) for more details.
@@ -0,0 +1,80 @@
---
title: 'Compliance'
---
Prowler allows you to execute checks based on requirements defined in compliance frameworks. By default, it will execute and give you an overview of the status of each compliance framework:
<img src="/images/cli/compliance/compliance.png" />
You can find CSVs containing detailed compliance results in the compliance folder within Prowler's output folder.
## Execute Prowler based on Compliance Frameworks
Prowler can analyze your environment based on a specific compliance framework and get more details, to do it, you can use option `--compliance`:
```sh
prowler <provider> --compliance <compliance_framework>
```
Standard results will be shown and additionally the framework information as the sample below for CIS AWS 2.0. For details a CSV file has been generated as well.
<img src="/images/cli/compliance/compliance-cis-sample1.png" />
<Note>
**If Prowler can't find a resource related with a check from a compliance requirement, this requirement won't appear on the output**
</Note>
## List Available Compliance Frameworks
To see which compliance frameworks are covered by Prowler, use the `--list-compliance` option:
```sh
prowler <provider> --list-compliance
```
Or you can visit [Prowler Hub](https://hub.prowler.com/compliance).
## List Requirements of Compliance Frameworks
To list requirements for a compliance framework, use the `--list-compliance-requirements` option:
```sh
prowler <provider> --list-compliance-requirements <compliance_framework(s)>
```
Example for the first requirements of CIS 1.5 for AWS:
```
Listing CIS 1.5 AWS Compliance Requirements:
Requirement Id: 1.1
- Description: Maintain current contact details
- Checks:
account_maintain_current_contact_details
Requirement Id: 1.2
- Description: Ensure security contact information is registered
- Checks:
account_security_contact_information_is_registered
Requirement Id: 1.3
- Description: Ensure security questions are registered in the AWS account
- Checks:
account_security_questions_are_registered_in_the_aws_account
Requirement Id: 1.4
- Description: Ensure no 'root' user account access key exists
- Checks:
iam_no_root_access_key
Requirement Id: 1.5
- Description: Ensure MFA is enabled for the 'root' user account
- Checks:
iam_root_mfa_enabled
[redacted]
```
## Create and contribute adding other Security Frameworks
This information is part of the Developer Guide and can be found [here](/developer-guide/security-compliance-framework).
@@ -56,7 +56,6 @@ The following list includes all the AWS checks with configurable variables that
| `elb_is_in_multiple_az` | `elb_min_azs` | Integer |
| `elbv2_is_in_multiple_az` | `elbv2_min_azs` | Integer |
| `guardduty_is_enabled` | `mute_non_default_regions` | Boolean |
| `iam_user_access_not_stale_to_sagemaker` | `max_unused_sagemaker_access_days` | Integer |
| `iam_user_accesskey_unused` | `max_unused_access_keys_days` | Integer |
| `iam_user_console_access_unused` | `max_console_access_days` | Integer |
| `organizations_delegated_administrators` | `organizations_trusted_delegated_administrators` | List of Strings |
@@ -158,15 +157,6 @@ The following list includes all the Vercel checks with configurable variables th
| `team_member_role_least_privilege` | `max_owners` | Integer |
| `team_no_stale_invitations` | `stale_invitation_threshold_days` | Integer |
## Okta
### Configurable Checks
The following list includes all the Okta checks with configurable variables that can be changed in the configuration YAML file:
| Check Name | Value | Type |
|---------------------------------------------------------------|------------------------------------|---------|
| `signon_global_session_idle_timeout_15min` | `okta_max_session_idle_minutes` | Integer |
## Config YAML File Structure
<Note>
@@ -196,8 +186,6 @@ aws:
max_unused_access_keys_days: 45
# aws.iam_user_console_access_unused --> CIS recommends 45 days
max_console_access_days: 45
# aws.iam_user_access_not_stale_to_sagemaker --> default 90 days
max_unused_sagemaker_access_days: 90
# AWS EC2 Configuration
# aws.ec2_elastic_ip_shodan

Some files were not shown because too many files have changed in this diff Show More