Compare commits

..

13 Commits

Author SHA1 Message Date
Andoni A. 60350794da feat(api): show only production URL in API reference playground 2025-11-12 09:47:28 +01:00
Andoni Alonso 6a876a3205 Merge branch 'master' into DEVREL-98-include-open-api-specification-in-mintlify 2025-11-12 09:22:08 +01:00
Andoni A. e9f6bc8604 chore: update CHANGELOG 2025-11-11 07:36:18 +01:00
Andoni A. dc71d86c35 chore(api): regenerate openapi schema 2025-11-10 08:57:34 +01:00
Andoni Alonso 7a54900a62 Merge branch 'master' into DEVREL-98-include-open-api-specification-in-mintlify 2025-11-10 08:39:59 +01:00
Andoni A. 1d62b8d64e fix(docs): json version is not needed 2025-11-05 14:52:23 +01:00
Andoni A. 0f5dc165bb fix(docs): use hardlink, mintlify can't follow the symlink 2025-11-05 12:51:27 +01:00
Andoni A. 676a11bb13 feat(docs): use symlink to point to openapi spec 2025-11-05 12:43:16 +01:00
Andoni A. 4b80544f0a chore: update CHANGELOG 2025-11-05 12:35:44 +01:00
Andoni A. 6815a9dd86 feat(api): add dynamic server URL configuration to OpenAPI schema
Add a new postprocessing hook 'add_api_servers' that dynamically configures
the servers array in the OpenAPI specification based on the request environment.

This hook:
- Detects the current environment (local, staging, or production) from the
  request host
- Adds the current server URL as the primary option
- Includes production (https://api.prowler.com) as a fallback option when
  generating from non-production environments
- Enables users to toggle between local and production servers in Mintlify's
  API playground via a dropdown

Server detection logic:
- localhost/127.0.0.1 → "Local Development Server"
- hosts with 'dev' or 'staging' → "Development/Staging API"
- api.prowler.com → "Prowler Cloud API"

This enables the "Try it out" feature in Mintlify documentation and allows
testing against different environments without modifying the spec.
2025-11-05 12:04:18 +01:00
Andoni A. 00441e776d feat(api): add OpenAPI schema postprocessing hooks for Mintlify compatibility
Add three postprocessing hooks to fix OpenAPI 3.0.x compatibility issues
generated by drf-spectacular-jsonapi:

1. fix_empty_id_fields: Fixes empty id field definitions ({}) in JSON:API
   request schemas (particularly PATCH/update requests) by replacing them
   with proper schema: {"type": "string", "format": "uuid", "description": "..."}

2. fix_pattern_properties: Converts patternProperties to additionalProperties
   for OpenAPI 3.0 compatibility. patternProperties is only available in
   OpenAPI 3.1+, but drf-spectacular generates 3.0.3 specs. This is needed
   for the Prowler mutelist configuration which uses dynamic keys.

3. fix_type_formats: Fixes invalid type values like "email" that
   drf-spectacular generates from Django's EmailField. Converts them to
   proper OpenAPI format: "type": "string" with "format": "email".
   Also handles "url" → "uri" and "uuid" formats.

These hooks ensure the generated OpenAPI schema validates correctly with
Mintlify and other OpenAPI 3.0.x tools.
2025-11-05 11:36:24 +01:00
Andoni A. 4037927c61 Merge branch 'master' into try-openapi 2025-11-05 10:36:07 +01:00
Andoni A. 20337b7e0c poc: test yq to create json schema 2025-10-31 08:24:21 +01:00
557 changed files with 32109 additions and 31322 deletions
+9 -2
View File
@@ -14,6 +14,14 @@ UI_PORT=3000
AUTH_SECRET="N/c6mnaS5+SWq81+819OrzQZlmx1Vxtp/orjttJSmw8="
# Google Tag Manager ID
NEXT_PUBLIC_GOOGLE_TAG_MANAGER_ID=""
# Sentry
SENTRY_DSN=
NEXT_PUBLIC_SENTRY_DSN=
SENTRY_ORG=
SENTRY_PROJECT=
SENTRY_AUTH_TOKEN=
SENTRY_ENVIRONMENT=production
NEXT_PUBLIC_SENTRY_ENVIRONMENT=production
#### Code Review Configuration ####
# Enable Claude Code standards validation on pre-push hook
@@ -108,8 +116,6 @@ DJANGO_THROTTLE_TOKEN_OBTAIN=50/minute
# Sentry settings
SENTRY_ENVIRONMENT=local
SENTRY_RELEASE=local
NEXT_PUBLIC_SENTRY_ENVIRONMENT=${SENTRY_ENVIRONMENT}
#### Prowler release version ####
NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v5.12.2
@@ -140,3 +146,4 @@ LANGCHAIN_PROJECT=""
RSS_FEED_SOURCES='[{"id":"prowler-releases","name":"Prowler Releases","type":"github_releases","url":"https://github.com/prowler-cloud/prowler/releases.atom","enabled":true}]'
# Example with multiple sources (no trailing comma after last item):
# RSS_FEED_SOURCES='[{"id":"prowler-releases","name":"Prowler Releases","type":"github_releases","url":"https://github.com/prowler-cloud/prowler/releases.atom","enabled":true},{"id":"prowler-blog","name":"Prowler Blog","type":"blog","url":"https://prowler.com/blog/rss","enabled":false}]'
+20 -62
View File
@@ -44,16 +44,7 @@ jobs:
container-build-push:
needs: setup
runs-on: ${{ matrix.runner }}
strategy:
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
arch: amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
arch: arm64
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
@@ -72,17 +63,17 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- name: Build and push API container for ${{ matrix.arch }}
if: github.event_name == 'push' || github.event_name == 'release'
- name: Build and push API container (latest)
if: github.event_name == 'push'
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
with:
context: ${{ env.WORKING_DIRECTORY }}
push: true
platforms: ${{ matrix.platform }}
tags: |
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-${{ matrix.arch }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }}
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Notify container push started
if: github.event_name == 'release'
@@ -98,6 +89,19 @@ jobs:
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
payload-file-path: "./.github/scripts/slack-messages/container-release-started.json"
- name: Build and push API container (release)
if: github.event_name == 'release'
id: container-push
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
with:
context: ${{ env.WORKING_DIRECTORY }}
push: true
tags: |
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.RELEASE_TAG }}
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Notify container push completed
if: github.event_name == 'release' && always()
uses: ./.github/actions/slack-notification
@@ -113,52 +117,6 @@ jobs:
payload-file-path: "./.github/scripts/slack-messages/container-release-completed.json"
step-outcome: ${{ steps.container-push.outcome }}
# Create and push multi-architecture manifest
create-manifest:
needs: [setup, container-build-push]
if: github.event_name == 'push' || github.event_name == 'release'
runs-on: ubuntu-latest
steps:
- name: Login to DockerHub
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- name: Create and push manifests for push event
if: github.event_name == 'push'
run: |
docker buildx imagetools create \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }} \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }} \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-amd64 \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-arm64
- name: Create and push manifests for release event
if: github.event_name == 'release'
run: |
docker buildx imagetools create \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.RELEASE_TAG }} \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }} \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-amd64 \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-arm64
- name: Install regctl
if: always()
uses: regclient/actions/regctl-installer@f61d18f46c86af724a9c804cb9ff2a6fec741c7c # main
- name: Cleanup intermediate architecture tags
if: always()
run: |
echo "Cleaning up intermediate tags..."
regctl tag delete "${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-amd64" || true
regctl tag delete "${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-arm64" || true
echo "Cleanup completed"
trigger-deployment:
if: github.event_name == 'push'
needs: [setup, container-build-push]
@@ -1,39 +0,0 @@
name: 'Tools: Comment Label Update'
on:
issue_comment:
types:
- 'created'
concurrency:
group: ${{ github.workflow }}-${{ github.event.issue.number }}
cancel-in-progress: false
jobs:
update-labels:
if: contains(github.event.issue.labels.*.name, 'status/awaiting-response')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
issues: write
pull-requests: write
steps:
- name: Remove 'status/awaiting-response' label
env:
GH_TOKEN: ${{ github.token }}
ISSUE_NUMBER: ${{ github.event.issue.number }}
run: |
echo "Removing 'status/awaiting-response' label from #$ISSUE_NUMBER"
gh api /repos/${{ github.repository }}/issues/$ISSUE_NUMBER/labels/status%2Fawaiting-response \
-X DELETE
- name: Add 'status/waiting-for-revision' label
env:
GH_TOKEN: ${{ github.token }}
ISSUE_NUMBER: ${{ github.event.issue.number }}
run: |
echo "Adding 'status/waiting-for-revision' label to #$ISSUE_NUMBER"
gh api /repos/${{ github.repository }}/issues/$ISSUE_NUMBER/labels \
-X POST \
-f labels[]='status/waiting-for-revision'
+29 -64
View File
@@ -43,16 +43,7 @@ jobs:
container-build-push:
needs: setup
runs-on: ${{ matrix.runner }}
strategy:
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
arch: amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
arch: arm64
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
@@ -70,25 +61,24 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- name: Build and push MCP container for ${{ matrix.arch }}
if: github.event_name == 'push' || github.event_name == 'release'
- name: Build and push MCP container (latest)
if: github.event_name == 'push'
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
with:
context: ${{ env.WORKING_DIRECTORY }}
push: true
platforms: ${{ matrix.platform }}
tags: |
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-${{ matrix.arch }}
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }}
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}
labels: |
org.opencontainers.image.title=Prowler MCP Server
org.opencontainers.image.description=Model Context Protocol server for Prowler
org.opencontainers.image.vendor=ProwlerPro, Inc.
org.opencontainers.image.source=https://github.com/${{ github.repository }}
org.opencontainers.image.revision=${{ github.sha }}
org.opencontainers.image.created=${{ github.event_name == 'release' && github.event.release.published_at || github.event.head_commit.timestamp }}
${{ github.event_name == 'release' && format('org.opencontainers.image.version={0}', env.RELEASE_TAG) || '' }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
org.opencontainers.image.created=${{ github.event.head_commit.timestamp }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Notify container push started
if: github.event_name == 'release'
@@ -104,6 +94,27 @@ jobs:
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
payload-file-path: "./.github/scripts/slack-messages/container-release-started.json"
- name: Build and push MCP container (release)
if: github.event_name == 'release'
id: container-push
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
with:
context: ${{ env.WORKING_DIRECTORY }}
push: true
tags: |
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.RELEASE_TAG }}
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }}
labels: |
org.opencontainers.image.title=Prowler MCP Server
org.opencontainers.image.description=Model Context Protocol server for Prowler
org.opencontainers.image.vendor=ProwlerPro, Inc.
org.opencontainers.image.version=${{ env.RELEASE_TAG }}
org.opencontainers.image.source=https://github.com/${{ github.repository }}
org.opencontainers.image.revision=${{ github.sha }}
org.opencontainers.image.created=${{ github.event.release.published_at }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Notify container push completed
if: github.event_name == 'release' && always()
uses: ./.github/actions/slack-notification
@@ -119,52 +130,6 @@ jobs:
payload-file-path: "./.github/scripts/slack-messages/container-release-completed.json"
step-outcome: ${{ steps.container-push.outcome }}
# Create and push multi-architecture manifest
create-manifest:
needs: [setup, container-build-push]
if: github.event_name == 'push' || github.event_name == 'release'
runs-on: ubuntu-latest
steps:
- name: Login to DockerHub
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- name: Create and push manifests for push event
if: github.event_name == 'push'
run: |
docker buildx imagetools create \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }} \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }} \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-amd64 \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-arm64
- name: Create and push manifests for release event
if: github.event_name == 'release'
run: |
docker buildx imagetools create \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.RELEASE_TAG }} \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }} \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-amd64 \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-arm64
- name: Install regctl
if: always()
uses: regclient/actions/regctl-installer@main
- name: Cleanup intermediate architecture tags
if: always()
run: |
echo "Cleaning up intermediate tags..."
regctl tag delete "${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-amd64" || true
regctl tag delete "${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-arm64" || true
echo "Cleanup completed"
trigger-deployment:
if: github.event_name == 'push'
needs: [setup, container-build-push]
+26 -84
View File
@@ -46,16 +46,7 @@ env:
jobs:
container-build-push:
if: github.repository == 'prowler-cloud/prowler'
runs-on: ${{ matrix.runner }}
strategy:
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
arch: amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
arch: arm64
runs-on: ubuntu-latest
timeout-minutes: 45
permissions:
contents: read
@@ -63,8 +54,6 @@ jobs:
outputs:
prowler_version: ${{ steps.get-prowler-version.outputs.prowler_version }}
prowler_version_major: ${{ steps.get-prowler-version.outputs.prowler_version_major }}
latest_tag: ${{ steps.get-prowler-version.outputs.latest_tag }}
stable_tag: ${{ steps.get-prowler-version.outputs.stable_tag }}
env:
POETRY_VIRTUALENVS_CREATE: 'false'
@@ -99,22 +88,16 @@ jobs:
3)
echo "LATEST_TAG=v3-latest" >> "${GITHUB_ENV}"
echo "STABLE_TAG=v3-stable" >> "${GITHUB_ENV}"
echo "latest_tag=v3-latest" >> "${GITHUB_OUTPUT}"
echo "stable_tag=v3-stable" >> "${GITHUB_OUTPUT}"
echo "✓ Prowler v3 detected - tags: v3-latest, v3-stable"
;;
4)
echo "LATEST_TAG=v4-latest" >> "${GITHUB_ENV}"
echo "STABLE_TAG=v4-stable" >> "${GITHUB_ENV}"
echo "latest_tag=v4-latest" >> "${GITHUB_OUTPUT}"
echo "stable_tag=v4-stable" >> "${GITHUB_OUTPUT}"
echo "✓ Prowler v4 detected - tags: v4-latest, v4-stable"
;;
5)
echo "LATEST_TAG=latest" >> "${GITHUB_ENV}"
echo "STABLE_TAG=stable" >> "${GITHUB_ENV}"
echo "latest_tag=latest" >> "${GITHUB_OUTPUT}"
echo "stable_tag=stable" >> "${GITHUB_OUTPUT}"
echo "✓ Prowler v5 detected - tags: latest, stable"
;;
*)
@@ -141,18 +124,19 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- name: Build and push SDK container for ${{ matrix.arch }}
if: github.event_name == 'push' || github.event_name == 'release'
- name: Build and push SDK container (latest)
if: github.event_name == 'push'
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
with:
context: .
file: ${{ env.DOCKERFILE_PATH }}
push: true
platforms: ${{ matrix.platform }}
tags: |
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }}-${{ matrix.arch }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.LATEST_TAG }}
${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.LATEST_TAG }}
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Notify container push started
if: github.event_name == 'release'
@@ -168,6 +152,24 @@ jobs:
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
payload-file-path: "./.github/scripts/slack-messages/container-release-started.json"
- name: Build and push SDK container (release)
if: github.event_name == 'release'
id: container-push
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
with:
context: .
file: ${{ env.DOCKERFILE_PATH }}
push: true
tags: |
${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.PROWLER_VERSION }}
${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.STABLE_TAG }}
${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.PROWLER_VERSION }}
${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.STABLE_TAG }}
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.PROWLER_VERSION }}
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Notify container push completed
if: github.event_name == 'release' && always()
uses: ./.github/actions/slack-notification
@@ -183,66 +185,6 @@ jobs:
payload-file-path: "./.github/scripts/slack-messages/container-release-completed.json"
step-outcome: ${{ steps.container-push.outcome }}
# Create and push multi-architecture manifest
create-manifest:
needs: [container-build-push]
if: github.event_name == 'push' || github.event_name == 'release'
runs-on: ubuntu-latest
steps:
- name: Login to DockerHub
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to Public ECR
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
with:
registry: public.ecr.aws
username: ${{ secrets.PUBLIC_ECR_AWS_ACCESS_KEY_ID }}
password: ${{ secrets.PUBLIC_ECR_AWS_SECRET_ACCESS_KEY }}
env:
AWS_REGION: ${{ env.AWS_REGION }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- name: Create and push manifests for push event
if: github.event_name == 'push'
run: |
docker buildx imagetools create \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.container-build-push.outputs.latest_tag }} \
-t ${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.container-build-push.outputs.latest_tag }} \
-t ${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.container-build-push.outputs.latest_tag }} \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.container-build-push.outputs.latest_tag }}-amd64 \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.container-build-push.outputs.latest_tag }}-arm64
- name: Create and push manifests for release event
if: github.event_name == 'release'
run: |
docker buildx imagetools create \
-t ${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ needs.container-build-push.outputs.prowler_version }} \
-t ${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ needs.container-build-push.outputs.stable_tag }} \
-t ${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ needs.container-build-push.outputs.prowler_version }} \
-t ${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ needs.container-build-push.outputs.stable_tag }} \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.container-build-push.outputs.prowler_version }} \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.container-build-push.outputs.stable_tag }} \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.container-build-push.outputs.latest_tag }}-amd64 \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.container-build-push.outputs.latest_tag }}-arm64
- name: Install regctl
if: always()
uses: regclient/actions/regctl-installer@main
- name: Cleanup intermediate architecture tags
if: always()
run: |
echo "Cleaning up intermediate tags..."
regctl tag delete "${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.container-build-push.outputs.latest_tag }}-amd64" || true
regctl tag delete "${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.container-build-push.outputs.latest_tag }}-arm64" || true
echo "Cleanup completed"
dispatch-v3-deployment:
if: needs.container-build-push.outputs.prowler_version_major == '3'
needs: container-build-push
+19 -72
View File
@@ -46,16 +46,7 @@ jobs:
container-build-push:
needs: setup
runs-on: ${{ matrix.runner }}
strategy:
matrix:
include:
- platform: linux/amd64
runner: ubuntu-latest
arch: amd64
- platform: linux/arm64
runner: ubuntu-24.04-arm
arch: arm64
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
@@ -74,7 +65,7 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- name: Build and push UI container for ${{ matrix.arch }}
- name: Build and push UI container (latest)
if: github.event_name == 'push'
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
with:
@@ -83,22 +74,8 @@ jobs:
NEXT_PUBLIC_PROWLER_RELEASE_VERSION=${{ needs.setup.outputs.short-sha }}
NEXT_PUBLIC_API_BASE_URL=${{ env.NEXT_PUBLIC_API_BASE_URL }}
push: true
platforms: ${{ matrix.platform }}
tags: |
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-${{ matrix.arch }}
cache-from: type=gha,scope=${{ matrix.arch }}
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
- name: Build and push UI container for ${{ matrix.arch }}
if: github.event_name == 'push' || github.event_name == 'release'
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
with:
context: ${{ env.WORKING_DIRECTORY }}
build-args: |
NEXT_PUBLIC_PROWLER_RELEASE_VERSION=${{ github.event_name == 'release' && format('v{0}', env.RELEASE_TAG) || needs.setup.outputs.short_sha }}
NEXT_PUBLIC_API_BASE_URL=${{ env.NEXT_PUBLIC_API_BASE_URL }}
push: true
tags: |
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }}
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}
cache-from: type=gha
cache-to: type=gha,mode=max
@@ -117,6 +94,22 @@ jobs:
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
payload-file-path: "./.github/scripts/slack-messages/container-release-started.json"
- name: Build and push UI container (release)
if: github.event_name == 'release'
id: container-push
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
with:
context: ${{ env.WORKING_DIRECTORY }}
build-args: |
NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${{ env.RELEASE_TAG }}
NEXT_PUBLIC_API_BASE_URL=${{ env.NEXT_PUBLIC_API_BASE_URL }}
push: true
tags: |
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.RELEASE_TAG }}
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Notify container push completed
if: github.event_name == 'release' && always()
uses: ./.github/actions/slack-notification
@@ -132,52 +125,6 @@ jobs:
payload-file-path: "./.github/scripts/slack-messages/container-release-completed.json"
step-outcome: ${{ steps.container-push.outcome }}
# Create and push multi-architecture manifest
create-manifest:
needs: [setup, container-build-push]
if: github.event_name == 'push' || github.event_name == 'release'
runs-on: ubuntu-latest
steps:
- name: Login to DockerHub
uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
- name: Create and push manifests for push event
if: github.event_name == 'push'
run: |
docker buildx imagetools create \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }} \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }} \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-amd64 \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-arm64
- name: Create and push manifests for release event
if: github.event_name == 'release'
run: |
docker buildx imagetools create \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.RELEASE_TAG }} \
-t ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }} \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-amd64 \
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-arm64
- name: Install regctl
if: always()
uses: regclient/actions/regctl-installer@main
- name: Cleanup intermediate architecture tags
if: always()
run: |
echo "Cleaning up intermediate tags..."
regctl tag delete "${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-amd64" || true
regctl tag delete "${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}-arm64" || true
echo "Cleanup completed"
trigger-deployment:
if: github.event_name == 'push'
needs: [setup, container-build-push]
+4 -69
View File
@@ -45,86 +45,21 @@ pytest_*.xml
.coverage
htmlcov/
# VSCode files and settings
# VSCode files
.vscode/
*.code-workspace
.vscode-test/
# VSCode extension settings and workspaces
.history/
.ionide/
# MCP Server Settings (various locations)
**/cline_mcp_settings.json
**/mcp_settings.json
**/mcp-config.json
**/mcpServers.json
.mcp/
# AI Coding Assistants - Cursor
# Cursor files
.cursorignore
.cursor/
.cursorrules
# AI Coding Assistants - RooCode
# RooCode files
.roo/
.rooignore
.roomodes
# AI Coding Assistants - Cline (formerly Claude Dev)
# Cline files
.cline/
.clineignore
.clinerules
# AI Coding Assistants - Continue
.continue/
continue.json
.continuerc
.continuerc.json
# AI Coding Assistants - GitHub Copilot
.copilot/
.github/copilot/
# AI Coding Assistants - Amazon Q Developer (formerly CodeWhisperer)
.aws/
.codewhisperer/
.amazonq/
.aws-toolkit/
# AI Coding Assistants - Tabnine
.tabnine/
tabnine_config.json
# AI Coding Assistants - Kiro
.kiro/
.kiroignore
kiro.config.json
# AI Coding Assistants - Aider
.aider/
.aider.chat.history.md
.aider.input.history
.aider.tags.cache.v3/
# AI Coding Assistants - Windsurf
.windsurf/
.windsurfignore
# AI Coding Assistants - Replit Agent
.replit
.replitignore
# AI Coding Assistants - Supermaven
.supermaven/
# AI Coding Assistants - Sourcegraph Cody
.cody/
# AI Coding Assistants - General
.ai/
.aiconfig
ai-config.json
# Terraform
.terraform*
-9
View File
@@ -126,12 +126,3 @@ repos:
entry: bash -c 'vulture --exclude "contrib,.venv,api/src/backend/api/tests/,api/src/backend/conftest.py,api/src/backend/tasks/tests/" --min-confidence 100 .'
language: system
files: '.*\.py'
- id: ui-checks
name: UI - Husky Pre-commit
description: "Run UI pre-commit checks (Claude Code validation + healthcheck)"
entry: bash -c 'cd ui && .husky/pre-commit'
language: system
files: '^ui/.*\.(ts|tsx|js|jsx|json|css)$'
pass_filenames: false
verbose: true
-22
View File
@@ -4,10 +4,6 @@ LABEL maintainer="https://github.com/prowler-cloud/prowler"
LABEL org.opencontainers.image.source="https://github.com/prowler-cloud/prowler"
ARG POWERSHELL_VERSION=7.5.0
ENV POWERSHELL_VERSION=${POWERSHELL_VERSION}
ARG TRIVY_VERSION=0.66.0
ENV TRIVY_VERSION=${TRIVY_VERSION}
# hadolint ignore=DL3008
RUN apt-get update && apt-get install -y --no-install-recommends \
@@ -29,24 +25,6 @@ RUN ARCH=$(uname -m) && \
ln -s /opt/microsoft/powershell/7/pwsh /usr/bin/pwsh && \
rm /tmp/powershell.tar.gz
# Install Trivy for IaC scanning
RUN ARCH=$(uname -m) && \
if [ "$ARCH" = "x86_64" ]; then \
TRIVY_ARCH="Linux-64bit" ; \
elif [ "$ARCH" = "aarch64" ]; then \
TRIVY_ARCH="Linux-ARM64" ; \
else \
echo "Unsupported architecture for Trivy: $ARCH" && exit 1 ; \
fi && \
wget --progress=dot:giga "https://github.com/aquasecurity/trivy/releases/download/v${TRIVY_VERSION}/trivy_${TRIVY_VERSION}_${TRIVY_ARCH}.tar.gz" -O /tmp/trivy.tar.gz && \
tar zxf /tmp/trivy.tar.gz -C /tmp && \
mv /tmp/trivy /usr/local/bin/trivy && \
chmod +x /usr/local/bin/trivy && \
rm /tmp/trivy.tar.gz && \
# Create trivy cache directory with proper permissions
mkdir -p /tmp/.cache/trivy && \
chmod 777 /tmp/.cache/trivy
# Add prowler user
RUN addgroup --gid 1000 prowler && \
adduser --uid 1000 --gid 1000 --disabled-password --gecos "" prowler
+15 -26
View File
@@ -2,7 +2,7 @@
All notable changes to the **Prowler API** are documented in this file.
## [1.15.0] (Prowler v5.14.0)
## [1.15.0] (Prowler UNRELEASED)
### Added
- IaC (Infrastructure as Code) provider support for remote repositories [(#8751)](https://github.com/prowler-cloud/prowler/pull/8751)
@@ -14,33 +14,22 @@ All notable changes to the **Prowler API** are documented in this file.
- Support muting findings based on simple rules with custom reason [(#9051)](https://github.com/prowler-cloud/prowler/pull/9051)
- Support C5 compliance framework for the GCP provider [(#9097)](https://github.com/prowler-cloud/prowler/pull/9097)
- Support for Amazon Bedrock and OpenAI compatible providers in Lighthouse AI [(#8957)](https://github.com/prowler-cloud/prowler/pull/8957)
- Support PDF reporting for ENS compliance framework [(#9158)](https://github.com/prowler-cloud/prowler/pull/9158)
- Support PDF reporting for NIS2 compliance framework [(#9170)](https://github.com/prowler-cloud/prowler/pull/9170)
- OpenAPI schema integration with Mintlify documentation including compatibility fixes and dynamic server URL configuration [(#9168)](https://github.com/prowler-cloud/prowler/pull/9168)
- Tenant-wide ThreatScore overview aggregation and snapshot persistence with backfill support [(#9148)](https://github.com/prowler-cloud/prowler/pull/9148)
- Added `metadata`, `details`, and `partition` attributes to `/resources` endpoint & `details`, and `partition` to `/findings` endpoint [(#9098)](https://github.com/prowler-cloud/prowler/pull/9098)
- Support for MongoDB Atlas provider [(#9167)](https://github.com/prowler-cloud/prowler/pull/9167)
- Support Prowler ThreatScore for the K8S provider [(#9235)](https://github.com/prowler-cloud/prowler/pull/9235)
- Enhanced compliance overview endpoint with provider filtering and latest scan aggregation [(#9244)](https://github.com/prowler-cloud/prowler/pull/9244)
- New endpoint `GET /api/v1/overview/regions` to retrieve aggregated findings data by region [(#9273)](https://github.com/prowler-cloud/prowler/pull/9273)
### Changed
- Optimized database write queries for scan related tasks [(#9190)](https://github.com/prowler-cloud/prowler/pull/9190)
- Date filters are now optional for `GET /api/v1/overviews/services` endpoint; returns latest scan data by default [(#9248)](https://github.com/prowler-cloud/prowler/pull/9248)
### Fixed
- Scans no longer fail when findings have UIDs exceeding 300 characters; such findings are now skipped with detailed logging [(#9246)](https://github.com/prowler-cloud/prowler/pull/9246)
- Updated unique constraint for `Provider` model to exclude soft-deleted entries, resolving duplicate errors when re-deleting providers [(#9054)](https://github.com/prowler-cloud/prowler/pull/9054)
- Removed compliance generation for providers without compliance frameworks [(#9208)](https://github.com/prowler-cloud/prowler/pull/9208)
- Refresh output report timestamps for each scan [(#9272)](https://github.com/prowler-cloud/prowler/pull/9272)
- Severity overview endpoint now ignores muted findings as expected [(#9283)](https://github.com/prowler-cloud/prowler/pull/9283)
- Fixed discrepancy between ThreatScore PDF report values and database calculations [(#9296)](https://github.com/prowler-cloud/prowler/pull/9296)
### Security
- Django updated to the latest 5.1 security release, 5.1.14, due to problems with potential [SQL injection](https://github.com/prowler-cloud/prowler/security/dependabot/113) and [denial-of-service vulnerability](https://github.com/prowler-cloud/prowler/security/dependabot/114) [(#9176)](https://github.com/prowler-cloud/prowler/pull/9176)
---
## [1.14.1] (Prowler v5.13.1)
## [1.14.2] (Prowler UNRELEASED)
### Fixed
- Update unique constraint for `Provider` model to exclude soft-deleted entries, resolving duplicate errors when re-deleting providers.[(#9054)](https://github.com/prowler-cloud/prowler/pull/9054)
- Remove compliance generation for providers without compliance frameworks [(#9208)](https://github.com/prowler-cloud/prowler/pull/9208)
## [1.14.1] (Prowler 5.13.1)
### Fixed
- `/api/v1/overviews/providers` collapses data by provider type so the UI receives a single aggregated record per cloud family even when multiple accounts exist [(#9053)](https://github.com/prowler-cloud/prowler/pull/9053)
@@ -49,7 +38,7 @@ All notable changes to the **Prowler API** are documented in this file.
---
## [1.14.0] (Prowler v5.13.0)
## [1.14.0] (Prowler 5.13.0)
### Added
- Default JWT keys are generated and stored if they are missing from configuration [(#8655)](https://github.com/prowler-cloud/prowler/pull/8655)
@@ -73,14 +62,14 @@ All notable changes to the **Prowler API** are documented in this file.
---
## [1.13.2] (Prowler v5.12.3)
## [1.13.2] (Prowler 5.12.3)
### Fixed
- 500 error when deleting user [(#8731)](https://github.com/prowler-cloud/prowler/pull/8731)
---
## [1.13.1] (Prowler v5.12.2)
## [1.13.1] (Prowler 5.12.2)
### Changed
- Renamed compliance overview task queue to `compliance` [(#8755)](https://github.com/prowler-cloud/prowler/pull/8755)
@@ -90,7 +79,7 @@ All notable changes to the **Prowler API** are documented in this file.
---
## [1.13.0] (Prowler v5.12.0)
## [1.13.0] (Prowler 5.12.0)
### Added
- Integration with JIRA, enabling sending findings to a JIRA project [(#8622)](https://github.com/prowler-cloud/prowler/pull/8622), [(#8637)](https://github.com/prowler-cloud/prowler/pull/8637)
@@ -99,7 +88,7 @@ All notable changes to the **Prowler API** are documented in this file.
---
## [1.12.0] (Prowler v5.11.0)
## [1.12.0] (Prowler 5.11.0)
### Added
- Lighthouse support for OpenAI GPT-5 [(#8527)](https://github.com/prowler-cloud/prowler/pull/8527)
@@ -111,7 +100,7 @@ All notable changes to the **Prowler API** are documented in this file.
---
## [1.11.0] (Prowler v5.10.0)
## [1.11.0] (Prowler 5.10.0)
### Added
- Github provider support [(#8271)](https://github.com/prowler-cloud/prowler/pull/8271)
+5 -79
View File
@@ -1,4 +1,4 @@
# This file is automatically @generated by Poetry 2.1.1 and should not be changed by hand.
# This file is automatically @generated by Poetry 2.1.4 and should not be changed by hand.
[[package]]
name = "about-time"
@@ -610,24 +610,6 @@ azure-mgmt-core = ">=1.3.2"
isodate = ">=0.6.1"
typing-extensions = ">=4.6.0"
[[package]]
name = "azure-mgmt-postgresqlflexibleservers"
version = "1.1.0"
description = "Microsoft Azure Postgresqlflexibleservers Management Client Library for Python"
optional = false
python-versions = ">=3.8"
groups = ["main"]
files = [
{file = "azure_mgmt_postgresqlflexibleservers-1.1.0-py3-none-any.whl", hash = "sha256:87ddb5a5e6d12c45769485d234cfe0322140e3a0a7636d0e61fb00ac544b5d20"},
{file = "azure_mgmt_postgresqlflexibleservers-1.1.0.tar.gz", hash = "sha256:9ede9d8ba63e9d2879cb74adc903c649af3bc5460a02787287b0cd18d754af14"},
]
[package.dependencies]
azure-common = ">=1.1"
azure-mgmt-core = ">=1.3.2"
isodate = ">=0.6.1"
typing-extensions = ">=4.6.0"
[[package]]
name = "azure-mgmt-rdbms"
version = "10.1.0"
@@ -1182,18 +1164,6 @@ files = [
{file = "charset_normalizer-3.4.3.tar.gz", hash = "sha256:6fce4b8500244f6fcb71465d4a4930d132ba9ab8e71a7859e6a5d59851068d14"},
]
[[package]]
name = "circuitbreaker"
version = "2.1.3"
description = "Python Circuit Breaker pattern implementation"
optional = false
python-versions = "*"
groups = ["main"]
files = [
{file = "circuitbreaker-2.1.3-py3-none-any.whl", hash = "sha256:87ba6a3ed03fdc7032bc175561c2b04d52ade9d5faf94ca2b035fbdc5e6b1dd1"},
{file = "circuitbreaker-2.1.3.tar.gz", hash = "sha256:1a4baee510f7bea3c91b194dcce7c07805fe96c4423ed5594b75af438531d084"},
]
[[package]]
name = "click"
version = "8.2.1"
@@ -4076,29 +4046,6 @@ rsa = ["cryptography (>=3.0.0)"]
signals = ["blinker (>=1.4.0)"]
signedtoken = ["cryptography (>=3.0.0)", "pyjwt (>=2.0.0,<3)"]
[[package]]
name = "oci"
version = "2.160.3"
description = "Oracle Cloud Infrastructure Python SDK"
optional = false
python-versions = "*"
groups = ["main"]
files = [
{file = "oci-2.160.3-py3-none-any.whl", hash = "sha256:858bff3e697098bdda44833d2476bfb4632126f0182178e7dbde4dbd156d71f0"},
{file = "oci-2.160.3.tar.gz", hash = "sha256:57514889be3b713a8385d86e3ba8a33cf46e3563c2a7e29a93027fb30b8a2537"},
]
[package.dependencies]
certifi = "*"
circuitbreaker = {version = ">=1.3.1,<3.0.0", markers = "python_version >= \"3.7\""}
cryptography = ">=3.2.1,<46.0.0"
pyOpenSSL = ">=17.5.0,<25.0.0"
python-dateutil = ">=2.5.3,<3.0.0"
pytz = ">=2016.10"
[package.extras]
adk = ["docstring-parser (>=0.16) ; python_version >= \"3.10\" and python_version < \"4\"", "mcp (>=1.6.0) ; python_version >= \"3.10\" and python_version < \"4\"", "pydantic (>=2.10.6) ; python_version >= \"3.10\" and python_version < \"4\"", "rich (>=13.9.4) ; python_version >= \"3.10\" and python_version < \"4\""]
[[package]]
name = "openai"
version = "1.101.0"
@@ -4633,7 +4580,7 @@ files = [
[[package]]
name = "prowler"
version = "5.14.0"
version = "5.13.0"
description = "Prowler is an Open Source security tool to perform AWS, GCP and Azure security best practices assessments, audits, incident response, continuous monitoring, hardening and forensics readiness. It contains hundreds of controls covering CIS, NIST 800, NIST CSF, CISA, RBI, FedRAMP, PCI-DSS, GDPR, HIPAA, FFIEC, SOC2, GXP, AWS Well-Architected Framework Security Pillar, AWS Foundational Technical Review (FTR), ENS (Spanish National Security Scheme) and your custom security frameworks."
optional = false
python-versions = ">3.9.1,<3.13"
@@ -4658,7 +4605,6 @@ azure-mgmt-keyvault = "10.3.1"
azure-mgmt-loganalytics = "12.0.0"
azure-mgmt-monitor = "6.0.2"
azure-mgmt-network = "28.1.0"
azure-mgmt-postgresqlflexibleservers = "1.1.0"
azure-mgmt-rdbms = "10.1.0"
azure-mgmt-recoveryservices = "3.1.0"
azure-mgmt-recoveryservicesbackup = "9.2.0"
@@ -4688,7 +4634,6 @@ markdown = "3.9.0"
microsoft-kiota-abstractions = "1.9.2"
msgraph-sdk = "1.23.0"
numpy = "2.0.2"
oci = "2.160.3"
pandas = "2.2.3"
py-iam-expand = "0.1.0"
py-ocsf-models = "0.5.0"
@@ -4705,8 +4650,8 @@ tzlocal = "5.3.1"
[package.source]
type = "git"
url = "https://github.com/prowler-cloud/prowler.git"
reference = "v5.14"
resolved_reference = "3b05a1430e016cee92b60973705cba400255d9e5"
reference = "master"
resolved_reference = "a52697bfdfee83d14a49c11dcbe96888b5cd767e"
[[package]]
name = "psutil"
@@ -5191,25 +5136,6 @@ cffi = ">=1.4.1"
docs = ["sphinx (>=1.6.5)", "sphinx-rtd-theme"]
tests = ["hypothesis (>=3.27.0)", "pytest (>=3.2.1,!=3.3.0)"]
[[package]]
name = "pyopenssl"
version = "24.3.0"
description = "Python wrapper module around the OpenSSL library"
optional = false
python-versions = ">=3.7"
groups = ["main"]
files = [
{file = "pyOpenSSL-24.3.0-py3-none-any.whl", hash = "sha256:e474f5a473cd7f92221cc04976e48f4d11502804657a08a989fb3be5514c904a"},
{file = "pyopenssl-24.3.0.tar.gz", hash = "sha256:49f7a019577d834746bc55c5fce6ecbcec0f2b4ec5ce1cf43a9a173b8138bb36"},
]
[package.dependencies]
cryptography = ">=41.0.5,<45"
[package.extras]
docs = ["sphinx (!=5.2.0,!=5.2.0.post0,!=7.2.5)", "sphinx_rtd_theme"]
test = ["pretend", "pytest (>=3.0.1)", "pytest-rerunfailures"]
[[package]]
name = "pyparsing"
version = "3.2.3"
@@ -6860,4 +6786,4 @@ type = ["pytest-mypy"]
[metadata]
lock-version = "2.1"
python-versions = ">=3.11,<3.13"
content-hash = "ed4c11443ea6a54da50a62c8c27efbf69608ef9f15894746696169fa010d04c9"
content-hash = "943e2cd6b87229704550d4e140b36509fb9f58896ebb5834b9fbabe28a9ee92f"
+1 -1
View File
@@ -24,7 +24,7 @@ dependencies = [
"drf-spectacular-jsonapi==0.5.1",
"gunicorn==23.0.0",
"lxml==5.3.2",
"prowler @ git+https://github.com/prowler-cloud/prowler.git@v5.14",
"prowler @ git+https://github.com/prowler-cloud/prowler.git@master",
"psycopg2-binary==2.9.9",
"pytest-celery[redis] (>=1.0.1,<2.0.0)",
"sentry-sdk[django] (>=2.20.0,<3.0.0)",
+24 -13
View File
@@ -761,14 +761,6 @@ class RoleFilter(FilterSet):
class ComplianceOverviewFilter(FilterSet):
inserted_at = DateFilter(field_name="inserted_at", lookup_expr="date")
scan_id = UUIDFilter(field_name="scan_id")
provider_id = UUIDFilter(field_name="scan__provider__id", lookup_expr="exact")
provider_id__in = UUIDInFilter(field_name="scan__provider__id", lookup_expr="in")
provider_type = ChoiceFilter(
field_name="scan__provider__provider", choices=Provider.ProviderChoices.choices
)
provider_type__in = ChoiceInFilter(
field_name="scan__provider__provider", choices=Provider.ProviderChoices.choices
)
region = CharFilter(field_name="region")
class Meta:
@@ -820,8 +812,7 @@ class ScanSummarySeverityFilter(ScanSummaryFilter):
elif value == OverviewStatusChoices.PASS:
return queryset.annotate(status_count=F("_pass"))
else:
# Exclude muted findings by default
return queryset.annotate(status_count=F("_pass") + F("fail"))
return queryset.annotate(status_count=F("total"))
def filter_status_in(self, queryset, name, value):
# Validate the status values
@@ -830,7 +821,7 @@ class ScanSummarySeverityFilter(ScanSummaryFilter):
if status_val not in valid_statuses:
raise ValidationError(f"Invalid status value: {status_val}")
# If all statuses or no valid statuses, exclude muted findings (pass + fail)
# If all statuses or no valid statuses, use total
if (
set(value)
>= {
@@ -839,7 +830,7 @@ class ScanSummarySeverityFilter(ScanSummaryFilter):
}
or not value
):
return queryset.annotate(status_count=F("_pass") + F("fail"))
return queryset.annotate(status_count=F("total"))
# Build the sum expression based on status values
sum_expression = None
@@ -857,7 +848,7 @@ class ScanSummarySeverityFilter(ScanSummaryFilter):
sum_expression = sum_expression + field_expr
if sum_expression is None:
return queryset.annotate(status_count=F("_pass") + F("fail"))
return queryset.annotate(status_count=F("total"))
return queryset.annotate(status_count=sum_expression)
@@ -869,6 +860,26 @@ class ScanSummarySeverityFilter(ScanSummaryFilter):
}
class ServiceOverviewFilter(ScanSummaryFilter):
def is_valid(self):
# Check if at least one of the inserted_at filters is present
inserted_at_filters = [
self.data.get("inserted_at"),
self.data.get("inserted_at__gte"),
self.data.get("inserted_at__lte"),
]
if not any(inserted_at_filters):
raise ValidationError(
{
"inserted_at": [
"At least one of filter[inserted_at], filter[inserted_at__gte], or "
"filter[inserted_at__lte] is required."
]
}
)
return super().is_valid()
class IntegrationFilter(FilterSet):
inserted_at = DateFilter(field_name="inserted_at", lookup_expr="date")
integration_type = ChoiceFilter(choices=Integration.IntegrationChoices.choices)
@@ -22,7 +22,7 @@ class Migration(migrations.Migration):
("kubernetes", "Kubernetes"),
("m365", "M365"),
("github", "GitHub"),
("oraclecloud", "Oracle Cloud Infrastructure"),
("oci", "Oracle Cloud Infrastructure"),
("iac", "IaC"),
],
default="aws",
@@ -29,8 +29,4 @@ class Migration(migrations.Migration):
default="aws",
),
),
migrations.RunSQL(
"ALTER TYPE provider ADD VALUE IF NOT EXISTS 'mongodbatlas';",
reverse_sql=migrations.RunSQL.noop,
),
]
@@ -1,29 +0,0 @@
from django.contrib.postgres.operations import RemoveIndexConcurrently
from django.db import migrations
class Migration(migrations.Migration):
atomic = False
dependencies = [
("api", "0057_threatscoresnapshot"),
]
operations = [
RemoveIndexConcurrently(
model_name="compliancerequirementoverview",
name="cro_tenant_scan_idx",
),
RemoveIndexConcurrently(
model_name="compliancerequirementoverview",
name="cro_scan_comp_idx",
),
RemoveIndexConcurrently(
model_name="compliancerequirementoverview",
name="cro_scan_comp_req_idx",
),
RemoveIndexConcurrently(
model_name="compliancerequirementoverview",
name="cro_scan_comp_req_reg_idx",
),
]
@@ -1,75 +0,0 @@
# Generated by Django 5.1.13 on 2025-10-30 15:23
import uuid
import django.db.models.deletion
from django.db import migrations, models
import api.rls
class Migration(migrations.Migration):
dependencies = [
("api", "0058_drop_redundant_compliance_requirement_indexes"),
]
operations = [
migrations.CreateModel(
name="ComplianceOverviewSummary",
fields=[
(
"id",
models.UUIDField(
default=uuid.uuid4,
editable=False,
primary_key=True,
serialize=False,
),
),
("inserted_at", models.DateTimeField(auto_now_add=True)),
("compliance_id", models.TextField()),
("requirements_passed", models.IntegerField(default=0)),
("requirements_failed", models.IntegerField(default=0)),
("requirements_manual", models.IntegerField(default=0)),
("total_requirements", models.IntegerField(default=0)),
(
"scan",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="compliance_summaries",
related_query_name="compliance_summary",
to="api.scan",
),
),
(
"tenant",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE, to="api.tenant"
),
),
],
options={
"db_table": "compliance_overview_summaries",
"abstract": False,
"indexes": [
models.Index(
fields=["tenant_id", "scan_id"], name="cos_tenant_scan_idx"
)
],
"constraints": [
models.UniqueConstraint(
fields=("tenant_id", "scan_id", "compliance_id"),
name="unique_compliance_summary_per_scan",
)
],
},
),
migrations.AddConstraint(
model_name="complianceoverviewsummary",
constraint=api.rls.RowLevelSecurityConstraint(
"tenant_id",
name="rls_on_complianceoverviewsummary",
statements=["SELECT", "INSERT", "UPDATE", "DELETE"],
),
),
]
+22 -54
View File
@@ -1371,70 +1371,35 @@ class ComplianceRequirementOverview(RowLevelSecurityProtectedModel):
),
]
indexes = [
models.Index(fields=["tenant_id", "scan_id"], name="cro_tenant_scan_idx"),
models.Index(
fields=["tenant_id", "scan_id", "compliance_id"],
name="cro_scan_comp_idx",
),
models.Index(
fields=["tenant_id", "scan_id", "compliance_id", "region"],
name="cro_scan_comp_reg_idx",
),
models.Index(
fields=["tenant_id", "scan_id", "compliance_id", "requirement_id"],
name="cro_scan_comp_req_idx",
),
models.Index(
fields=[
"tenant_id",
"scan_id",
"compliance_id",
"requirement_id",
"region",
],
name="cro_scan_comp_req_reg_idx",
),
]
class JSONAPIMeta:
resource_name = "compliance-requirements-overviews"
class ComplianceOverviewSummary(RowLevelSecurityProtectedModel):
"""
Pre-aggregated compliance overview aggregated across ALL regions.
One row per (scan_id, compliance_id) combination.
This table optimizes the common case where users view overall compliance
without filtering by region. For region-specific views, the detailed
ComplianceRequirementOverview table is used instead.
"""
id = models.UUIDField(primary_key=True, default=uuid4, editable=False)
inserted_at = models.DateTimeField(auto_now_add=True, editable=False)
scan = models.ForeignKey(
Scan,
on_delete=models.CASCADE,
related_name="compliance_summaries",
related_query_name="compliance_summary",
)
compliance_id = models.TextField(blank=False)
# Pre-aggregated scores (computed across ALL regions)
requirements_passed = models.IntegerField(default=0)
requirements_failed = models.IntegerField(default=0)
requirements_manual = models.IntegerField(default=0)
total_requirements = models.IntegerField(default=0)
class Meta(RowLevelSecurityProtectedModel.Meta):
db_table = "compliance_overview_summaries"
constraints = [
models.UniqueConstraint(
fields=("tenant_id", "scan_id", "compliance_id"),
name="unique_compliance_summary_per_scan",
),
RowLevelSecurityConstraint(
field="tenant_id",
name="rls_on_%(class)s",
statements=["SELECT", "INSERT", "DELETE"],
),
]
indexes = [
models.Index(
fields=["tenant_id", "scan_id"],
name="cos_tenant_scan_idx",
),
]
class JSONAPIMeta:
resource_name = "compliance-overview-summaries"
class ScanSummary(RowLevelSecurityProtectedModel):
objects = ActiveProviderManager()
all_objects = models.Manager()
@@ -2282,6 +2247,9 @@ class ThreatScoreSnapshot(RowLevelSecurityProtectedModel):
Snapshots are created automatically after each ThreatScore report generation.
"""
objects = models.Manager()
all_objects = models.Manager()
id = models.UUIDField(primary_key=True, default=uuid4, editable=False)
inserted_at = models.DateTimeField(auto_now_add=True, editable=False)
+137
View File
@@ -36,6 +36,143 @@ def _extract_task_example_from_components(components):
}
def fix_empty_id_fields(result, generator, request, public): # noqa: F841
"""
Fix empty id fields in JSON:API request schemas.
drf-spectacular-jsonapi sometimes generates empty id field definitions ({})
which cause validation errors in Mintlify and other OpenAPI validators.
"""
if not isinstance(result, dict):
return result
components = result.get("components", {}) or {}
schemas = components.get("schemas", {}) or {}
for schema_name, schema in schemas.items():
if not isinstance(schema, dict):
continue
# Check if this is a JSON:API request schema with a data object
properties = schema.get("properties", {})
if not isinstance(properties, dict):
continue
data_prop = properties.get("data")
if not isinstance(data_prop, dict):
continue
data_properties = data_prop.get("properties", {})
if not isinstance(data_properties, dict):
continue
# Fix empty id field
id_field = data_properties.get("id")
if id_field == {} or (isinstance(id_field, dict) and not id_field):
data_properties["id"] = {
"type": "string",
"format": "uuid",
"description": "Unique identifier for this resource object.",
}
return result
def convert_pattern_properties_to_additional(obj):
"""
Recursively convert patternProperties to additionalProperties.
OpenAPI 3.0.x doesn't support patternProperties (only available in 3.1+).
"""
if isinstance(obj, dict):
if "patternProperties" in obj:
# Get the pattern and its schema
pattern_props = obj.pop("patternProperties")
# Use the first pattern's schema as additionalProperties
if pattern_props:
first_pattern_schema = next(iter(pattern_props.values()))
obj["additionalProperties"] = first_pattern_schema
# Recursively process all nested objects
for key, value in obj.items():
obj[key] = convert_pattern_properties_to_additional(value)
elif isinstance(obj, list):
return [convert_pattern_properties_to_additional(item) for item in obj]
return obj
def fix_pattern_properties(result, generator, request, public): # noqa: F841
"""
Convert patternProperties to additionalProperties for OpenAPI 3.0 compatibility.
patternProperties is only supported in OpenAPI 3.1+, but drf-spectacular
generates OpenAPI 3.0.x specs.
"""
if not isinstance(result, dict):
return result
return convert_pattern_properties_to_additional(result)
def fix_invalid_types(obj):
"""
Recursively fix invalid type values in OpenAPI schemas.
Converts invalid types like "email" to proper OpenAPI format.
"""
if isinstance(obj, dict):
# Fix invalid "type" values
if "type" in obj:
type_value = obj["type"]
if type_value == "email":
obj["type"] = "string"
obj["format"] = "email"
elif type_value == "url":
obj["type"] = "string"
obj["format"] = "uri"
elif type_value == "uuid":
obj["type"] = "string"
obj["format"] = "uuid"
# Recursively process all nested objects
for key, value in list(obj.items()):
obj[key] = fix_invalid_types(value)
elif isinstance(obj, list):
return [fix_invalid_types(item) for item in obj]
return obj
def fix_type_formats(result, generator, request, public): # noqa: F841
"""
Fix invalid type values in OpenAPI schemas.
drf-spectacular sometimes generates invalid type values like "email"
instead of "type": "string" with "format": "email".
"""
if not isinstance(result, dict):
return result
return fix_invalid_types(result)
def add_api_servers(result, generator, request, public): # noqa: F841
"""
Add servers configuration to OpenAPI spec for Mintlify API playground.
This enables the "Try it out" feature in the documentation.
Only adds the production URL to ensure consistent documentation.
"""
if not isinstance(result, dict):
return result
# Add servers array if not already present
if "servers" not in result:
result["servers"] = [
{
"url": "https://api.prowler.com",
"description": "Prowler Cloud API",
}
]
return result
def attach_task_202_examples(result, generator, request, public): # noqa: F841
if not isinstance(result, dict):
return result
File diff suppressed because it is too large Load Diff
+20 -202
View File
@@ -35,8 +35,6 @@ from rest_framework.response import Response
from api.compliance import get_compliance_frameworks
from api.db_router import MainRouter
from api.models import (
ComplianceOverviewSummary,
ComplianceRequirementOverview,
Finding,
Integration,
Invitation,
@@ -57,7 +55,6 @@ from api.models import (
Scan,
ScanSummary,
StateChoices,
StatusChoices,
Task,
TenantAPIKey,
ThreatScoreSnapshot,
@@ -3545,9 +3542,6 @@ class TestResourceViewSet:
)
assert response.status_code == status.HTTP_200_OK
assert len(response.json()["data"]) == len(resources_fixture)
assert "metadata" in response.json()["data"][0]["attributes"]
assert "details" in response.json()["data"][0]["attributes"]
assert "partition" in response.json()["data"][0]["attributes"]
@pytest.mark.parametrize(
"include_values, expected_resources",
@@ -5820,44 +5814,16 @@ class TestProviderGroupMembershipViewSet:
@pytest.mark.django_db
class TestComplianceOverviewViewSet:
@pytest.fixture(autouse=True)
def mock_backfill_task(self):
with patch("api.v1.views.backfill_compliance_summaries_task.delay") as mock:
yield mock
def test_compliance_overview_list_none(
self,
authenticated_client,
tenants_fixture,
providers_fixture,
mock_backfill_task,
):
tenant = tenants_fixture[0]
provider = providers_fixture[0]
scan = Scan.objects.create(
name="empty-compliance-scan",
provider=provider,
trigger=Scan.TriggerChoices.MANUAL,
state=StateChoices.COMPLETED,
tenant=tenant,
)
def test_compliance_overview_list_none(self, authenticated_client):
response = authenticated_client.get(
reverse("complianceoverview-list"),
{"filter[scan_id]": str(scan.id)},
{"filter[scan_id]": "8d20ac7d-4cbc-435e-85f4-359be37af821"},
)
assert response.status_code == status.HTTP_200_OK
assert len(response.json()["data"]) == 0
mock_backfill_task.assert_called_once()
_, kwargs = mock_backfill_task.call_args
assert kwargs["scan_id"] == str(scan.id)
assert str(kwargs["tenant_id"]) == str(tenant.id)
def test_compliance_overview_list(
self,
authenticated_client,
compliance_requirements_overviews_fixture,
mock_backfill_task,
self, authenticated_client, compliance_requirements_overviews_fixture
):
# List compliance overviews with existing data
requirement_overview1 = compliance_requirements_overviews_fixture[0]
@@ -5887,112 +5853,6 @@ class TestComplianceOverviewViewSet:
assert "requirements_failed" in attributes
assert "requirements_manual" in attributes
assert "total_requirements" in attributes
mock_backfill_task.assert_called_once()
_, kwargs = mock_backfill_task.call_args
assert kwargs["scan_id"] == scan_id
def test_compliance_overview_list_uses_preaggregated_summaries(
self,
authenticated_client,
tenants_fixture,
providers_fixture,
mock_backfill_task,
):
tenant = tenants_fixture[0]
provider = providers_fixture[0]
scan = Scan.objects.create(
name="preaggregated-scan",
provider=provider,
trigger=Scan.TriggerChoices.MANUAL,
state=StateChoices.COMPLETED,
tenant=tenant,
)
ComplianceRequirementOverview.objects.create(
tenant=tenant,
scan=scan,
compliance_id="cis_1.4_aws",
framework="CIS-1.4-AWS",
version="1.4",
description="CIS AWS Foundations Benchmark v1.4.0",
region="eu-west-1",
requirement_id="framework-metadata",
requirement_status=StatusChoices.PASS,
passed_checks=1,
failed_checks=0,
total_checks=1,
)
ComplianceOverviewSummary.objects.create(
tenant=tenant,
scan=scan,
compliance_id="cis_1.4_aws",
requirements_passed=5,
requirements_failed=1,
requirements_manual=2,
total_requirements=8,
)
response = authenticated_client.get(
reverse("complianceoverview-list"),
{"filter[scan_id]": str(scan.id)},
)
assert response.status_code == status.HTTP_200_OK
data = response.json()["data"]
assert len(data) == 1
overview = data[0]
assert overview["id"] == "cis_1.4_aws"
assert overview["attributes"]["requirements_passed"] == 5
assert overview["attributes"]["requirements_failed"] == 1
assert overview["attributes"]["requirements_manual"] == 2
assert overview["attributes"]["total_requirements"] == 8
assert "framework" in overview["attributes"]
assert "version" in overview["attributes"]
mock_backfill_task.assert_not_called()
def test_compliance_overview_region_filter_skips_backfill(
self,
authenticated_client,
compliance_requirements_overviews_fixture,
mock_backfill_task,
):
requirement_overview = compliance_requirements_overviews_fixture[0]
scan_id = str(requirement_overview.scan.id)
response = authenticated_client.get(
reverse("complianceoverview-list"),
{
"filter[scan_id]": scan_id,
"filter[region]": requirement_overview.region,
},
)
assert response.status_code == status.HTTP_200_OK
assert len(response.json()["data"]) >= 1
mock_backfill_task.assert_not_called()
def test_compliance_overview_list_without_scan_id(
self, authenticated_client, compliance_requirements_overviews_fixture
):
# Ensure the endpoint works without passing a scan filter
response = authenticated_client.get(reverse("complianceoverview-list"))
assert response.status_code == status.HTTP_200_OK
data = response.json()["data"]
assert len(data) == 3
# Validate payload structure
first_item = data[0]
assert "id" in first_item
assert "attributes" in first_item
attributes = first_item["attributes"]
assert "framework" in attributes
assert "version" in attributes
assert "requirements_passed" in attributes
assert "requirements_failed" in attributes
assert "requirements_manual" in attributes
assert "total_requirements" in attributes
def test_compliance_overview_metadata(
self, authenticated_client, compliance_requirements_overviews_fixture
@@ -6146,11 +6006,6 @@ class TestComplianceOverviewViewSet:
requirement_overview1 = compliance_requirements_overviews_fixture[0]
scan_id = str(requirement_overview1.scan.id)
# Remove existing compliance data so the view falls back to task checks
scan = requirement_overview1.scan
ComplianceOverviewSummary.objects.filter(scan=scan).delete()
ComplianceRequirementOverview.objects.filter(scan=scan).delete()
# Mock a running task
with patch.object(
ComplianceOverviewViewSet, "get_task_response_if_running"
@@ -6178,11 +6033,6 @@ class TestComplianceOverviewViewSet:
requirement_overview1 = compliance_requirements_overviews_fixture[0]
scan_id = str(requirement_overview1.scan.id)
# Remove existing compliance data so the view falls back to task checks
scan = requirement_overview1.scan
ComplianceOverviewSummary.objects.filter(scan=scan).delete()
ComplianceRequirementOverview.objects.filter(scan=scan).delete()
# Mock a failed task
with patch.object(
ComplianceOverviewViewSet, "get_task_response_if_running"
@@ -6206,8 +6056,6 @@ class TestComplianceOverviewViewSet:
("framework", "framework", 1),
("version", "version", 1),
("region", "region", 1),
("region__in", "region", 1),
("region.in", "region", 1),
],
)
def test_compliance_overview_filters(
@@ -6280,10 +6128,10 @@ class TestOverviewViewSet:
response = authenticated_client.get(reverse("overview-providers"))
assert response.status_code == status.HTTP_200_OK
assert len(response.json()["data"]) == 1
assert response.json()["data"][0]["attributes"]["findings"]["total"] == 9
assert response.json()["data"][0]["attributes"]["findings"]["total"] == 4
assert response.json()["data"][0]["attributes"]["findings"]["pass"] == 2
assert response.json()["data"][0]["attributes"]["findings"]["fail"] == 1
assert response.json()["data"][0]["attributes"]["findings"]["muted"] == 6
assert response.json()["data"][0]["attributes"]["findings"]["muted"] == 1
# Aggregated resources include all AWS providers present in the tenant
assert response.json()["data"][0]["attributes"]["resources"]["total"] == 3
@@ -6335,10 +6183,10 @@ class TestOverviewViewSet:
assert len(data) == 1
attributes = data[0]["attributes"]
assert attributes["findings"]["total"] == 15
assert attributes["findings"]["total"] == 10
assert attributes["findings"]["pass"] == 5
assert attributes["findings"]["fail"] == 3
assert attributes["findings"]["muted"] == 7
assert attributes["findings"]["muted"] == 2
assert attributes["resources"]["total"] == 4
def test_overview_providers_count(
@@ -6780,35 +6628,7 @@ class TestOverviewViewSet:
self, authenticated_client, scan_summaries_fixture
):
response = authenticated_client.get(reverse("overview-services"))
assert response.status_code == status.HTTP_200_OK
# Should return services from latest scans
assert len(response.json()["data"]) == 2
def test_overview_regions_list(self, authenticated_client, scan_summaries_fixture):
response = authenticated_client.get(
reverse("overview-regions"), {"filter[inserted_at]": TODAY}
)
assert response.status_code == status.HTTP_200_OK
# Only two different regions in the fixture (region1, region2)
assert len(response.json()["data"]) == 2
data = response.json()["data"]
regions = {item["id"]: item["attributes"] for item in data}
assert "aws:region1" in regions
assert "aws:region2" in regions
# region1 has 5 findings (2 pass, 0 fail, 3 muted)
assert regions["aws:region1"]["total"] == 5
assert regions["aws:region1"]["pass"] == 2
assert regions["aws:region1"]["fail"] == 0
assert regions["aws:region1"]["muted"] == 3
# region2 has 4 findings (0 pass, 1 fail, 3 muted)
assert regions["aws:region2"]["total"] == 4
assert regions["aws:region2"]["pass"] == 0
assert regions["aws:region2"]["fail"] == 1
assert regions["aws:region2"]["muted"] == 3
assert response.status_code == status.HTTP_400_BAD_REQUEST
def test_overview_services_list(self, authenticated_client, scan_summaries_fixture):
response = authenticated_client.get(
@@ -6817,14 +6637,15 @@ class TestOverviewViewSet:
assert response.status_code == status.HTTP_200_OK
# Only two different services
assert len(response.json()["data"]) == 2
# Fixed data from the fixture
# Fixed data from the fixture, TODO improve this at some point with something more dynamic
service1_data = response.json()["data"][0]
service2_data = response.json()["data"][1]
assert service1_data["id"] == "service1"
assert service2_data["id"] == "service2"
assert service1_data["attributes"]["total"] == 7
assert service2_data["attributes"]["total"] == 2
# TODO fix numbers when muted_findings filter is fixed
assert service1_data["attributes"]["total"] == 3
assert service2_data["attributes"]["total"] == 1
assert service1_data["attributes"]["pass"] == 1
assert service2_data["attributes"]["pass"] == 1
@@ -6832,8 +6653,8 @@ class TestOverviewViewSet:
assert service1_data["attributes"]["fail"] == 1
assert service2_data["attributes"]["fail"] == 0
assert service1_data["attributes"]["muted"] == 5
assert service2_data["attributes"]["muted"] == 1
assert service1_data["attributes"]["muted"] == 1
assert service2_data["attributes"]["muted"] == 0
def test_overview_findings_provider_id_in_filter(
self, authenticated_client, tenants_fixture, providers_fixture
@@ -6943,7 +6764,6 @@ class TestOverviewViewSet:
tenant=tenant,
)
# Muted findings should be excluded from severity counts
ScanSummary.objects.create(
tenant=tenant,
scan=scan1,
@@ -6953,8 +6773,8 @@ class TestOverviewViewSet:
region="region-a",
_pass=4,
fail=4,
muted=3,
total=11,
muted=0,
total=8,
)
ScanSummary.objects.create(
tenant=tenant,
@@ -6965,8 +6785,8 @@ class TestOverviewViewSet:
region="region-b",
_pass=2,
fail=2,
muted=2,
total=6,
muted=0,
total=4,
)
ScanSummary.objects.create(
tenant=tenant,
@@ -6977,8 +6797,8 @@ class TestOverviewViewSet:
region="region-c",
_pass=1,
fail=2,
muted=5,
total=8,
muted=0,
total=3,
)
single_response = authenticated_client.get(
@@ -6987,7 +6807,6 @@ class TestOverviewViewSet:
)
assert single_response.status_code == status.HTTP_200_OK
single_attributes = single_response.json()["data"]["attributes"]
# Should only count pass + fail, excluding muted (3 muted in high, 2 in medium)
assert single_attributes["high"] == 8
assert single_attributes["medium"] == 4
assert single_attributes["critical"] == 0
@@ -6998,7 +6817,6 @@ class TestOverviewViewSet:
)
assert combined_response.status_code == status.HTTP_200_OK
combined_attributes = combined_response.json()["data"]["attributes"]
# Should only count pass + fail, excluding muted (5 muted in critical)
assert combined_attributes["high"] == 8
assert combined_attributes["medium"] == 4
assert combined_attributes["critical"] == 3
-34
View File
@@ -1167,17 +1167,11 @@ class ResourceSerializer(RLSSerializer):
"findings",
"failed_findings_count",
"url",
"metadata",
"details",
"partition",
]
extra_kwargs = {
"id": {"read_only": True},
"inserted_at": {"read_only": True},
"updated_at": {"read_only": True},
"metadata": {"read_only": True},
"details": {"read_only": True},
"partition": {"read_only": True},
}
included_serializers = {
@@ -1234,15 +1228,11 @@ class ResourceIncludeSerializer(RLSSerializer):
"service",
"type_",
"tags",
"details",
"partition",
]
extra_kwargs = {
"id": {"read_only": True},
"inserted_at": {"read_only": True},
"updated_at": {"read_only": True},
"details": {"read_only": True},
"partition": {"read_only": True},
}
@extend_schema_field(
@@ -2229,30 +2219,6 @@ class OverviewServiceSerializer(serializers.Serializer):
return {"version": "v1"}
class OverviewRegionSerializer(serializers.Serializer):
id = serializers.SerializerMethodField()
provider_type = serializers.CharField()
region = serializers.CharField()
total = serializers.IntegerField()
_pass = serializers.IntegerField()
fail = serializers.IntegerField()
muted = serializers.IntegerField()
class JSONAPIMeta:
resource_name = "regions-overview"
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.fields["pass"] = self.fields.pop("_pass")
def get_id(self, obj):
"""Generate unique ID from provider_type and region."""
return f"{obj['provider_type']}:{obj['region']}"
def get_root_meta(self, _resource, _many):
return {"version": "v1"}
# Schedules
+152 -452
View File
@@ -75,7 +75,6 @@ from rest_framework_simplejwt.exceptions import InvalidToken, TokenError
from tasks.beat import schedule_provider_scan
from tasks.jobs.export import get_s3_client
from tasks.tasks import (
backfill_compliance_summaries_task,
backfill_scan_resource_summaries_task,
check_integration_connection_task,
check_lighthouse_connection_task,
@@ -119,6 +118,7 @@ from api.filters import (
ScanFilter,
ScanSummaryFilter,
ScanSummarySeverityFilter,
ServiceOverviewFilter,
TaskFilter,
TenantApiKeyFilter,
TenantFilter,
@@ -126,7 +126,6 @@ from api.filters import (
UserFilter,
)
from api.models import (
ComplianceOverviewSummary,
ComplianceRequirementOverview,
Finding,
Integration,
@@ -204,7 +203,6 @@ from api.v1.serializers import (
OverviewFindingSerializer,
OverviewProviderCountSerializer,
OverviewProviderSerializer,
OverviewRegionSerializer,
OverviewServiceSerializer,
OverviewSeveritySerializer,
ProcessorCreateSerializer,
@@ -1662,44 +1660,6 @@ class ProviderViewSet(DisablePaginationMixin, BaseRLSViewSet):
),
},
),
ens=extend_schema(
tags=["Scan"],
summary="Retrieve ENS RD2022 compliance report",
description="Download ENS RD2022 compliance report (e.g., 'ens_rd2022_aws') as a PDF file.",
request=None,
responses={
200: OpenApiResponse(
description="PDF file containing the ENS compliance report"
),
202: OpenApiResponse(description="The task is in progress"),
401: OpenApiResponse(
description="API key missing or user not Authenticated"
),
403: OpenApiResponse(description="There is a problem with credentials"),
404: OpenApiResponse(
description="The scan has no ENS reports, or the ENS report generation task has not started yet"
),
},
),
nis2=extend_schema(
tags=["Scan"],
summary="Retrieve NIS2 compliance report",
description="Download NIS2 compliance report (Directive (EU) 2022/2555) as a PDF file.",
request=None,
responses={
200: OpenApiResponse(
description="PDF file containing the NIS2 compliance report"
),
202: OpenApiResponse(description="The task is in progress"),
401: OpenApiResponse(
description="API key missing or user not Authenticated"
),
403: OpenApiResponse(description="There is a problem with credentials"),
404: OpenApiResponse(
description="The scan has no NIS2 reports, or the NIS2 report generation task has not started yet"
),
},
),
)
@method_decorator(CACHE_DECORATOR, name="list")
@method_decorator(CACHE_DECORATOR, name="retrieve")
@@ -1759,12 +1719,6 @@ class ScanViewSet(BaseRLSViewSet):
elif self.action == "threatscore":
if hasattr(self, "response_serializer_class"):
return self.response_serializer_class
elif self.action == "ens":
if hasattr(self, "response_serializer_class"):
return self.response_serializer_class
elif self.action == "nis2":
if hasattr(self, "response_serializer_class"):
return self.response_serializer_class
return super().get_serializer_class()
def partial_update(self, request, *args, **kwargs):
@@ -2018,7 +1972,6 @@ class ScanViewSet(BaseRLSViewSet):
if running_resp:
return running_resp
# TODO: add detailed response if the compliance framework is not supported for the provider
if not scan.output_location:
return Response(
{
@@ -2047,85 +2000,6 @@ class ScanViewSet(BaseRLSViewSet):
content, filename = loader
return self._serve_file(content, filename, "application/pdf")
@action(
detail=True,
methods=["get"],
url_name="ens",
)
def ens(self, request, pk=None):
scan = self.get_object()
running_resp = self._get_task_status(scan)
if running_resp:
return running_resp
# TODO: add detailed response if the compliance framework is not supported for the provider
if not scan.output_location:
return Response(
{
"detail": "The scan has no reports, or the ENS report generation task has not started yet."
},
status=status.HTTP_404_NOT_FOUND,
)
if scan.output_location.startswith("s3://"):
bucket = env.str("DJANGO_OUTPUT_S3_AWS_OUTPUT_BUCKET", "")
key_prefix = scan.output_location.removeprefix(f"s3://{bucket}/")
prefix = os.path.join(
os.path.dirname(key_prefix),
"ens",
"*_ens_report.pdf",
)
loader = self._load_file(prefix, s3=True, bucket=bucket, list_objects=True)
else:
base = os.path.dirname(scan.output_location)
pattern = os.path.join(base, "ens", "*_ens_report.pdf")
loader = self._load_file(pattern, s3=False)
if isinstance(loader, Response):
return loader
content, filename = loader
return self._serve_file(content, filename, "application/pdf")
@action(
detail=True,
methods=["get"],
url_name="nis2",
)
def nis2(self, request, pk=None):
scan = self.get_object()
running_resp = self._get_task_status(scan)
if running_resp:
return running_resp
if not scan.output_location:
return Response(
{
"detail": "The scan has no reports, or the NIS2 report generation task has not started yet."
},
status=status.HTTP_404_NOT_FOUND,
)
if scan.output_location.startswith("s3://"):
bucket = env.str("DJANGO_OUTPUT_S3_AWS_OUTPUT_BUCKET", "")
key_prefix = scan.output_location.removeprefix(f"s3://{bucket}/")
prefix = os.path.join(
os.path.dirname(key_prefix),
"nis2",
"*_nis2_report.pdf",
)
loader = self._load_file(prefix, s3=True, bucket=bucket, list_objects=True)
else:
base = os.path.dirname(scan.output_location)
pattern = os.path.join(base, "nis2", "*_nis2_report.pdf")
loader = self._load_file(pattern, s3=False)
if isinstance(loader, Response):
return loader
content, filename = loader
return self._serve_file(content, filename, "application/pdf")
def create(self, request, *args, **kwargs):
input_serializer = self.get_serializer(data=request.data)
input_serializer.is_valid(raise_exception=True)
@@ -2138,7 +2012,7 @@ class ScanViewSet(BaseRLSViewSet):
"scan_id": str(scan.id),
"provider_id": str(scan.provider_id),
# Disabled for now
# checks_to_execute=scan.scanner_args.get("checks_to_execute")
# checks_to_execute=scan.scanner_args.get("checks_to_execute"),
},
)
@@ -3359,50 +3233,15 @@ class RoleProviderGroupRelationshipView(RelationshipView, BaseRLSViewSet):
@extend_schema_view(
list=extend_schema(
tags=["Compliance Overview"],
summary="List compliance overviews",
description=(
"Retrieve an overview of all compliance frameworks. "
"If scan_id is provided, returns compliance data for that specific scan. "
"If scan_id is omitted, returns compliance data aggregated from the latest completed scan of each provider."
),
summary="List compliance overviews for a scan",
description="Retrieve an overview of all the compliance in a given scan.",
parameters=[
OpenApiParameter(
name="filter[scan_id]",
required=False,
required=True,
type=OpenApiTypes.UUID,
location=OpenApiParameter.QUERY,
description=(
"Optional scan ID. If provided, returns compliance for that scan. "
"If omitted, returns compliance for the latest completed scan per provider."
),
),
OpenApiParameter(
name="filter[provider_id]",
required=False,
type=OpenApiTypes.UUID,
location=OpenApiParameter.QUERY,
description="Filter by specific provider ID.",
),
OpenApiParameter(
name="filter[provider_id__in]",
required=False,
type={"type": "array", "items": {"type": "string", "format": "uuid"}},
location=OpenApiParameter.QUERY,
description="Filter by multiple provider IDs (comma-separated).",
),
OpenApiParameter(
name="filter[provider_type]",
required=False,
type=OpenApiTypes.STR,
location=OpenApiParameter.QUERY,
description="Filter by provider type (e.g., aws, azure, gcp).",
),
OpenApiParameter(
name="filter[provider_type__in]",
required=False,
type={"type": "array", "items": {"type": "string"}},
location=OpenApiParameter.QUERY,
description="Filter by multiple provider types (comma-separated).",
description="Related scan ID.",
),
],
responses={
@@ -3559,45 +3398,33 @@ class ComplianceOverviewViewSet(BaseRLSViewSet, TaskManagementMixin):
def retrieve(self, request, *args, **kwargs):
raise MethodNotAllowed(method="GET")
def _compliance_summaries_queryset(self, scan_id):
"""Return pre-aggregated summaries constrained by RBAC visibility."""
role = get_role(self.request.user)
unlimited_visibility = getattr(
role, Permissions.UNLIMITED_VISIBILITY.value, False
)
summaries = ComplianceOverviewSummary.objects.filter(
tenant_id=self.request.tenant_id,
scan_id=scan_id,
)
def list(self, request, *args, **kwargs):
scan_id = request.query_params.get("filter[scan_id]")
if not scan_id:
raise ValidationError(
[
{
"detail": "This query parameter is required.",
"status": 400,
"source": {"pointer": "filter[scan_id]"},
"code": "required",
}
]
)
try:
if task := self.get_task_response_if_running(
task_name="scan-compliance-overviews",
task_kwargs={"tenant_id": self.request.tenant_id, "scan_id": scan_id},
raise_on_not_found=False,
):
return task
except TaskFailedException:
return Response(
{"detail": "Task failed to generate compliance overview data."},
status=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
queryset = self.filter_queryset(self.filter_queryset(self.get_queryset()))
if not unlimited_visibility:
providers = Provider.all_objects.filter(
provider_groups__in=role.provider_groups.all()
).distinct()
summaries = summaries.filter(scan__provider__in=providers)
return summaries
def _get_compliance_template(self, *, provider=None, scan_id=None):
"""Return the compliance template for the given provider or scan."""
if provider is None and scan_id is not None:
scan = Scan.all_objects.select_related("provider").get(pk=scan_id)
provider = scan.provider
if not provider:
return {}
return PROWLER_COMPLIANCE_OVERVIEW_TEMPLATE.get(provider.provider, {})
def _aggregate_compliance_overview(self, queryset, template_metadata=None):
"""
Aggregate requirement rows into compliance overview dictionaries.
Args:
queryset: ComplianceRequirementOverview queryset already filtered.
template_metadata: Optional dict mapping compliance_id -> metadata.
"""
template_metadata = template_metadata or {}
requirement_status_subquery = queryset.values(
"compliance_id", "requirement_id"
).annotate(
@@ -3607,15 +3434,13 @@ class ComplianceOverviewViewSet(BaseRLSViewSet, TaskManagementMixin):
)
compliance_data = {}
fallback_metadata = {
item["compliance_id"]: {
framework_info = {}
for item in queryset.values("compliance_id", "framework", "version").distinct():
framework_info[item["compliance_id"]] = {
"framework": item["framework"],
"version": item["version"],
}
for item in queryset.values(
"compliance_id", "framework", "version"
).distinct()
}
for item in requirement_status_subquery:
compliance_id = item["compliance_id"]
@@ -3627,36 +3452,32 @@ class ComplianceOverviewViewSet(BaseRLSViewSet, TaskManagementMixin):
else:
req_status = "MANUAL"
compliance_status = compliance_data.setdefault(
compliance_id,
{
if compliance_id not in compliance_data:
compliance_data[compliance_id] = {
"total_requirements": 0,
"requirements_passed": 0,
"requirements_failed": 0,
"requirements_manual": 0,
},
)
}
compliance_status["total_requirements"] += 1
compliance_data[compliance_id]["total_requirements"] += 1
if req_status == "PASS":
compliance_status["requirements_passed"] += 1
compliance_data[compliance_id]["requirements_passed"] += 1
elif req_status == "FAIL":
compliance_status["requirements_failed"] += 1
compliance_data[compliance_id]["requirements_failed"] += 1
else:
compliance_status["requirements_manual"] += 1
compliance_data[compliance_id]["requirements_manual"] += 1
response_data = []
for compliance_id, data in compliance_data.items():
template = template_metadata.get(compliance_id, {})
fallback = fallback_metadata.get(compliance_id, {})
framework = framework_info.get(compliance_id, {})
response_data.append(
{
"id": compliance_id,
"compliance_id": compliance_id,
"framework": template.get("framework")
or fallback.get("framework", ""),
"version": template.get("version") or fallback.get("version", ""),
"framework": framework.get("framework", ""),
"version": framework.get("version", ""),
"requirements_passed": data["requirements_passed"],
"requirements_failed": data["requirements_failed"],
"requirements_manual": data["requirements_manual"],
@@ -3665,151 +3486,7 @@ class ComplianceOverviewViewSet(BaseRLSViewSet, TaskManagementMixin):
)
serializer = self.get_serializer(response_data, many=True)
return serializer.data
def _task_response_if_running(self, scan_id):
"""Check for an in-progress task only when no compliance data exists."""
try:
return self.get_task_response_if_running(
task_name="scan-compliance-overviews",
task_kwargs={"tenant_id": self.request.tenant_id, "scan_id": scan_id},
raise_on_not_found=False,
)
except TaskFailedException:
return Response(
{"detail": "Task failed to generate compliance overview data."},
status=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
def _list_with_region_filter(self, scan_id, region_filter):
"""
Fall back to detailed ComplianceRequirementOverview query when region filter is applied.
This uses the original aggregation logic across filtered regions.
"""
regions = region_filter.split(",") if "," in region_filter else [region_filter]
queryset = self.filter_queryset(self.get_queryset()).filter(
scan_id=scan_id,
region__in=regions,
)
data = self._aggregate_compliance_overview(queryset)
if data:
return Response(data)
task_response = self._task_response_if_running(scan_id)
if task_response:
return task_response
return Response(data)
def _list_without_region_aggregation(self, scan_id):
"""
Fall back aggregation when compliance summaries don't exist yet.
Aggregates ComplianceRequirementOverview data across ALL regions.
"""
queryset = self.filter_queryset(self.get_queryset()).filter(scan_id=scan_id)
compliance_template = self._get_compliance_template(scan_id=scan_id)
data = self._aggregate_compliance_overview(
queryset, template_metadata=compliance_template
)
if data:
return Response(data)
task_response = self._task_response_if_running(scan_id)
if task_response:
return task_response
return Response(data)
def list(self, request, *args, **kwargs):
scan_id = request.query_params.get("filter[scan_id]")
tenant_id = self.request.tenant_id
if scan_id:
# Specific scan requested - use optimized summaries with region support
region_filter = request.query_params.get(
"filter[region]"
) or request.query_params.get("filter[region__in]")
if region_filter:
# Fall back to detailed query with region filtering
return self._list_with_region_filter(scan_id, region_filter)
summaries = list(self._compliance_summaries_queryset(scan_id))
if not summaries:
# Trigger async backfill for next time
backfill_compliance_summaries_task.delay(
tenant_id=self.request.tenant_id, scan_id=scan_id
)
# Use fallback aggregation for this request
return self._list_without_region_aggregation(scan_id)
# Get compliance template for provider to enrich with framework/version
compliance_template = self._get_compliance_template(scan_id=scan_id)
# Convert to response format with framework/version enrichment
response_data = []
for summary in summaries:
compliance_metadata = compliance_template.get(summary.compliance_id, {})
response_data.append(
{
"id": summary.compliance_id,
"compliance_id": summary.compliance_id,
"framework": compliance_metadata.get("framework", ""),
"version": compliance_metadata.get("version", ""),
"requirements_passed": summary.requirements_passed,
"requirements_failed": summary.requirements_failed,
"requirements_manual": summary.requirements_manual,
"total_requirements": summary.total_requirements,
}
)
serializer = self.get_serializer(response_data, many=True)
return Response(serializer.data)
else:
# No scan_id provided - use latest scans per provider
# First, check if provider filters are present
provider_id = request.query_params.get("filter[provider_id]")
provider_id__in = request.query_params.get("filter[provider_id__in]")
provider_type = request.query_params.get("filter[provider_type]")
provider_type__in = request.query_params.get("filter[provider_type__in]")
scan_filters = {"tenant_id": tenant_id, "state": StateChoices.COMPLETED}
# Apply provider ID filters
if provider_id:
scan_filters["provider_id"] = provider_id
elif provider_id__in:
# Convert comma-separated string to list
provider_ids = [pid.strip() for pid in provider_id__in.split(",")]
scan_filters["provider_id__in"] = provider_ids
# Apply provider type filters
if provider_type:
scan_filters["provider__provider"] = provider_type
elif provider_type__in:
# Convert comma-separated string to list
provider_types = [pt.strip() for pt in provider_type__in.split(",")]
scan_filters["provider__provider__in"] = provider_types
latest_scan_ids = (
Scan.all_objects.filter(**scan_filters)
.order_by("provider_id", "-inserted_at")
.distinct("provider_id")
.values_list("id", flat=True)
)
base_queryset = self.get_queryset()
queryset = self.filter_queryset(
base_queryset.filter(scan_id__in=latest_scan_ids)
)
# Aggregate compliance data across latest scans
compliance_template = self._get_compliance_template()
data = self._aggregate_compliance_overview(
queryset, template_metadata=compliance_template
)
return Response(data)
return Response(serializer.data)
@action(detail=False, methods=["get"], url_name="metadata")
def metadata(self, request):
@@ -3825,6 +3502,18 @@ class ComplianceOverviewViewSet(BaseRLSViewSet, TaskManagementMixin):
}
]
)
try:
if task := self.get_task_response_if_running(
task_name="scan-compliance-overviews",
task_kwargs={"tenant_id": self.request.tenant_id, "scan_id": scan_id},
raise_on_not_found=False,
):
return task
except TaskFailedException:
return Response(
{"detail": "Task failed to generate compliance overview data."},
status=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
regions = list(
self.get_queryset()
.filter(scan_id=scan_id)
@@ -3834,15 +3523,6 @@ class ComplianceOverviewViewSet(BaseRLSViewSet, TaskManagementMixin):
)
result = {"regions": regions}
if regions:
serializer = self.get_serializer(data=result)
serializer.is_valid(raise_exception=True)
return Response(serializer.data, status=status.HTTP_200_OK)
task_response = self._task_response_if_running(scan_id)
if task_response:
return task_response
serializer = self.get_serializer(data=result)
serializer.is_valid(raise_exception=True)
return Response(serializer.data, status=status.HTTP_200_OK)
@@ -3875,6 +3555,18 @@ class ComplianceOverviewViewSet(BaseRLSViewSet, TaskManagementMixin):
}
]
)
try:
if task := self.get_task_response_if_running(
task_name="scan-compliance-overviews",
task_kwargs={"tenant_id": self.request.tenant_id, "scan_id": scan_id},
raise_on_not_found=False,
):
return task
except TaskFailedException:
return Response(
{"detail": "Task failed to generate compliance overview data."},
status=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
filtered_queryset = self.filter_queryset(self.get_queryset())
all_requirements = filtered_queryset.values(
@@ -3934,13 +3626,6 @@ class ComplianceOverviewViewSet(BaseRLSViewSet, TaskManagementMixin):
requirements_summary, many=True
)
if requirements_summary:
return Response(serializer.data, status=status.HTTP_200_OK)
task_response = self._task_response_if_running(scan_id)
if task_response:
return task_response
return Response(serializer.data, status=status.HTTP_200_OK)
@action(detail=False, methods=["get"], url_name="attributes")
@@ -3959,6 +3644,15 @@ class ComplianceOverviewViewSet(BaseRLSViewSet, TaskManagementMixin):
)
provider_type = None
try:
sample_requirement = (
self.get_queryset().filter(compliance_id=compliance_id).first()
)
if sample_requirement:
provider_type = sample_requirement.scan.provider.provider
except Exception:
pass
# If we couldn't determine from database, try each provider type
if not provider_type:
@@ -4061,16 +3755,8 @@ class ComplianceOverviewViewSet(BaseRLSViewSet, TaskManagementMixin):
summary="Get findings data by service",
description=(
"Retrieve an aggregated summary of findings grouped by service. The response includes the total count "
"of findings for each service, as long as there are at least one finding for that service."
),
filters=True,
),
regions=extend_schema(
summary="Get findings data by region",
description=(
"Retrieve an aggregated summary of findings grouped by region. The response includes the total, passed, "
"failed, and muted findings for each region based on the latest completed scans per provider. "
"Standard overview filters (inserted_at, provider filters, region filters, etc.) are supported."
"of findings for each service, as long as there are at least one finding for that service. At least "
"one of the `inserted_at` filters must be provided."
),
filters=True,
),
@@ -4104,8 +3790,6 @@ class OverviewViewSet(BaseRLSViewSet):
return OverviewSeveritySerializer
elif self.action == "services":
return OverviewServiceSerializer
elif self.action == "regions":
return OverviewRegionSerializer
elif self.action == "threatscore":
return ThreatScoreSnapshotSerializer
return super().get_serializer_class()
@@ -4113,10 +3797,12 @@ class OverviewViewSet(BaseRLSViewSet):
def get_filterset_class(self):
if self.action == "providers":
return None
elif self.action in ["findings", "services", "regions"]:
elif self.action == "findings":
return ScanSummaryFilter
elif self.action == "findings_severity":
return ScanSummarySeverityFilter
elif self.action == "services":
return ServiceOverviewFilter
return None
@extend_schema(exclude=True)
@@ -4127,35 +3813,6 @@ class OverviewViewSet(BaseRLSViewSet):
def retrieve(self, request, *args, **kwargs):
raise MethodNotAllowed(method="GET")
def _get_latest_scans_queryset(self):
"""
Get filtered queryset for the latest completed scans per provider.
Returns:
Filtered ScanSummary queryset with latest scan IDs applied.
"""
tenant_id = self.request.tenant_id
queryset = self.get_queryset()
filtered_queryset = self.filter_queryset(queryset)
provider_filter = (
{"provider__in": self.allowed_providers}
if hasattr(self, "allowed_providers")
else {}
)
latest_scan_ids = (
Scan.all_objects.filter(
tenant_id=tenant_id, state=StateChoices.COMPLETED, **provider_filter
)
.order_by("provider_id", "-inserted_at")
.distinct("provider_id")
.values_list("id", flat=True)
)
return filtered_queryset.filter(
tenant_id=tenant_id, scan_id__in=latest_scan_ids
)
@action(detail=False, methods=["get"], url_name="providers")
def providers(self, request):
tenant_id = self.request.tenant_id
@@ -4248,7 +3905,26 @@ class OverviewViewSet(BaseRLSViewSet):
@action(detail=False, methods=["get"], url_name="findings")
def findings(self, request):
filtered_queryset = self._get_latest_scans_queryset()
tenant_id = self.request.tenant_id
queryset = self.get_queryset()
filtered_queryset = self.filter_queryset(queryset)
provider_filter = (
{"provider__in": self.allowed_providers}
if hasattr(self, "allowed_providers")
else {}
)
latest_scan_ids = (
Scan.all_objects.filter(
tenant_id=tenant_id, state=StateChoices.COMPLETED, **provider_filter
)
.order_by("provider_id", "-inserted_at")
.distinct("provider_id")
.values_list("id", flat=True)
)
filtered_queryset = filtered_queryset.filter(
tenant_id=tenant_id, scan_id__in=latest_scan_ids
)
aggregated_totals = filtered_queryset.aggregate(
_pass=Sum("_pass") or 0,
@@ -4275,14 +3951,37 @@ class OverviewViewSet(BaseRLSViewSet):
@action(detail=False, methods=["get"], url_name="findings_severity")
def findings_severity(self, request):
filtered_queryset = self._get_latest_scans_queryset()
tenant_id = self.request.tenant_id
# Load only required fields
queryset = self.get_queryset().only(
"tenant_id", "scan_id", "severity", "fail", "_pass", "total"
)
filtered_queryset = self.filter_queryset(queryset)
provider_filter = (
{"provider__in": self.allowed_providers}
if hasattr(self, "allowed_providers")
else {}
)
latest_scan_ids = (
Scan.all_objects.filter(
tenant_id=tenant_id, state=StateChoices.COMPLETED, **provider_filter
)
.order_by("provider_id", "-inserted_at")
.distinct("provider_id")
.values_list("id", flat=True)
)
filtered_queryset = filtered_queryset.filter(
tenant_id=tenant_id, scan_id__in=latest_scan_ids
)
# The filter will have added a status_count annotation if any status filter was used
if "status_count" in filtered_queryset.query.annotations:
sum_expression = Sum("status_count")
else:
# Exclude muted findings by default
sum_expression = Sum(F("_pass") + F("fail"))
sum_expression = Sum("total")
severity_counts = (
filtered_queryset.values("severity")
@@ -4300,7 +3999,26 @@ class OverviewViewSet(BaseRLSViewSet):
@action(detail=False, methods=["get"], url_name="services")
def services(self, request):
filtered_queryset = self._get_latest_scans_queryset()
tenant_id = self.request.tenant_id
queryset = self.get_queryset()
filtered_queryset = self.filter_queryset(queryset)
provider_filter = (
{"provider__in": self.allowed_providers}
if hasattr(self, "allowed_providers")
else {}
)
latest_scan_ids = (
Scan.all_objects.filter(
tenant_id=tenant_id, state=StateChoices.COMPLETED, **provider_filter
)
.order_by("provider_id", "-inserted_at")
.distinct("provider_id")
.values_list("id", flat=True)
)
filtered_queryset = filtered_queryset.filter(
tenant_id=tenant_id, scan_id__in=latest_scan_ids
)
services_data = (
filtered_queryset.values("service")
@@ -4315,31 +4033,13 @@ class OverviewViewSet(BaseRLSViewSet):
return Response(serializer.data, status=status.HTTP_200_OK)
@action(detail=False, methods=["get"], url_name="regions")
def regions(self, request):
filtered_queryset = self._get_latest_scans_queryset()
regions_data = (
filtered_queryset.annotate(provider_type=F("scan__provider__provider"))
.values("provider_type", "region")
.annotate(_pass=Sum("_pass"))
.annotate(fail=Sum("fail"))
.annotate(muted=Sum("muted"))
.annotate(total=Sum("total"))
.order_by("provider_type", "region")
)
serializer = self.get_serializer(regions_data, many=True)
return Response(serializer.data, status=status.HTTP_200_OK)
@extend_schema(
summary="Get ThreatScore snapshots",
description=(
"Retrieve ThreatScore metrics. By default, returns the latest snapshot for each provider. "
"Use snapshot_id to retrieve a specific historical snapshot."
),
tags=["Overview"],
tags=["Overviews"],
parameters=[
OpenApiParameter(
name="snapshot_id",
+4
View File
@@ -125,6 +125,10 @@ SPECTACULAR_SETTINGS = {
"drf_spectacular_jsonapi.hooks.fix_nested_path_parameters",
],
"POSTPROCESSING_HOOKS": [
"api.schema_hooks.fix_empty_id_fields",
"api.schema_hooks.fix_pattern_properties",
"api.schema_hooks.fix_type_formats",
"api.schema_hooks.add_api_servers",
"api.schema_hooks.attach_task_202_examples",
],
"TITLE": "API Reference - Prowler",
+9 -9
View File
@@ -1108,8 +1108,8 @@ def scan_summaries_fixture(tenants_fixture, providers_fixture):
region="region1",
_pass=1,
fail=0,
muted=2,
total=3,
muted=0,
total=1,
new=1,
changed=0,
unchanged=0,
@@ -1117,7 +1117,7 @@ def scan_summaries_fixture(tenants_fixture, providers_fixture):
fail_changed=0,
pass_new=1,
pass_changed=0,
muted_new=2,
muted_new=0,
muted_changed=0,
scan=scan,
)
@@ -1130,8 +1130,8 @@ def scan_summaries_fixture(tenants_fixture, providers_fixture):
region="region2",
_pass=0,
fail=1,
muted=3,
total=4,
muted=1,
total=2,
new=2,
changed=0,
unchanged=0,
@@ -1139,7 +1139,7 @@ def scan_summaries_fixture(tenants_fixture, providers_fixture):
fail_changed=0,
pass_new=0,
pass_changed=0,
muted_new=3,
muted_new=1,
muted_changed=0,
scan=scan,
)
@@ -1152,8 +1152,8 @@ def scan_summaries_fixture(tenants_fixture, providers_fixture):
region="region1",
_pass=1,
fail=0,
muted=1,
total=2,
muted=0,
total=1,
new=1,
changed=0,
unchanged=0,
@@ -1161,7 +1161,7 @@ def scan_summaries_fixture(tenants_fixture, providers_fixture):
fail_changed=0,
pass_new=1,
pass_changed=0,
muted_new=1,
muted_new=0,
muted_changed=0,
scan=scan,
)
Binary file not shown.

Before

Width:  |  Height:  |  Size: 95 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 94 KiB

+1 -117
View File
@@ -1,10 +1,5 @@
from collections import defaultdict
from api.db_router import READ_REPLICA_ALIAS
from api.db_utils import rls_transaction
from api.models import (
ComplianceOverviewSummary,
ComplianceRequirementOverview,
Resource,
ResourceFindingMapping,
ResourceScanSummary,
@@ -14,7 +9,7 @@ from api.models import (
def backfill_resource_scan_summaries(tenant_id: str, scan_id: str):
with rls_transaction(tenant_id, using=READ_REPLICA_ALIAS):
with rls_transaction(tenant_id):
if ResourceScanSummary.objects.filter(
tenant_id=tenant_id, scan_id=scan_id
).exists():
@@ -64,114 +59,3 @@ def backfill_resource_scan_summaries(tenant_id: str, scan_id: str):
)
return {"status": "backfilled", "inserted": len(summaries)}
def backfill_compliance_summaries(tenant_id: str, scan_id: str):
"""
Backfill ComplianceOverviewSummary records for a completed scan.
This function checks if summary records already exist for the scan.
If not, it aggregates compliance requirement data and creates the summaries.
Args:
tenant_id: Target tenant UUID
scan_id: Scan UUID to backfill
Returns:
dict: Status indicating whether backfill was performed
"""
with rls_transaction(tenant_id):
if ComplianceOverviewSummary.objects.filter(
tenant_id=tenant_id, scan_id=scan_id
).exists():
return {"status": "already backfilled"}
with rls_transaction(tenant_id):
if not Scan.objects.filter(
tenant_id=tenant_id,
id=scan_id,
state__in=(StateChoices.COMPLETED, StateChoices.FAILED),
).exists():
return {"status": "scan is not completed"}
# Fetch all compliance requirement overview rows for this scan
requirement_rows = ComplianceRequirementOverview.objects.filter(
tenant_id=tenant_id, scan_id=scan_id
).values(
"compliance_id",
"requirement_id",
"requirement_status",
)
if not requirement_rows:
return {"status": "no compliance data to backfill"}
# Group by (compliance_id, requirement_id) across regions
requirement_statuses = defaultdict(
lambda: {"fail_count": 0, "pass_count": 0, "total_count": 0}
)
for row in requirement_rows:
compliance_id = row["compliance_id"]
requirement_id = row["requirement_id"]
requirement_status = row["requirement_status"]
# Aggregate requirement status across regions
key = (compliance_id, requirement_id)
requirement_statuses[key]["total_count"] += 1
if requirement_status == "FAIL":
requirement_statuses[key]["fail_count"] += 1
elif requirement_status == "PASS":
requirement_statuses[key]["pass_count"] += 1
# Determine per-requirement status and aggregate to compliance level
compliance_summaries = defaultdict(
lambda: {
"total_requirements": 0,
"requirements_passed": 0,
"requirements_failed": 0,
"requirements_manual": 0,
}
)
for (compliance_id, requirement_id), counts in requirement_statuses.items():
# Apply business rule: any FAIL → requirement fails
if counts["fail_count"] > 0:
req_status = "FAIL"
elif counts["pass_count"] == counts["total_count"]:
req_status = "PASS"
else:
req_status = "MANUAL"
# Aggregate to compliance level
compliance_summaries[compliance_id]["total_requirements"] += 1
if req_status == "PASS":
compliance_summaries[compliance_id]["requirements_passed"] += 1
elif req_status == "FAIL":
compliance_summaries[compliance_id]["requirements_failed"] += 1
else:
compliance_summaries[compliance_id]["requirements_manual"] += 1
# Create summary objects
summary_objects = []
for compliance_id, data in compliance_summaries.items():
summary_objects.append(
ComplianceOverviewSummary(
tenant_id=tenant_id,
scan_id=scan_id,
compliance_id=compliance_id,
requirements_passed=data["requirements_passed"],
requirements_failed=data["requirements_failed"],
requirements_manual=data["requirements_manual"],
total_requirements=data["total_requirements"],
)
)
# Bulk insert summaries
if summary_objects:
ComplianceOverviewSummary.objects.bulk_create(
summary_objects, batch_size=500, ignore_conflicts=True
)
return {"status": "backfilled", "inserted": len(summary_objects)}
+33 -122
View File
@@ -15,7 +15,6 @@ from prowler.config.config import (
html_file_suffix,
json_asff_file_suffix,
json_ocsf_file_suffix,
set_output_timestamp,
)
from prowler.lib.outputs.asff.asff import ASFF
from prowler.lib.outputs.compliance.aws_well_architected.aws_well_architected import (
@@ -59,9 +58,6 @@ from prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_azur
from prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_gcp import (
ProwlerThreatScoreGCP,
)
from prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_kubernetes import (
ProwlerThreatScoreKubernetes,
)
from prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_m365 import (
ProwlerThreatScoreM365,
)
@@ -108,10 +104,6 @@ COMPLIANCE_CLASS_MAP = {
"kubernetes": [
(lambda name: name.startswith("cis_"), KubernetesCIS),
(lambda name: name.startswith("iso27001_"), KubernetesISO27001),
(
lambda name: name == "prowler_threatscore_kubernetes",
ProwlerThreatScoreKubernetes,
),
],
"m365": [
(lambda name: name.startswith("cis_"), M365CIS),
@@ -242,33 +234,36 @@ def _upload_to_s3(
logger.error(f"S3 upload failed: {str(e)}")
def _build_output_path(
output_directory: str,
prowler_provider: str,
tenant_id: str,
scan_id: str,
subdirectory: str = None,
) -> str:
def _generate_output_directory(
output_directory, prowler_provider: object, tenant_id: str, scan_id: str
) -> tuple[str, str, str]:
"""
Build a file system path for the output directory of a prowler scan.
Generate a file system path for the output directory of a prowler scan.
This function constructs the output directory path by combining a base
temporary output directory, the tenant ID, the scan ID, and details about
the prowler provider along with a timestamp. The resulting path is used to
store the output files of a prowler scan.
Note:
This function depends on one external variable:
- `output_file_timestamp`: A timestamp (as a string) used to uniquely identify the output.
Args:
output_directory (str): The base output directory.
prowler_provider (str): An identifier or descriptor for the prowler provider.
Typically, this is a string indicating the provider (e.g., "aws").
prowler_provider (object): An identifier or descriptor for the prowler provider.
Typically, this is a string indicating the provider (e.g., "aws").
tenant_id (str): The unique identifier for the tenant.
scan_id (str): The unique identifier for the scan.
subdirectory (str, optional): Optional subdirectory to include in the path
(e.g., "compliance", "threatscore", "ens").
Returns:
str: The constructed path with directory created.
str: The constructed file system path for the prowler scan output directory.
Example:
>>> _build_output_path("/tmp", "aws", "tenant-1234", "scan-5678")
'/tmp/tenant-1234/scan-5678/prowler-output-aws-20230215123456'
>>> _build_output_path("/tmp", "aws", "tenant-1234", "scan-5678", "threatscore")
'/tmp/tenant-1234/scan-5678/threatscore/prowler-output-aws-20230215123456'
>>> _generate_output_directory("/tmp", "aws", "tenant-1234", "scan-5678")
'/tmp/tenant-1234/aws/scan-5678/prowler-output-2023-02-15T12:34:56',
'/tmp/tenant-1234/aws/scan-5678/compliance/prowler-output-2023-02-15T12:34:56'
'/tmp/tenant-1234/aws/scan-5678/threatscore/prowler-output-2023-02-15T12:34:56'
"""
# Sanitize the prowler provider name to ensure it is a valid directory name
prowler_provider_sanitized = re.sub(r"[^\w\-]", "-", prowler_provider)
@@ -276,107 +271,23 @@ def _build_output_path(
with rls_transaction(tenant_id):
started_at = Scan.objects.get(id=scan_id).started_at
set_output_timestamp(started_at)
timestamp = started_at.strftime("%Y%m%d%H%M%S")
if subdirectory:
path = (
f"{output_directory}/{tenant_id}/{scan_id}/{subdirectory}/prowler-output-"
f"{prowler_provider_sanitized}-{timestamp}"
)
else:
path = (
f"{output_directory}/{tenant_id}/{scan_id}/prowler-output-"
f"{prowler_provider_sanitized}-{timestamp}"
)
# Create directory for the path if it doesn't exist
path = (
f"{output_directory}/{tenant_id}/{scan_id}/prowler-output-"
f"{prowler_provider_sanitized}-{timestamp}"
)
os.makedirs("/".join(path.split("/")[:-1]), exist_ok=True)
return path
def _generate_compliance_output_directory(
output_directory: str,
prowler_provider: str,
tenant_id: str,
scan_id: str,
compliance_framework: str,
) -> str:
"""
Generate a file system path for a compliance framework output directory.
This function constructs the output directory path specifically for a compliance
framework (e.g., "threatscore", "ens") by combining a base temporary output directory,
the tenant ID, the scan ID, the compliance framework name, and details about the
prowler provider along with a timestamp.
Args:
output_directory (str): The base output directory.
prowler_provider (str): An identifier or descriptor for the prowler provider.
Typically, this is a string indicating the provider (e.g., "aws").
tenant_id (str): The unique identifier for the tenant.
scan_id (str): The unique identifier for the scan.
compliance_framework (str): The compliance framework name (e.g., "threatscore", "ens").
Returns:
str: The path for the compliance framework output directory.
Example:
>>> _generate_compliance_output_directory("/tmp", "aws", "tenant-1234", "scan-5678", "threatscore")
'/tmp/tenant-1234/scan-5678/threatscore/prowler-output-aws-20230215123456'
>>> _generate_compliance_output_directory("/tmp", "aws", "tenant-1234", "scan-5678", "ens")
'/tmp/tenant-1234/scan-5678/ens/prowler-output-aws-20230215123456'
>>> _generate_compliance_output_directory("/tmp", "aws", "tenant-1234", "scan-5678", "nis2")
'/tmp/tenant-1234/scan-5678/nis2/prowler-output-aws-20230215123456'
"""
return _build_output_path(
output_directory,
prowler_provider,
tenant_id,
scan_id,
subdirectory=compliance_framework,
compliance_path = (
f"{output_directory}/{tenant_id}/{scan_id}/compliance/prowler-output-"
f"{prowler_provider_sanitized}-{timestamp}"
)
os.makedirs("/".join(compliance_path.split("/")[:-1]), exist_ok=True)
def _generate_output_directory(
output_directory: str,
prowler_provider: str,
tenant_id: str,
scan_id: str,
) -> tuple[str, str]:
"""
Generate file system paths for the standard and compliance output directories of a prowler scan.
This function constructs both the standard output directory path and the compliance
output directory path by combining a base temporary output directory, the tenant ID,
the scan ID, and details about the prowler provider along with a timestamp.
Args:
output_directory (str): The base output directory.
prowler_provider (str): An identifier or descriptor for the prowler provider.
Typically, this is a string indicating the provider (e.g., "aws").
tenant_id (str): The unique identifier for the tenant.
scan_id (str): The unique identifier for the scan.
Returns:
tuple[str, str]: A tuple containing (standard_path, compliance_path).
Example:
>>> _generate_output_directory("/tmp", "aws", "tenant-1234", "scan-5678")
('/tmp/tenant-1234/scan-5678/prowler-output-aws-20230215123456',
'/tmp/tenant-1234/scan-5678/compliance/prowler-output-aws-20230215123456')
"""
standard_path = _build_output_path(
output_directory, prowler_provider, tenant_id, scan_id
)
compliance_path = _build_output_path(
output_directory,
prowler_provider,
tenant_id,
scan_id,
subdirectory="compliance",
threatscore_path = (
f"{output_directory}/{tenant_id}/{scan_id}/threatscore/prowler-output-"
f"{prowler_provider_sanitized}-{timestamp}"
)
os.makedirs("/".join(threatscore_path.split("/")[:-1]), exist_ok=True)
return standard_path, compliance_path
return path, compliance_path, threatscore_path
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+2 -114
View File
@@ -1,14 +1,9 @@
from collections import defaultdict
from celery.utils.log import get_task_logger
from config.django.base import DJANGO_FINDINGS_BATCH_SIZE
from django.db.models import Count, Q
from tasks.utils import batched
from api.db_router import READ_REPLICA_ALIAS
from api.db_utils import rls_transaction
from api.models import Finding, StatusChoices
from prowler.lib.outputs.finding import Finding as FindingOutput
logger = get_task_logger(__name__)
@@ -41,15 +36,10 @@ def _aggregate_requirement_statistics_from_database(
with rls_transaction(tenant_id, using=READ_REPLICA_ALIAS):
aggregated_statistics_queryset = (
Finding.all_objects.filter(
tenant_id=tenant_id, scan_id=scan_id, muted=False
)
Finding.all_objects.filter(tenant_id=tenant_id, scan_id=scan_id)
.values("check_id")
.annotate(
total_findings=Count(
"id",
filter=Q(status__in=[StatusChoices.PASS, StatusChoices.FAIL]),
),
total_findings=Count("id"),
passed_findings=Count("id", filter=Q(status=StatusChoices.PASS)),
)
)
@@ -135,105 +125,3 @@ def _calculate_requirements_data_from_statistics(
)
return attributes_by_requirement_id, requirements_list
def _load_findings_for_requirement_checks(
tenant_id: str,
scan_id: str,
check_ids: list[str],
prowler_provider,
findings_cache: dict[str, list[FindingOutput]] | None = None,
) -> dict[str, list[FindingOutput]]:
"""
Load findings for specific check IDs on-demand with optional caching.
This function loads only the findings needed for a specific set of checks,
minimizing memory usage by avoiding loading all findings at once. This is used
when generating detailed findings tables for specific requirements in the PDF.
Supports optional caching to avoid duplicate queries when generating multiple
reports for the same scan.
Args:
tenant_id (str): The tenant ID for Row-Level Security context.
scan_id (str): The ID of the scan to retrieve findings for.
check_ids (list[str]): List of check IDs to load findings for.
prowler_provider: The initialized Prowler provider instance.
findings_cache (dict, optional): Cache of already loaded findings.
If provided, checks are first looked up in cache before querying database.
Returns:
dict[str, list[FindingOutput]]: Dictionary mapping check_id to list of FindingOutput objects.
Example:
{
'aws_iam_user_mfa_enabled': [FindingOutput(...), FindingOutput(...)],
'aws_s3_bucket_public_access': [FindingOutput(...)]
}
"""
findings_by_check_id = defaultdict(list)
if not check_ids:
return dict(findings_by_check_id)
# Initialize cache if not provided
if findings_cache is None:
findings_cache = {}
# Separate cached and non-cached check_ids
check_ids_to_load = []
cache_hits = 0
cache_misses = 0
for check_id in check_ids:
if check_id in findings_cache:
# Reuse from cache
findings_by_check_id[check_id] = findings_cache[check_id]
cache_hits += 1
else:
# Need to load from database
check_ids_to_load.append(check_id)
cache_misses += 1
if cache_hits > 0:
logger.info(
f"Findings cache: {cache_hits} hits, {cache_misses} misses "
f"({cache_hits / (cache_hits + cache_misses) * 100:.1f}% hit rate)"
)
# If all check_ids were in cache, return early
if not check_ids_to_load:
return dict(findings_by_check_id)
logger.info(f"Loading findings for {len(check_ids_to_load)} checks on-demand")
findings_queryset = (
Finding.all_objects.filter(
tenant_id=tenant_id, scan_id=scan_id, check_id__in=check_ids_to_load
)
.order_by("uid")
.iterator()
)
with rls_transaction(tenant_id, using=READ_REPLICA_ALIAS):
for batch, is_last_batch in batched(
findings_queryset, DJANGO_FINDINGS_BATCH_SIZE
):
for finding_model in batch:
finding_output = FindingOutput.transform_api_finding(
finding_model, prowler_provider
)
findings_by_check_id[finding_output.check_id].append(finding_output)
# Update cache with newly loaded findings
if finding_output.check_id not in findings_cache:
findings_cache[finding_output.check_id] = []
findings_cache[finding_output.check_id].append(finding_output)
total_findings_loaded = sum(
len(findings) for findings in findings_by_check_id.values()
)
logger.info(
f"Loaded {total_findings_loaded} findings for {len(findings_by_check_id)} checks"
)
return dict(findings_by_check_id)
+9 -42
View File
@@ -8,10 +8,7 @@ from celery.utils.log import get_task_logger
from config.celery import RLSTask
from config.django.base import DJANGO_FINDINGS_BATCH_SIZE, DJANGO_TMP_OUTPUT_DIRECTORY
from django_celery_beat.models import PeriodicTask
from tasks.jobs.backfill import (
backfill_compliance_summaries,
backfill_resource_scan_summaries,
)
from tasks.jobs.backfill import backfill_resource_scan_summaries
from tasks.jobs.connection import (
check_integration_connection,
check_lighthouse_connection,
@@ -35,7 +32,7 @@ from tasks.jobs.lighthouse_providers import (
refresh_lighthouse_provider_models,
)
from tasks.jobs.muting import mute_historical_findings
from tasks.jobs.report import generate_compliance_reports_job
from tasks.jobs.report import generate_threatscore_report_job
from tasks.jobs.scan import (
aggregate_findings,
create_compliance_requirements,
@@ -75,8 +72,7 @@ def _perform_scan_complete_tasks(tenant_id: str, scan_id: str, provider_id: str)
scan_id=scan_id, provider_id=provider_id, tenant_id=tenant_id
),
group(
# Use optimized task that generates both reports with shared queries
generate_compliance_reports_task.si(
generate_threatscore_report_task.si(
tenant_id=tenant_id, scan_id=scan_id, provider_id=provider_id
),
check_integrations_task.si(
@@ -320,7 +316,7 @@ def generate_outputs_task(scan_id: str, provider_id: str, tenant_id: str):
frameworks_bulk = Compliance.get_bulk(provider_type)
frameworks_avail = get_compliance_frameworks(provider_type)
out_dir, comp_dir = _generate_output_directory(
out_dir, comp_dir, _ = _generate_output_directory(
DJANGO_TMP_OUTPUT_DIRECTORY, provider_uid, tenant_id, scan_id
)
@@ -498,21 +494,6 @@ def backfill_scan_resource_summaries_task(tenant_id: str, scan_id: str):
return backfill_resource_scan_summaries(tenant_id=tenant_id, scan_id=scan_id)
@shared_task(name="backfill-compliance-summaries", queue="backfill")
def backfill_compliance_summaries_task(tenant_id: str, scan_id: str):
"""
Tries to backfill compliance overview summaries for a completed scan.
This task aggregates compliance requirement data across regions
to create pre-computed summary records for fast compliance overview queries.
Args:
tenant_id (str): The tenant identifier.
scan_id (str): The scan identifier.
"""
return backfill_compliance_summaries(tenant_id=tenant_id, scan_id=scan_id)
@shared_task(base=RLSTask, name="scan-compliance-overviews", queue="compliance")
def create_compliance_requirements_task(tenant_id: str, scan_id: str):
"""
@@ -687,33 +668,19 @@ def jira_integration_task(
@shared_task(
base=RLSTask,
name="scan-compliance-reports",
name="scan-threatscore-report",
queue="scan-reports",
)
def generate_compliance_reports_task(tenant_id: str, scan_id: str, provider_id: str):
def generate_threatscore_report_task(tenant_id: str, scan_id: str, provider_id: str):
"""
Optimized task to generate ThreatScore, ENS, and NIS2 reports with shared queries.
This task is more efficient than running separate report tasks because it reuses database queries:
- Provider object fetched once (instead of three times)
- Requirement statistics aggregated once (instead of three times)
- Can reduce database load by up to 50-70%
Task to generate a threatscore report for a given scan.
Args:
tenant_id (str): The tenant identifier.
scan_id (str): The scan identifier.
provider_id (str): The provider identifier.
Returns:
dict: Results for all reports containing upload status and paths.
"""
return generate_compliance_reports_job(
tenant_id=tenant_id,
scan_id=scan_id,
provider_id=provider_id,
generate_threatscore=True,
generate_ens=True,
generate_nis2=True,
return generate_threatscore_report_job(
tenant_id=tenant_id, scan_id=scan_id, provider_id=provider_id
)
+32 -127
View File
@@ -1,53 +1,43 @@
from uuid import uuid4
import pytest
from tasks.jobs.backfill import (
backfill_compliance_summaries,
backfill_resource_scan_summaries,
)
from tasks.jobs.backfill import backfill_resource_scan_summaries
from api.models import (
ComplianceOverviewSummary,
ResourceScanSummary,
Scan,
StateChoices,
)
@pytest.fixture(scope="function")
def resource_scan_summary_data(scans_fixture):
scan = scans_fixture[0]
return ResourceScanSummary.objects.create(
tenant_id=scan.tenant_id,
scan_id=scan.id,
resource_id=str(uuid4()),
service="aws",
region="us-east-1",
resource_type="instance",
)
@pytest.fixture(scope="function")
def get_not_completed_scans(providers_fixture):
provider_id = providers_fixture[0].id
tenant_id = providers_fixture[0].tenant_id
scan_1 = Scan.objects.create(
tenant_id=tenant_id,
trigger=Scan.TriggerChoices.MANUAL,
state=StateChoices.EXECUTING,
provider_id=provider_id,
)
scan_2 = Scan.objects.create(
tenant_id=tenant_id,
trigger=Scan.TriggerChoices.MANUAL,
state=StateChoices.AVAILABLE,
provider_id=provider_id,
)
return scan_1, scan_2
from api.models import ResourceScanSummary, Scan, StateChoices
@pytest.mark.django_db
class TestBackfillResourceScanSummaries:
@pytest.fixture(scope="function")
def resource_scan_summary_data(self, scans_fixture):
scan = scans_fixture[0]
return ResourceScanSummary.objects.create(
tenant_id=scan.tenant_id,
scan_id=scan.id,
resource_id=str(uuid4()),
service="aws",
region="us-east-1",
resource_type="instance",
)
@pytest.fixture(scope="function")
def get_not_completed_scans(self, providers_fixture):
provider_id = providers_fixture[0].id
tenant_id = providers_fixture[0].tenant_id
scan_1 = Scan.objects.create(
tenant_id=tenant_id,
trigger=Scan.TriggerChoices.MANUAL,
state=StateChoices.EXECUTING,
provider_id=provider_id,
)
scan_2 = Scan.objects.create(
tenant_id=tenant_id,
trigger=Scan.TriggerChoices.MANUAL,
state=StateChoices.AVAILABLE,
provider_id=provider_id,
)
return scan_1, scan_2
def test_already_backfilled(self, resource_scan_summary_data):
tenant_id = resource_scan_summary_data.tenant_id
scan_id = resource_scan_summary_data.scan_id
@@ -87,88 +77,3 @@ class TestBackfillResourceScanSummaries:
assert summary.service == resource.service
assert summary.region == resource.region
assert summary.resource_type == resource.type
def test_no_resources_to_backfill(self, scans_fixture):
scan = scans_fixture[1] # Failed scan with no findings/resources
tenant_id = str(scan.tenant_id)
scan_id = str(scan.id)
result = backfill_resource_scan_summaries(tenant_id, scan_id)
assert result == {"status": "no resources to backfill"}
@pytest.mark.django_db
class TestBackfillComplianceSummaries:
def test_already_backfilled(self, scans_fixture):
scan = scans_fixture[0]
tenant_id = str(scan.tenant_id)
ComplianceOverviewSummary.objects.create(
tenant_id=scan.tenant_id,
scan=scan,
compliance_id="aws_account_security_onboarding_aws",
requirements_passed=1,
requirements_failed=0,
requirements_manual=0,
total_requirements=1,
)
result = backfill_compliance_summaries(tenant_id, str(scan.id))
assert result == {"status": "already backfilled"}
def test_not_completed_scan(self, get_not_completed_scans):
for scan in get_not_completed_scans:
result = backfill_compliance_summaries(str(scan.tenant_id), str(scan.id))
assert result == {"status": "scan is not completed"}
def test_no_compliance_data(self, scans_fixture):
scan = scans_fixture[1] # Failed scan with no compliance rows
result = backfill_compliance_summaries(str(scan.tenant_id), str(scan.id))
assert result == {"status": "no compliance data to backfill"}
def test_backfill_creates_compliance_summaries(
self, tenants_fixture, scans_fixture, compliance_requirements_overviews_fixture
):
tenant = tenants_fixture[0]
scan = scans_fixture[0]
result = backfill_compliance_summaries(str(tenant.id), str(scan.id))
expected = {
"aws_account_security_onboarding_aws": {
"requirements_passed": 1,
"requirements_failed": 1,
"requirements_manual": 1,
"total_requirements": 3,
},
"cis_1.4_aws": {
"requirements_passed": 0,
"requirements_failed": 1,
"requirements_manual": 0,
"total_requirements": 1,
},
"mitre_attack_aws": {
"requirements_passed": 0,
"requirements_failed": 1,
"requirements_manual": 0,
"total_requirements": 1,
},
}
assert result == {"status": "backfilled", "inserted": len(expected)}
summaries = ComplianceOverviewSummary.objects.filter(
tenant_id=str(tenant.id), scan_id=str(scan.id)
)
assert summaries.count() == len(expected)
for summary in summaries:
assert summary.compliance_id in expected
expected_counts = expected[summary.compliance_id]
assert summary.requirements_passed == expected_counts["requirements_passed"]
assert summary.requirements_failed == expected_counts["requirements_failed"]
assert summary.requirements_manual == expected_counts["requirements_manual"]
assert summary.total_requirements == expected_counts["total_requirements"]
+8 -39
View File
@@ -9,7 +9,6 @@ import pytest
from botocore.exceptions import ClientError
from tasks.jobs.export import (
_compress_output_files,
_generate_compliance_output_directory,
_generate_output_directory,
_upload_to_s3,
get_s3_client,
@@ -148,11 +147,10 @@ class TestOutputs:
)
mock_logger.assert_called()
@patch("tasks.jobs.export.set_output_timestamp")
@patch("tasks.jobs.export.rls_transaction")
@patch("tasks.jobs.export.Scan")
def test_generate_output_directory_creates_paths(
self, mock_scan, mock_rls_transaction, mock_set_timestamp, tmpdir
self, mock_scan, mock_rls_transaction, tmpdir
):
# Mock the scan object with a started_at timestamp
mock_scan_instance = MagicMock()
@@ -170,40 +168,22 @@ class TestOutputs:
provider = "aws"
expected_timestamp = "20230615103045"
# Test _generate_output_directory (returns standard and compliance paths)
path, compliance = _generate_output_directory(
path, compliance, threatscore = _generate_output_directory(
base_dir, provider, tenant_id, scan_id
)
assert os.path.isdir(os.path.dirname(path))
assert os.path.isdir(os.path.dirname(compliance))
assert os.path.isdir(os.path.dirname(threatscore))
assert path.endswith(f"{provider}-{expected_timestamp}")
assert compliance.endswith(f"{provider}-{expected_timestamp}")
assert "/compliance/" in compliance
# Test _generate_compliance_output_directory with "threatscore"
threatscore = _generate_compliance_output_directory(
base_dir, provider, tenant_id, scan_id, compliance_framework="threatscore"
)
assert os.path.isdir(os.path.dirname(threatscore))
assert threatscore.endswith(f"{provider}-{expected_timestamp}")
assert "/threatscore/" in threatscore
# Test _generate_compliance_output_directory with "ens"
ens = _generate_compliance_output_directory(
base_dir, provider, tenant_id, scan_id, compliance_framework="ens"
)
assert os.path.isdir(os.path.dirname(ens))
assert ens.endswith(f"{provider}-{expected_timestamp}")
assert "/ens/" in ens
@patch("tasks.jobs.export.set_output_timestamp")
@patch("tasks.jobs.export.rls_transaction")
@patch("tasks.jobs.export.Scan")
def test_generate_output_directory_invalid_character(
self, mock_scan, mock_rls_transaction, mock_set_timestamp, tmpdir
self, mock_scan, mock_rls_transaction, tmpdir
):
# Mock the scan object with a started_at timestamp
mock_scan_instance = MagicMock()
@@ -221,25 +201,14 @@ class TestOutputs:
provider = "aws/test@check"
expected_timestamp = "20230615103045"
# Test provider name sanitization with _generate_output_directory
path, compliance = _generate_output_directory(
path, compliance, threatscore = _generate_output_directory(
base_dir, provider, tenant_id, scan_id
)
assert os.path.isdir(os.path.dirname(path))
assert os.path.isdir(os.path.dirname(compliance))
assert os.path.isdir(os.path.dirname(threatscore))
assert path.endswith(f"aws-test-check-{expected_timestamp}")
assert compliance.endswith(f"aws-test-check-{expected_timestamp}")
# Test provider name sanitization with _generate_compliance_output_directory
threatscore = _generate_compliance_output_directory(
base_dir, provider, tenant_id, scan_id, compliance_framework="threatscore"
)
ens = _generate_compliance_output_directory(
base_dir, provider, tenant_id, scan_id, compliance_framework="ens"
)
assert os.path.isdir(os.path.dirname(threatscore))
assert os.path.isdir(os.path.dirname(ens))
assert threatscore.endswith(f"aws-test-check-{expected_timestamp}")
assert ens.endswith(f"aws-test-check-{expected_timestamp}")
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+15 -23
View File
@@ -109,6 +109,7 @@ class TestGenerateOutputs:
return_value=(
"/tmp/test/out-dir",
"/tmp/test/comp-dir",
"/tmp/test/threat-dir",
),
),
patch("tasks.tasks.Scan.all_objects.filter") as mock_scan_update,
@@ -138,7 +139,7 @@ class TestGenerateOutputs:
patch("tasks.tasks.Finding.all_objects.filter") as mock_findings,
patch(
"tasks.tasks._generate_output_directory",
return_value=("/tmp/test/out", "/tmp/test/comp"),
return_value=("/tmp/test/out", "/tmp/test/comp", "/tmp/test/threat"),
),
patch("tasks.tasks.FindingOutput._transform_findings_stats"),
patch("tasks.tasks.FindingOutput.transform_api_finding"),
@@ -208,7 +209,7 @@ class TestGenerateOutputs:
patch("tasks.tasks.Finding.all_objects.filter") as mock_findings,
patch(
"tasks.tasks._generate_output_directory",
return_value=("/tmp/test/out", "/tmp/test/comp"),
return_value=("/tmp/test/out", "/tmp/test/comp", "/tmp/test/threat"),
),
patch(
"tasks.tasks.FindingOutput._transform_findings_stats",
@@ -288,6 +289,7 @@ class TestGenerateOutputs:
return_value=(
"/tmp/test/outdir",
"/tmp/test/compdir",
"/tmp/test/threatdir",
),
),
patch("tasks.tasks._compress_output_files", return_value="outdir.zip"),
@@ -366,6 +368,7 @@ class TestGenerateOutputs:
return_value=(
"/tmp/test/outdir",
"/tmp/test/compdir",
"/tmp/test/threatdir",
),
),
patch("tasks.tasks.FindingOutput._transform_findings_stats"),
@@ -433,7 +436,7 @@ class TestGenerateOutputs:
patch("tasks.tasks.Finding.all_objects.filter") as mock_findings,
patch(
"tasks.tasks._generate_output_directory",
return_value=("/tmp/test/out", "/tmp/test/comp"),
return_value=("/tmp/test/out", "/tmp/test/comp", "/tmp/test/threat"),
),
patch(
"tasks.tasks.FindingOutput._transform_findings_stats",
@@ -491,7 +494,7 @@ class TestGenerateOutputs:
patch("tasks.tasks.Finding.all_objects.filter") as mock_findings,
patch(
"tasks.tasks._generate_output_directory",
return_value=("/tmp/test/out", "/tmp/test/comp"),
return_value=("/tmp/test/out", "/tmp/test/comp", "/tmp/test/threat"),
),
patch("tasks.tasks.FindingOutput._transform_findings_stats"),
patch("tasks.tasks.FindingOutput.transform_api_finding"),
@@ -532,45 +535,34 @@ class TestScanCompleteTasks:
@patch("tasks.tasks.create_compliance_requirements_task.apply_async")
@patch("tasks.tasks.perform_scan_summary_task.si")
@patch("tasks.tasks.generate_outputs_task.si")
@patch("tasks.tasks.generate_compliance_reports_task.si")
@patch("tasks.tasks.generate_threatscore_report_task.si")
@patch("tasks.tasks.check_integrations_task.si")
def test_scan_complete_tasks(
self,
mock_check_integrations_task,
mock_compliance_reports_task,
mock_threatscore_task,
mock_outputs_task,
mock_scan_summary_task,
mock_compliance_requirements_task,
mock_compliance_tasks,
):
"""Test that scan complete tasks are properly orchestrated with optimized reports."""
_perform_scan_complete_tasks("tenant-id", "scan-id", "provider-id")
# Verify compliance requirements task is called
mock_compliance_requirements_task.assert_called_once_with(
mock_compliance_tasks.assert_called_once_with(
kwargs={"tenant_id": "tenant-id", "scan_id": "scan-id"},
)
# Verify scan summary task is called
mock_scan_summary_task.assert_called_once_with(
scan_id="scan-id",
tenant_id="tenant-id",
)
# Verify outputs task is called
mock_outputs_task.assert_called_once_with(
scan_id="scan-id",
provider_id="provider-id",
tenant_id="tenant-id",
)
# Verify optimized compliance reports task is called (replaces individual tasks)
mock_compliance_reports_task.assert_called_once_with(
mock_threatscore_task.assert_called_once_with(
tenant_id="tenant-id",
scan_id="scan-id",
provider_id="provider-id",
)
# Verify integrations task is called
mock_check_integrations_task.assert_called_once_with(
tenant_id="tenant-id",
provider_id="provider-id",
@@ -746,7 +738,7 @@ class TestCheckIntegrationsTask:
mock_initialize_provider.return_value = MagicMock()
mock_compliance_bulk.return_value = {}
mock_get_frameworks.return_value = []
mock_generate_dir.return_value = ("out-dir", "comp-dir")
mock_generate_dir.return_value = ("out-dir", "comp-dir", "threat-dir")
mock_transform_stats.return_value = {"stats": "data"}
# Mock findings
@@ -871,7 +863,7 @@ class TestCheckIntegrationsTask:
mock_initialize_provider.return_value = MagicMock()
mock_compliance_bulk.return_value = {}
mock_get_frameworks.return_value = []
mock_generate_dir.return_value = ("out-dir", "comp-dir")
mock_generate_dir.return_value = ("out-dir", "comp-dir", "threat-dir")
mock_transform_stats.return_value = {"stats": "data"}
# Mock findings
@@ -987,7 +979,7 @@ class TestCheckIntegrationsTask:
mock_initialize_provider.return_value = MagicMock()
mock_compliance_bulk.return_value = {}
mock_get_frameworks.return_value = []
mock_generate_dir.return_value = ("out-dir", "comp-dir")
mock_generate_dir.return_value = ("out-dir", "comp-dir", "threat-dir")
mock_transform_stats.return_value = {"stats": "data"}
# Mock findings
@@ -1,28 +0,0 @@
import warnings
from dashboard.common_methods import get_section_containers_threatscore
warnings.filterwarnings("ignore")
def get_table(data):
aux = data[
[
"REQUIREMENTS_ID",
"REQUIREMENTS_DESCRIPTION",
"REQUIREMENTS_ATTRIBUTES_SECTION",
"REQUIREMENTS_ATTRIBUTES_SUBSECTION",
"CHECKID",
"STATUS",
"REGION",
"ACCOUNTID",
"RESOURCEID",
]
].copy()
return get_section_containers_threatscore(
aux,
"REQUIREMENTS_ATTRIBUTES_SECTION",
"REQUIREMENTS_ATTRIBUTES_SUBSECTION",
"REQUIREMENTS_ID",
)
File diff suppressed because it is too large Load Diff
+6 -8
View File
@@ -52,7 +52,7 @@
{
"group": "Prowler Lighthouse AI",
"pages": [
"getting-started/products/prowler-lighthouse-ai"
"user-guide/tutorials/prowler-app-lighthouse"
]
},
{
@@ -109,13 +109,7 @@
"user-guide/tutorials/prowler-app-jira-integration"
]
},
{
"group": "Lighthouse AI",
"pages": [
"user-guide/tutorials/prowler-app-lighthouse",
"user-guide/tutorials/prowler-app-lighthouse-multi-llm"
]
},
"user-guide/tutorials/prowler-app-lighthouse",
"user-guide/tutorials/prowler-cloud-public-ips",
{
"group": "Tutorials",
@@ -335,6 +329,10 @@
{
"tab": "Public Roadmap",
"href": "https://roadmap.prowler.com/"
},
{
"tab": "API Reference",
"openapi": "api-reference/openapi.yml"
}
],
"global": {
@@ -1,180 +0,0 @@
---
title: 'Overview'
---
import { VersionBadge } from "/snippets/version-badge.mdx"
<VersionBadge version="5.8.0" />
Prowler Lighthouse AI is a Cloud Security Analyst chatbot that helps you understand, prioritize, and remediate security findings in your cloud environments. It's designed to provide security expertise for teams without dedicated resources, acting as your 24/7 virtual cloud security analyst.
<img src="/images/prowler-app/lighthouse-intro.png" alt="Prowler Lighthouse" />
<Card title="Set Up Lighthouse AI" icon="rocket" href="/user-guide/tutorials/prowler-app-lighthouse#set-up">
Learn how to configure Lighthouse AI with your preferred LLM provider
</Card>
## Capabilities
Prowler Lighthouse AI is designed to be your AI security team member, with capabilities including:
### Natural Language Querying
Ask questions in plain English about your security findings. Examples:
- "What are my highest risk findings?"
- "Show me all S3 buckets with public access."
- "What security issues were found in my production accounts?"
<img src="/images/prowler-app/lighthouse-feature1.png" alt="Natural language querying" />
### Detailed Remediation Guidance
Get tailored step-by-step instructions for fixing security issues:
- Clear explanations of the problem and its impact
- Commands or console steps to implement fixes
- Alternative approaches with different solutions
<img src="/images/prowler-app/lighthouse-feature2.png" alt="Detailed Remediation" />
### Enhanced Context and Analysis
Lighthouse AI can provide additional context to help you understand the findings:
- Explain security concepts related to findings in simple terms
- Provide risk assessments based on your environment and context
- Connect related findings to show broader security patterns
<img src="/images/prowler-app/lighthouse-config.png" alt="Business Context" />
<img src="/images/prowler-app/lighthouse-feature3.png" alt="Contextual Responses" />
## Important Notes
Prowler Lighthouse AI is powerful, but there are limitations:
- **Continuous improvement**: Please report any issues, as the feature may make mistakes or encounter errors, despite extensive testing.
- **Access limitations**: Lighthouse AI can only access data the logged-in user can view. If you can't see certain information, Lighthouse AI can't see it either.
- **NextJS session dependence**: If your Prowler application session expires or logs out, Lighthouse AI will error out. Refresh and log back in to continue.
- **Response quality**: The response quality depends on the selected LLM provider and model. Choose models with strong tool-calling capabilities for best results. We recommend `gpt-5` model from OpenAI.
### Getting Help
If you encounter issues with Prowler Lighthouse AI or have suggestions for improvements, please [reach out through our Slack channel](https://goto.prowler.com/slack).
### What Data Is Shared to LLM Providers?
The following API endpoints are accessible to Prowler Lighthouse AI. Data from the following API endpoints could be shared with LLM provider depending on the scope of user's query:
#### Accessible API Endpoints
**User Management:**
- List all users - `/api/v1/users`
- Retrieve the current user's information - `/api/v1/users/me`
**Provider Management:**
- List all providers - `/api/v1/providers`
- Retrieve data from a provider - `/api/v1/providers/{id}`
**Scan Management:**
- List all scans - `/api/v1/scans`
- Retrieve data from a specific scan - `/api/v1/scans/{id}`
**Resource Management:**
- List all resources - `/api/v1/resources`
- Retrieve data for a resource - `/api/v1/resources/{id}`
**Findings Management:**
- List all findings - `/api/v1/findings`
- Retrieve data from a specific finding - `/api/v1/findings/{id}`
- Retrieve metadata values from findings - `/api/v1/findings/metadata`
**Overview Data:**
- Get aggregated findings data - `/api/v1/overviews/findings`
- Get findings data by severity - `/api/v1/overviews/findings_severity`
- Get aggregated provider data - `/api/v1/overviews/providers`
- Get findings data by service - `/api/v1/overviews/services`
**Compliance Management:**
- List compliance overviews (optionally filter by scan) - `/api/v1/compliance-overviews`
- Retrieve data from a specific compliance overview - `/api/v1/compliance-overviews/{id}`
#### Excluded API Endpoints
Not all Prowler API endpoints are integrated with Lighthouse AI. They are intentionally excluded for the following reasons:
- OpenAI/other LLM providers shouldn't have access to sensitive data (like fetching provider secrets and other sensitive config)
- Users queries don't need responses from those API endpoints (ex: tasks, tenant details, downloading zip file, etc.)
**Excluded Endpoints:**
**User Management:**
- List specific users information - `/api/v1/users/{id}`
- List user memberships - `/api/v1/users/{user_pk}/memberships`
- Retrieve membership data from the user - `/api/v1/users/{user_pk}/memberships/{id}`
**Tenant Management:**
- List all tenants - `/api/v1/tenants`
- Retrieve data from a tenant - `/api/v1/tenants/{id}`
- List tenant memberships - `/api/v1/tenants/{tenant_pk}/memberships`
- List all invitations - `/api/v1/tenants/invitations`
- Retrieve data from tenant invitation - `/api/v1/tenants/invitations/{id}`
**Security and Configuration:**
- List all secrets - `/api/v1/providers/secrets`
- Retrieve data from a secret - `/api/v1/providers/secrets/{id}`
- List all provider groups - `/api/v1/provider-groups`
- Retrieve data from a provider group - `/api/v1/provider-groups/{id}`
**Reports and Tasks:**
- Download zip report - `/api/v1/scans/{v1}/report`
- List all tasks - `/api/v1/tasks`
- Retrieve data from a specific task - `/api/v1/tasks/{id}`
**Lighthouse AI Configuration:**
- List LLM providers - `/api/v1/lighthouse/providers`
- Retrieve LLM provider - `/api/v1/lighthouse/providers/{id}`
- List available models - `/api/v1/lighthouse/models`
- Retrieve tenant configuration - `/api/v1/lighthouse/configuration`
<Note>
Agents only have access to hit GET endpoints. They don't have access to other HTTP methods.
</Note>
## FAQs
**1. Which LLM providers are supported?**
Lighthouse AI supports three providers:
- **OpenAI** - GPT models (GPT-5, GPT-4o, etc.)
- **Amazon Bedrock** - Claude, Llama, Titan, and other models via AWS
- **OpenAI Compatible** - Custom endpoints like OpenRouter, Ollama, or any OpenAI-compatible service
For detailed configuration instructions, see [Using Multiple LLM Providers with Lighthouse](/user-guide/tutorials/prowler-app-lighthouse-multi-llm).
**2. Why a multi-agent supervisor model?**
Context windows are limited. While demo data fits inside the context window, querying real-world data often exceeds it. A multi-agent architecture is used so different agents fetch different sizes of data and respond with the minimum required data to the supervisor. This spreads the context window usage across agents.
**3. Is my security data shared with LLM providers?**
Minimal data is shared to generate useful responses. Agents can access security findings and remediation details when needed. Provider secrets are protected by design and cannot be read. The LLM provider credentials configured with Lighthouse AI are only accessible to our NextJS server and are never sent to the LLM providers. Resource metadata (names, tags, account/project IDs, etc) may be shared with the configured LLM provider based on query requirements.
**4. Can the Lighthouse AI change my cloud environment?**
No. The agent doesn't have the tools to make the changes, even if the configured cloud provider API keys contain permissions to modify resources.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 96 KiB

After

Width:  |  Height:  |  Size: 197 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 540 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 136 KiB

After

Width:  |  Height:  |  Size: 204 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 147 KiB

After

Width:  |  Height:  |  Size: 241 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 180 KiB

After

Width:  |  Height:  |  Size: 268 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 165 KiB

After

Width:  |  Height:  |  Size: 404 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 347 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 173 KiB

+2 -2
View File
@@ -24,7 +24,7 @@ Standard results will be shown and additionally the framework information as the
**If Prowler can't find a resource related with a check from a compliance requirement, this requirement won't appear on the output**
</Note>
## List Available Compliance Frameworks
## List Available Compliance Frameworks
To see which compliance frameworks are covered by Prowler, use the `--list-compliance` option:
@@ -34,7 +34,7 @@ prowler <provider> --list-compliance
Or you can visit [Prowler Hub](https://hub.prowler.com/compliance).
## List Requirements of Compliance Frameworks
## List Requirements of Compliance Frameworks
To list requirements for a compliance framework, use the `--list-compliance-requirements` option:
```sh
@@ -94,7 +94,7 @@ The following list includes all the Azure checks with configurable variables tha
### Configurable Checks
## Kubernetes
## Kubernetes
### Configurable Checks
The following list includes all the Kubernetes checks with configurable variables that can be changed in the configuration yaml file:
@@ -2,7 +2,7 @@
title: 'Integrations'
---
## Integration with Slack
## Integration with Slack
Prowler can be integrated with [Slack](https://slack.com/) to send a summary of the execution having configured a Slack APP in your channel with the following command:
+3 -3
View File
@@ -14,7 +14,7 @@ prowler <provider> -V/-v/--version
Prowler provides various execution settings.
### Verbose Execution
### Verbose Execution
To enable verbose mode in Prowler, similar to Version 2, use:
@@ -54,7 +54,7 @@ To run Prowler without color formatting:
prowler <provider> --no-color
```
### Checks in Prowler
### Checks in Prowler
Prowler provides various security checks per cloud provider. Use the following options to list, execute, or exclude specific checks:
@@ -96,7 +96,7 @@ prowler <provider> -e/--excluded-checks ec2 rds
prowler <provider> -C/--checks-file <checks_list>.json
```
## Custom Checks in Prowler
## Custom Checks in Prowler
Prowler supports custom security checks, allowing users to define their own logic.
+2 -2
View File
@@ -57,7 +57,7 @@ Tags have special matching behavior:
Remember that mutelist can be used with regular expressions.
</Note>
## Mutelist Specification
## Mutelist Specification
<Note>
- For Azure provider, the Account ID is the Subscription Name and the Region is the Location.
@@ -72,7 +72,7 @@ The Mutelist file uses the [YAML](https://en.wikipedia.org/wiki/YAML) format wit
### Resources and tags are lists that can have either Regex or Keywords.
### Multiple tags in the list are "ANDed" together (ALL must match).
### Use regex alternation (|) within a single tag for "OR" logic (e.g., "env=dev|env=stg").
### For each check you can use Exceptions to unmute specific Accounts, Regions, Resources and/or Tags.
### For each check you can use Exceptions to unmute specific Accounts, Regions, Resources and/or Tags.
### All conditions (Account, Check, Region, Resource, Tags) are ANDed together.
########################### MUTELIST EXAMPLE ###########################
Mutelist:
@@ -10,7 +10,7 @@ This can help for really large accounts, but please be aware of AWS API rate lim
2. **API Rate Limits**: Most of the rate limits in AWS are applied at the API level. Each API call to an AWS service counts towards the rate limit for that service.
3. **Throttling Responses**: When you exceed the rate limit for a service, AWS responds with a throttling error. In AWS SDKs, these are typically represented as `ThrottlingException` or `RateLimitExceeded` errors.
For information on Prowler's retrier configuration please refer to this [page](https://docs.prowler.com/user-guide/providers/aws/boto3-configuration/).
For information on Prowler's retrier configuration please refer to this [page](https://docs.prowler.cloud/en/latest/tutorials/aws/boto3-configuration/).
<Note>
You might need to increase the `--aws-retries-max-attempts` parameter from the default value of 3. The retrier follows an exponential backoff strategy.
@@ -24,6 +24,6 @@ By default, it extracts resources from all the regions, you could use `-f`/`--fi
![Quick Inventory Example](/images/quick-inventory.jpg)
## Objections
## Objections
The inventorying process is carried out with `resourcegroupstaggingapi` calls, which means that only resources they have or have had tags will appear (except for the IAM and S3 resources which are done with Boto3 API calls).
+5 -5
View File
@@ -22,7 +22,7 @@ prowler <provider> --output-formats json-asff
All compliance-related reports are automatically generated when Prowler is executed. These outputs are stored in the `/output/compliance` directory.
## Custom Output Flags
## Custom Output Flags
By default, Prowler creates a file inside the `output` directory named: `prowler-output-ACCOUNT_NUM-OUTPUT_DATE.format`.
@@ -53,13 +53,13 @@ Both flags can be used simultaneously to provide a custom directory and filename
By default, the timestamp format of the output files is ISO 8601. This can be changed with the flag `--unix-timestamp` generating the timestamp fields in pure unix timestamp format.
## Supported Output Formats
## Supported Output Formats
Prowler natively supports the following reporting output formats:
- CSV
- JSON-OCSF
- JSON-ASFF (AWS only)
- JSON-ASFF
- HTML
Hereunder is the structure for each of the supported report formats by Prowler:
@@ -285,10 +285,10 @@ The JSON-OCSF output format implements the [Detection Finding](https://schema.oc
Each finding is a `json` object within a list.
</Note>
### JSON-ASFF (AWS Only)
### JSON-ASFF
<Note>
Only available when using `--security-hub` or `--output-formats json-asff` with the AWS provider.
Only available when using `--security-hub` or `--output-formats json-asff`
</Note>
The following code is an example output of the [JSON-ASFF](https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-findings-format-syntax.html) format:
@@ -14,7 +14,7 @@ prowler <provider> --scan-unused-services
## Services Ignored
### AWS
### AWS
#### ACM (AWS Certificate Manager)
@@ -22,21 +22,21 @@ Certificates stored in ACM without active usage in AWS resources are excluded. B
- `acm_certificates_expiration_check`
#### Athena
#### Athena
Upon AWS account creation, Athena provisions a default primary workgroup for the user. Prowler verifies if this workgroup is enabled and used by checking for queries within the last 45 days. If Athena is unused, findings related to its checks will not appear.
- `athena_workgroup_encryption`
- `athena_workgroup_enforce_configuration`
#### AWS CloudTrail
#### AWS CloudTrail
AWS CloudTrail should have at least one trail with a data event to record all S3 object-level API operations. Before flagging this issue, Prowler verifies if S3 buckets exist in the account.
- `cloudtrail_s3_dataevents_read_enabled`
- `cloudtrail_s3_dataevents_write_enabled`
#### AWS Elastic Compute Cloud (EC2)
#### AWS Elastic Compute Cloud (EC2)
If Amazon Elastic Block Store (EBS) default encyption is not enabled, sensitive data at rest will remain unprotected in EC2. However, Prowler will only generate a finding if EBS volumes exist where default encryption could be enforced.
@@ -56,7 +56,7 @@ Prowler scans only attached security groups to report vulnerabilities in activel
- `ec2_networkacl_allow_ingress_X_port`
#### AWS Glue
#### AWS Glue
AWS Glue best practices recommend encrypting metadata and connection passwords in Data Catalogs.
@@ -71,7 +71,7 @@ Amazon Inspector is a vulnerability discovery service that automates continuous
- `inspector2_is_enabled`
#### Amazon Macie
#### Amazon Macie
Amazon Macie leverages machine learning to automatically discover, classify, and protect sensitive data in S3 buckets. Prowler only generates findings if Macie is disabled and there are S3 buckets in the AWS account.
@@ -83,7 +83,7 @@ A network firewall is essential for monitoring and controlling traffic within a
- `networkfirewall_in_all_vpc`
#### Amazon S3
#### Amazon S3
To prevent unintended data exposure:
@@ -91,7 +91,7 @@ Public Access Block should be enabled at the account level. Prowler only checks
- `s3_account_level_public_access_blocks`
#### Virtual Private Cloud (VPC)
#### Virtual Private Cloud (VPC)
VPC settings directly impact network security and availability.
@@ -4,6 +4,7 @@ title: "Prowler ThreatScore Documentation"
<Info>This feature is only available in Prowler Cloud/App.</Info>
## Introduction
@@ -84,17 +84,6 @@ For detailed instructions on how to create the role, see [Authentication > Assum
![Next button in Prowler Cloud](/images/providers/next-button-prowler-cloud.png)
![Launch Scan](/images/providers/launch-scan-button-prowler-cloud.png)
<Note>
Check if your AWS Security Token Service (STS) has the EU (Ireland) endpoint active. If not, we will not be able to connect to your AWS account.
If that is the case your STS configuration may look like this:
<img src="/images/sts-configuration.png" alt="AWS Role" width="800" />
To solve this issue, please activate the EU (Ireland) STS endpoint.
</Note>
---
#### Credentials (Static Access Keys)
@@ -14,10 +14,7 @@ Prowler for Google Cloud supports multiple authentication methods. To use a spec
Prowler for Google Cloud requires the following permissions:
### IAM Roles
- **Viewer (`roles/viewer`)** Must be granted at the **project, folder, or organization** level to allow scanning of target projects.
- **Service Usage Consumer (`roles/serviceusage.serviceUsageConsumer`)** IAM Role Required for resource scanning.
- **Custom `ProwlerRole`** Include granular permissions that are not included in the Viewer role:
- `storage.buckets.getIamPolicy`
- **Reader (`roles/reader`)** Must be granted at the **project, folder, or organization** level to allow scanning of target projects.
### Project-Level Settings
@@ -109,46 +106,18 @@ prowler gcp --project-ids <project-id>
This method uses a service account with a downloaded key file for authentication.
### Step 1: Create ProwlerRole
### Create Service Account and Key
To keep permissions focused:
1. Create a custom role named **ProwlerRole** that explicitly includes the permissions your compliance team approves. Click **Create role**, set the title to *ProwlerRole*, keep the ID readable (for example, `prowler_role`)
2. Add the required permission `storage.buckets.getIamPolicy` (the permission highlighted in the screenshots). To make it easier, filter the permissions by `Storage Admin` role.
![Create a custom Prowler role](/user-guide/providers/gcp/img/roles-section.png)
![Sample permissions for a custom Prowler role](/user-guide/providers/gcp/img/prowler-role.png)
### Step 2: Create the Service Account
1. Navigate to **IAM & Admin > Service Accounts** and make sure the correct project is selected.
![Service accounts landing page](/user-guide/providers/gcp/img/service-account-page.png)
2. Select **Create service account**, provide a name, ID, and a short description that states the purpose (for example, “Service account to execute Prowler”), then click **Create and continue**.
![Create service account wizard](/user-guide/providers/gcp/img/create-service-account.png)
3. Assign the roles you prepared earlier:
- **ProwlerRole** for `cloudstorage` service checks.
- **Viewer** for broad read-only visibility.
- **Service Usage Consumer** so Prowler can inspect API states.
![Assign roles to the service account](/user-guide/providers/gcp/img/service-account-permissions.png)
4. Continue through the wizard and finish. No principals need to be granted access in step 3 unless you want other identities to impersonate this account.
### Step 3: Generate a JSON Key
1. Open the newly created service account, move to the **Keys** tab, and choose **Add key > Create new key**.
![Add a new key to the service account](/user-guide/providers/gcp/img/create-new-key.png)
2. Select **JSON** as the key type and click **Create**. The browser downloads the file exactly once.
![Select JSON as the key type](/user-guide/providers/gcp/img/json-key.png)
3. Once created, make sure to store the Key securely.
1. Go to the [Service Accounts page](https://console.cloud.google.com/iam-admin/serviceaccounts) in the GCP Console
2. Click "Create Service Account"
3. Fill in the service account details and click "Create and Continue"
4. Grant the service account the "Reader" role
5. Click "Done"
6. Find your service account in the list and click on it
7. Go to the "Keys" tab
8. Click "Add Key" > "Create new key"
9. Select "JSON" and click "Create"
10. Save the downloaded key file securely
### Using with Prowler CLI
@@ -55,7 +55,6 @@ For Google Cloud, first enter your `GCP Project ID` and then select the authenti
- [Generate a key for a service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys)
<img src="/images/prowler-app/gcp-service-account-creds.png" alt="GCP Service Account Credentials" width="700" />
For detailed instructions on how to setup Service Account authentication, see the [Authentication](/user-guide/providers/gcp/authentication#service-account-authentication) page.
</Tab>
<Tab title="Application Default Credentials">
1. Run the following command in your terminal to authenticate with GCP:
@@ -69,7 +68,6 @@ For Google Cloud, first enter your `GCP Project ID` and then select the authenti
3. Paste the `Client ID`, `Client Secret` and `Refresh Token` into Prowler App.
<img src="/images/gcp-credentials.png" alt="GCP Credentials" width="700" />
</Tab>
</Tabs>
Binary file not shown.

Before

Width:  |  Height:  |  Size: 185 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 369 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 587 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 403 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 299 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 472 KiB

@@ -64,8 +64,6 @@ Prowler App now separates Microsoft 365 authentication into two app-only options
Use this method whenever possible to avoid managing client secrets and to unlock every Microsoft 365 check, including those that require PowerShell modules.
For detailed instructions on how to setup Application Certificate Authentication, see the [Authentication](/user-guide/providers/microsoft365/authentication#application-certificate-authentication-recommended) page.
#### Application Client Secret Authentication
1. Enter your **tenant ID**: This is the unique identifier for your Microsoft Entra ID directory.
@@ -74,7 +72,6 @@ For detailed instructions on how to setup Application Certificate Authentication
<img src="/images/providers/secret-form.png" alt="M365 client secret authentication form" width="700" />
For detailed instructions on how to setup Application Client Secret Authentication, see the [Authentication](/user-guide/providers/microsoft365/authentication#application-client-secret-authentication) page.
### Step 4: Launch the Scan
@@ -1,128 +0,0 @@
---
title: 'Using Multiple LLM Providers with Lighthouse'
---
import { VersionBadge } from "/snippets/version-badge.mdx"
<VersionBadge version="5.14.0" />
Prowler Lighthouse AI supports multiple Large Language Model (LLM) providers, offering flexibility to choose the provider that best fits infrastructure, compliance requirements, and cost considerations. This guide explains how to configure and use different LLM providers with Lighthouse AI.
## Supported Providers
Lighthouse AI supports the following LLM providers:
- **OpenAI**: Provides access to GPT models (GPT-4o, GPT-4, etc.)
- **Amazon Bedrock**: Offers AWS-hosted access to Claude, Llama, Titan, and other models
- **OpenAI Compatible**: Supports custom endpoints like OpenRouter, Ollama, or any OpenAI-compatible service
## How Default Providers Work
All three providers can be configured for a tenant, but only one can be set as the default provider. The first configured provider automatically becomes the default.
When visiting Lighthouse AI chat, the default provider's default model loads automatically. Users can switch to any available LLM model (including those from non-default providers) using the dropdown in chat.
<img src="/images/prowler-app/lighthouse-switch-models.png" alt="Switch models in Lighthouse AI chat interface" />
## Configuring Providers
Navigate to **Configuration** → **Lighthouse AI** to see all three provider options with a **Connect** button under each.
<img src="/images/prowler-app/lighthouse-configuration.png" alt="Prowler Lighthouse Configuration" />
### Connecting a Provider
To connect a provider:
1. Click **Connect** under the desired provider
2. Enter the required credentials
3. Select a default model for that provider
4. Click **Connect** to save
### OpenAI
#### Required Information
- **API Key**: OpenAI API key (starts with `sk-` or `sk-proj-`)
<Note>
To generate an OpenAI API key, visit https://platform.openai.com/api-keys
</Note>
### Amazon Bedrock
#### Required Information
- **AWS Access Key ID**: AWS access key ID
- **AWS Secret Access Key**: AWS secret access key
- **AWS Region**: Region where Bedrock is available (e.g., `us-east-1`, `us-west-2`)
#### Required Permissions
The AWS user must have the `AmazonBedrockLimitedAccess` managed policy attached:
```text
arn:aws:iam::aws:policy/AmazonBedrockLimitedAccess
```
<Note>
Currently, only AWS access key and secret key authentication is supported. Amazon Bedrock API key support will be available soon.
</Note>
<Note>
Available models depend on AWS region and account entitlements. Lighthouse AI displays only accessible models.
</Note>
### OpenAI Compatible
Use this option to connect to any LLM provider exposing OpenAI compatible API endpoint (OpenRouter, Ollama, etc.).
#### Required Information
- **API Key**: API key from the compatible service
- **Base URL**: API endpoint URL including the API version (e.g., `https://openrouter.ai/api/v1`)
#### Example: OpenRouter
1. Create an account at [OpenRouter](https://openrouter.ai/)
2. [Generate an API key](https://openrouter.ai/docs/guides/overview/auth/provisioning-api-keys) from the OpenRouter dashboard
3. Configure in Lighthouse AI:
- **API Key**: OpenRouter API key
- **Base URL**: `https://openrouter.ai/api/v1`
## Changing the Default Provider
To set a different provider as default:
1. Navigate to **Configuration** → **Lighthouse AI**
2. Click **Configure** under the provider you want as default
3. Click **Set as Default**
<img src="/images/prowler-app/lighthouse-set-default-provider.png" alt="Set default LLM provider" />
## Updating Provider Credentials
To update credentials for a connected provider:
1. Navigate to **Configuration** → **Lighthouse AI**
2. Click **Configure** under the provider
3. Enter the new credentials
4. Click **Update**
## Deleting a Provider
To remove a configured provider:
1. Navigate to **Configuration** → **Lighthouse AI**
2. Click **Configure** under the provider
3. Click **Delete**
## Model Recommendations
For best results with Lighthouse AI, the recommended model is `gpt-5` from OpenAI.
Models from other providers such as Amazon Bedrock and OpenAI Compatible endpoints can be connected and used, but performance is not guaranteed.
## Getting Help
For issues or suggestions, [reach out through our Slack channel](https://goto.prowler.com/slack).
@@ -1,20 +1,26 @@
---
title: 'How It Works'
title: 'Prowler Lighthouse AI'
---
import { VersionBadge } from "/snippets/version-badge.mdx"
<VersionBadge version="5.8.0" />
Prowler Lighthouse AI integrates Large Language Models (LLMs) with Prowler security findings data.
Prowler Lighthouse AI is a Cloud Security Analyst chatbot that helps you understand, prioritize, and remediate security findings in your cloud environments. It's designed to provide security expertise for teams without dedicated resources, acting as your 24/7 virtual cloud security analyst.
<img src="/images/prowler-app/lighthouse-intro.png" alt="Prowler Lighthouse" />
## How It Works
Prowler Lighthouse AI uses OpenAI's language models and integrates with your Prowler security findings data.
Here's what's happening behind the scenes:
- The system uses a multi-agent architecture built with [LanggraphJS](https://github.com/langchain-ai/langgraphjs) for LLM logic and [Vercel AI SDK UI](https://sdk.vercel.ai/docs/ai-sdk-ui/overview) for frontend chatbot.
- It uses a ["supervisor" architecture](https://langchain-ai.lang.chat/langgraphjs/tutorials/multi_agent/agent_supervisor/) that interacts with different agents for specialized tasks. For example, `findings_agent` can analyze detected security findings, while `overview_agent` provides a summary of connected cloud accounts.
- The system connects to the configured LLM provider to understand user's query, fetches the right data, and responds to the query.
- The system connects to OpenAI models to understand, fetch the right data, and respond to the user's query.
<Note>
Lighthouse AI supports multiple LLM providers including OpenAI, Amazon Bedrock, and OpenAI-compatible services. For configuration details, see [Using Multiple LLM Providers with Lighthouse](/user-guide/tutorials/prowler-app-lighthouse-multi-llm).
Lighthouse AI is tested against `gpt-4o` and `gpt-4o-mini` OpenAI models.
</Note>
- The supervisor agent is the main contact point. It is what users interact with directly from the chat interface. It coordinates with other agents to answer users' questions comprehensively.
@@ -24,22 +30,16 @@ Lighthouse AI supports multiple LLM providers including OpenAI, Amazon Bedrock,
All agents can only read relevant security data. They cannot modify your data or access sensitive information like configured secrets or tenant details.
</Note>
## Set up
Getting started with Prowler Lighthouse AI is easy:
1. Navigate to **Configuration** → **Lighthouse AI**
2. Click **Connect** under the desired provider (OpenAI, Amazon Bedrock, or OpenAI Compatible)
3. Enter the required credentials
4. Select a default model
5. Click **Connect** to save
1. Go to the configuration page in your Prowler dashboard.
2. Enter your OpenAI API key.
3. Select your preferred model. The recommended one for best results is `gpt-4o`.
4. (Optional) Add business context to improve response quality and prioritization.
<Note>
For detailed configuration instructions for each provider, see [Using Multiple LLM Providers with Lighthouse](/user-guide/tutorials/prowler-app-lighthouse-multi-llm).
</Note>
<img src="/images/prowler-app/lighthouse-configuration.png" alt="Lighthouse AI Configuration" />
<img src="/images/prowler-app/lighthouse-config.png" alt="Lighthouse AI Configuration" />
### Adding Business Context
@@ -51,3 +51,163 @@ The optional business context field lets you provide additional information to h
- Current security initiatives or focus areas
Better context leads to more relevant responses and prioritization that aligns with your needs.
## Capabilities
Prowler Lighthouse AI is designed to be your AI security team member, with capabilities including:
### Natural Language Querying
Ask questions in plain English about your security findings. Examples:
- "What are my highest risk findings?"
- "Show me all S3 buckets with public access."
- "What security issues were found in my production accounts?"
<img src="/images/prowler-app/lighthouse-feature1.png" alt="Natural language querying" />
### Detailed Remediation Guidance
Get tailored step-by-step instructions for fixing security issues:
- Clear explanations of the problem and its impact
- Commands or console steps to implement fixes
- Alternative approaches with different solutions
<img src="/images/prowler-app/lighthouse-feature2.png" alt="Detailed Remediation" />
### Enhanced Context and Analysis
Lighthouse AI can provide additional context to help you understand the findings:
- Explain security concepts related to findings in simple terms
- Provide risk assessments based on your environment and context
- Connect related findings to show broader security patterns
<img src="/images/prowler-app/lighthouse-config.png" alt="Business Context" />
<img src="/images/prowler-app/lighthouse-feature3.png" alt="Contextual Responses" />
## Important Notes
Prowler Lighthouse AI is powerful, but there are limitations:
- **Continuous improvement**: Please report any issues, as the feature may make mistakes or encounter errors, despite extensive testing.
- **Access limitations**: Lighthouse AI can only access data the logged-in user can view. If you can't see certain information, Lighthouse AI can't see it either.
- **NextJS session dependence**: If your Prowler application session expires or logs out, Lighthouse AI will error out. Refresh and log back in to continue.
- **Response quality**: The response quality depends on the selected OpenAI model. For best results, use gpt-4o.
### Getting Help
If you encounter issues with Prowler Lighthouse AI or have suggestions for improvements, please [reach out through our Slack channel](https://goto.prowler.com/slack).
### What Data Is Shared to OpenAI?
The following API endpoints are accessible to Prowler Lighthouse AI. Data from the following API endpoints could be shared with OpenAI depending on the scope of user's query:
#### Accessible API Endpoints
**User Management:**
- List all users - `/api/v1/users`
- Retrieve the current user's information - `/api/v1/users/me`
**Provider Management:**
- List all providers - `/api/v1/providers`
- Retrieve data from a provider - `/api/v1/providers/{id}`
**Scan Management:**
- List all scans - `/api/v1/scans`
- Retrieve data from a specific scan - `/api/v1/scans/{id}`
**Resource Management:**
- List all resources - `/api/v1/resources`
- Retrieve data for a resource - `/api/v1/resources/{id}`
**Findings Management:**
- List all findings - `/api/v1/findings`
- Retrieve data from a specific finding - `/api/v1/findings/{id}`
- Retrieve metadata values from findings - `/api/v1/findings/metadata`
**Overview Data:**
- Get aggregated findings data - `/api/v1/overviews/findings`
- Get findings data by severity - `/api/v1/overviews/findings_severity`
- Get aggregated provider data - `/api/v1/overviews/providers`
- Get findings data by service - `/api/v1/overviews/services`
**Compliance Management:**
- List compliance overviews for a scan - `/api/v1/compliance-overviews`
- Retrieve data from a specific compliance overview - `/api/v1/compliance-overviews/{id}`
#### Excluded API Endpoints
Not all Prowler API endpoints are integrated with Lighthouse AI. They are intentionally excluded for the following reasons:
- OpenAI/other LLM providers shouldn't have access to sensitive data (like fetching provider secrets and other sensitive config)
- Users queries don't need responses from those API endpoints (ex: tasks, tenant details, downloading zip file, etc.)
**Excluded Endpoints:**
**User Management:**
- List specific users information - `/api/v1/users/{id}`
- List user memberships - `/api/v1/users/{user_pk}/memberships`
- Retrieve membership data from the user - `/api/v1/users/{user_pk}/memberships/{id}`
**Tenant Management:**
- List all tenants - `/api/v1/tenants`
- Retrieve data from a tenant - `/api/v1/tenants/{id}`
- List tenant memberships - `/api/v1/tenants/{tenant_pk}/memberships`
- List all invitations - `/api/v1/tenants/invitations`
- Retrieve data from tenant invitation - `/api/v1/tenants/invitations/{id}`
**Security and Configuration:**
- List all secrets - `/api/v1/providers/secrets`
- Retrieve data from a secret - `/api/v1/providers/secrets/{id}`
- List all provider groups - `/api/v1/provider-groups`
- Retrieve data from a provider group - `/api/v1/provider-groups/{id}`
**Reports and Tasks:**
- Download zip report - `/api/v1/scans/{v1}/report`
- List all tasks - `/api/v1/tasks`
- Retrieve data from a specific task - `/api/v1/tasks/{id}`
**Lighthouse AI Configuration:**
- List OpenAI configuration - `/api/v1/lighthouse-config`
- Retrieve OpenAI key and configuration - `/api/v1/lighthouse-config/{id}`
<Note>
Agents only have access to hit GET endpoints. They don't have access to other HTTP methods.
</Note>
## FAQs
**1. Why only OpenAI models?**
During feature development, we evaluated other LLM models.
- **Claude AI** - Claude models have [tier-based ratelimits](https://docs.anthropic.com/en/api/rate-limits#requirements-to-advance-tier). For Lighthouse AI to answer slightly complex questions, there are a handful of API calls to the LLM provider within few seconds. With Claude's tiering system, users must purchase $400 credits or convert their subscription to monthly invoicing after talking to their sales team. This pricing may not suit all Prowler users.
- **Gemini Models** - Gemini lacks a solid tool calling feature like OpenAI. It calls functions recursively until exceeding limits. Gemini-2.5-Pro-Experimental is better than previous models regarding tool calling and responding, but it's still experimental.
- **Deepseek V3** - Doesn't support system prompt messages.
**2. Why a multi-agent supervisor model?**
Context windows are limited. While demo data fits inside the context window, querying real-world data often exceeds it. A multi-agent architecture is used so different agents fetch different sizes of data and respond with the minimum required data to the supervisor. This spreads the context window usage across agents.
**3. Is my security data shared with OpenAI?**
Minimal data is shared to generate useful responses. Agents can access security findings and remediation details when needed. Provider secrets are protected by design and cannot be read. The OpenAI key configured with Lighthouse AI is only accessible to our NextJS server and is never sent to LLMs. Resource metadata (names, tags, account/project IDs, etc) may be shared with OpenAI based on your query requirements.
**4. Can the Lighthouse AI change my cloud environment?**
No. The agent doesn't have the tools to make the changes, even if the configured cloud provider API keys contain permissions to modify resources.
@@ -67,7 +67,7 @@ The Mutelist configuration takes effect on the next scans.
</Note>
## Mutelist Ready To Use Examples
Below are examples for different cloud providers supported by Prowler App. Check how the mutelist works [here](/user-guide/cli/tutorials/mutelist#how-the-mutelist-works).
Below are examples for different cloud providers supported by Prowler App. Check how the mutelist works [here](https://docs.prowler.com/projects/prowler-open-source/en/latest/tutorials/mutelist/#how-the-mutelist-works).
### AWS Provider
+33 -41
View File
@@ -10,9 +10,9 @@ This guide provides comprehensive instructions to configure SAML-based Single Si
This document is divided into two main sections:
- **[User Guide](#user-guide-configuration)**: For organization administrators to configure SAML SSO through Prowler App.
- **User Guide**: For organization administrators to configure SAML SSO through Prowler App.
- **[Developer and Administrator Guide](#developer-and-administrator-guide)**: For developers and system administrators running self-hosted Prowler App instances, providing technical details on environment configuration, API usage, and testing.
- **Developer and Administrator Guide**: For developers and system administrators running self-hosted Prowler App instances, providing technical details on environment configuration, API usage, and testing.
---
@@ -53,65 +53,58 @@ On the profile page, find the "SAML SSO Integration" card and click "Enable" to
![Enable SAML Integration](/images/prowler-app/saml/saml-step-2.png)
Next section will explain how to fill the IdP configuration based on your Identity Provider.
<Info>
**Choose Your Method**
#### Step 3: Configure the Identity Provider (IdP)
Choose a Method:
**Use Step 3A (Generic Method)** for any SAML 2.0 compliant Identity Provider or when you need custom configuration.
- Use [**Generic Method**](#generic-method) for any SAML 2.0 compliant Identity Provider or when you need custom configuration.
- Use [**Okta App Catalog**](#okta-app-catalog) if you're using Okta and want a simplified setup process with pre-configured settings.
**Use Step 3B (Okta App Catalog)** if you're using Okta and want a simplified setup process with pre-configured settings.
<Tabs>
<Tab title="Generic Method">
Prowler App displays the SAML configuration information needed to configure the IdP. Use this information to create a new SAML application in the IdP.
</Info>
#### Step 3A: Configure the Identity Provider (IdP) - Generic
1. **Assertion Consumer Service (ACS) URL**: The endpoint in Prowler that will receive the SAML assertion from the IdP.
2. **Audience URI (Entity ID)**: A unique identifier for the Prowler application (Service Provider).
Prowler App displays the SAML configuration information needed to configure the IdP. Use this information to create a new SAML application in the IdP.
To configure the IdP, copy the **ACS URL** and **Audience URI** from Prowler App and use them to set up a new SAML application.
1. **Assertion Consumer Service (ACS) URL**: The endpoint in Prowler that will receive the SAML assertion from the IdP.
2. **Audience URI (Entity ID)**: A unique identifier for the Prowler application (Service Provider).
![IdP configuration](/images/prowler-app/saml/idp_config.png)
To configure the IdP, copy the **ACS URL** and **Audience URI** from Prowler App and use them to set up a new SAML application.
<Info>
**IdP Configuration**
![IdP configuration](/images/prowler-app/saml/idp_config.png)
The exact steps for configuring an IdP vary depending on the provider (Okta, Azure AD, etc.). Please refer to the IdP's documentation for instructions on creating a SAML application. For SSO integration with Azure AD / Entra ID, see our [Entra ID configuration instructions](/user-guide/tutorials/prowler-app-sso-entra).
<Info>
**IdP Configuration**
</Info>
The exact steps for configuring an IdP vary depending on the provider (Okta, Azure AD, etc.). Please refer to the IdP's documentation for instructions on creating a SAML application. For SSO integration with Azure AD / Entra ID, see our [Entra ID configuration instructions](/user-guide/tutorials/prowler-app-sso-entra).
</Tab>
<Tab title="Okta App Catalog">
Instead of creating a custom SAML integration, Okta administrators can configure Prowler Cloud directly from Okta's application catalog.
</Info>
#### Step 3B: Configure Prowler from App Catalog - Okta
You can find a walkthrough video [here](https://youtu.be/NjSp5owvCdY).
Instead of creating a custom SAML integration, Okta administrators can configure Prowler Cloud directly from Okta's application catalog:
1. **Access App Catalog**: Navigate to the IdP's application catalog (e.g., [Browse App Catalog](https://www.okta.com/integrations/) in Okta).
1. **Access App Catalog**: Navigate to the IdP's application catalog (e.g., [Browse App Catalog](https://www.okta.com/integrations/) in Okta).
![Browse App Catalog](/images/prowler-app/saml/app-catalog-browse.png)
![Browse App Catalog](/images/prowler-app/saml/app-catalog-browse.png)
2. **Search for Prowler Cloud**: Use the search functionality to find "Prowler Cloud" in the app catalog. The official Prowler Cloud application will appear in the search results.
2. **Search for Prowler Cloud**: Use the search functionality to find "Prowler Cloud" in the app catalog. The official Prowler Cloud application will appear in the search results.
![Search for Prowler](/images/prowler-app/saml/app-catalog-browse-prowler.png)
![Search for Prowler](/images/prowler-app/saml/app-catalog-browse-prowler.png)
3. **Select Prowler Cloud Application**: Click the Prowler Cloud application from the search results to view its details page.
3. **Select Prowler Cloud Application**: Click on the Prowler Cloud application from the search results to view its details page.
![Prowler Application Details](/images/prowler-app/saml/app-catalog-browse-prowler-add.png)
![Prowler Application Details](/images/prowler-app/saml/app-catalog-browse-prowler-add.png)
4. **Add Integration**: Click the "Add Integration" button to begin adding Prowler Cloud to the organization's applications.
4. **Add Integration**: Click the "Add Integration" button to begin adding Prowler Cloud to the organization's applications.
5. **Configure General Settings**: In the "Add Prowler Cloud" configuration screen, the integration automatically configures the necessary settings.
5. **Configure General Settings**: In the "Add Prowler Cloud" configuration screen, the integration automatically configures the necessary settings.
![Add Prowler Configuration](/images/prowler-app/saml/app-catalog-browse-prowler-configure.png)
![Add Prowler Configuration](/images/prowler-app/saml/app-catalog-browse-prowler-configure.png)
6. **Assign Users**: Navigate to the "Assignments" tab and assign the appropriate users or groups to the Prowler application by clicking "Assign" and selecting "Assign to People" or "Assign to Groups".
6. **Assign Users**: Navigate to the **Assignments** tab and assign the appropriate users or groups to the Prowler application by clicking "Assign" and selecting "Assign to People" or "Assign to Groups".
With this step, the Okta app catalog configuration is complete. Users can now access Prowler Cloud using either [IdP-initiated](#idp-initiated-sso) or [SP-initiated SSO](#sp-initiated-sso) flows.
With this step, the Okta app catalog configuration is complete. Users can now access Prowler Cloud using either [IdP-initiated](#idp-initiated-sso) or [SP-initiated SSO](#sp-initiated-sso) flows.
7. **Download Metadata XML**: Inside the "Sign On" section, go to the "Metadata URL" and download the metadata XML file.
Jump to [Step 5: Upload IdP Metadata to Prowler](#step-5:-upload-idp-metadata-to-prowler).
</Tab>
</Tabs>
**If you used Step 3B (Okta App Catalog)**, jump to [Step 6: Save and Verify Configuration](#step-6-save-and-verify-configuration).
#### Step 4: Configure Attribute Mapping in the IdP
@@ -127,7 +120,7 @@ For Prowler App to correctly identify and provision users, configure the IdP to
<Info>
**IdP Attribute Mapping**
Note that the attribute name is just an example and may be different depending on the IdP. For instance, if the IdP provides a `division` attribute, it can be mapped to `userType`.
Note that the attribute name is just an example and may be different depending on the IdP. For instance, if the IdP provides a 'division' attribute, it can be mapped to 'userType'.
![IdP configuration](/images/prowler-app/saml/saml_attribute_statements.png)
</Info>
@@ -163,8 +156,7 @@ Click the "Save" button to complete the setup. The "SAML Integration" card will
The exact steps for configuring an IdP vary depending on the provider (Okta, Azure AD, etc.). Please refer to the IdP's documentation for instructions on creating a SAML application.
</Info>
### Remove SAML Configuration
##### Remove SAML Configuration
SAML SSO can be disabled by removing the existing configuration from the integration panel.
![Remove SAML configuration](/images/prowler-app/saml/saml-step-remove.png)
-5
View File
@@ -2,11 +2,6 @@
All notable changes to the **Prowler MCP Server** are documented in this file.
## [0.1.1] (Prowler 5.14.0)
### Fixed
- Fix documentation MCP Server to return list of dictionaries [(#9205)](https://github.com/prowler-cloud/prowler/pull/9205)
## [0.1.0] (Prowler 5.13.0)
### Added
@@ -1,8 +1,9 @@
from typing import Any, List
from typing import List
from fastmcp import FastMCP
from prowler_mcp_server.prowler_documentation.search_engine import (
ProwlerDocsSearchEngine,
SearchResult,
)
# Initialize FastMCP server
@@ -14,7 +15,7 @@ prowler_docs_search_engine = ProwlerDocsSearchEngine()
def search(
query: str,
page_size: int = 5,
) -> List[dict[str, Any]]:
) -> List[SearchResult]:
"""
Search in Prowler documentation.
Generated
+9 -20
View File
@@ -621,24 +621,6 @@ azure-mgmt-core = ">=1.3.2"
isodate = ">=0.6.1"
typing-extensions = ">=4.6.0"
[[package]]
name = "azure-mgmt-postgresqlflexibleservers"
version = "1.1.0"
description = "Microsoft Azure Postgresqlflexibleservers Management Client Library for Python"
optional = false
python-versions = ">=3.8"
groups = ["main"]
files = [
{file = "azure_mgmt_postgresqlflexibleservers-1.1.0-py3-none-any.whl", hash = "sha256:87ddb5a5e6d12c45769485d234cfe0322140e3a0a7636d0e61fb00ac544b5d20"},
{file = "azure_mgmt_postgresqlflexibleservers-1.1.0.tar.gz", hash = "sha256:9ede9d8ba63e9d2879cb74adc903c649af3bc5460a02787287b0cd18d754af14"},
]
[package.dependencies]
azure-common = ">=1.1"
azure-mgmt-core = ">=1.3.2"
isodate = ">=0.6.1"
typing-extensions = ">=4.6.0"
[[package]]
name = "azure-mgmt-rdbms"
version = "10.1.0"
@@ -2366,6 +2348,8 @@ python-versions = "*"
groups = ["dev"]
files = [
{file = "jsonpath-ng-1.7.0.tar.gz", hash = "sha256:f6f5f7fd4e5ff79c785f1573b394043b39849fb2bb47bcead935d12b00beab3c"},
{file = "jsonpath_ng-1.7.0-py2-none-any.whl", hash = "sha256:898c93fc173f0c336784a3fa63d7434297544b7198124a68f9a3ef9597b0ae6e"},
{file = "jsonpath_ng-1.7.0-py3-none-any.whl", hash = "sha256:f3d7f9e848cba1b6da28c55b1c26ff915dc9e0b1ba7e752a53d6da8d5cbd00b6"},
]
[package.dependencies]
@@ -4440,7 +4424,7 @@ version = "2025.9.18"
description = "Alternative regular expression module, to replace re."
optional = false
python-versions = ">=3.9"
groups = ["dev"]
groups = ["dev", "docs"]
files = [
{file = "regex-2025.9.18-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:12296202480c201c98a84aecc4d210592b2f55e200a1d193235c4db92b9f6788"},
{file = "regex-2025.9.18-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:220381f1464a581f2ea988f2220cf2a67927adcef107d47d6897ba5a2f6d51a4"},
@@ -4884,6 +4868,7 @@ files = [
{file = "ruamel.yaml.clib-0.2.12-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f66efbc1caa63c088dead1c4170d148eabc9b80d95fb75b6c92ac0aad2437d76"},
{file = "ruamel.yaml.clib-0.2.12-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:22353049ba4181685023b25b5b51a574bce33e7f51c759371a7422dcae5402a6"},
{file = "ruamel.yaml.clib-0.2.12-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:932205970b9f9991b34f55136be327501903f7c66830e9760a8ffb15b07f05cd"},
{file = "ruamel.yaml.clib-0.2.12-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:a52d48f4e7bf9005e8f0a89209bf9a73f7190ddf0489eee5eb51377385f59f2a"},
{file = "ruamel.yaml.clib-0.2.12-cp310-cp310-win32.whl", hash = "sha256:3eac5a91891ceb88138c113f9db04f3cebdae277f5d44eaa3651a4f573e6a5da"},
{file = "ruamel.yaml.clib-0.2.12-cp310-cp310-win_amd64.whl", hash = "sha256:ab007f2f5a87bd08ab1499bdf96f3d5c6ad4dcfa364884cb4549aa0154b13a28"},
{file = "ruamel.yaml.clib-0.2.12-cp311-cp311-macosx_13_0_arm64.whl", hash = "sha256:4a6679521a58256a90b0d89e03992c15144c5f3858f40d7c18886023d7943db6"},
@@ -4892,6 +4877,7 @@ files = [
{file = "ruamel.yaml.clib-0.2.12-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:811ea1594b8a0fb466172c384267a4e5e367298af6b228931f273b111f17ef52"},
{file = "ruamel.yaml.clib-0.2.12-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:cf12567a7b565cbf65d438dec6cfbe2917d3c1bdddfce84a9930b7d35ea59642"},
{file = "ruamel.yaml.clib-0.2.12-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:7dd5adc8b930b12c8fc5b99e2d535a09889941aa0d0bd06f4749e9a9397c71d2"},
{file = "ruamel.yaml.clib-0.2.12-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1492a6051dab8d912fc2adeef0e8c72216b24d57bd896ea607cb90bb0c4981d3"},
{file = "ruamel.yaml.clib-0.2.12-cp311-cp311-win32.whl", hash = "sha256:bd0a08f0bab19093c54e18a14a10b4322e1eacc5217056f3c063bd2f59853ce4"},
{file = "ruamel.yaml.clib-0.2.12-cp311-cp311-win_amd64.whl", hash = "sha256:a274fb2cb086c7a3dea4322ec27f4cb5cc4b6298adb583ab0e211a4682f241eb"},
{file = "ruamel.yaml.clib-0.2.12-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:20b0f8dc160ba83b6dcc0e256846e1a02d044e13f7ea74a3d1d56ede4e48c632"},
@@ -4900,6 +4886,7 @@ files = [
{file = "ruamel.yaml.clib-0.2.12-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:749c16fcc4a2b09f28843cda5a193e0283e47454b63ec4b81eaa2242f50e4ccd"},
{file = "ruamel.yaml.clib-0.2.12-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:bf165fef1f223beae7333275156ab2022cffe255dcc51c27f066b4370da81e31"},
{file = "ruamel.yaml.clib-0.2.12-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:32621c177bbf782ca5a18ba4d7af0f1082a3f6e517ac2a18b3974d4edf349680"},
{file = "ruamel.yaml.clib-0.2.12-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:b82a7c94a498853aa0b272fd5bc67f29008da798d4f93a2f9f289feb8426a58d"},
{file = "ruamel.yaml.clib-0.2.12-cp312-cp312-win32.whl", hash = "sha256:e8c4ebfcfd57177b572e2040777b8abc537cdef58a2120e830124946aa9b42c5"},
{file = "ruamel.yaml.clib-0.2.12-cp312-cp312-win_amd64.whl", hash = "sha256:0467c5965282c62203273b838ae77c0d29d7638c8a4e3a1c8bdd3602c10904e4"},
{file = "ruamel.yaml.clib-0.2.12-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:4c8c5d82f50bb53986a5e02d1b3092b03622c02c2eb78e29bec33fd9593bae1a"},
@@ -4908,6 +4895,7 @@ files = [
{file = "ruamel.yaml.clib-0.2.12-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:96777d473c05ee3e5e3c3e999f5d23c6f4ec5b0c38c098b3a5229085f74236c6"},
{file = "ruamel.yaml.clib-0.2.12-cp313-cp313-musllinux_1_1_i686.whl", hash = "sha256:3bc2a80e6420ca8b7d3590791e2dfc709c88ab9152c00eeb511c9875ce5778bf"},
{file = "ruamel.yaml.clib-0.2.12-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:e188d2699864c11c36cdfdada94d781fd5d6b0071cd9c427bceb08ad3d7c70e1"},
{file = "ruamel.yaml.clib-0.2.12-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4f6f3eac23941b32afccc23081e1f50612bdbe4e982012ef4f5797986828cd01"},
{file = "ruamel.yaml.clib-0.2.12-cp313-cp313-win32.whl", hash = "sha256:6442cb36270b3afb1b4951f060eccca1ce49f3d087ca1ca4563a6eb479cb3de6"},
{file = "ruamel.yaml.clib-0.2.12-cp313-cp313-win_amd64.whl", hash = "sha256:e5b8daf27af0b90da7bb903a876477a9e6d7270be6146906b276605997c7e9a3"},
{file = "ruamel.yaml.clib-0.2.12-cp39-cp39-macosx_12_0_arm64.whl", hash = "sha256:fc4b630cd3fa2cf7fce38afa91d7cfe844a9f75d7f0f36393fa98815e911d987"},
@@ -4916,6 +4904,7 @@ files = [
{file = "ruamel.yaml.clib-0.2.12-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e2f1c3765db32be59d18ab3953f43ab62a761327aafc1594a2a1fbe038b8b8a7"},
{file = "ruamel.yaml.clib-0.2.12-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:d85252669dc32f98ebcd5d36768f5d4faeaeaa2d655ac0473be490ecdae3c285"},
{file = "ruamel.yaml.clib-0.2.12-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:e143ada795c341b56de9418c58d028989093ee611aa27ffb9b7f609c00d813ed"},
{file = "ruamel.yaml.clib-0.2.12-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:2c59aa6170b990d8d2719323e628aaf36f3bfbc1c26279c0eeeb24d05d2d11c7"},
{file = "ruamel.yaml.clib-0.2.12-cp39-cp39-win32.whl", hash = "sha256:beffaed67936fbbeffd10966a4eb53c402fafd3d6833770516bf7314bc6ffa12"},
{file = "ruamel.yaml.clib-0.2.12-cp39-cp39-win_amd64.whl", hash = "sha256:040ae85536960525ea62868b642bdb0c2cc6021c9f9d507810c0c604e66f5a7b"},
{file = "ruamel.yaml.clib-0.2.12.tar.gz", hash = "sha256:6c8fbb13ec503f99a91901ab46e0b07ae7941cd527393187039aec586fdfd36f"},
@@ -5688,4 +5677,4 @@ type = ["pytest-mypy"]
[metadata]
lock-version = "2.1"
python-versions = ">3.9.1,<3.13"
content-hash = "a367e65bc43c0a16495a3d0f6eab8b356cc49b509e329b61c6641cd87f374ff4"
content-hash = "ea79d82b4e255ec4604f440a507da6dac38b57af93356761ac793678aa615cf5"
+6 -27
View File
@@ -2,7 +2,7 @@
All notable changes to the **Prowler SDK** are documented in this file.
## [v5.14.0] (Prowler v5.14.0)
## [v5.14.0] (Prowler UNRELEASED)
### Added
- GitHub provider check `organization_default_repository_permission_strict` [(#8785)](https://github.com/prowler-cloud/prowler/pull/8785)
@@ -11,23 +11,16 @@ All notable changes to the **Prowler SDK** are documented in this file.
- `cloudstorage_bucket_versioning_enabled` check for GCP provider [(#9014)](https://github.com/prowler-cloud/prowler/pull/9014)
- `cloudstorage_bucket_soft_delete_enabled` check for GCP provider [(#9028)](https://github.com/prowler-cloud/prowler/pull/9028)
- `cloudstorage_bucket_logging_enabled` check for GCP provider [(#9091)](https://github.com/prowler-cloud/prowler/pull/9091)
- `cloudstorage_audit_logs_enabled` check for GCP provider [(#9220)](https://github.com/prowler-cloud/prowler/pull/9220)
- `cloudstorage_bucket_sufficient_retention_period` check for GCP provider [(#9149)](https://github.com/prowler-cloud/prowler/pull/9149)
- C5 compliance framework for Azure provider [(#9081)](https://github.com/prowler-cloud/prowler/pull/9081)
- C5 compliance framework for the GCP provider [(#9097)](https://github.com/prowler-cloud/prowler/pull/9097)
- `organization_repository_creation_limited` check for GitHub provider [(#8844)](https://github.com/prowler-cloud/prowler/pull/8844)
- HIPAA compliance framework for the GCP provider [(#8955)](https://github.com/prowler-cloud/prowler/pull/8955)
- Support PDF reporting for ENS compliance framework [(#9158)](https://github.com/prowler-cloud/prowler/pull/9158)
- PDF reporting for NIS2 compliance framework [(#9170)](https://github.com/prowler-cloud/prowler/pull/9170)
- Add organization ID parameter for MongoDB Atlas provider [(#9167)](https://github.com/prowler-cloud/prowler/pull/9167)
- Add multiple compliance improvements [(#9145)](https://github.com/prowler-cloud/prowler/pull/9145)
- Added validation for invalid checks, services, and categories in `load_checks_to_execute` function [(#8971)](https://github.com/prowler-cloud/prowler/pull/8971)
- NIST CSF 2.0 compliance framework for the AWS provider [(#9185)](https://github.com/prowler-cloud/prowler/pull/9185)
- Add FedRAMP 20x KSI Low for AWS, Azure and GCP [(#9198)](https://github.com/prowler-cloud/prowler/pull/9198)
- Add verification for provider ID in MongoDB Atlas provider [(#9211)](https://github.com/prowler-cloud/prowler/pull/9211)
- Add Prowler ThreatScore for the K8S provider [(#9235)](https://github.com/prowler-cloud/prowler/pull/9235)
- Add `postgresql_flexible_server_entra_id_authentication_enabled` check for Azure provider [(#8764)](https://github.com/prowler-cloud/prowler/pull/8764)
- Add branch name to IaC provider region [(#9296)](https://github.com/prowler-cloud/prowler/pull/9295)
### Changed
- Update AWS Direct Connect service metadata to new format [(#8855)](https://github.com/prowler-cloud/prowler/pull/8855)
@@ -38,7 +31,6 @@ All notable changes to the **Prowler SDK** are documented in this file.
- Update AWS EKS service metadata to new format [(#8890)](https://github.com/prowler-cloud/prowler/pull/8890)
- Update AWS Elastic Beanstalk service metadata to new format [(#8934)](https://github.com/prowler-cloud/prowler/pull/8934)
- Update AWS ElastiCache service metadata to new format [(#8933)](https://github.com/prowler-cloud/prowler/pull/8933)
- Update Kubernetes etcd service metadata to new format [(#9096)](https://github.com/prowler-cloud/prowler/pull/9096)
- Update MongoDB Atlas projects service metadata to new format [(#9093)](https://github.com/prowler-cloud/prowler/pull/9093)
- Update GitHub Organization service metadata to new format [(#9094)](https://github.com/prowler-cloud/prowler/pull/9094)
- Update AWS CodeBuild service metadata to new format [(#8851)](https://github.com/prowler-cloud/prowler/pull/8851)
@@ -50,19 +42,13 @@ All notable changes to the **Prowler SDK** are documented in this file.
- Update AWS FSx service metadata to new format [(#9006)](https://github.com/prowler-cloud/prowler/pull/9006)
- Update AWS Glacier service metadata to new format [(#9007)](https://github.com/prowler-cloud/prowler/pull/9007)
- Update oraclecloud analytics service metadata to new format [(#9114)](https://github.com/prowler-cloud/prowler/pull/9114)
- Update AWS ELB service metadata to new format [(#8935)](https://github.com/prowler-cloud/prowler/pull/8935)
- Update AWS CodeArtifact service metadata to new format [(#8850)](https://github.com/prowler-cloud/prowler/pull/8850)
- Rename OCI provider to oraclecloud with oci alias [(#9126)](https://github.com/prowler-cloud/prowler/pull/9126)
- Remove unnecessary tests for M365_PowerShell module [(#9204)](https://github.com/prowler-cloud/prowler/pull/9204)
- Update AWS ELB v2 service metadata to new format [(#9001)](https://github.com/prowler-cloud/prowler/pull/9001)
- Update oraclecloud cloudguard service metadata to new format [(#9223)](https://github.com/prowler-cloud/prowler/pull/9223)
- Update oraclecloud blockstorage service metadata to new format [(#9222)](https://github.com/prowler-cloud/prowler/pull/9222)
- Update oraclecloud audit service metadata to new format [(#9221)](https://github.com/prowler-cloud/prowler/pull/9221)
- Raise ASFF output error for non-AWS providers [(#9225)](https://github.com/prowler-cloud/prowler/pull/9225)
- Update AWS ECR service metadata to new format [(#8872)](https://github.com/prowler-cloud/prowler/pull/8872)
- Update AWS ECS service metadata to new format [(#8888)](https://github.com/prowler-cloud/prowler/pull/8888)
- Update AWS Kinesis service metadata to new format [(#9262)](https://github.com/prowler-cloud/prowler/pull/9262)
- Update AWS DocumentDB service metadata to new format [(#8862)](https://github.com/prowler-cloud/prowler/pull/8862)
---
## [v5.13.2] (Prowler UNRELEASED)
### Fixed
- Check `check_name` has no `resource_name` error for GCP provider [(#9169)](https://github.com/prowler-cloud/prowler/pull/9169)
@@ -70,12 +56,6 @@ All notable changes to the **Prowler SDK** are documented in this file.
- False negative in `iam_role_cross_service_confused_deputy_prevention` check [(#9213)](https://github.com/prowler-cloud/prowler/pull/9213)
- Fix M365 Teams `--sp-env-auth` connection error and enhanced timeout logging [(#9191)](https://github.com/prowler-cloud/prowler/pull/9191)
- Rename `get_oci_assessment_summary` to `get_oraclecloud_assessment_summary` in HTML output [(#9200)](https://github.com/prowler-cloud/prowler/pull/9200)
- Fix Validation and other errors in Azure provider [(#8915)](https://github.com/prowler-cloud/prowler/pull/8915)
- Update documentation URLs from docs.prowler.cloud to docs.prowler.com [(#9240)](https://github.com/prowler-cloud/prowler/pull/9240)
- Refresh output report timestamps for each scan [(#9272)](https://github.com/prowler-cloud/prowler/pull/9272)
- Fix file name parsing for checks on Windows [(#9268)](https://github.com/prowler-cloud/prowler/pull/9268)
- Remove typo for Prowler ThreatScore - M365 [(#9274)](https://github.com/prowler-cloud/prowler/pull/9274)
- Point HTML logo to the one present in the Github repository [(#9282)](https://github.com/prowler-cloud/prowler/pull/9282)
---
@@ -138,7 +118,6 @@ All notable changes to the **Prowler SDK** are documented in this file.
- Update AWS CloudFront service metadata to new format [(#8829)](https://github.com/prowler-cloud/prowler/pull/8829)
- Deprecate user authentication for M365 provider [(#8865)](https://github.com/prowler-cloud/prowler/pull/8865)
### Fixed
- Fix SNS topics showing empty AWS_ResourceID in Quick Inventory output [(#8762)](https://github.com/prowler-cloud/prowler/issues/8762)
- Fix HTML Markdown output for long strings [(#8803)](https://github.com/prowler-cloud/prowler/pull/8803)
@@ -403,7 +382,7 @@ All notable changes to the **Prowler SDK** are documented in this file.
---
## [v5.7.5] (Prowler v5.7.5)
## [v5.7.5] (Prowler 5.7.5)
### Fixed
- Use unified timestamp for all requirements [(#8059)](https://github.com/prowler-cloud/prowler/pull/8059)
+2 -17
View File
@@ -90,9 +90,6 @@ from prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_azur
from prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_gcp import (
ProwlerThreatScoreGCP,
)
from prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_kubernetes import (
ProwlerThreatScoreKubernetes,
)
from prowler.lib.outputs.compliance.prowler_threatscore.prowler_threatscore_m365 import (
ProwlerThreatScoreM365,
)
@@ -433,7 +430,7 @@ def prowler():
else:
# Refactor(CLI)
logger.critical(
"Slack integration needs SLACK_API_TOKEN and SLACK_CHANNEL_NAME environment variables (see more in https://docs.prowler.com/user-guide/cli/tutorials/integrations#configuration-of-the-integration-with-slack)."
"Slack integration needs SLACK_API_TOKEN and SLACK_CHANNEL_NAME environment variables (see more in https://docs.prowler.cloud/en/latest/tutorials/integrations/#slack)."
)
sys.exit(1)
@@ -576,6 +573,7 @@ def prowler():
generated_outputs["compliance"].append(prowler_threatscore)
prowler_threatscore.batch_write_data_to_file()
elif compliance_name.startswith("ccc_"):
filename = (
f"{output_options.output_directory}/compliance/"
f"{output_options.output_filename}_{compliance_name}.csv"
@@ -848,19 +846,6 @@ def prowler():
)
generated_outputs["compliance"].append(iso27001)
iso27001.batch_write_data_to_file()
elif compliance_name == "prowler_threatscore_kubernetes":
# Generate Prowler ThreatScore Finding Object
filename = (
f"{output_options.output_directory}/compliance/"
f"{output_options.output_filename}_{compliance_name}.csv"
)
prowler_threatscore = ProwlerThreatScoreKubernetes(
findings=finding_outputs,
compliance=bulk_compliance_frameworks[compliance_name],
file_path=filename,
)
generated_outputs["compliance"].append(prowler_threatscore)
prowler_threatscore.batch_write_data_to_file()
else:
filename = (
f"{output_options.output_directory}/compliance/"
File diff suppressed because it is too large Load Diff
@@ -1024,7 +1024,7 @@
"Attributes": [
{
"Title": "AuditDisabled organizationally is set to False",
"Section": "3. Logging and Monitoring",
"Section": "3. Logging and monitoring",
"SubSection": "3.1 Logging",
"AttributeDescription": "The setting “Mailbox auditing on by default” determines whether mailbox auditing is automatically enabled across all mailboxes in the organization, regardless of their individual auditing configuration. When this setting is configured as False, it enables auditing at the organization level, overriding the AuditEnabled property for individual mailboxes—even if it is explicitly set to False. With this setting enabled, default audit actions are automatically recorded for all mailboxes without requiring manual configuration. Conversely, disabling this setting (True) effectively turns off mailbox auditing across the organization and overrides any mailbox-level auditing settings. The consequences of disabling this setting include: • Mailbox auditing is completely disabled organization-wide. • No mailbox actions are logged, even if AuditEnabled is set to True for individual mailboxes. • New mailboxes do not inherit auditing, and setting AuditEnabled=True has no effect. • Bypass audit rules set via Set-MailboxAuditBypassAssociation are ignored. • Existing audit records remain in place until they expire based on the audit log retention policy. The recommended configuration is to set this value to False at the organization level to ensure auditing is enforced consistently.",
"AdditionalInformation": "Enforcing mailbox auditing by default ensures that audit logging cannot be unintentionally or maliciously disabled on individual mailboxes. This setting provides vital visibility for forensic investigations and incident response (IR) teams, allowing them to trace suspicious or malicious activity—such as unauthorized inbox access, message deletion, or rule manipulation—that may signal account compromise. Consistent auditing across all mailboxes is critical for detecting threat actor behaviors (TTPs) and correlating events across users. While organizations without Microsoft 365 E5 licenses are limited to 90 days of audit log retention, enabling this setting still significantly improves detection and accountability within that window.",
@@ -1042,7 +1042,7 @@
"Attributes": [
{
"Title": "Mailbox auditing for E3 users is Enabled",
"Section": "3. Logging and Monitoring",
"Section": "3. Logging and monitoring",
"SubSection": "3.1 Logging",
"AttributeDescription": "As of January 2019, Microsoft enables mailbox audit logging by default across all organizations. This feature ensures that specific actions performed by mailbox owners, delegates, and administrators are automatically captured and recorded. These audit records can then be searched by administrators through the mailbox audit log in Microsoft 365. Each mailbox type—whether user, shared, resource, or public folder—can have tailored audit settings to track activities that are most relevant to the organization. While audit logging is enabled by default at the organizational level, it is important to explicitly configure the AuditEnabled property to True on all user mailboxes, and to expand the list of audited actions beyond the Microsoft defaults to meet specific visibility or compliance needs. Note: This recommendation is particularly relevant to users with Microsoft 365 E3 licenses, where audit actions differ slightly from the default configurations in E5.",
"AdditionalInformation": "Mailbox auditing plays a critical role in supporting both regulatory compliance and security monitoring. Whether investigating unauthorized configuration changes, potential account compromise, or insider threats, detailed mailbox audit logs provide essential evidence for security operations, forensic analysis, and general administrative oversight. While mailbox auditing is enabled by default for most user mailboxes, certain mailbox types—such as Resource Mailboxes, Public Folder Mailboxes, and the DiscoverySearch Mailbox—do not inherit the organizational auditing default. For these mailboxes, AuditEnabled must be manually set to True to ensure relevant activities are captured. Note: Organizations without Microsoft 365 E5 licenses are subject to a 90-day audit log retention limit, but enabling comprehensive mailbox auditing remains a best practice for operational readiness and incident response.",
@@ -1060,7 +1060,7 @@
"Attributes": [
{
"Title": "Mailbox auditing for E5 users is Enabled",
"Section": "3. Logging and Monitoring",
"Section": "3. Logging and monitoring",
"SubSection": "3.1 Logging",
"AttributeDescription": "Since January 2019, mailbox audit logging has been enabled by default in all Microsoft 365 organizations. This feature ensures that specific actions performed by mailbox owners, delegates, and administrators are automatically captured and stored as audit records. These logs are accessible to administrators through the Microsoft 365 mailbox audit log, enabling visibility into key mailbox-level activity. Although logging is enabled by default, each mailbox—particularly user and shared mailboxes—can have custom audit actions assigned to capture the specific types of events deemed valuable by the organization. For environments with Microsoft 365 E5 licenses or the advanced auditing add-on, it is recommended to explicitly set AuditEnabled to True on all user mailboxes and to configure additional audit actions beyond Microsofts default settings for enhanced visibility. Note: This recommendation specifically applies to E5 or equivalent auditing-enabled license holders, as the available audit depth and event coverage differ from E3.",
"AdditionalInformation": "Mailbox audit logging is essential for supporting security investigations, regulatory compliance, and operational forensics in Microsoft 365. Whether youre tracking unauthorized changes, detecting suspicious access, or conducting post-incident analysis, having a complete and accurate mailbox audit trail is critical. While audit logging is broadly applied by default, certain mailbox types bypass the organizational setting and require manual configuration to enable auditing. These include: • Resource Mailboxes • Public Folder Mailboxes • DiscoverySearch Mailboxes For these mailbox types, the AuditEnabled property must be explicitly set to True to ensure that audit events are captured. Important: Without advanced auditing (included in E5 or via add-on), mailbox audit logs are retained for only 90 days, limiting the historical window for investigations. Nonetheless, enabling detailed auditing remains a key best practice for maintaining strong visibility and compliance readiness.",
@@ -1078,7 +1078,7 @@
"Attributes": [
{
"Title": "AuditBypassEnabled is not enabled on mailboxes",
"Section": "3. Logging and Monitoring",
"Section": "3. Logging and monitoring",
"SubSection": "3.1 Logging",
"AttributeDescription": "The AuditBypassEnabled setting in Microsoft 365 allows specific user or computer accounts to bypass mailbox audit logging, meaning that any actions they perform on mailboxes will not be recorded in the audit logs. This includes actions such as reading, deleting, moving, or modifying messages.",
"AdditionalInformation": "Allowing an account to bypass mailbox audit logging creates a blind spot in security monitoring. If the account is compromised, misused, or maliciously configured, it can access and interact with mailboxes without leaving any trace in the logs. This significantly undermines the organizations ability to conduct forensic investigations, detect insider threats, or comply with audit requirements.",
@@ -1096,7 +1096,7 @@
"Attributes": [
{
"Title": "Microsoft 365 audit log search is Enabled ",
"Section": "3. Logging and Monitoring",
"Section": "3. Logging and monitoring",
"SubSection": "3.2 Retention",
"AttributeDescription": "Audit log search in the Microsoft Purview compliance portal allows organizations to track and retain user and administrator activities across Microsoft 365 services. When enabled, audit events—such as sign-ins, file access, configuration changes, and other operational actions—are captured and stored for up to 90 days by default. While some organizations may choose to integrate auditing data with third-party Security Information and Event Management (SIEM) systems, audit log search in Microsoft Purview remains a critical native capability for centralized visibility and incident response. Although global administrators have the ability to disable audit log search, it is generally recommended to keep it enabled to maintain full visibility into user and system activity.",
"AdditionalInformation": "Activating audit log search provides essential forensic and compliance value. It enables organizations to detect anomalous behavior, investigate potential security incidents, and demonstrate adherence to regulatory and legal requirements. In addition, it supports operational monitoring, internal audits, and proactive threat detection. By retaining and centralizing audit data within the Microsoft 365 ecosystem, security and compliance teams gain faster access to actionable insights, reducing response times and strengthening the organizations overall security posture.",
@@ -1114,7 +1114,7 @@
"Attributes": [
{
"Title": "Notifications for internal users sending malware is Enabled",
"Section": "3. Logging and Monitoring",
"Section": "3. Logging and monitoring",
"SubSection": "3.3 Monitoring",
"AttributeDescription": "Exchange Online Protection (EOP) is Microsofts cloud-based email filtering service designed to safeguard organizations against spam, malware, and other email-borne threats. It is included by default in all Microsoft 365 tenants with Exchange Online mailboxes. EOP provides customizable anti-malware policies that allow administrators to define protection settings and configure alerts for detected malicious activity.",
"AdditionalInformation": "Enabling notifications for malware detections ensures that administrators are alerted when an internal user sends a message containing malware. Such incidents may signal a compromised user account or infected device, requiring immediate investigation to mitigate potential security breaches.",
+3 -57
View File
@@ -3,7 +3,6 @@ import pathlib
from datetime import datetime, timezone
from enum import Enum
from os import getcwd
from typing import Tuple
import requests
import yaml
@@ -11,36 +10,11 @@ from packaging import version
from prowler.lib.logger import logger
class _MutableTimestamp:
"""Lightweight proxy to keep timestamp references in sync across modules."""
def __init__(self, value: datetime) -> None:
self.value = value
def set(self, value: datetime) -> None:
self.value = value
def __getattr__(self, name):
return getattr(self.value, name)
def __str__(self) -> str: # pragma: no cover - trivial forwarder
return str(self.value)
def __repr__(self) -> str: # pragma: no cover - trivial forwarder
return repr(self.value)
def __eq__(self, other) -> bool:
if isinstance(other, _MutableTimestamp):
return self.value == other.value
return self.value == other
timestamp = _MutableTimestamp(datetime.today())
timestamp_utc = _MutableTimestamp(datetime.now(timezone.utc))
timestamp = datetime.today()
timestamp_utc = datetime.now(timezone.utc).replace(tzinfo=timezone.utc)
prowler_version = "5.14.0"
html_logo_url = "https://github.com/prowler-cloud/prowler/"
square_logo_img = "https://raw.githubusercontent.com/prowler-cloud/prowler/dc7d2d5aeb92fdf12e8604f42ef6472cd3e8e889/docs/img/prowler-logo-black.png"
square_logo_img = "https://prowler.com/wp-content/uploads/logo-html.png"
aws_logo = "https://user-images.githubusercontent.com/38561120/235953920-3e3fba08-0795-41dc-b480-9bea57db9f2e.png"
azure_logo = "https://user-images.githubusercontent.com/38561120/235927375-b23e2e0f-8932-49ec-b59c-d89f61c8041d.png"
gcp_logo = "https://user-images.githubusercontent.com/38561120/235928332-eb4accdc-c226-4391-8e97-6ca86a91cf50.png"
@@ -110,34 +84,6 @@ encoding_format_utf_8 = "utf-8"
available_output_formats = ["csv", "json-asff", "json-ocsf", "html"]
def set_output_timestamp(
new_timestamp: datetime,
) -> Tuple[datetime, datetime, str, str]:
"""
Override the global output timestamps so generated artifacts reflect a specific scan.
Returns the previous values so callers can restore them afterwards.
"""
global timestamp, timestamp_utc, output_file_timestamp, timestamp_iso
previous_values = (
timestamp.value,
timestamp_utc.value,
output_file_timestamp,
timestamp_iso,
)
timestamp.set(new_timestamp)
timestamp_utc.set(
new_timestamp.astimezone(timezone.utc)
if new_timestamp.tzinfo
else new_timestamp.replace(tzinfo=timezone.utc)
)
output_file_timestamp = timestamp.strftime("%Y%m%d%H%M%S")
timestamp_iso = timestamp.isoformat(sep=" ", timespec="seconds")
return previous_values
def get_default_mute_file_path(provider: str):
"""
get_default_mute_file_path returns the default mute file path for the provider
+1 -2
View File
@@ -457,8 +457,7 @@ class Check(ABC, CheckMetadata):
# Verify names consistency
check_id = self.CheckID
class_name = self.__class__.__name__
# os.path.basename handles Windows and POSIX paths reliably
file_name = os.path.basename(file_path)
file_name = file_path.split(sep="/")[-1]
errors = []
if check_id != class_name:
+4 -11
View File
@@ -15,7 +15,6 @@ from prowler.lib.check.models import Severity
from prowler.lib.outputs.common import Status
from prowler.providers.common.arguments import (
init_providers_parser,
validate_asff_usage,
validate_provider_arguments,
)
@@ -136,12 +135,6 @@ Detailed documentation at https://docs.prowler.com
if not valid:
self.parser.error(f"{args.provider}: {message}")
asff_is_valid, asff_error = validate_asff_usage(
args.provider, getattr(args, "output_formats", None)
)
if not asff_is_valid:
self.parser.error(asff_error)
return args
def __set_default_provider__(self, args: list) -> list:
@@ -311,7 +304,7 @@ Detailed documentation at https://docs.prowler.com
"--checks-folder",
"-x",
nargs="?",
help="Specify external directory with custom checks (each check must have a folder with the required files, see more in https://docs.prowler.com/user-guide/cli/tutorials/misc#custom-checks-in-prowler).",
help="Specify external directory with custom checks (each check must have a folder with the required files, see more in https://docs.prowler.cloud/en/latest/tutorials/misc/#custom-checks).",
)
def __init_list_checks_parser__(self):
@@ -364,7 +357,7 @@ Detailed documentation at https://docs.prowler.com
"--mutelist-file",
"-w",
nargs="?",
help="Path for mutelist YAML file. See example prowler/config/<provider>_mutelist.yaml for reference and format. For AWS provider, it also accepts AWS DynamoDB Table, Lambda ARNs or S3 URIs, see more in https://docs.prowler.com/user-guide/cli/tutorials/mutelist",
help="Path for mutelist YAML file. See example prowler/config/<provider>_mutelist.yaml for reference and format. For AWS provider, it also accepts AWS DynamoDB Table, Lambda ARNs or S3 URIs, see more in https://docs.prowler.cloud/en/latest/tutorials/mutelist/",
)
def __init_config_parser__(self):
@@ -391,7 +384,7 @@ Detailed documentation at https://docs.prowler.com
"--custom-checks-metadata-file",
nargs="?",
default=None,
help="Path for the custom checks metadata YAML file. See example prowler/config/custom_checks_metadata_example.yaml for reference and format. See more in https://docs.prowler.com/user-guide/cli/tutorials/custom-checks-metadata/",
help="Path for the custom checks metadata YAML file. See example prowler/config/custom_checks_metadata_example.yaml for reference and format. See more in https://docs.prowler.cloud/en/latest/tutorials/custom-checks-metadata/",
)
def __init_third_party_integrations_parser__(self):
@@ -409,5 +402,5 @@ Detailed documentation at https://docs.prowler.com
third_party_subparser.add_argument(
"--slack",
action="store_true",
help="Send a summary of the execution with a Slack APP in your channel. Environment variables SLACK_API_TOKEN and SLACK_CHANNEL_NAME are required (see more in https://docs.prowler.com/user-guide/cli/tutorials/integrations#configuration-of-the-integration-with-slack/).",
help="Send a summary of the execution with a Slack APP in your channel. Environment variables SLACK_API_TOKEN and SLACK_CHANNEL_NAME are required (see more in https://docs.prowler.cloud/en/latest/tutorials/integrations/#slack).",
)
@@ -117,32 +117,3 @@ class ProwlerThreatScoreM365Model(BaseModel):
Muted: bool
Framework: str
Name: str
class ProwlerThreatScoreKubernetesModel(BaseModel):
"""
ProwlerThreatScoreKubernetesModel generates a finding's output in Kubernetes Prowler ThreatScore Compliance format.
"""
Provider: str
Description: str
Context: str
Namespace: str
AssessmentDate: str
Requirements_Id: str
Requirements_Description: str
Requirements_Attributes_Title: str
Requirements_Attributes_Section: str
Requirements_Attributes_SubSection: Optional[str] = None
Requirements_Attributes_AttributeDescription: str
Requirements_Attributes_AdditionalInformation: str
Requirements_Attributes_LevelOfRisk: int
Requirements_Attributes_Weight: int
Status: str
StatusExtended: str
ResourceId: str
ResourceName: str
CheckId: str
Muted: bool
Framework: str
Name: str
@@ -1,98 +0,0 @@
from prowler.config.config import timestamp
from prowler.lib.check.compliance_models import Compliance
from prowler.lib.outputs.compliance.compliance_output import ComplianceOutput
from prowler.lib.outputs.compliance.prowler_threatscore.models import (
ProwlerThreatScoreKubernetesModel,
)
from prowler.lib.outputs.finding import Finding
class ProwlerThreatScoreKubernetes(ComplianceOutput):
"""
This class represents the Kubernetes Prowler ThreatScore compliance output.
Attributes:
- _data (list): A list to store transformed data from findings.
- _file_descriptor (TextIOWrapper): A file descriptor to write data to a file.
Methods:
- transform: Transforms findings into Kubernetes Prowler ThreatScore compliance format.
"""
def transform(
self,
findings: list[Finding],
compliance: Compliance,
compliance_name: str,
) -> None:
"""
Transforms a list of findings into Kubernetes Prowler ThreatScore compliance format.
Parameters:
- findings (list): A list of findings.
- compliance (Compliance): A compliance model.
- compliance_name (str): The name of the compliance model.
Returns:
- None
"""
for finding in findings:
# Get the compliance requirements for the finding
finding_requirements = finding.compliance.get(compliance_name, [])
for requirement in compliance.Requirements:
if requirement.Id in finding_requirements:
for attribute in requirement.Attributes:
compliance_row = ProwlerThreatScoreKubernetesModel(
Provider=finding.provider,
Description=compliance.Description,
Context=finding.account_name,
Namespace=finding.region,
AssessmentDate=str(timestamp),
Requirements_Id=requirement.Id,
Requirements_Description=requirement.Description,
Requirements_Attributes_Title=attribute.Title,
Requirements_Attributes_Section=attribute.Section,
Requirements_Attributes_SubSection=attribute.SubSection,
Requirements_Attributes_AttributeDescription=attribute.AttributeDescription,
Requirements_Attributes_AdditionalInformation=attribute.AdditionalInformation,
Requirements_Attributes_LevelOfRisk=attribute.LevelOfRisk,
Requirements_Attributes_Weight=attribute.Weight,
Status=finding.status,
StatusExtended=finding.status_extended,
ResourceId=finding.resource_uid,
ResourceName=finding.resource_name,
CheckId=finding.check_id,
Muted=finding.muted,
Framework=compliance.Framework,
Name=compliance.Name,
)
self._data.append(compliance_row)
# Add manual requirements to the compliance output
for requirement in compliance.Requirements:
if not requirement.Checks:
for attribute in requirement.Attributes:
compliance_row = ProwlerThreatScoreKubernetesModel(
Provider=compliance.Provider.lower(),
Description=compliance.Description,
Context="",
Namespace="",
AssessmentDate=str(timestamp),
Requirements_Id=requirement.Id,
Requirements_Description=requirement.Description,
Requirements_Attributes_Title=attribute.Title,
Requirements_Attributes_Section=attribute.Section,
Requirements_Attributes_SubSection=attribute.SubSection,
Requirements_Attributes_AttributeDescription=attribute.AttributeDescription,
Requirements_Attributes_AdditionalInformation=attribute.AdditionalInformation,
Requirements_Attributes_LevelOfRisk=attribute.LevelOfRisk,
Requirements_Attributes_Weight=attribute.Weight,
Status="MANUAL",
StatusExtended="Manual check",
ResourceId="manual_check",
ResourceName="Manual check",
CheckId="manual",
Muted=False,
Framework=compliance.Framework,
Name=compliance.Name,
)
self._data.append(compliance_row)
+3 -1
View File
@@ -314,7 +314,9 @@ class Finding(BaseModel):
)
output_data["resource_uid"] = getattr(check_output, "resource_name", "")
# For IaC, resource_line_range only exists on CheckReportIAC, not on Finding objects
output_data["region"] = getattr(check_output, "region", "global")
output_data["region"] = getattr(
check_output, "resource_line_range", "file"
)
output_data["resource_line_range"] = getattr(
check_output, "resource_line_range", ""
)
+1 -1
View File
@@ -241,7 +241,7 @@ class HTML(Output):
<th scope="col">Status</th>
<th scope="col">Severity</th>
<th scope="col">Service Name</th>
<th scope="col">Region</th>
<th scope="col">{"Line Range" if provider.type == "iac" else "Region"}</th>
<th style="width:20%" scope="col">Check ID</th>
<th style="width:20%" scope="col">Check Title</th>
<th scope="col">Resource ID</th>
+1 -1
View File
@@ -325,7 +325,7 @@ class Scan:
resource_name=report.resource_name,
resource_details=report.resource_details,
resource_tags={}, # IaC doesn't have resource tags
region=report.region, # IaC region is the branch name
region="global", # IaC doesn't have regions
compliance={}, # IaC doesn't have compliance mappings yet
raw=report.resource, # The raw finding dict
)
+1 -1
View File
@@ -64,7 +64,7 @@ def open_file(input_file: str, mode: str = "r") -> TextIOWrapper:
except OSError as os_error:
if os_error.strerror == "Too many open files":
logger.critical(
"Ooops! You reached your user session maximum open files. To solve this issue, increase the shell session limit by running this command `ulimit -n 4096`. For more info visit https://docs.prowler.com/troubleshooting/"
"Ooops! You reached your user session maximum open files. To solve this issue, increase the shell session limit by running this command `ulimit -n 4096`. For more info visit https://docs.prowler.cloud/en/latest/troubleshooting/"
)
else:
logger.critical(
@@ -1454,23 +1454,6 @@
]
}
},
"bedrock-agentcore": {
"regions": {
"aws": [
"ap-northeast-1",
"ap-south-1",
"ap-southeast-1",
"ap-southeast-2",
"eu-central-1",
"eu-west-1",
"us-east-1",
"us-east-2",
"us-west-2"
],
"aws-cn": [],
"aws-us-gov": []
}
},
"bedrock-data-automation": {
"regions": {
"aws": [
@@ -6990,6 +6973,21 @@
"aws-us-gov": []
}
},
"lookoutvision": {
"regions": {
"aws": [
"ap-northeast-1",
"ap-northeast-2",
"eu-central-1",
"eu-west-1",
"us-east-1",
"us-east-2",
"us-west-2"
],
"aws-cn": [],
"aws-us-gov": []
}
},
"lumberyard": {
"regions": {
"aws": [
@@ -7901,7 +7899,6 @@
"ap-southeast-3",
"ap-southeast-4",
"ap-southeast-5",
"ap-southeast-6",
"ap-southeast-7",
"ca-central-1",
"ca-west-1",
@@ -297,7 +297,7 @@ def create_output(resources: list, provider: AwsProvider, args):
csv_file.close()
print(
f"\n{Fore.YELLOW}WARNING: Only resources that have or have had tags will appear (except for IAM and S3).\nSee more in https://docs.prowler.com/user-guide/cli/tutorials/quick-inventory/#objections{Style.RESET_ALL}"
f"\n{Fore.YELLOW}WARNING: Only resources that have or have had tags will appear (except for IAM and S3).\nSee more in https://docs.prowler.cloud/en/latest/tutorials/quick-inventory/#objections{Style.RESET_ALL}"
)
print("\nMore details in files:")
print(f" - CSV: {args.output_directory}/{output_file + csv_file_suffix}")
@@ -256,7 +256,7 @@ class SecurityHub:
security_hub_client.list_enabled_products_for_import()
):
logger.warning(
f"Security Hub is enabled in {region} but Prowler integration does not accept findings. More info: https://docs.prowler.com/user-guide/providers/aws/securityhub#aws-security-hub-integration-with-prowler"
f"Security Hub is enabled in {region} but Prowler integration does not accept findings. More info: https://docs.prowler.cloud/en/latest/tutorials/aws/securityhub/"
)
return region, None
else:
@@ -32,7 +32,9 @@
"Url": "https://hub.prowler.com/check/awslambda_function_using_supported_runtimes"
}
},
"Categories": [],
"Categories": [
"container-security"
],
"DependsOn": [],
"RelatedTo": [],
"Notes": ""
@@ -1,40 +1,29 @@
{
"Provider": "aws",
"CheckID": "documentdb_cluster_backup_enabled",
"CheckTitle": "DocumentDB cluster has automated backups enabled with retention period of at least 7 days",
"CheckType": [
"Software and Configuration Checks/AWS Security Best Practices",
"Industry and Regulatory Standards/AWS Foundational Security Best Practices",
"Effects/Data Destruction"
],
"CheckTitle": "Check if DocumentDB Clusters have backup enabled.",
"CheckType": [],
"ServiceName": "documentdb",
"SubServiceName": "",
"ResourceIdTemplate": "",
"ResourceIdTemplate": "arn:aws:rds:region:account-id:db-cluster",
"Severity": "medium",
"ResourceType": "AwsRdsDbCluster",
"Description": "**Amazon DocumentDB clusters** are evaluated for **automated backups** and an adequate **backup retention period**. Clusters should have `backup_retention_period` set to at least the configured minimum (default `7` days). Values of `0` indicate backups are disabled; values below the threshold are considered insufficient.",
"Risk": "Without adequate backups, clusters can't be reliably restored. Accidental deletes, logical corruption, or ransomware may cause irreversible data loss once a short retention window expires, leading to prolonged outages, missed RPO/RTO, and limited ability to roll back malicious or erroneous changes.",
"RelatedUrl": "",
"AdditionalURLs": [
"https://docs.amazonaws.cn/en_us/documentdb/latest/developerguide/what-is.html",
"https://www.trendmicro.com/cloudoneconformity-staging/knowledge-base/aws/DocumentDB/sufficient-backup-retention-period.html#",
"https://docs.aws.amazon.com/systems-manager-automation-runbooks/latest/userguide/aws-enabledocdbclusterbackupretentionperiod.html"
],
"Description": "Check if DocumentDB Clusters have backup enabled.",
"Risk": "Ensure that your Amazon DocumentDB database clusters have set a minimum backup retention period in order to achieve compliance requirements in your organization.",
"RelatedUrl": "https://docs.aws.amazon.com/securityhub/latest/userguide/documentdb-controls.html#documentdb-2",
"Remediation": {
"Code": {
"CLI": "aws docdb modify-db-cluster --db-cluster-identifier <DB_CLUSTER_ID> --backup-retention-period 7 --apply-immediately",
"NativeIaC": "```yaml\n# CloudFormation: Set DocumentDB backup retention to at least 7 days\nResources:\n <example_resource_name>:\n Type: AWS::DocDB::DBCluster\n Properties:\n BackupRetentionPeriod: 7 # CRITICAL: enables automated backups and sets retention to >=7 days\n```",
"Other": "1. Open the Amazon DocumentDB console\n2. Go to Clusters and select <example_resource_id>\n3. Click Modify\n4. Set Backup retention period to 7 (or higher)\n5. Check Apply immediately\n6. Click Continue and then Modify cluster",
"Terraform": "```hcl\n# Terraform: Ensure DocumentDB backup retention is at least 7 days\nresource \"aws_docdb_cluster\" \"<example_resource_name>\" {\n cluster_identifier = \"<example_resource_id>\"\n backup_retention_period = 7 # CRITICAL: enables automated backups and sets retention to >=7 days\n}\n```"
"CLI": "aws docdb modify-db-cluster --region <REGION> --db-cluster-identifier <DB_CLUSTER_ID> --backup-retention-period 7 --apply-immediately",
"NativeIaC": "https://www.trendmicro.com/cloudoneconformity-staging/knowledge-base/aws/DocumentDB/sufficient-backup-retention-period.html#",
"Other": "https://www.trendmicro.com/cloudoneconformity-staging/knowledge-base/aws/DocumentDB/sufficient-backup-retention-period.html#",
"Terraform": "https://www.trendmicro.com/cloudoneconformity-staging/knowledge-base/aws/DocumentDB/sufficient-backup-retention-period.html#"
},
"Recommendation": {
"Text": "Enable **automated backups** and set retention to meet RPO/RTO (typically `7-35` days).\n- Regularly test point-in-time restores\n- Apply **least privilege** to backup/snapshot management\n- Protect backup artifacts and define stable backup windows\n- Include restores in a tested **disaster recovery** plan",
"Url": "https://hub.prowler.com/check/documentdb_cluster_backup_enabled"
"Text": "Enable automated backup for production data. Define a retention period and periodically test backup restoration. A Disaster Recovery process should be in place to govern Data Protection approach.",
"Url": "https://docs.aws.amazon.com/securityhub/latest/userguide/documentdb-controls.html#documentdb-2"
}
},
"Categories": [
"resilience"
],
"Categories": [],
"DependsOn": [],
"RelatedTo": [],
"Notes": ""
@@ -1,39 +1,29 @@
{
"Provider": "aws",
"CheckID": "documentdb_cluster_cloudwatch_log_export",
"CheckTitle": "DocumentDB cluster exports audit and profiler logs to CloudWatch Logs",
"CheckType": [
"Software and Configuration Checks/AWS Security Best Practices/Runtime Behavior Analysis",
"Software and Configuration Checks/Industry and Regulatory Standards/AWS Foundational Security Best Practices"
],
"CheckTitle": "Check if DocumentDB clusters are using the log export feature.",
"CheckType": [],
"ServiceName": "documentdb",
"SubServiceName": "",
"ResourceIdTemplate": "",
"ResourceIdTemplate": "arn:partition:service:region:account-id:resource-id",
"Severity": "medium",
"ResourceType": "AwsRdsDbCluster",
"Description": "Amazon DocumentDB clusters are evaluated for exporting `audit` and `profiler` logs to **CloudWatch Logs**.\nClusters missing one or both log types are identified as lacking complete log export configuration.",
"Risk": "Missing **audit** and/or **profiler** exports reduces observability of authentication, authorization, and data definition activity.\nAttacks like brute-force logins, privilege abuse, or destructive schema changes can go unnoticed, degrading **confidentiality** and **integrity** and delaying incident response.",
"RelatedUrl": "",
"AdditionalURLs": [
"https://docs.aws.amazon.com/securityhub/latest/userguide/documentdb-controls.html#documentdb-4",
"https://www.trendmicro.com/cloudoneconformity/knowledge-base/aws/DocumentDB/enable-profiler.html",
"https://docs.aws.amazon.com/cli/latest/reference/docdb/create-db-cluster.html"
],
"Description": "Check if DocumentDB clusters are using the log export feature.",
"Risk": "Ensure that all your Amazon DocumentDB clusters are using the Log Exports feature in order to publish audit logs directly to CloudWatch Logs. The events recorded by Log Exports include events such as successful and failed authentication attempts, creating indexes, or dropping collections in DocumentDB databases.",
"RelatedUrl": "https://docs.aws.amazon.com/securityhub/latest/userguide/documentdb-controls.html#documentdb-4",
"Remediation": {
"Code": {
"CLI": "aws docdb modify-db-cluster --db-cluster-identifier <DB_CLUSTER_ID> --cloudwatch-logs-export-configuration '{\"EnableLogTypes\":[\"audit\",\"profiler\"]}' --apply-immediately",
"NativeIaC": "```yaml\n# CloudFormation: enable DocumentDB log exports\nResources:\n <example_resource_name>:\n Type: AWS::DocDB::DBCluster\n Properties:\n EnableCloudwatchLogsExports:\n - audit # Critical: export audit logs to CloudWatch Logs\n - profiler # Critical: export profiler logs to CloudWatch Logs\n```",
"Other": "1. In AWS Console, go to Amazon DocumentDB > Clusters\n2. Select the cluster and choose Actions > Modify\n3. In Log exports, check Audit and Profiler\n4. Check Apply immediately and click Modify cluster",
"Terraform": "```hcl\n# Enable DocumentDB log exports\nresource \"aws_docdb_cluster\" \"<example_resource_name>\" {\n enabled_cloudwatch_logs_exports = [\"audit\", \"profiler\"] # Critical: export both logs to CloudWatch Logs\n}\n```"
"CLI": "aws docdb modify-db-cluster --region <REGION> --db-cluster-identifier <DB_CLUSTER_ID> --db-cluster-parameter-group-name <DB_CLUSTER_PARAMETER_GROUP_NAME> --cloudwatch-logs-export-configuration '{EnableLogTypes:[profiler]}' --apply-immediately",
"NativeIaC": "https://www.trendmicro.com/cloudoneconformity-staging/knowledge-base/aws/DocumentDB/enable-profiler.html",
"Other": "https://www.trendmicro.com/cloudoneconformity-staging/knowledge-base/aws/DocumentDB/enable-profiler.html",
"Terraform": "https://www.trendmicro.com/cloudoneconformity-staging/knowledge-base/aws/DocumentDB/enable-profiler.html"
},
"Recommendation": {
"Text": "Enable export of both `audit` and `profiler` logs to **CloudWatch Logs** for all clusters and centralize analysis.\nApply **least privilege** to log access, define retention and immutability, integrate with alerting, and use **separation of duties** to protect and regularly review logs for **defense in depth**.",
"Url": "https://hub.prowler.com/check/documentdb_cluster_cloudwatch_log_export"
"Text": "Enabled DocumentDB Log export functionality to analyze, monitor, and archive auditing events for security and compliance requirements.",
"Url": "https://docs.aws.amazon.com/securityhub/latest/userguide/documentdb-controls.html#documentdb-4"
}
},
"Categories": [
"logging"
],
"Categories": [],
"DependsOn": [],
"RelatedTo": [],
"Notes": ""
@@ -1,40 +1,29 @@
{
"Provider": "aws",
"CheckID": "documentdb_cluster_deletion_protection",
"CheckTitle": "DocumentDB cluster has deletion protection enabled",
"CheckType": [
"Software and Configuration Checks/AWS Security Best Practices",
"Software and Configuration Checks/Industry and Regulatory Standards/AWS Foundational Security Best Practices"
],
"CheckTitle": "Check if DocumentDB Clusters has deletion protection enabled.",
"CheckType": [],
"ServiceName": "documentdb",
"SubServiceName": "",
"ResourceIdTemplate": "",
"ResourceIdTemplate": "arn:aws:rds:region:account-id:db-cluster",
"Severity": "medium",
"ResourceType": "AwsRdsDbCluster",
"Description": "**Amazon DocumentDB clusters** are evaluated for the `deletion_protection` setting on the cluster configuration.\n\nThe finding highlights clusters where this protection is not enabled.",
"Risk": "Without **deletion protection**, clusters can be deleted by mistake or misuse, causing sudden outage and loss of recovery points, impacting **availability** and **data integrity**.\n\nCompromised accounts or faulty automation can remove databases or skip final snapshots, hindering restoration.",
"RelatedUrl": "",
"AdditionalURLs": [
"https://support.icompaas.com/support/solutions/articles/62000233689-ensure-documentdb-clusters-has-deletion-protection-enabled",
"https://www.trendmicro.com/cloudoneconformity/knowledge-base/aws/DocumentDB/deletion-protection.html",
"https://docs.aws.amazon.com/documentdb/latest/developerguide/db-cluster-delete.html",
"https://docs.aws.amazon.com/securityhub/latest/userguide/documentdb-controls.html#documentdb-5"
],
"Description": "Check if DocumentDB Clusters has deletion protection enabled.",
"Risk": "Enabling cluster deletion protection offers an additional layer of protection against accidental database deletion or deletion by an unauthorized user. A DocumentDB cluster can't be deleted while deletion protection is enabled. You must first disable deletion protection before a delete request can succeed.",
"RelatedUrl": "https://docs.aws.amazon.com/securityhub/latest/userguide/documentdb-controls.html#documentdb-5",
"Remediation": {
"Code": {
"CLI": "aws docdb modify-db-cluster --db-cluster-identifier <DB_CLUSTER_ID> --deletion-protection --apply-immediately",
"NativeIaC": "```yaml\n# CloudFormation: Enable deletion protection on a DocumentDB cluster\nResources:\n <example_resource_name>:\n Type: AWS::DocDB::DBCluster\n Properties:\n MasterUsername: \"<MASTER_USERNAME>\"\n MasterUserPassword: \"<MASTER_USER_PASSWORD>\"\n DeletionProtection: true # CRITICAL: Prevents cluster deletion until disabled\n```",
"Other": "1. In the AWS Console, go to Amazon DocumentDB > Clusters\n2. Select the target cluster and click Modify\n3. Enable Deletion protection\n4. Check Apply immediately and click Save changes",
"Terraform": "```hcl\n# Terraform: Enable deletion protection on a DocumentDB cluster\nresource \"aws_docdb_cluster\" \"<example_resource_name>\" {\n master_username = \"<MASTER_USERNAME>\"\n master_password = \"<MASTER_USER_PASSWORD>\"\n deletion_protection = true # CRITICAL: Prevents cluster deletion until disabled\n}\n```"
"CLI": "aws aws docdb modify-db-cluster --region <REGION> --db-cluster-identifier <DB_CLUSTER_ID> --deletion-protection --apply-immediately",
"NativeIaC": "https://www.trendmicro.com/cloudoneconformity-staging/knowledge-base/aws/DocumentDB/deletion-protection.html#",
"Other": "https://www.trendmicro.com/cloudoneconformity-staging/knowledge-base/aws/DocumentDB/deletion-protection.html#",
"Terraform": "https://www.trendmicro.com/cloudoneconformity-staging/knowledge-base/aws/DocumentDB/deletion-protection.html#"
},
"Recommendation": {
"Text": "Enable **deletion protection** on all non-ephemeral clusters, prioritizing production.\n\nEnforce **least privilege** for delete and modify actions, require change control to toggle protection, and implement **defense in depth** with automation that continuously enforces this setting. *Before decommissioning*, take a final snapshot.",
"Url": "https://hub.prowler.com/check/documentdb_cluster_deletion_protection"
"Text": "Enable deletion protection for production DocumentDB Clusters.",
"Url": "https://docs.aws.amazon.com/securityhub/latest/userguide/documentdb-controls.html#documentdb-5"
}
},
"Categories": [
"resilience"
],
"Categories": [],
"DependsOn": [],
"RelatedTo": [],
"Notes": ""
@@ -1,36 +1,30 @@
{
"Provider": "aws",
"CheckID": "documentdb_cluster_multi_az_enabled",
"CheckTitle": "DocumentDB cluster has Multi-AZ enabled",
"CheckType": [
"Software and Configuration Checks/AWS Security Best Practices"
],
"CheckTitle": "Ensure DocumentDB Cluster have Multi-AZ enabled.",
"CheckType": [],
"ServiceName": "documentdb",
"SubServiceName": "",
"ResourceIdTemplate": "",
"ResourceIdTemplate": "arn:aws:rds:region:account-id:db-cluster",
"Severity": "medium",
"ResourceType": "AwsRdsDbCluster",
"Description": "**Amazon DocumentDB clusters** with **Multi-AZ** (`multi_az`) indicate deployment of a primary and one or more replicas across Availability Zones.",
"Risk": "Without Multi-AZ, the cluster depends on a single AZ/instance. An AZ or node failure-or maintenance-can stop reads and writes, causing downtime, timeouts, and SLA breaches. Availability degrades, RTO rises, and applications may experience failed or retried transactions until replacement capacity is created.",
"RelatedUrl": "",
"AdditionalURLs": [
"https://docs.aws.amazon.com/documentdb/latest/developerguide/failover.html",
"https://support.icompaas.com/support/solutions/articles/62000233690-ensure-documentdb-cluster-have-multi-az-enabled"
],
"Description": "Ensure DocumentDB Cluster have Multi-AZ enabled.",
"Risk": "Ensure that your Amazon DocumentDB Clusters are using Multi-AZ deployment configurations to provide High Availability (HA) through automatic failover to standby replicas in the event of a failure such as an Availability Zone (AZ) outage, an internal hardware or network outage, a software failure or in case of a planned maintenance session.",
"RelatedUrl": "https://docs.aws.amazon.com/documentdb/latest/developerguide/failover.html",
"Remediation": {
"Code": {
"CLI": "aws docdb create-db-instance --db-instance-identifier <example_resource_id> --db-cluster-identifier <example_resource_id> --db-instance-class <INSTANCE_CLASS> --engine docdb --availability-zone <OTHER_AZ>",
"NativeIaC": "```yaml\n# CloudFormation: add a replica to enable Multi-AZ for an existing DocumentDB cluster\nResources:\n DocDBReplica:\n Type: AWS::DocDB::DBInstance\n Properties:\n DBClusterIdentifier: \"<example_resource_id>\" # CRITICAL: adds a new instance to the cluster to achieve Multi-AZ\n DBInstanceClass: \"<INSTANCE_CLASS>\"\n AvailabilityZone: \"<OTHER_AZ>\" # CRITICAL: place in a different AZ to provide Multi-AZ failover\n```",
"Other": "1. In the AWS Console, go to Amazon DocumentDB and open your cluster\n2. Click Create instance\n3. Set Instance class and choose an Availability Zone different from the primary\n4. Click Create to add the replica\n5. Verify the cluster now shows Multi-AZ enabled",
"Terraform": "```hcl\n# Add a replica to enable Multi-AZ for an existing DocumentDB cluster\nresource \"aws_docdb_cluster_instance\" \"<example_resource_name>\" {\n cluster_identifier = \"<example_resource_id>\" # CRITICAL: adds a new instance to the cluster to achieve Multi-AZ\n instance_class = \"<INSTANCE_CLASS>\"\n availability_zone = \"<OTHER_AZ>\" # CRITICAL: different AZ ensures Multi-AZ failover\n}\n```"
"CLI": "",
"NativeIaC": "",
"Other": "",
"Terraform": ""
},
"Recommendation": {
"Text": "Enable **Multi-AZ** for DocumentDB and distribute instances across distinct AZs.\n- Maintain at least one replica\n- Set promotion priorities to guide failover\n- Test failover regularly and use resilient client retries\n\nThis builds **fault tolerance** and preserves service availability.",
"Url": "https://hub.prowler.com/check/documentdb_cluster_multi_az_enabled"
"Text": "Enable Multi-AZ for all DocumentDB Clusters.",
"Url": "https://docs.aws.amazon.com/documentdb/latest/developerguide/failover.html"
}
},
"Categories": [
"resilience"
"redundancy"
],
"DependsOn": [],
"RelatedTo": [],
@@ -1,36 +1,26 @@
{
"Provider": "aws",
"CheckID": "documentdb_cluster_public_snapshot",
"CheckTitle": "DocumentDB manual cluster snapshot is not shared publicly",
"CheckType": [
"Software and Configuration Checks/AWS Security Best Practices",
"Software and Configuration Checks/Industry and Regulatory Standards/AWS Foundational Security Best Practices",
"Effects/Data Exposure",
"TTPs/Initial Access"
],
"CheckTitle": "Check if DocumentDB manual cluster snapshot is public.",
"CheckType": [],
"ServiceName": "documentdb",
"SubServiceName": "",
"ResourceIdTemplate": "",
"ResourceIdTemplate": "arn:partition:service:region:account-id:resource-id",
"Severity": "critical",
"ResourceType": "AwsRdsDbClusterSnapshot",
"Description": "**Amazon DocumentDB** manual cluster snapshot visibility is evaluated to detect snapshots marked as **public** instead of limited to specified AWS accounts.",
"Risk": "**Public snapshots** weaken **confidentiality**: any AWS account can restore and read database contents, enabling data exfiltration.\n\nThey also aid **lateral movement** by revealing embedded secrets/config and reduce accountability when restores occur outside your account.",
"RelatedUrl": "",
"AdditionalURLs": [
"https://docs.aws.amazon.com/documentdb/latest/developerguide/backup_restore-share_cluster_snapshots.html#backup_restore-share_snapshots",
"https://docs.aws.amazon.com/securityhub/latest/userguide/documentdb-controls.html#documentdb-3",
"https://docs.aws.amazon.com/config/latest/developerguide/docdb-cluster-snapshot-public-prohibited.html"
],
"Description": "Check if DocumentDB manual cluster snapshot is public.",
"Risk": "If you share an unencrypted manual snapshot as public, the snapshot is available to all AWS accounts. Public snapshots may result in unintended data exposure.",
"RelatedUrl": "https://docs.aws.amazon.com/config/latest/developerguide/docdb-cluster-snapshot-public-prohibited.html",
"Remediation": {
"Code": {
"CLI": "aws docdb modify-db-cluster-snapshot-attribute --db-cluster-snapshot-identifier <snapshot_id> --attribute-name restore --values-to-remove all",
"CLI": "aws docdb modify-db-snapshot-attribute --db-snapshot-identifier <snapshot_id> --attribute-name restore --values-to-remove all",
"NativeIaC": "",
"Other": "1. Open the Amazon DocumentDB console and go to Snapshots\n2. Select the public manual cluster snapshot\n3. Click Actions > Share\n4. Set DB snapshot visibility to Private (remove \"all\" if listed)\n5. Click Save",
"Other": "https://docs.aws.amazon.com/securityhub/latest/userguide/documentdb-controls.html#documentdb-3",
"Terraform": ""
},
"Recommendation": {
"Text": "Keep snapshot visibility `Private` and share only with trusted accounts under **least privilege**. Prefer **CMEK encryption** to enforce key-based access and prevent public sharing. Periodically review sharing lists, restrict IAM permissions that alter visibility, and monitor for exposure as **defense in depth**.",
"Url": "https://hub.prowler.com/check/documentdb_cluster_public_snapshot"
"Text": "To remove public access from a manual snapshot, follow the Sharing a snapshot tutorial.",
"Url": "https://docs.aws.amazon.com/documentdb/latest/developerguide/backup_restore-share_cluster_snapshots.html#backup_restore-share_snapshots"
}
},
"Categories": [

Some files were not shown because too many files have changed in this diff Show More