mirror of
https://github.com/prowler-cloud/prowler.git
synced 2026-05-18 02:03:26 +00:00
Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 3066d82863 | |||
| 969ca8863a | |||
| 03c6f98db4 | |||
| 8ebefb8aa1 | |||
| c3694fdc5b | |||
| df10bc0c4c | |||
| e694b0f634 | |||
| 81e3f87003 | |||
| 7ffe2aeec9 | |||
| 672aa6eb2f | |||
| 2e999f55f9 | |||
| 18998b8867 | |||
| ff4a186df6 | |||
| b8dab5e0ed | |||
| 0b3142f7a8 | |||
| f5dc0c9ee0 | |||
| a230809095 | |||
| e6d1b5639b |
@@ -35,8 +35,6 @@ POSTGRES_DB=prowler_db
|
||||
# POSTGRES_REPLICA_USER=prowler
|
||||
# POSTGRES_REPLICA_PASSWORD=postgres
|
||||
# POSTGRES_REPLICA_DB=prowler_db
|
||||
# POSTGRES_REPLICA_MAX_ATTEMPTS=3
|
||||
# POSTGRES_REPLICA_RETRY_BASE_DELAY=0.5
|
||||
|
||||
# Celery-Prowler task settings
|
||||
TASK_RETRY_DELAY_SECONDS=0.1
|
||||
|
||||
@@ -22,8 +22,8 @@ inputs:
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
- name: Replace @master with current branch in pyproject.toml (prowler repo only)
|
||||
if: github.event_name == 'pull_request' && github.base_ref == 'master' && github.repository == 'prowler-cloud/prowler'
|
||||
- name: Replace @master with current branch in pyproject.toml
|
||||
if: github.event_name == 'pull_request' && github.base_ref == 'master'
|
||||
shell: bash
|
||||
working-directory: ${{ inputs.working-directory }}
|
||||
run: |
|
||||
@@ -37,8 +37,8 @@ runs:
|
||||
python -m pip install --upgrade pip
|
||||
pipx install poetry==${{ inputs.poetry-version }}
|
||||
|
||||
- name: Update poetry.lock with latest Prowler commit
|
||||
if: github.repository_owner == 'prowler-cloud' && github.repository != 'prowler-cloud/prowler'
|
||||
- name: Update SDK resolved_reference to latest commit
|
||||
if: github.event_name == 'push' && github.ref == 'refs/heads/master'
|
||||
shell: bash
|
||||
working-directory: ${{ inputs.working-directory }}
|
||||
run: |
|
||||
@@ -50,21 +50,7 @@ runs:
|
||||
echo "Updated resolved_reference:"
|
||||
grep -A2 -B2 "resolved_reference" poetry.lock
|
||||
|
||||
- name: Update SDK resolved_reference to latest commit (prowler repo on push)
|
||||
if: github.event_name == 'push' && github.ref == 'refs/heads/master' && github.repository == 'prowler-cloud/prowler'
|
||||
shell: bash
|
||||
working-directory: ${{ inputs.working-directory }}
|
||||
run: |
|
||||
LATEST_COMMIT=$(curl -s "https://api.github.com/repos/prowler-cloud/prowler/commits/master" | jq -r '.sha')
|
||||
echo "Latest commit hash: $LATEST_COMMIT"
|
||||
sed -i '/url = "https:\/\/github\.com\/prowler-cloud\/prowler\.git"/,/resolved_reference = / {
|
||||
s/resolved_reference = "[a-f0-9]\{40\}"/resolved_reference = "'"$LATEST_COMMIT"'"/
|
||||
}' poetry.lock
|
||||
echo "Updated resolved_reference:"
|
||||
grep -A2 -B2 "resolved_reference" poetry.lock
|
||||
|
||||
- name: Update poetry.lock (prowler repo only)
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
- name: Update poetry.lock
|
||||
shell: bash
|
||||
working-directory: ${{ inputs.working-directory }}
|
||||
run: poetry lock
|
||||
@@ -83,11 +69,3 @@ runs:
|
||||
run: |
|
||||
poetry install --no-root
|
||||
poetry run pip list
|
||||
|
||||
- name: Update Prowler Cloud API Client
|
||||
if: github.repository_owner == 'prowler-cloud' && github.repository != 'prowler-cloud/prowler'
|
||||
shell: bash
|
||||
working-directory: ${{ inputs.working-directory }}
|
||||
run: |
|
||||
poetry remove prowler-cloud-api-client
|
||||
poetry add ./prowler-cloud-api-client
|
||||
|
||||
@@ -1,198 +0,0 @@
|
||||
# Slack Notification Action
|
||||
|
||||
A generic and flexible GitHub composite action for sending Slack notifications using JSON template files. Supports both standalone messages and message updates, with automatic status detection.
|
||||
|
||||
## Features
|
||||
|
||||
- **Template-based**: All messages use JSON template files for consistency
|
||||
- **Automatic status detection**: Pass `step-outcome` to auto-calculate success/failure
|
||||
- **Message updates**: Supports updating existing messages (using `chat.update`)
|
||||
- **Simple API**: Clean and minimal interface
|
||||
- **Reusable**: Use across all workflows and scenarios
|
||||
- **Maintainable**: Centralized message templates
|
||||
|
||||
## Use Cases
|
||||
|
||||
1. **Container releases**: Track push start and completion with automatic status
|
||||
2. **Deployments**: Track deployment progress with rich Block Kit formatting
|
||||
3. **Custom notifications**: Any scenario where you need to notify Slack
|
||||
|
||||
## Inputs
|
||||
|
||||
| Input | Description | Required | Default |
|
||||
|-------|-------------|----------|---------|
|
||||
| `slack-bot-token` | Slack bot token for authentication | Yes | - |
|
||||
| `payload-file-path` | Path to JSON file with the Slack message payload | Yes | - |
|
||||
| `update-ts` | Message timestamp to update (leave empty for new messages) | No | `''` |
|
||||
| `step-outcome` | Step outcome for automatic status detection (sets STATUS_EMOJI and STATUS_TEXT env vars) | No | `''` |
|
||||
|
||||
## Outputs
|
||||
|
||||
| Output | Description |
|
||||
|--------|-------------|
|
||||
| `ts` | Timestamp of the Slack message (use for updates) |
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Example 1: Container Release with Automatic Status Detection
|
||||
|
||||
Using JSON template files with automatic status detection:
|
||||
|
||||
```yaml
|
||||
# Send start notification
|
||||
- name: Notify container push started
|
||||
if: github.event_name == 'release'
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_CHANNEL_ID }}
|
||||
COMPONENT: API
|
||||
RELEASE_TAG: ${{ env.RELEASE_TAG }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-started.json"
|
||||
|
||||
# Do the work
|
||||
- name: Build and push container
|
||||
if: github.event_name == 'release'
|
||||
id: container-push
|
||||
uses: docker/build-push-action@...
|
||||
with:
|
||||
push: true
|
||||
tags: ...
|
||||
|
||||
# Send completion notification with automatic status detection
|
||||
- name: Notify container push completed
|
||||
if: github.event_name == 'release' && always()
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_CHANNEL_ID }}
|
||||
COMPONENT: API
|
||||
RELEASE_TAG: ${{ env.RELEASE_TAG }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-completed.json"
|
||||
step-outcome: ${{ steps.container-push.outcome }}
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- No status calculation needed in workflow
|
||||
- Reusable template files
|
||||
- Clean and concise
|
||||
- Automatic `STATUS_EMOJI` and `STATUS_TEXT` env vars set by action
|
||||
- Consistent message format across all workflows
|
||||
|
||||
### Example 2: Deployment with Message Update Pattern
|
||||
|
||||
```yaml
|
||||
# Send initial deployment message
|
||||
- name: Notify deployment started
|
||||
id: slack-start
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_CHANNEL_ID }}
|
||||
COMPONENT: API
|
||||
ENVIRONMENT: PRODUCTION
|
||||
COMMIT_HASH: ${{ github.sha }}
|
||||
VERSION_DEPLOYED: latest
|
||||
GITHUB_ACTOR: ${{ github.actor }}
|
||||
GITHUB_WORKFLOW: ${{ github.workflow }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/deployment-started.json"
|
||||
|
||||
# Run deployment
|
||||
- name: Deploy
|
||||
id: deploy
|
||||
run: terraform apply -auto-approve
|
||||
|
||||
# Determine additional status variables
|
||||
- name: Determine deployment status
|
||||
if: always()
|
||||
id: deploy-status
|
||||
run: |
|
||||
if [[ "${{ steps.deploy.outcome }}" == "success" ]]; then
|
||||
echo "STATUS_COLOR=28a745" >> $GITHUB_ENV
|
||||
echo "STATUS=Completed" >> $GITHUB_ENV
|
||||
else
|
||||
echo "STATUS_COLOR=fc3434" >> $GITHUB_ENV
|
||||
echo "STATUS=Failed" >> $GITHUB_ENV
|
||||
fi
|
||||
|
||||
# Update the same message with final status
|
||||
- name: Update deployment notification
|
||||
if: always()
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_CHANNEL_ID }}
|
||||
MESSAGE_TS: ${{ steps.slack-start.outputs.ts }}
|
||||
COMPONENT: API
|
||||
ENVIRONMENT: PRODUCTION
|
||||
COMMIT_HASH: ${{ github.sha }}
|
||||
VERSION_DEPLOYED: latest
|
||||
GITHUB_ACTOR: ${{ github.actor }}
|
||||
GITHUB_WORKFLOW: ${{ github.workflow }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
STATUS: ${{ env.STATUS }}
|
||||
STATUS_COLOR: ${{ env.STATUS_COLOR }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
update-ts: ${{ steps.slack-start.outputs.ts }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/deployment-completed.json"
|
||||
step-outcome: ${{ steps.deploy.outcome }}
|
||||
```
|
||||
|
||||
## Automatic Status Detection
|
||||
|
||||
When you provide `step-outcome` input, the action automatically sets these environment variables:
|
||||
|
||||
| Outcome | STATUS_EMOJI | STATUS_TEXT |
|
||||
|---------|--------------|-------------|
|
||||
| success | `[✓]` | `completed successfully!` |
|
||||
| failure | `[✗]` | `failed` |
|
||||
|
||||
These variables are then available in your payload template files.
|
||||
|
||||
## Template File Format
|
||||
|
||||
All template files must be valid JSON and support environment variable substitution. Example:
|
||||
|
||||
```json
|
||||
{
|
||||
"channel": "$SLACK_CHANNEL_ID",
|
||||
"text": "$STATUS_EMOJI $COMPONENT container release $RELEASE_TAG push $STATUS_TEXT <$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID|View run>"
|
||||
}
|
||||
```
|
||||
|
||||
See available templates in [`.github/scripts/slack-messages/`](../../scripts/slack-messages/).
|
||||
|
||||
## Requirements
|
||||
|
||||
- Slack Bot Token with scopes: `chat:write`, `chat:write.public`
|
||||
- Slack Channel ID where messages will be posted
|
||||
- JSON template files for your messages
|
||||
|
||||
## Benefits
|
||||
|
||||
- **Consistency**: All notifications use standardized templates
|
||||
- **Automatic status handling**: No need to calculate success/failure in workflows
|
||||
- **Clean workflows**: Minimal boilerplate code
|
||||
- **Reusable templates**: One template for all components
|
||||
- **Easy to maintain**: Change template once, applies everywhere
|
||||
- **Version controlled**: All message formats in git
|
||||
|
||||
## Related Resources
|
||||
|
||||
- [Slack Block Kit Builder](https://app.slack.com/block-kit-builder)
|
||||
- [Slack API Method Documentation](https://docs.slack.dev/tools/slack-github-action/sending-techniques/sending-data-slack-api-method/)
|
||||
- [Message templates documentation](../../scripts/slack-messages/README.md)
|
||||
@@ -1,74 +0,0 @@
|
||||
name: 'Slack Notification'
|
||||
description: 'Generic action to send Slack notifications with optional message updates and automatic status detection'
|
||||
inputs:
|
||||
slack-bot-token:
|
||||
description: 'Slack bot token for authentication'
|
||||
required: true
|
||||
payload-file-path:
|
||||
description: 'Path to JSON file with the Slack message payload'
|
||||
required: true
|
||||
update-ts:
|
||||
description: 'Message timestamp to update (only for updates, leave empty for new messages)'
|
||||
required: false
|
||||
default: ''
|
||||
step-outcome:
|
||||
description: 'Outcome of a step to determine status (success/failure) - automatically sets STATUS_TEXT and STATUS_COLOR env vars'
|
||||
required: false
|
||||
default: ''
|
||||
outputs:
|
||||
ts:
|
||||
description: 'Timestamp of the Slack message'
|
||||
value: ${{ steps.slack-notification.outputs.ts }}
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
- name: Determine status
|
||||
id: status
|
||||
shell: bash
|
||||
run: |
|
||||
if [[ "${{ inputs.step-outcome }}" == "success" ]]; then
|
||||
echo "STATUS_TEXT=Completed" >> $GITHUB_ENV
|
||||
echo "STATUS_COLOR=#6aa84f" >> $GITHUB_ENV
|
||||
elif [[ "${{ inputs.step-outcome }}" == "failure" ]]; then
|
||||
echo "STATUS_TEXT=Failed" >> $GITHUB_ENV
|
||||
echo "STATUS_COLOR=#fc3434" >> $GITHUB_ENV
|
||||
else
|
||||
# No outcome provided - pending/in progress state
|
||||
echo "STATUS_COLOR=#dbab09" >> $GITHUB_ENV
|
||||
fi
|
||||
|
||||
- name: Send Slack notification (new message)
|
||||
if: inputs.update-ts == ''
|
||||
id: slack-notification-post
|
||||
uses: slackapi/slack-github-action@91efab103c0de0a537f72a35f6b8cda0ee76bf0a # v2.1.1
|
||||
env:
|
||||
SLACK_PAYLOAD_FILE_PATH: ${{ inputs.payload-file-path }}
|
||||
with:
|
||||
method: chat.postMessage
|
||||
token: ${{ inputs.slack-bot-token }}
|
||||
payload-file-path: ${{ inputs.payload-file-path }}
|
||||
payload-templated: true
|
||||
errors: true
|
||||
|
||||
- name: Update Slack notification
|
||||
if: inputs.update-ts != ''
|
||||
id: slack-notification-update
|
||||
uses: slackapi/slack-github-action@91efab103c0de0a537f72a35f6b8cda0ee76bf0a # v2.1.1
|
||||
env:
|
||||
SLACK_PAYLOAD_FILE_PATH: ${{ inputs.payload-file-path }}
|
||||
with:
|
||||
method: chat.update
|
||||
token: ${{ inputs.slack-bot-token }}
|
||||
payload-file-path: ${{ inputs.payload-file-path }}
|
||||
payload-templated: true
|
||||
errors: true
|
||||
|
||||
- name: Set output
|
||||
id: slack-notification
|
||||
shell: bash
|
||||
run: |
|
||||
if [[ "${{ inputs.update-ts }}" == "" ]]; then
|
||||
echo "ts=${{ steps.slack-notification-post.outputs.ts }}" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "ts=${{ inputs.update-ts }}" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
@@ -45,13 +45,22 @@ outputs:
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
- name: Cache Trivy vulnerability database
|
||||
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
|
||||
- name: Run Trivy vulnerability scan (SARIF)
|
||||
if: inputs.upload-sarif == 'true'
|
||||
uses: aquasecurity/trivy-action@b6643a29fecd7f34b3597bc6acb0a98b03d33ff8 # v0.33.1
|
||||
with:
|
||||
path: ~/.cache/trivy
|
||||
key: trivy-db-${{ runner.os }}-${{ github.run_id }}
|
||||
restore-keys: |
|
||||
trivy-db-${{ runner.os }}-
|
||||
image-ref: ${{ inputs.image-name }}:${{ inputs.image-tag }}
|
||||
format: 'sarif'
|
||||
output: 'trivy-results.sarif'
|
||||
severity: 'CRITICAL,HIGH'
|
||||
exit-code: '0'
|
||||
|
||||
- name: Upload Trivy results to GitHub Security tab
|
||||
if: inputs.upload-sarif == 'true'
|
||||
uses: github/codeql-action/upload-sarif@3599b3baa15b485a2e49ef411a7a4bb2452e7f93 # v3.30.5
|
||||
with:
|
||||
sarif_file: 'trivy-results.sarif'
|
||||
category: 'trivy-container'
|
||||
|
||||
- name: Run Trivy vulnerability scan (JSON)
|
||||
uses: aquasecurity/trivy-action@b6643a29fecd7f34b3597bc6acb0a98b03d33ff8 # v0.33.1
|
||||
@@ -61,27 +70,6 @@ runs:
|
||||
output: 'trivy-report.json'
|
||||
severity: ${{ inputs.severity }}
|
||||
exit-code: '0'
|
||||
scanners: 'vuln'
|
||||
timeout: '5m'
|
||||
|
||||
- name: Run Trivy vulnerability scan (SARIF)
|
||||
if: inputs.upload-sarif == 'true' && github.event_name == 'push'
|
||||
uses: aquasecurity/trivy-action@b6643a29fecd7f34b3597bc6acb0a98b03d33ff8 # v0.33.1
|
||||
with:
|
||||
image-ref: ${{ inputs.image-name }}:${{ inputs.image-tag }}
|
||||
format: 'sarif'
|
||||
output: 'trivy-results.sarif'
|
||||
severity: 'CRITICAL,HIGH'
|
||||
exit-code: '0'
|
||||
scanners: 'vuln'
|
||||
timeout: '5m'
|
||||
|
||||
- name: Upload Trivy results to GitHub Security tab
|
||||
if: inputs.upload-sarif == 'true' && github.event_name == 'push'
|
||||
uses: github/codeql-action/upload-sarif@3599b3baa15b485a2e49ef411a7a4bb2452e7f93 # v3.30.5
|
||||
with:
|
||||
sarif_file: 'trivy-results.sarif'
|
||||
category: 'trivy-container'
|
||||
|
||||
- name: Upload Trivy report artifact
|
||||
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
|
||||
@@ -121,20 +109,20 @@ runs:
|
||||
with:
|
||||
script: |
|
||||
const comment = require('./.github/scripts/trivy-pr-comment.js');
|
||||
|
||||
|
||||
// Unique identifier to find our comment
|
||||
const marker = '<!-- trivy-scan-comment:${{ inputs.image-name }} -->';
|
||||
const body = marker + '\n' + comment;
|
||||
|
||||
|
||||
// Find existing comment
|
||||
const { data: comments } = await github.rest.issues.listComments({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: context.issue.number,
|
||||
});
|
||||
|
||||
|
||||
const existingComment = comments.find(c => c.body?.includes(marker));
|
||||
|
||||
|
||||
if (existingComment) {
|
||||
// Update existing comment
|
||||
await github.rest.issues.updateComment({
|
||||
|
||||
@@ -1,462 +0,0 @@
|
||||
# Slack Message Templates
|
||||
|
||||
This directory contains reusable message templates for Slack notifications sent from GitHub Actions workflows.
|
||||
|
||||
## Usage
|
||||
|
||||
These JSON templates are used with the `slackapi/slack-github-action` using the Slack API method (`chat.postMessage` and `chat.update`). All templates support rich Block Kit formatting and message updates.
|
||||
|
||||
### Available Templates
|
||||
|
||||
**Container Releases**
|
||||
- `container-release-started.json`: Simple one-line notification when container push starts
|
||||
- `container-release-completed.json`: Simple one-line notification when container release completes
|
||||
|
||||
**Deployments**
|
||||
- `deployment-started.json`: Deployment start notification with Block Kit formatting
|
||||
- `deployment-completed.json`: Deployment completion notification (updates the start message)
|
||||
|
||||
All templates use the Slack API method and require a Slack Bot Token.
|
||||
|
||||
## Setup Requirements
|
||||
|
||||
1. Create a Slack App (or use existing)
|
||||
2. Add Bot Token Scopes: `chat:write`, `chat:write.public`
|
||||
3. Install the app to your workspace
|
||||
4. Get the Bot Token from OAuth & Permissions page
|
||||
5. Add secrets:
|
||||
- `SLACK_BOT_TOKEN`: Your bot token
|
||||
- `SLACK_CHANNEL_ID`: The channel ID where messages will be posted
|
||||
|
||||
Reference: [Sending data using a Slack API method](https://docs.slack.dev/tools/slack-github-action/sending-techniques/sending-data-slack-api-method/)
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Required Secrets (GitHub Secrets)
|
||||
- `SLACK_BOT_TOKEN`: Passed as `token` parameter to the action (not as env variable)
|
||||
- `SLACK_CHANNEL_ID`: Used in payload as env variable
|
||||
|
||||
### Container Release Variables (configured as env)
|
||||
- `COMPONENT`: Component name (e.g., "API", "SDK", "UI", "MCP")
|
||||
- `RELEASE_TAG` / `PROWLER_VERSION`: The release tag or version being deployed
|
||||
- `GITHUB_SERVER_URL`: Provided by GitHub context
|
||||
- `GITHUB_REPOSITORY`: Provided by GitHub context
|
||||
- `GITHUB_RUN_ID`: Provided by GitHub context
|
||||
- `STATUS_EMOJI`: Status symbol (calculated: `[✓]` for success, `[✗]` for failure)
|
||||
- `STATUS_TEXT`: Status text (calculated: "completed successfully!" or "failed")
|
||||
|
||||
### Deployment Variables (configured as env)
|
||||
- `COMPONENT`: Component name (e.g., "API", "SDK", "UI", "MCP")
|
||||
- `ENVIRONMENT`: Environment name (e.g., "DEVELOPMENT", "PRODUCTION")
|
||||
- `COMMIT_HASH`: Commit hash being deployed
|
||||
- `VERSION_DEPLOYED`: Version being deployed
|
||||
- `GITHUB_ACTOR`: User who triggered the workflow
|
||||
- `GITHUB_WORKFLOW`: Workflow name
|
||||
- `GITHUB_SERVER_URL`: Provided by GitHub context
|
||||
- `GITHUB_REPOSITORY`: Provided by GitHub context
|
||||
- `GITHUB_RUN_ID`: Provided by GitHub context
|
||||
|
||||
All other variables (MESSAGE_TS, STATUS, STATUS_COLOR, STATUS_EMOJI, etc.) are calculated internally within the workflow and should NOT be configured as environment variables.
|
||||
|
||||
## Example Workflow Usage
|
||||
|
||||
### Using the Generic Slack Notification Action (Recommended)
|
||||
|
||||
**Recommended approach**: Use the generic reusable action `.github/actions/slack-notification` which provides maximum flexibility:
|
||||
|
||||
#### Example 1: Container Release (Start + Completion)
|
||||
|
||||
```yaml
|
||||
# Send start notification
|
||||
- name: Notify container push started
|
||||
if: github.event_name == 'release'
|
||||
uses: ./.github/actions/slack-notification
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload: |
|
||||
{
|
||||
"channel": "${{ secrets.SLACK_CHANNEL_ID }}",
|
||||
"text": "API container release ${{ env.RELEASE_TAG }} push started... <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run>"
|
||||
}
|
||||
|
||||
# Build and push container
|
||||
- name: Build and push container
|
||||
if: github.event_name == 'release'
|
||||
id: container-push
|
||||
uses: docker/build-push-action@...
|
||||
with:
|
||||
push: true
|
||||
tags: ...
|
||||
|
||||
# Calculate status
|
||||
- name: Determine push status
|
||||
if: github.event_name == 'release' && always()
|
||||
id: push-status
|
||||
run: |
|
||||
if [[ "${{ steps.container-push.outcome }}" == "success" ]]; then
|
||||
echo "emoji=[✓]" >> $GITHUB_OUTPUT
|
||||
echo "text=completed successfully!" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "emoji=[✗]" >> $GITHUB_OUTPUT
|
||||
echo "text=failed" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
# Send completion notification
|
||||
- name: Notify container push completed
|
||||
if: github.event_name == 'release' && always()
|
||||
uses: ./.github/actions/slack-notification
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload: |
|
||||
{
|
||||
"channel": "${{ secrets.SLACK_CHANNEL_ID }}",
|
||||
"text": "${{ steps.push-status.outputs.emoji }} API container release ${{ env.RELEASE_TAG }} push ${{ steps.push-status.outputs.text }} <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View run>"
|
||||
}
|
||||
```
|
||||
|
||||
#### Example 2: Simple One-Time Message
|
||||
|
||||
```yaml
|
||||
- name: Send notification
|
||||
uses: ./.github/actions/slack-notification
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload: |
|
||||
{
|
||||
"channel": "${{ secrets.SLACK_CHANNEL_ID }}",
|
||||
"text": "Deployment completed successfully!"
|
||||
}
|
||||
```
|
||||
|
||||
#### Example 3: Deployment with Message Update Pattern
|
||||
|
||||
```yaml
|
||||
# Send initial deployment message
|
||||
- name: Notify deployment started
|
||||
id: slack-start
|
||||
uses: ./.github/actions/slack-notification
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload: |
|
||||
{
|
||||
"channel": "${{ secrets.SLACK_CHANNEL_ID }}",
|
||||
"text": "API deployment to PRODUCTION started",
|
||||
"attachments": [
|
||||
{
|
||||
"color": "dbab09",
|
||||
"blocks": [
|
||||
{
|
||||
"type": "header",
|
||||
"text": {
|
||||
"type": "plain_text",
|
||||
"text": "API | Deployment to PRODUCTION"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "section",
|
||||
"fields": [
|
||||
{
|
||||
"type": "mrkdwn",
|
||||
"text": "*Status:*\nIn Progress"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Run deployment
|
||||
- name: Deploy
|
||||
id: deploy
|
||||
run: terraform apply -auto-approve
|
||||
|
||||
# Calculate status
|
||||
- name: Determine status
|
||||
if: always()
|
||||
id: status
|
||||
run: |
|
||||
if [[ "${{ steps.deploy.outcome }}" == "success" ]]; then
|
||||
echo "color=28a745" >> $GITHUB_OUTPUT
|
||||
echo "emoji=[✓]" >> $GITHUB_OUTPUT
|
||||
echo "status=Completed" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "color=fc3434" >> $GITHUB_OUTPUT
|
||||
echo "emoji=[✗]" >> $GITHUB_OUTPUT
|
||||
echo "status=Failed" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
# Update the same message with final status
|
||||
- name: Update deployment notification
|
||||
if: always()
|
||||
uses: ./.github/actions/slack-notification
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
update-ts: ${{ steps.slack-start.outputs.ts }}
|
||||
payload: |
|
||||
{
|
||||
"channel": "${{ secrets.SLACK_CHANNEL_ID }}",
|
||||
"ts": "${{ steps.slack-start.outputs.ts }}",
|
||||
"text": "${{ steps.status.outputs.emoji }} API deployment to PRODUCTION ${{ steps.status.outputs.status }}",
|
||||
"attachments": [
|
||||
{
|
||||
"color": "${{ steps.status.outputs.color }}",
|
||||
"blocks": [
|
||||
{
|
||||
"type": "header",
|
||||
"text": {
|
||||
"type": "plain_text",
|
||||
"text": "API | Deployment to PRODUCTION"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "section",
|
||||
"fields": [
|
||||
{
|
||||
"type": "mrkdwn",
|
||||
"text": "*Status:*\n${{ steps.status.outputs.emoji }} ${{ steps.status.outputs.status }}"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Benefits of using the generic action:**
|
||||
- Maximum flexibility: Build any payload you need directly in the workflow
|
||||
- No template files needed: Everything inline
|
||||
- Supports all scenarios: one-time messages, start/update patterns, rich Block Kit
|
||||
- Easy to customize per use case
|
||||
- Generic: Works for containers, deployments, or any notification type
|
||||
|
||||
For more details, see [Slack Notification Action](../../actions/slack-notification/README.md).
|
||||
|
||||
### Using Message Templates (Alternative Approach)
|
||||
|
||||
Simple one-line notifications for container releases:
|
||||
|
||||
```yaml
|
||||
# Step 1: Notify when push starts
|
||||
- name: Notify container push started
|
||||
if: github.event_name == 'release'
|
||||
uses: slackapi/slack-github-action@91efab103c0de0a537f72a35f6b8cda0ee76bf0a # v2.1.1
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_CHANNEL_ID }}
|
||||
COMPONENT: API
|
||||
RELEASE_TAG: ${{ env.RELEASE_TAG }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
method: chat.postMessage
|
||||
token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-started.json"
|
||||
|
||||
# Step 2: Build and push container
|
||||
- name: Build and push container
|
||||
id: container-push
|
||||
uses: docker/build-push-action@...
|
||||
with:
|
||||
push: true
|
||||
tags: ...
|
||||
|
||||
# Step 3: Determine push status
|
||||
- name: Determine push status
|
||||
if: github.event_name == 'release' && always()
|
||||
id: push-status
|
||||
run: |
|
||||
if [[ "${{ steps.container-push.outcome }}" == "success" ]]; then
|
||||
echo "status-emoji=[✓]" >> $GITHUB_OUTPUT
|
||||
echo "status-text=completed successfully!" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "status-emoji=[✗]" >> $GITHUB_OUTPUT
|
||||
echo "status-text=failed" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
# Step 4: Notify when push completes (success or failure)
|
||||
- name: Notify container push completed
|
||||
if: github.event_name == 'release' && always()
|
||||
uses: slackapi/slack-github-action@91efab103c0de0a537f72a35f6b8cda0ee76bf0a # v2.1.1
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_CHANNEL_ID }}
|
||||
COMPONENT: API
|
||||
RELEASE_TAG: ${{ env.RELEASE_TAG }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
STATUS_EMOJI: ${{ steps.push-status.outputs.status-emoji }}
|
||||
STATUS_TEXT: ${{ steps.push-status.outputs.status-text }}
|
||||
with:
|
||||
method: chat.postMessage
|
||||
token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-completed.json"
|
||||
```
|
||||
|
||||
### Deployment with Update Pattern
|
||||
|
||||
For deployments that start with one message and update it with the final status:
|
||||
|
||||
```yaml
|
||||
# Step 1: Send deployment start notification
|
||||
- name: Notify Deployment Start
|
||||
id: slack-notification-start
|
||||
uses: slackapi/slack-github-action@91efab103c0de0a537f72a35f6b8cda0ee76bf0a # v2.1.1
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_CHANNEL_ID }}
|
||||
COMPONENT: API
|
||||
ENVIRONMENT: PRODUCTION
|
||||
COMMIT_HASH: ${{ github.sha }}
|
||||
VERSION_DEPLOYED: latest
|
||||
GITHUB_ACTOR: ${{ github.actor }}
|
||||
GITHUB_WORKFLOW: ${{ github.workflow }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
method: chat.postMessage
|
||||
token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/deployment-started.json"
|
||||
|
||||
# Step 2: Run your deployment steps
|
||||
- name: Terraform Plan
|
||||
id: terraform-plan
|
||||
run: terraform plan
|
||||
|
||||
- name: Terraform Apply
|
||||
id: terraform-apply
|
||||
run: terraform apply -auto-approve
|
||||
|
||||
# Step 3: Determine status (calculated internally, not configured)
|
||||
- name: Determine Status
|
||||
if: always()
|
||||
id: determine-status
|
||||
run: |
|
||||
if [[ "${{ steps.terraform-apply.outcome }}" == "success" ]]; then
|
||||
echo "status=Completed" >> $GITHUB_OUTPUT
|
||||
echo "status-color=28a745" >> $GITHUB_OUTPUT
|
||||
echo "status-emoji=[✓]" >> $GITHUB_OUTPUT
|
||||
echo "plan-emoji=[✓]" >> $GITHUB_OUTPUT
|
||||
echo "apply-emoji=[✓]" >> $GITHUB_OUTPUT
|
||||
elif [[ "${{ steps.terraform-plan.outcome }}" == "failure" || "${{ steps.terraform-apply.outcome }}" == "failure" ]]; then
|
||||
echo "status=Failed" >> $GITHUB_OUTPUT
|
||||
echo "status-color=fc3434" >> $GITHUB_OUTPUT
|
||||
echo "status-emoji=[✗]" >> $GITHUB_OUTPUT
|
||||
if [[ "${{ steps.terraform-plan.outcome }}" == "failure" ]]; then
|
||||
echo "plan-emoji=[✗]" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "plan-emoji=[✓]" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
if [[ "${{ steps.terraform-apply.outcome }}" == "failure" ]]; then
|
||||
echo "apply-emoji=[✗]" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "apply-emoji=[✓]" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
else
|
||||
echo "status=Failed" >> $GITHUB_OUTPUT
|
||||
echo "status-color=fc3434" >> $GITHUB_OUTPUT
|
||||
echo "status-emoji=[✗]" >> $GITHUB_OUTPUT
|
||||
echo "plan-emoji=[?]" >> $GITHUB_OUTPUT
|
||||
echo "apply-emoji=[?]" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
# Step 4: Update the same Slack message (using calculated values)
|
||||
- name: Notify Deployment Result
|
||||
if: always()
|
||||
uses: slackapi/slack-github-action@91efab103c0de0a537f72a35f6b8cda0ee76bf0a # v2.1.1
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_CHANNEL_ID }}
|
||||
MESSAGE_TS: ${{ steps.slack-notification-start.outputs.ts }}
|
||||
COMPONENT: API
|
||||
ENVIRONMENT: PRODUCTION
|
||||
COMMIT_HASH: ${{ github.sha }}
|
||||
VERSION_DEPLOYED: latest
|
||||
GITHUB_ACTOR: ${{ github.actor }}
|
||||
GITHUB_WORKFLOW: ${{ github.workflow }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
STATUS: ${{ steps.determine-status.outputs.status }}
|
||||
STATUS_COLOR: ${{ steps.determine-status.outputs.status-color }}
|
||||
STATUS_EMOJI: ${{ steps.determine-status.outputs.status-emoji }}
|
||||
PLAN_EMOJI: ${{ steps.determine-status.outputs.plan-emoji }}
|
||||
APPLY_EMOJI: ${{ steps.determine-status.outputs.apply-emoji }}
|
||||
TERRAFORM_PLAN_OUTCOME: ${{ steps.terraform-plan.outcome }}
|
||||
TERRAFORM_APPLY_OUTCOME: ${{ steps.terraform-apply.outcome }}
|
||||
with:
|
||||
method: chat.update
|
||||
token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/deployment-completed.json"
|
||||
```
|
||||
|
||||
**Note**: Variables like `STATUS`, `STATUS_COLOR`, `STATUS_EMOJI`, `PLAN_EMOJI`, `APPLY_EMOJI` are calculated by the `determine-status` step based on the outcomes of previous steps. They should NOT be manually configured.
|
||||
|
||||
## Key Features
|
||||
|
||||
### Benefits of Using Slack API Method
|
||||
|
||||
- **Rich Block Kit Formatting**: Full support for Slack's Block Kit including headers, sections, fields, colors, and attachments
|
||||
- **Message Updates**: Update the same message instead of posting multiple messages (using `chat.update` with `ts`)
|
||||
- **Consistent Experience**: Same look and feel as Prowler Cloud notifications
|
||||
- **Flexible**: Easy to customize message appearance by editing JSON templates
|
||||
|
||||
### Differences from Webhook Method
|
||||
|
||||
| Feature | webhook-trigger | Slack API (chat.postMessage) |
|
||||
|---------|-----------------|------------------------------|
|
||||
| Setup | Workflow Builder webhook | Slack Bot Token + Channel ID |
|
||||
| Formatting | Plain text/simple | Full Block Kit support |
|
||||
| Message Update | No | Yes (with chat.update) |
|
||||
| Authentication | Webhook URL | Bot Token |
|
||||
| Scopes Required | None | chat:write, chat:write.public |
|
||||
|
||||
## Message Appearance
|
||||
|
||||
### Container Release (Simple One-Line)
|
||||
|
||||
**Start message:**
|
||||
```
|
||||
API container release 4.5.0 push started... View run
|
||||
```
|
||||
|
||||
**Completion message (success):**
|
||||
```
|
||||
[✓] API container release 4.5.0 push completed successfully! View run
|
||||
```
|
||||
|
||||
**Completion message (failure):**
|
||||
```
|
||||
[✗] API container release 4.5.0 push failed View run
|
||||
```
|
||||
|
||||
All messages are simple one-liners with a clickable "View run" link. The completion message adapts to show success `[✓]` or failure `[✗]` based on the outcome of the container push.
|
||||
|
||||
### Deployment Start
|
||||
- Header: Component and environment
|
||||
- Yellow bar (color: `dbab09`)
|
||||
- Status: In Progress
|
||||
- Details: Commit, version, actor, workflow
|
||||
- Link: Direct link to deployment run
|
||||
|
||||
### Deployment Completion
|
||||
- Header: Component and environment
|
||||
- Green bar for success (color: `28a745`) / Red bar for failure (color: `fc3434`)
|
||||
- Status: [✓] Completed or [✗] Failed
|
||||
- Details: All deployment info plus terraform outcomes
|
||||
- Link: Direct link to deployment run
|
||||
|
||||
## Adding New Templates
|
||||
|
||||
1. Create a new JSON file with Block Kit structure
|
||||
2. Use environment variable placeholders (e.g., `$VAR_NAME`)
|
||||
3. Include `channel` and `text` fields (required)
|
||||
4. Add `blocks` or `attachments` for rich formatting
|
||||
5. For update templates, include `ts` field as `$MESSAGE_TS`
|
||||
6. Document the template in this README
|
||||
7. Reference it in your workflow using `payload-file-path`
|
||||
|
||||
## Reference
|
||||
|
||||
- [Slack Block Kit Builder](https://app.slack.com/block-kit-builder)
|
||||
- [Slack API Method Documentation](https://docs.slack.dev/tools/slack-github-action/sending-techniques/sending-data-slack-api-method/)
|
||||
@@ -1,17 +0,0 @@
|
||||
{
|
||||
"channel": "${{ env.SLACK_CHANNEL_ID }}",
|
||||
"attachments": [
|
||||
{
|
||||
"color": "${{ env.STATUS_COLOR }}",
|
||||
"blocks": [
|
||||
{
|
||||
"type": "section",
|
||||
"text": {
|
||||
"type": "mrkdwn",
|
||||
"text": "*Status:*\n${{ env.STATUS_TEXT }}\n\n${{ env.COMPONENT }} container release ${{ env.RELEASE_TAG }} push ${{ env.STATUS_TEXT }}\n\n<${{ env.GITHUB_SERVER_URL }}/${{ env.GITHUB_REPOSITORY }}/actions/runs/${{ env.GITHUB_RUN_ID }}|View run>"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,17 +0,0 @@
|
||||
{
|
||||
"channel": "${{ env.SLACK_CHANNEL_ID }}",
|
||||
"attachments": [
|
||||
{
|
||||
"color": "${{ env.STATUS_COLOR }}",
|
||||
"blocks": [
|
||||
{
|
||||
"type": "section",
|
||||
"text": {
|
||||
"type": "mrkdwn",
|
||||
"text": "*Status:*\nStarted\n\n${{ env.COMPONENT }} container release ${{ env.RELEASE_TAG }} push started...\n\n<${{ env.GITHUB_SERVER_URL }}/${{ env.GITHUB_REPOSITORY }}/actions/runs/${{ env.GITHUB_RUN_ID }}|View run>"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
name: API - Build and Push containers
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- "master"
|
||||
paths:
|
||||
- "api/**"
|
||||
- "prowler/**"
|
||||
- ".github/workflows/api-build-lint-push-containers.yml"
|
||||
|
||||
# Uncomment the code below to test this action on PRs
|
||||
# pull_request:
|
||||
# branches:
|
||||
# - "master"
|
||||
# paths:
|
||||
# - "api/**"
|
||||
# - ".github/workflows/api-build-lint-push-containers.yml"
|
||||
|
||||
release:
|
||||
types: [published]
|
||||
|
||||
env:
|
||||
# Tags
|
||||
LATEST_TAG: latest
|
||||
RELEASE_TAG: ${{ github.event.release.tag_name }}
|
||||
STABLE_TAG: stable
|
||||
|
||||
WORKING_DIRECTORY: ./api
|
||||
|
||||
# Container Registries
|
||||
PROWLERCLOUD_DOCKERHUB_REPOSITORY: prowlercloud
|
||||
PROWLERCLOUD_DOCKERHUB_IMAGE: prowler-api
|
||||
|
||||
jobs:
|
||||
repository-check:
|
||||
name: Repository check
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
is_repo: ${{ steps.repository_check.outputs.is_repo }}
|
||||
steps:
|
||||
- name: Repository check
|
||||
id: repository_check
|
||||
working-directory: /tmp
|
||||
run: |
|
||||
if [[ ${{ github.repository }} == "prowler-cloud/prowler" ]]
|
||||
then
|
||||
echo "is_repo=true" >> "${GITHUB_OUTPUT}"
|
||||
else
|
||||
echo "This action only runs for prowler-cloud/prowler"
|
||||
echo "is_repo=false" >> "${GITHUB_OUTPUT}"
|
||||
fi
|
||||
|
||||
# Build Prowler OSS container
|
||||
container-build-push:
|
||||
needs: repository-check
|
||||
if: needs.repository-check.outputs.is_repo == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: ${{ env.WORKING_DIRECTORY }}
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Set short git commit SHA
|
||||
id: vars
|
||||
run: |
|
||||
shortSha=$(git rev-parse --short ${{ github.sha }})
|
||||
echo "SHORT_SHA=${shortSha}" >> $GITHUB_ENV
|
||||
|
||||
- name: Login to DockerHub
|
||||
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
|
||||
- name: Build and push container image (latest)
|
||||
# Comment the following line for testing
|
||||
if: github.event_name == 'push'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.WORKING_DIRECTORY }}
|
||||
# Set push: false for testing
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.SHORT_SHA }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Build and push container image (release)
|
||||
if: github.event_name == 'release'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.WORKING_DIRECTORY }}
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.RELEASE_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Trigger deployment
|
||||
if: github.event_name == 'push'
|
||||
uses: peter-evans/repository-dispatch@5fc4efd1a4797ddb68ffd0714a238564e4cc0e6f # v4.0.0
|
||||
with:
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
repository: ${{ secrets.CLOUD_DISPATCH }}
|
||||
event-type: prowler-api-deploy
|
||||
client-payload: '{"sha": "${{ github.sha }}", "short_sha": "${{ env.SHORT_SHA }}"}'
|
||||
@@ -1,71 +0,0 @@
|
||||
name: 'API: Code Quality'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
API_WORKING_DIR: ./api
|
||||
|
||||
jobs:
|
||||
api-code-quality:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
matrix:
|
||||
python-version:
|
||||
- '3.12'
|
||||
defaults:
|
||||
run:
|
||||
working-directory: ./api
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check for API changes
|
||||
id: check-changes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
api/**
|
||||
.github/workflows/api-code-quality.yml
|
||||
files_ignore: |
|
||||
api/docs/**
|
||||
api/README.md
|
||||
api/CHANGELOG.md
|
||||
|
||||
- name: Setup Python with Poetry
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: ./.github/actions/setup-python-poetry
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
working-directory: ./api
|
||||
|
||||
- name: Poetry check
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry check --lock
|
||||
|
||||
- name: Ruff lint
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run ruff check . --exclude contrib
|
||||
|
||||
- name: Ruff format
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run ruff format --check . --exclude contrib
|
||||
|
||||
- name: Pylint
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run pylint --disable=W,C,R,E -j 0 -rn -sn src/
|
||||
@@ -25,7 +25,7 @@ concurrency:
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
api-analyze:
|
||||
analyze:
|
||||
name: CodeQL Security Analysis
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
|
||||
@@ -1,135 +0,0 @@
|
||||
name: 'API: Container Build and Push'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
paths:
|
||||
- 'api/**'
|
||||
- 'prowler/**'
|
||||
- '.github/workflows/api-build-lint-push-containers.yml'
|
||||
release:
|
||||
types:
|
||||
- 'published'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
# Tags
|
||||
LATEST_TAG: latest
|
||||
RELEASE_TAG: ${{ github.event.release.tag_name }}
|
||||
STABLE_TAG: stable
|
||||
WORKING_DIRECTORY: ./api
|
||||
|
||||
# Container registries
|
||||
PROWLERCLOUD_DOCKERHUB_REPOSITORY: prowlercloud
|
||||
PROWLERCLOUD_DOCKERHUB_IMAGE: prowler-api
|
||||
|
||||
jobs:
|
||||
setup:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
outputs:
|
||||
short-sha: ${{ steps.set-short-sha.outputs.short-sha }}
|
||||
steps:
|
||||
- name: Calculate short SHA
|
||||
id: set-short-sha
|
||||
run: echo "short-sha=${GITHUB_SHA::7}" >> $GITHUB_OUTPUT
|
||||
|
||||
container-build-push:
|
||||
needs: setup
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Login to DockerHub
|
||||
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
|
||||
- name: Build and push API container (latest)
|
||||
if: github.event_name == 'push'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.WORKING_DIRECTORY }}
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Notify container push started
|
||||
if: github.event_name == 'release'
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_PLATFORM_DEPLOYMENTS }}
|
||||
COMPONENT: API
|
||||
RELEASE_TAG: ${{ env.RELEASE_TAG }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-started.json"
|
||||
|
||||
- name: Build and push API container (release)
|
||||
if: github.event_name == 'release'
|
||||
id: container-push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.WORKING_DIRECTORY }}
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.RELEASE_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Notify container push completed
|
||||
if: github.event_name == 'release' && always()
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_PLATFORM_DEPLOYMENTS }}
|
||||
COMPONENT: API
|
||||
RELEASE_TAG: ${{ env.RELEASE_TAG }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-completed.json"
|
||||
step-outcome: ${{ steps.container-push.outcome }}
|
||||
|
||||
trigger-deployment:
|
||||
if: github.event_name == 'push'
|
||||
needs: [setup, container-build-push]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Trigger API deployment
|
||||
uses: peter-evans/repository-dispatch@5fc4efd1a4797ddb68ffd0714a238564e4cc0e6f # v4.0.0
|
||||
with:
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
repository: ${{ secrets.CLOUD_DISPATCH }}
|
||||
event-type: api-prowler-deployment
|
||||
client-payload: '{"sha": "${{ github.sha }}", "short_sha": "${{ needs.setup.outputs.short-sha }}"}'
|
||||
@@ -1,89 +0,0 @@
|
||||
name: 'API: Container Checks'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
API_WORKING_DIR: ./api
|
||||
IMAGE_NAME: prowler-api
|
||||
|
||||
jobs:
|
||||
api-dockerfile-lint:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check if Dockerfile changed
|
||||
id: dockerfile-changed
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: api/Dockerfile
|
||||
|
||||
- name: Lint Dockerfile with Hadolint
|
||||
if: steps.dockerfile-changed.outputs.any_changed == 'true'
|
||||
uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
|
||||
with:
|
||||
dockerfile: api/Dockerfile
|
||||
ignore: DL3013
|
||||
|
||||
api-container-build-and-scan:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
pull-requests: write
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check for API changes
|
||||
id: check-changes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: api/**
|
||||
files_ignore: |
|
||||
api/docs/**
|
||||
api/README.md
|
||||
api/CHANGELOG.md
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
|
||||
- name: Build container
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.API_WORKING_DIR }}
|
||||
push: false
|
||||
load: true
|
||||
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Scan container with Trivy
|
||||
if: github.repository == 'prowler-cloud/prowler' && steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: ./.github/actions/trivy-scan
|
||||
with:
|
||||
image-name: ${{ env.IMAGE_NAME }}
|
||||
image-tag: ${{ github.sha }}
|
||||
fail-on-critical: 'false'
|
||||
severity: 'CRITICAL'
|
||||
@@ -0,0 +1,228 @@
|
||||
name: 'API: Pull Request'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
paths:
|
||||
- '.github/workflows/api-pull-request.yml'
|
||||
- 'api/**'
|
||||
- '!api/docs/**'
|
||||
- '!api/README.md'
|
||||
- '!api/CHANGELOG.md'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
paths:
|
||||
- '.github/workflows/api-pull-request.yml'
|
||||
- 'api/**'
|
||||
- '!api/docs/**'
|
||||
- '!api/README.md'
|
||||
- '!api/CHANGELOG.md'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
POSTGRES_HOST: localhost
|
||||
POSTGRES_PORT: 5432
|
||||
POSTGRES_ADMIN_USER: prowler
|
||||
POSTGRES_ADMIN_PASSWORD: S3cret
|
||||
POSTGRES_USER: prowler_user
|
||||
POSTGRES_PASSWORD: prowler
|
||||
POSTGRES_DB: postgres-db
|
||||
VALKEY_HOST: localhost
|
||||
VALKEY_PORT: 6379
|
||||
VALKEY_DB: 0
|
||||
API_WORKING_DIR: ./api
|
||||
IMAGE_NAME: prowler-api
|
||||
|
||||
jobs:
|
||||
code-quality:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
matrix:
|
||||
python-version:
|
||||
- '3.12'
|
||||
defaults:
|
||||
run:
|
||||
working-directory: ./api
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Setup Python with Poetry
|
||||
uses: ./.github/actions/setup-python-poetry
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
working-directory: ./api
|
||||
|
||||
- name: Poetry check
|
||||
run: poetry check --lock
|
||||
|
||||
- name: Ruff lint
|
||||
run: poetry run ruff check . --exclude contrib
|
||||
|
||||
- name: Ruff format
|
||||
run: poetry run ruff format --check . --exclude contrib
|
||||
|
||||
- name: Pylint
|
||||
run: poetry run pylint --disable=W,C,R,E -j 0 -rn -sn src/
|
||||
|
||||
security-scans:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
matrix:
|
||||
python-version:
|
||||
- '3.12'
|
||||
defaults:
|
||||
run:
|
||||
working-directory: ./api
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Setup Python with Poetry
|
||||
uses: ./.github/actions/setup-python-poetry
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
working-directory: ./api
|
||||
|
||||
- name: Bandit
|
||||
run: poetry run bandit -q -lll -x '*_test.py,./contrib/' -r .
|
||||
|
||||
- name: Safety
|
||||
# 76352, 76353, 77323 come from SDK, but they cannot upgrade it yet. It does not affect API
|
||||
# TODO: Botocore needs urllib3 1.X so we need to ignore these vulnerabilities 77744,77745. Remove this once we upgrade to urllib3 2.X
|
||||
run: poetry run safety check --ignore 70612,66963,74429,76352,76353,77323,77744,77745
|
||||
|
||||
- name: Vulture
|
||||
run: poetry run vulture --exclude "contrib,tests,conftest.py" --min-confidence 100 .
|
||||
|
||||
tests:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
matrix:
|
||||
python-version:
|
||||
- '3.12'
|
||||
defaults:
|
||||
run:
|
||||
working-directory: ./api
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres
|
||||
env:
|
||||
POSTGRES_HOST: ${{ env.POSTGRES_HOST }}
|
||||
POSTGRES_PORT: ${{ env.POSTGRES_PORT }}
|
||||
POSTGRES_USER: ${{ env.POSTGRES_USER }}
|
||||
POSTGRES_PASSWORD: ${{ env.POSTGRES_PASSWORD }}
|
||||
POSTGRES_DB: ${{ env.POSTGRES_DB }}
|
||||
ports:
|
||||
- 5432:5432
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
valkey:
|
||||
image: valkey/valkey:7-alpine3.19
|
||||
env:
|
||||
VALKEY_HOST: ${{ env.VALKEY_HOST }}
|
||||
VALKEY_PORT: ${{ env.VALKEY_PORT }}
|
||||
VALKEY_DB: ${{ env.VALKEY_DB }}
|
||||
ports:
|
||||
- 6379:6379
|
||||
options: >-
|
||||
--health-cmd "valkey-cli ping"
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Setup Python with Poetry
|
||||
uses: ./.github/actions/setup-python-poetry
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
working-directory: ./api
|
||||
|
||||
- name: Run tests with pytest
|
||||
run: poetry run pytest --cov=./src/backend --cov-report=xml src/backend
|
||||
|
||||
- name: Upload coverage reports to Codecov
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: api
|
||||
|
||||
dockerfile-lint:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Lint Dockerfile with Hadolint
|
||||
uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
|
||||
with:
|
||||
dockerfile: api/Dockerfile
|
||||
ignore: DL3013
|
||||
|
||||
container-build-and-scan:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
pull-requests: write
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
|
||||
- name: Build container
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.API_WORKING_DIR }}
|
||||
push: false
|
||||
load: true
|
||||
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Scan container with Trivy
|
||||
uses: ./.github/actions/trivy-scan
|
||||
with:
|
||||
image-name: ${{ env.IMAGE_NAME }}
|
||||
image-tag: ${{ github.sha }}
|
||||
fail-on-critical: 'false'
|
||||
severity: 'CRITICAL'
|
||||
@@ -1,69 +0,0 @@
|
||||
name: 'API: Security'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
API_WORKING_DIR: ./api
|
||||
|
||||
jobs:
|
||||
api-security-scans:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
matrix:
|
||||
python-version:
|
||||
- '3.12'
|
||||
defaults:
|
||||
run:
|
||||
working-directory: ./api
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check for API changes
|
||||
id: check-changes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
api/**
|
||||
.github/workflows/api-security.yml
|
||||
files_ignore: |
|
||||
api/docs/**
|
||||
api/README.md
|
||||
api/CHANGELOG.md
|
||||
|
||||
- name: Setup Python with Poetry
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: ./.github/actions/setup-python-poetry
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
working-directory: ./api
|
||||
|
||||
- name: Bandit
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run bandit -q -lll -x '*_test.py,./contrib/' -r .
|
||||
|
||||
- name: Safety
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
# 76352, 76353, 77323 come from SDK, but they cannot upgrade it yet. It does not affect API
|
||||
# TODO: Botocore needs urllib3 1.X so we need to ignore these vulnerabilities 77744,77745. Remove this once we upgrade to urllib3 2.X
|
||||
run: poetry run safety check --ignore 70612,66963,74429,76352,76353,77323,77744,77745
|
||||
|
||||
- name: Vulture
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run vulture --exclude "contrib,tests,conftest.py" --min-confidence 100 .
|
||||
@@ -1,107 +0,0 @@
|
||||
name: 'API: Tests'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
POSTGRES_HOST: localhost
|
||||
POSTGRES_PORT: 5432
|
||||
POSTGRES_ADMIN_USER: prowler
|
||||
POSTGRES_ADMIN_PASSWORD: S3cret
|
||||
POSTGRES_USER: prowler_user
|
||||
POSTGRES_PASSWORD: prowler
|
||||
POSTGRES_DB: postgres-db
|
||||
VALKEY_HOST: localhost
|
||||
VALKEY_PORT: 6379
|
||||
VALKEY_DB: 0
|
||||
API_WORKING_DIR: ./api
|
||||
|
||||
jobs:
|
||||
api-tests:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
matrix:
|
||||
python-version:
|
||||
- '3.12'
|
||||
defaults:
|
||||
run:
|
||||
working-directory: ./api
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres
|
||||
env:
|
||||
POSTGRES_HOST: ${{ env.POSTGRES_HOST }}
|
||||
POSTGRES_PORT: ${{ env.POSTGRES_PORT }}
|
||||
POSTGRES_USER: ${{ env.POSTGRES_USER }}
|
||||
POSTGRES_PASSWORD: ${{ env.POSTGRES_PASSWORD }}
|
||||
POSTGRES_DB: ${{ env.POSTGRES_DB }}
|
||||
ports:
|
||||
- 5432:5432
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
valkey:
|
||||
image: valkey/valkey:7-alpine3.19
|
||||
env:
|
||||
VALKEY_HOST: ${{ env.VALKEY_HOST }}
|
||||
VALKEY_PORT: ${{ env.VALKEY_PORT }}
|
||||
VALKEY_DB: ${{ env.VALKEY_DB }}
|
||||
ports:
|
||||
- 6379:6379
|
||||
options: >-
|
||||
--health-cmd "valkey-cli ping"
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check for API changes
|
||||
id: check-changes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
api/**
|
||||
.github/workflows/api-tests.yml
|
||||
files_ignore: |
|
||||
api/docs/**
|
||||
api/README.md
|
||||
api/CHANGELOG.md
|
||||
|
||||
- name: Setup Python with Poetry
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: ./.github/actions/setup-python-poetry
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
working-directory: ./api
|
||||
|
||||
- name: Run tests with pytest
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run pytest --cov=./src/backend --cov-report=xml src/backend
|
||||
|
||||
- name: Upload coverage reports to Codecov
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: api
|
||||
@@ -7,6 +7,8 @@ on:
|
||||
types:
|
||||
- 'labeled'
|
||||
- 'closed'
|
||||
paths:
|
||||
- '.github/workflows/backport.yml'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
|
||||
|
||||
@@ -1,45 +1,34 @@
|
||||
name: Community PR labelling
|
||||
name: Label Community Contributors PRs
|
||||
|
||||
on:
|
||||
# We need "write" permissions on the PR to be able to add a label.
|
||||
pull_request_target: # We need this to have labelling permissions. There are no user inputs here, so we should be fine.
|
||||
types:
|
||||
- opened
|
||||
|
||||
permissions: {}
|
||||
pull_request:
|
||||
types: [opened]
|
||||
|
||||
jobs:
|
||||
label-if-community:
|
||||
name: Add 'community' label if the PR is from a community contributor
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
add-community-label:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
issues: write
|
||||
pull-requests: write
|
||||
|
||||
steps:
|
||||
- name: Check if author is org member
|
||||
id: check_membership
|
||||
- name: Label community contributors
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
AUTHOR: ${{ github.event.pull_request.user.login }}
|
||||
ORG: ${{ github.repository_owner }}
|
||||
run: |
|
||||
echo "Checking if $AUTHOR is a member of $ORG"
|
||||
if gh api --method GET "orgs/$ORG/members/$AUTHOR" >/dev/null 2>&1; then
|
||||
echo "is_member=true" >> $GITHUB_OUTPUT
|
||||
echo "$AUTHOR is an organization member"
|
||||
else
|
||||
echo "is_member=false" >> $GITHUB_OUTPUT
|
||||
echo "$AUTHOR is not an organization member"
|
||||
fi
|
||||
# Fetch fresh PR data to get current author_association
|
||||
ASSOCIATION=$(gh api /repos/${{ github.repository }}/pulls/${{ github.event.number }} --jq '.author_association')
|
||||
AUTHOR=$(gh api /repos/${{ github.repository }}/pulls/${{ github.event.number }} --jq '.user.login')
|
||||
|
||||
- name: Add community label
|
||||
if: steps.check_membership.outputs.is_member == 'false'
|
||||
env:
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
run: |
|
||||
echo "Adding 'community' label to PR #$PR_NUMBER"
|
||||
gh api /repos/${{ github.repository }}/issues/${{ github.event.number }}/labels \
|
||||
-X POST \
|
||||
-f labels[]='community'
|
||||
echo "Author: $AUTHOR, Association: $ASSOCIATION"
|
||||
|
||||
# Members have associations like: OWNER, MEMBER, COLLABORATOR
|
||||
# Non-members have: CONTRIBUTOR, FIRST_TIME_CONTRIBUTOR, FIRST_TIMER, NONE
|
||||
if [[ "$ASSOCIATION" != "OWNER" && "$ASSOCIATION" != "MEMBER" && "$ASSOCIATION" != "COLLABORATOR" ]]; then
|
||||
gh api /repos/${{ github.repository }}/issues/${{ github.event.number }}/labels \
|
||||
-X POST \
|
||||
-f labels[]='community'
|
||||
echo "Added 'community' label for $ASSOCIATION contributor"
|
||||
else
|
||||
echo "Skipped labeling for $ASSOCIATION"
|
||||
fi
|
||||
|
||||
@@ -16,7 +16,7 @@ permissions:
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
# Tags
|
||||
@@ -80,23 +80,8 @@ jobs:
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Notify container push started
|
||||
if: github.event_name == 'release'
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_PLATFORM_DEPLOYMENTS }}
|
||||
COMPONENT: MCP
|
||||
RELEASE_TAG: ${{ env.RELEASE_TAG }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-started.json"
|
||||
|
||||
- name: Build and push MCP container (release)
|
||||
if: github.event_name == 'release'
|
||||
id: container-push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.WORKING_DIRECTORY }}
|
||||
@@ -115,31 +100,8 @@ jobs:
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Notify container push completed
|
||||
if: github.event_name == 'release' && always()
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_PLATFORM_DEPLOYMENTS }}
|
||||
COMPONENT: MCP
|
||||
RELEASE_TAG: ${{ env.RELEASE_TAG }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-completed.json"
|
||||
step-outcome: ${{ steps.container-push.outcome }}
|
||||
|
||||
trigger-deployment:
|
||||
if: github.event_name == 'push'
|
||||
needs: [setup, container-build-push]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Trigger MCP deployment
|
||||
if: github.event_name == 'push'
|
||||
uses: peter-evans/repository-dispatch@5fc4efd1a4797ddb68ffd0714a238564e4cc0e6f # v4.0.0
|
||||
with:
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
|
||||
+15
-22
@@ -1,14 +1,24 @@
|
||||
name: 'MCP: Container Checks'
|
||||
name: 'MCP: Pull Request'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
paths:
|
||||
- '.github/workflows/mcp-pull-request.yml'
|
||||
- 'mcp_server/**'
|
||||
- '!mcp_server/README.md'
|
||||
- '!mcp_server/CHANGELOG.md'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
paths:
|
||||
- '.github/workflows/mcp-pull-request.yml'
|
||||
- 'mcp_server/**'
|
||||
- '!mcp_server/README.md'
|
||||
- '!mcp_server/CHANGELOG.md'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
@@ -19,7 +29,8 @@ env:
|
||||
IMAGE_NAME: prowler-mcp
|
||||
|
||||
jobs:
|
||||
mcp-dockerfile-lint:
|
||||
dockerfile-lint:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
@@ -29,19 +40,13 @@ jobs:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check if Dockerfile changed
|
||||
id: dockerfile-changed
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: mcp_server/Dockerfile
|
||||
|
||||
- name: Lint Dockerfile with Hadolint
|
||||
if: steps.dockerfile-changed.outputs.any_changed == 'true'
|
||||
uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
|
||||
with:
|
||||
dockerfile: mcp_server/Dockerfile
|
||||
|
||||
mcp-container-build-and-scan:
|
||||
container-build-and-scan:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
@@ -53,21 +58,10 @@ jobs:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check for MCP changes
|
||||
id: check-changes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: mcp_server/**
|
||||
files_ignore: |
|
||||
mcp_server/README.md
|
||||
mcp_server/CHANGELOG.md
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
|
||||
- name: Build MCP container
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.MCP_WORKING_DIR }}
|
||||
@@ -78,7 +72,6 @@ jobs:
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Scan MCP container with Trivy
|
||||
if: github.repository == 'prowler-cloud/prowler' && steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: ./.github/actions/trivy-scan
|
||||
with:
|
||||
image-name: ${{ env.IMAGE_NAME }}
|
||||
@@ -47,7 +47,7 @@ jobs:
|
||||
git config --global user.name 'prowler-bot'
|
||||
git config --global user.email '179230569+prowler-bot@users.noreply.github.com'
|
||||
|
||||
- name: Parse version and determine branch
|
||||
- name: Parse version and read changelogs
|
||||
run: |
|
||||
# Validate version format (reusing pattern from sdk-bump-version.yml)
|
||||
if [[ $PROWLER_VERSION =~ ^([0-9]+)\.([0-9]+)\.([0-9]+)$ ]]; then
|
||||
@@ -64,83 +64,66 @@ jobs:
|
||||
BRANCH_NAME="v${MAJOR_VERSION}.${MINOR_VERSION}"
|
||||
echo "BRANCH_NAME=${BRANCH_NAME}" >> "${GITHUB_ENV}"
|
||||
|
||||
# Function to extract the latest version from changelog
|
||||
extract_latest_version() {
|
||||
local changelog_file="$1"
|
||||
if [ -f "$changelog_file" ]; then
|
||||
# Extract the first version entry (most recent) from changelog
|
||||
# Format: ## [version] (1.2.3) or ## [vversion] (v1.2.3)
|
||||
local version=$(grep -m 1 '^## \[' "$changelog_file" | sed 's/^## \[\(.*\)\].*/\1/' | sed 's/^v//' | tr -d '[:space:]')
|
||||
echo "$version"
|
||||
else
|
||||
echo ""
|
||||
fi
|
||||
}
|
||||
|
||||
# Read actual versions from changelogs (source of truth)
|
||||
UI_VERSION=$(extract_latest_version "ui/CHANGELOG.md")
|
||||
API_VERSION=$(extract_latest_version "api/CHANGELOG.md")
|
||||
SDK_VERSION=$(extract_latest_version "prowler/CHANGELOG.md")
|
||||
MCP_VERSION=$(extract_latest_version "mcp_server/CHANGELOG.md")
|
||||
|
||||
echo "UI_VERSION=${UI_VERSION}" >> "${GITHUB_ENV}"
|
||||
echo "API_VERSION=${API_VERSION}" >> "${GITHUB_ENV}"
|
||||
echo "SDK_VERSION=${SDK_VERSION}" >> "${GITHUB_ENV}"
|
||||
echo "MCP_VERSION=${MCP_VERSION}" >> "${GITHUB_ENV}"
|
||||
|
||||
if [ -n "$UI_VERSION" ]; then
|
||||
echo "Read UI version from changelog: $UI_VERSION"
|
||||
else
|
||||
echo "Warning: No UI version found in ui/CHANGELOG.md"
|
||||
fi
|
||||
|
||||
if [ -n "$API_VERSION" ]; then
|
||||
echo "Read API version from changelog: $API_VERSION"
|
||||
else
|
||||
echo "Warning: No API version found in api/CHANGELOG.md"
|
||||
fi
|
||||
|
||||
if [ -n "$SDK_VERSION" ]; then
|
||||
echo "Read SDK version from changelog: $SDK_VERSION"
|
||||
else
|
||||
echo "Warning: No SDK version found in prowler/CHANGELOG.md"
|
||||
fi
|
||||
|
||||
if [ -n "$MCP_VERSION" ]; then
|
||||
echo "Read MCP version from changelog: $MCP_VERSION"
|
||||
else
|
||||
echo "Warning: No MCP version found in mcp_server/CHANGELOG.md"
|
||||
fi
|
||||
|
||||
echo "Prowler version: $PROWLER_VERSION"
|
||||
echo "Branch name: $BRANCH_NAME"
|
||||
echo "UI version: $UI_VERSION"
|
||||
echo "API version: $API_VERSION"
|
||||
echo "SDK version: $SDK_VERSION"
|
||||
echo "MCP version: $MCP_VERSION"
|
||||
echo "Is minor release: $([ $PATCH_VERSION -eq 0 ] && echo 'true' || echo 'false')"
|
||||
else
|
||||
echo "Invalid version syntax: '$PROWLER_VERSION' (must be N.N.N)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Checkout release branch
|
||||
run: |
|
||||
echo "Checking out branch $BRANCH_NAME for release $PROWLER_VERSION..."
|
||||
if git show-ref --verify --quiet "refs/heads/$BRANCH_NAME"; then
|
||||
echo "Branch $BRANCH_NAME exists locally, checking out..."
|
||||
git checkout "$BRANCH_NAME"
|
||||
elif git show-ref --verify --quiet "refs/remotes/origin/$BRANCH_NAME"; then
|
||||
echo "Branch $BRANCH_NAME exists remotely, checking out..."
|
||||
git checkout -b "$BRANCH_NAME" "origin/$BRANCH_NAME"
|
||||
else
|
||||
echo "ERROR: Branch $BRANCH_NAME does not exist. For minor releases (X.Y.0), create it manually first. For patch releases (X.Y.Z), the branch should already exist."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Read changelog versions from release branch
|
||||
run: |
|
||||
# Function to extract the latest version from changelog
|
||||
extract_latest_version() {
|
||||
local changelog_file="$1"
|
||||
if [ -f "$changelog_file" ]; then
|
||||
# Extract the first version entry (most recent) from changelog
|
||||
# Format: ## [version] (1.2.3) or ## [vversion] (v1.2.3)
|
||||
local version=$(grep -m 1 '^## \[' "$changelog_file" | sed 's/^## \[\(.*\)\].*/\1/' | sed 's/^v//' | tr -d '[:space:]')
|
||||
echo "$version"
|
||||
else
|
||||
echo ""
|
||||
fi
|
||||
}
|
||||
|
||||
# Read actual versions from changelogs (source of truth)
|
||||
UI_VERSION=$(extract_latest_version "ui/CHANGELOG.md")
|
||||
API_VERSION=$(extract_latest_version "api/CHANGELOG.md")
|
||||
SDK_VERSION=$(extract_latest_version "prowler/CHANGELOG.md")
|
||||
MCP_VERSION=$(extract_latest_version "mcp_server/CHANGELOG.md")
|
||||
|
||||
echo "UI_VERSION=${UI_VERSION}" >> "${GITHUB_ENV}"
|
||||
echo "API_VERSION=${API_VERSION}" >> "${GITHUB_ENV}"
|
||||
echo "SDK_VERSION=${SDK_VERSION}" >> "${GITHUB_ENV}"
|
||||
echo "MCP_VERSION=${MCP_VERSION}" >> "${GITHUB_ENV}"
|
||||
|
||||
if [ -n "$UI_VERSION" ]; then
|
||||
echo "Read UI version from changelog: $UI_VERSION"
|
||||
else
|
||||
echo "Warning: No UI version found in ui/CHANGELOG.md"
|
||||
fi
|
||||
|
||||
if [ -n "$API_VERSION" ]; then
|
||||
echo "Read API version from changelog: $API_VERSION"
|
||||
else
|
||||
echo "Warning: No API version found in api/CHANGELOG.md"
|
||||
fi
|
||||
|
||||
if [ -n "$SDK_VERSION" ]; then
|
||||
echo "Read SDK version from changelog: $SDK_VERSION"
|
||||
else
|
||||
echo "Warning: No SDK version found in prowler/CHANGELOG.md"
|
||||
fi
|
||||
|
||||
if [ -n "$MCP_VERSION" ]; then
|
||||
echo "Read MCP version from changelog: $MCP_VERSION"
|
||||
else
|
||||
echo "Warning: No MCP version found in mcp_server/CHANGELOG.md"
|
||||
fi
|
||||
|
||||
echo "UI version: $UI_VERSION"
|
||||
echo "API version: $API_VERSION"
|
||||
echo "SDK version: $SDK_VERSION"
|
||||
echo "MCP version: $MCP_VERSION"
|
||||
|
||||
- name: Extract and combine changelog entries
|
||||
run: |
|
||||
set -e
|
||||
@@ -272,6 +255,21 @@ jobs:
|
||||
echo "Combined changelog preview:"
|
||||
cat combined_changelog.md
|
||||
|
||||
- name: Checkout release branch for patch release
|
||||
if: ${{ env.PATCH_VERSION != '0' }}
|
||||
run: |
|
||||
echo "Patch release detected, checking out existing branch $BRANCH_NAME..."
|
||||
if git show-ref --verify --quiet "refs/heads/$BRANCH_NAME"; then
|
||||
echo "Branch $BRANCH_NAME exists locally, checking out..."
|
||||
git checkout "$BRANCH_NAME"
|
||||
elif git show-ref --verify --quiet "refs/remotes/origin/$BRANCH_NAME"; then
|
||||
echo "Branch $BRANCH_NAME exists remotely, checking out..."
|
||||
git checkout -b "$BRANCH_NAME" "origin/$BRANCH_NAME"
|
||||
else
|
||||
echo "ERROR: Branch $BRANCH_NAME should exist for patch release $PROWLER_VERSION"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Verify SDK version in pyproject.toml
|
||||
run: |
|
||||
CURRENT_VERSION=$(grep '^version = ' pyproject.toml | sed -E 's/version = "([^"]+)"/\1/' | tr -d '[:space:]')
|
||||
@@ -325,6 +323,18 @@ jobs:
|
||||
fi
|
||||
echo "✓ api/src/backend/api/v1/views.py version: $CURRENT_API_VERSION"
|
||||
|
||||
- name: Checkout release branch for minor release
|
||||
if: ${{ env.PATCH_VERSION == '0' }}
|
||||
run: |
|
||||
echo "Minor release detected (patch = 0), checking out existing branch $BRANCH_NAME..."
|
||||
if git show-ref --verify --quiet "refs/remotes/origin/$BRANCH_NAME"; then
|
||||
echo "Branch $BRANCH_NAME exists remotely, checking out..."
|
||||
git checkout -b "$BRANCH_NAME" "origin/$BRANCH_NAME"
|
||||
else
|
||||
echo "ERROR: Branch $BRANCH_NAME should exist for minor release $PROWLER_VERSION. Please create it manually first."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Update API prowler dependency for minor release
|
||||
if: ${{ env.PATCH_VERSION == '0' }}
|
||||
run: |
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
name: SDK - Build and Push containers
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
# For `v3-latest`
|
||||
- "v3"
|
||||
# For `v4-latest`
|
||||
- "v4.6"
|
||||
# For `latest`
|
||||
- "master"
|
||||
paths-ignore:
|
||||
- ".github/**"
|
||||
- "README.md"
|
||||
- "docs/**"
|
||||
- "ui/**"
|
||||
- "api/**"
|
||||
|
||||
release:
|
||||
types: [published]
|
||||
|
||||
env:
|
||||
# AWS Configuration
|
||||
AWS_REGION_STG: eu-west-1
|
||||
AWS_REGION_PLATFORM: eu-west-1
|
||||
AWS_REGION: us-east-1
|
||||
|
||||
# Container's configuration
|
||||
IMAGE_NAME: prowler
|
||||
DOCKERFILE_PATH: ./Dockerfile
|
||||
|
||||
# Tags
|
||||
LATEST_TAG: latest
|
||||
STABLE_TAG: stable
|
||||
# The RELEASE_TAG is set during runtime in releases
|
||||
RELEASE_TAG: ""
|
||||
# The PROWLER_VERSION and PROWLER_VERSION_MAJOR are set during runtime in releases
|
||||
PROWLER_VERSION: ""
|
||||
PROWLER_VERSION_MAJOR: ""
|
||||
# TEMPORARY_TAG: temporary
|
||||
|
||||
# Python configuration
|
||||
PYTHON_VERSION: 3.12
|
||||
|
||||
# Container Registries
|
||||
PROWLERCLOUD_DOCKERHUB_REPOSITORY: prowlercloud
|
||||
PROWLERCLOUD_DOCKERHUB_IMAGE: prowler
|
||||
|
||||
jobs:
|
||||
# Build Prowler OSS container
|
||||
container-build-push:
|
||||
# needs: dockerfile-linter
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
prowler_version_major: ${{ steps.get-prowler-version.outputs.PROWLER_VERSION_MAJOR }}
|
||||
prowler_version: ${{ steps.get-prowler-version.outputs.PROWLER_VERSION }}
|
||||
env:
|
||||
POETRY_VIRTUALENVS_CREATE: "false"
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
|
||||
with:
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
|
||||
- name: Install Poetry
|
||||
run: |
|
||||
pipx install poetry==2.*
|
||||
pipx inject poetry poetry-bumpversion
|
||||
|
||||
- name: Get Prowler version
|
||||
id: get-prowler-version
|
||||
run: |
|
||||
PROWLER_VERSION="$(poetry version -s 2>/dev/null)"
|
||||
echo "PROWLER_VERSION=${PROWLER_VERSION}" >> "${GITHUB_ENV}"
|
||||
echo "PROWLER_VERSION=${PROWLER_VERSION}" >> "${GITHUB_OUTPUT}"
|
||||
|
||||
# Store prowler version major just for the release
|
||||
PROWLER_VERSION_MAJOR="${PROWLER_VERSION%%.*}"
|
||||
echo "PROWLER_VERSION_MAJOR=${PROWLER_VERSION_MAJOR}" >> "${GITHUB_ENV}"
|
||||
echo "PROWLER_VERSION_MAJOR=${PROWLER_VERSION_MAJOR}" >> "${GITHUB_OUTPUT}"
|
||||
|
||||
case ${PROWLER_VERSION_MAJOR} in
|
||||
3)
|
||||
echo "LATEST_TAG=v3-latest" >> "${GITHUB_ENV}"
|
||||
echo "STABLE_TAG=v3-stable" >> "${GITHUB_ENV}"
|
||||
;;
|
||||
|
||||
|
||||
4)
|
||||
echo "LATEST_TAG=v4-latest" >> "${GITHUB_ENV}"
|
||||
echo "STABLE_TAG=v4-stable" >> "${GITHUB_ENV}"
|
||||
;;
|
||||
|
||||
5)
|
||||
echo "LATEST_TAG=latest" >> "${GITHUB_ENV}"
|
||||
echo "STABLE_TAG=stable" >> "${GITHUB_ENV}"
|
||||
;;
|
||||
|
||||
*)
|
||||
# Fallback if any other version is present
|
||||
echo "Releasing another Prowler major version, aborting..."
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
- name: Login to DockerHub
|
||||
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to Public ECR
|
||||
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
|
||||
with:
|
||||
registry: public.ecr.aws
|
||||
username: ${{ secrets.PUBLIC_ECR_AWS_ACCESS_KEY_ID }}
|
||||
password: ${{ secrets.PUBLIC_ECR_AWS_SECRET_ACCESS_KEY }}
|
||||
env:
|
||||
AWS_REGION: ${{ env.AWS_REGION }}
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
|
||||
- name: Build and push container image (latest)
|
||||
if: github.event_name == 'push'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
push: true
|
||||
tags: |
|
||||
${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.LATEST_TAG }}
|
||||
${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.LATEST_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }}
|
||||
file: ${{ env.DOCKERFILE_PATH }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Build and push container image (release)
|
||||
if: github.event_name == 'release'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
# Use local context to get changes
|
||||
# https://github.com/docker/build-push-action#path-context
|
||||
context: .
|
||||
push: true
|
||||
tags: |
|
||||
${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.PROWLER_VERSION }}
|
||||
${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.STABLE_TAG }}
|
||||
${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.PROWLER_VERSION }}
|
||||
${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.STABLE_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.PROWLER_VERSION }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }}
|
||||
file: ${{ env.DOCKERFILE_PATH }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
# - name: Push README to Docker Hub (toniblyx)
|
||||
# uses: peter-evans/dockerhub-description@432a30c9e07499fd01da9f8a49f0faf9e0ca5b77 # v4.0.2
|
||||
# with:
|
||||
# username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
# password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
# repository: ${{ env.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}
|
||||
# readme-filepath: ./README.md
|
||||
#
|
||||
# - name: Push README to Docker Hub (prowlercloud)
|
||||
# uses: peter-evans/dockerhub-description@432a30c9e07499fd01da9f8a49f0faf9e0ca5b77 # v4.0.2
|
||||
# with:
|
||||
# username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
# password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
# repository: ${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}
|
||||
# readme-filepath: ./README.md
|
||||
|
||||
dispatch-action:
|
||||
needs: container-build-push
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Get latest commit info (latest)
|
||||
if: github.event_name == 'push'
|
||||
run: |
|
||||
LATEST_COMMIT_HASH=$(echo ${{ github.event.after }} | cut -b -7)
|
||||
echo "LATEST_COMMIT_HASH=${LATEST_COMMIT_HASH}" >> $GITHUB_ENV
|
||||
|
||||
- name: Dispatch event (latest)
|
||||
if: github.event_name == 'push' && needs.container-build-push.outputs.prowler_version_major == '3'
|
||||
run: |
|
||||
curl https://api.github.com/repos/${{ secrets.DISPATCH_OWNER }}/${{ secrets.DISPATCH_REPO }}/dispatches \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
-H "Authorization: Bearer ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}" \
|
||||
-H "X-GitHub-Api-Version: 2022-11-28" \
|
||||
--data '{"event_type":"dispatch","client_payload":{"version":"v3-latest", "tag": "${{ env.LATEST_COMMIT_HASH }}"}}'
|
||||
|
||||
- name: Dispatch event (release)
|
||||
if: github.event_name == 'release' && needs.container-build-push.outputs.prowler_version_major == '3'
|
||||
run: |
|
||||
curl https://api.github.com/repos/${{ secrets.DISPATCH_OWNER }}/${{ secrets.DISPATCH_REPO }}/dispatches \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
-H "Authorization: Bearer ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}" \
|
||||
-H "X-GitHub-Api-Version: 2022-11-28" \
|
||||
--data '{"event_type":"dispatch","client_payload":{"version":"release", "tag":"${{ needs.container-build-push.outputs.prowler_version }}"}}'
|
||||
@@ -1,218 +1,146 @@
|
||||
name: 'SDK: Bump Version'
|
||||
name: SDK - Bump Version
|
||||
|
||||
on:
|
||||
release:
|
||||
types:
|
||||
- 'published'
|
||||
types: [published]
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.release.tag_name }}
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
PROWLER_VERSION: ${{ github.event.release.tag_name }}
|
||||
BASE_BRANCH: master
|
||||
|
||||
jobs:
|
||||
detect-release-type:
|
||||
bump-version:
|
||||
name: Bump Version
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
outputs:
|
||||
is_minor: ${{ steps.detect.outputs.is_minor }}
|
||||
is_patch: ${{ steps.detect.outputs.is_patch }}
|
||||
major_version: ${{ steps.detect.outputs.major_version }}
|
||||
minor_version: ${{ steps.detect.outputs.minor_version }}
|
||||
patch_version: ${{ steps.detect.outputs.patch_version }}
|
||||
steps:
|
||||
- name: Detect release type and parse version
|
||||
id: detect
|
||||
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Get Prowler version
|
||||
shell: bash
|
||||
run: |
|
||||
if [[ $PROWLER_VERSION =~ ^([0-9]+)\.([0-9]+)\.([0-9]+)$ ]]; then
|
||||
MAJOR_VERSION=${BASH_REMATCH[1]}
|
||||
MINOR_VERSION=${BASH_REMATCH[2]}
|
||||
PATCH_VERSION=${BASH_REMATCH[3]}
|
||||
FIX_VERSION=${BASH_REMATCH[3]}
|
||||
|
||||
echo "major_version=${MAJOR_VERSION}" >> "${GITHUB_OUTPUT}"
|
||||
echo "minor_version=${MINOR_VERSION}" >> "${GITHUB_OUTPUT}"
|
||||
echo "patch_version=${PATCH_VERSION}" >> "${GITHUB_OUTPUT}"
|
||||
# Export version components to GitHub environment
|
||||
echo "MAJOR_VERSION=${MAJOR_VERSION}" >> "${GITHUB_ENV}"
|
||||
echo "MINOR_VERSION=${MINOR_VERSION}" >> "${GITHUB_ENV}"
|
||||
echo "FIX_VERSION=${FIX_VERSION}" >> "${GITHUB_ENV}"
|
||||
|
||||
if (( MAJOR_VERSION != 5 )); then
|
||||
echo "::error::Releasing another Prowler major version, aborting..."
|
||||
exit 1
|
||||
fi
|
||||
if (( MAJOR_VERSION == 5 )); then
|
||||
if (( FIX_VERSION == 0 )); then
|
||||
echo "Minor Release: $PROWLER_VERSION"
|
||||
|
||||
if (( PATCH_VERSION == 0 )); then
|
||||
echo "is_minor=true" >> "${GITHUB_OUTPUT}"
|
||||
echo "is_patch=false" >> "${GITHUB_OUTPUT}"
|
||||
echo "✓ Minor release detected: $PROWLER_VERSION"
|
||||
# Set up next minor version for master
|
||||
BUMP_VERSION_TO=${MAJOR_VERSION}.$((MINOR_VERSION + 1)).${FIX_VERSION}
|
||||
echo "BUMP_VERSION_TO=${BUMP_VERSION_TO}" >> "${GITHUB_ENV}"
|
||||
|
||||
TARGET_BRANCH=${BASE_BRANCH}
|
||||
echo "TARGET_BRANCH=${TARGET_BRANCH}" >> "${GITHUB_ENV}"
|
||||
|
||||
# Set up patch version for version branch
|
||||
PATCH_VERSION_TO=${MAJOR_VERSION}.${MINOR_VERSION}.1
|
||||
echo "PATCH_VERSION_TO=${PATCH_VERSION_TO}" >> "${GITHUB_ENV}"
|
||||
|
||||
VERSION_BRANCH=v${MAJOR_VERSION}.${MINOR_VERSION}
|
||||
echo "VERSION_BRANCH=${VERSION_BRANCH}" >> "${GITHUB_ENV}"
|
||||
|
||||
echo "Bumping to next minor version: ${BUMP_VERSION_TO} in branch ${TARGET_BRANCH}"
|
||||
echo "Bumping to next patch version: ${PATCH_VERSION_TO} in branch ${VERSION_BRANCH}"
|
||||
else
|
||||
echo "Patch Release: $PROWLER_VERSION"
|
||||
|
||||
BUMP_VERSION_TO=${MAJOR_VERSION}.${MINOR_VERSION}.$((FIX_VERSION + 1))
|
||||
echo "BUMP_VERSION_TO=${BUMP_VERSION_TO}" >> "${GITHUB_ENV}"
|
||||
|
||||
TARGET_BRANCH=v${MAJOR_VERSION}.${MINOR_VERSION}
|
||||
echo "TARGET_BRANCH=${TARGET_BRANCH}" >> "${GITHUB_ENV}"
|
||||
|
||||
echo "Bumping to next patch version: ${BUMP_VERSION_TO} in branch ${TARGET_BRANCH}"
|
||||
fi
|
||||
else
|
||||
echo "is_minor=false" >> "${GITHUB_OUTPUT}"
|
||||
echo "is_patch=true" >> "${GITHUB_OUTPUT}"
|
||||
echo "✓ Patch release detected: $PROWLER_VERSION"
|
||||
echo "Releasing another Prowler major version, aborting..."
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
echo "::error::Invalid version syntax: '$PROWLER_VERSION' (must be X.Y.Z)"
|
||||
echo "Invalid version syntax: '$PROWLER_VERSION' (must be N.N.N)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
bump-minor-version:
|
||||
needs: detect-release-type
|
||||
if: needs.detect-release-type.outputs.is_minor == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Calculate next minor version
|
||||
- name: Bump versions in files
|
||||
run: |
|
||||
MAJOR_VERSION=${{ needs.detect-release-type.outputs.major_version }}
|
||||
MINOR_VERSION=${{ needs.detect-release-type.outputs.minor_version }}
|
||||
echo "Using PROWLER_VERSION=$PROWLER_VERSION"
|
||||
echo "Using BUMP_VERSION_TO=$BUMP_VERSION_TO"
|
||||
|
||||
NEXT_MINOR_VERSION=${MAJOR_VERSION}.$((MINOR_VERSION + 1)).0
|
||||
echo "NEXT_MINOR_VERSION=${NEXT_MINOR_VERSION}" >> "${GITHUB_ENV}"
|
||||
set -e
|
||||
|
||||
echo "Current version: $PROWLER_VERSION"
|
||||
echo "Next minor version: $NEXT_MINOR_VERSION"
|
||||
echo "Bumping version in pyproject.toml ..."
|
||||
sed -i "s|version = \"${PROWLER_VERSION}\"|version = \"${BUMP_VERSION_TO}\"|" pyproject.toml
|
||||
|
||||
- name: Bump versions in files for master
|
||||
run: |
|
||||
set -e
|
||||
echo "Bumping version in prowler/config/config.py ..."
|
||||
sed -i "s|prowler_version = \"${PROWLER_VERSION}\"|prowler_version = \"${BUMP_VERSION_TO}\"|" prowler/config/config.py
|
||||
|
||||
sed -i "s|version = \"${PROWLER_VERSION}\"|version = \"${NEXT_MINOR_VERSION}\"|" pyproject.toml
|
||||
sed -i "s|prowler_version = \"${PROWLER_VERSION}\"|prowler_version = \"${NEXT_MINOR_VERSION}\"|" prowler/config/config.py
|
||||
sed -i "s|NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${PROWLER_VERSION}|NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${NEXT_MINOR_VERSION}|" .env
|
||||
echo "Bumping version in .env ..."
|
||||
sed -i "s|NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${PROWLER_VERSION}|NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${BUMP_VERSION_TO}|" .env
|
||||
|
||||
echo "Files modified:"
|
||||
git --no-pager diff
|
||||
git --no-pager diff
|
||||
|
||||
- name: Create PR for next minor version to master
|
||||
- name: Create Pull Request
|
||||
uses: peter-evans/create-pull-request@271a8d0340265f705b14b6d32b9829c1cb33d45e # v7.0.8
|
||||
with:
|
||||
author: prowler-bot <179230569+prowler-bot@users.noreply.github.com>
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
base: master
|
||||
commit-message: 'chore(release): Bump version to v${{ env.NEXT_MINOR_VERSION }}'
|
||||
branch: version-bump-to-v${{ env.NEXT_MINOR_VERSION }}
|
||||
title: 'chore(release): Bump version to v${{ env.NEXT_MINOR_VERSION }}'
|
||||
labels: no-changelog
|
||||
body: |
|
||||
### Description
|
||||
author: prowler-bot <179230569+prowler-bot@users.noreply.github.com>
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
base: ${{ env.TARGET_BRANCH }}
|
||||
commit-message: "chore(release): Bump version to v${{ env.BUMP_VERSION_TO }}"
|
||||
branch: "version-bump-to-v${{ env.BUMP_VERSION_TO }}"
|
||||
title: "chore(release): Bump version to v${{ env.BUMP_VERSION_TO }}"
|
||||
labels: no-changelog
|
||||
body: |
|
||||
### Description
|
||||
|
||||
Bump Prowler version to v${{ env.NEXT_MINOR_VERSION }} after releasing v${{ env.PROWLER_VERSION }}.
|
||||
Bump Prowler version to v${{ env.BUMP_VERSION_TO }}
|
||||
|
||||
### License
|
||||
### License
|
||||
|
||||
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
|
||||
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
|
||||
|
||||
- name: Checkout version branch
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
with:
|
||||
ref: v${{ needs.detect-release-type.outputs.major_version }}.${{ needs.detect-release-type.outputs.minor_version }}
|
||||
|
||||
- name: Calculate first patch version
|
||||
- name: Handle patch version for minor release
|
||||
if: env.FIX_VERSION == '0'
|
||||
run: |
|
||||
MAJOR_VERSION=${{ needs.detect-release-type.outputs.major_version }}
|
||||
MINOR_VERSION=${{ needs.detect-release-type.outputs.minor_version }}
|
||||
echo "Using PROWLER_VERSION=$PROWLER_VERSION"
|
||||
echo "Using PATCH_VERSION_TO=$PATCH_VERSION_TO"
|
||||
|
||||
FIRST_PATCH_VERSION=${MAJOR_VERSION}.${MINOR_VERSION}.1
|
||||
VERSION_BRANCH=v${MAJOR_VERSION}.${MINOR_VERSION}
|
||||
set -e
|
||||
|
||||
echo "FIRST_PATCH_VERSION=${FIRST_PATCH_VERSION}" >> "${GITHUB_ENV}"
|
||||
echo "VERSION_BRANCH=${VERSION_BRANCH}" >> "${GITHUB_ENV}"
|
||||
echo "Bumping version in pyproject.toml ..."
|
||||
sed -i "s|version = \"${PROWLER_VERSION}\"|version = \"${PATCH_VERSION_TO}\"|" pyproject.toml
|
||||
|
||||
echo "First patch version: $FIRST_PATCH_VERSION"
|
||||
echo "Version branch: $VERSION_BRANCH"
|
||||
echo "Bumping version in prowler/config/config.py ..."
|
||||
sed -i "s|prowler_version = \"${PROWLER_VERSION}\"|prowler_version = \"${PATCH_VERSION_TO}\"|" prowler/config/config.py
|
||||
|
||||
- name: Bump versions in files for version branch
|
||||
run: |
|
||||
set -e
|
||||
echo "Bumping version in .env ..."
|
||||
sed -i "s|NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${PROWLER_VERSION}|NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${PATCH_VERSION_TO}|" .env
|
||||
|
||||
sed -i "s|version = \"${PROWLER_VERSION}\"|version = \"${FIRST_PATCH_VERSION}\"|" pyproject.toml
|
||||
sed -i "s|prowler_version = \"${PROWLER_VERSION}\"|prowler_version = \"${FIRST_PATCH_VERSION}\"|" prowler/config/config.py
|
||||
sed -i "s|NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${PROWLER_VERSION}|NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${FIRST_PATCH_VERSION}|" .env
|
||||
git --no-pager diff
|
||||
|
||||
echo "Files modified:"
|
||||
git --no-pager diff
|
||||
|
||||
- name: Create PR for first patch version to version branch
|
||||
- name: Create Pull Request for patch version
|
||||
if: env.FIX_VERSION == '0'
|
||||
uses: peter-evans/create-pull-request@271a8d0340265f705b14b6d32b9829c1cb33d45e # v7.0.8
|
||||
with:
|
||||
author: prowler-bot <179230569+prowler-bot@users.noreply.github.com>
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
base: ${{ env.VERSION_BRANCH }}
|
||||
commit-message: 'chore(release): Bump version to v${{ env.FIRST_PATCH_VERSION }}'
|
||||
branch: version-bump-to-v${{ env.FIRST_PATCH_VERSION }}
|
||||
title: 'chore(release): Bump version to v${{ env.FIRST_PATCH_VERSION }}'
|
||||
labels: no-changelog
|
||||
body: |
|
||||
### Description
|
||||
author: prowler-bot <179230569+prowler-bot@users.noreply.github.com>
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
base: ${{ env.VERSION_BRANCH }}
|
||||
commit-message: "chore(release): Bump version to v${{ env.PATCH_VERSION_TO }}"
|
||||
branch: "version-bump-to-v${{ env.PATCH_VERSION_TO }}"
|
||||
title: "chore(release): Bump version to v${{ env.PATCH_VERSION_TO }}"
|
||||
labels: no-changelog
|
||||
body: |
|
||||
### Description
|
||||
|
||||
Bump Prowler version to v${{ env.FIRST_PATCH_VERSION }} in version branch after releasing v${{ env.PROWLER_VERSION }}.
|
||||
Bump Prowler version to v${{ env.PATCH_VERSION_TO }}
|
||||
|
||||
### License
|
||||
### License
|
||||
|
||||
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
|
||||
|
||||
bump-patch-version:
|
||||
needs: detect-release-type
|
||||
if: needs.detect-release-type.outputs.is_patch == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Calculate next patch version
|
||||
run: |
|
||||
MAJOR_VERSION=${{ needs.detect-release-type.outputs.major_version }}
|
||||
MINOR_VERSION=${{ needs.detect-release-type.outputs.minor_version }}
|
||||
PATCH_VERSION=${{ needs.detect-release-type.outputs.patch_version }}
|
||||
|
||||
NEXT_PATCH_VERSION=${MAJOR_VERSION}.${MINOR_VERSION}.$((PATCH_VERSION + 1))
|
||||
VERSION_BRANCH=v${MAJOR_VERSION}.${MINOR_VERSION}
|
||||
|
||||
echo "NEXT_PATCH_VERSION=${NEXT_PATCH_VERSION}" >> "${GITHUB_ENV}"
|
||||
echo "VERSION_BRANCH=${VERSION_BRANCH}" >> "${GITHUB_ENV}"
|
||||
|
||||
echo "Current version: $PROWLER_VERSION"
|
||||
echo "Next patch version: $NEXT_PATCH_VERSION"
|
||||
echo "Target branch: $VERSION_BRANCH"
|
||||
|
||||
- name: Bump versions in files for version branch
|
||||
run: |
|
||||
set -e
|
||||
|
||||
sed -i "s|version = \"${PROWLER_VERSION}\"|version = \"${NEXT_PATCH_VERSION}\"|" pyproject.toml
|
||||
sed -i "s|prowler_version = \"${PROWLER_VERSION}\"|prowler_version = \"${NEXT_PATCH_VERSION}\"|" prowler/config/config.py
|
||||
sed -i "s|NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${PROWLER_VERSION}|NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${NEXT_PATCH_VERSION}|" .env
|
||||
|
||||
echo "Files modified:"
|
||||
git --no-pager diff
|
||||
|
||||
- name: Create PR for next patch version to version branch
|
||||
uses: peter-evans/create-pull-request@271a8d0340265f705b14b6d32b9829c1cb33d45e # v7.0.8
|
||||
with:
|
||||
author: prowler-bot <179230569+prowler-bot@users.noreply.github.com>
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
base: ${{ env.VERSION_BRANCH }}
|
||||
commit-message: 'chore(release): Bump version to v${{ env.NEXT_PATCH_VERSION }}'
|
||||
branch: version-bump-to-v${{ env.NEXT_PATCH_VERSION }}
|
||||
title: 'chore(release): Bump version to v${{ env.NEXT_PATCH_VERSION }}'
|
||||
labels: no-changelog
|
||||
body: |
|
||||
### Description
|
||||
|
||||
Bump Prowler version to v${{ env.NEXT_PATCH_VERSION }} after releasing v${{ env.PROWLER_VERSION }}.
|
||||
|
||||
### License
|
||||
|
||||
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
|
||||
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
|
||||
|
||||
@@ -1,90 +0,0 @@
|
||||
name: 'SDK: Code Quality'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
sdk-code-quality:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
matrix:
|
||||
python-version:
|
||||
- '3.9'
|
||||
- '3.10'
|
||||
- '3.11'
|
||||
- '3.12'
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check for SDK changes
|
||||
id: check-changes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: ./**
|
||||
files_ignore: |
|
||||
.github/**
|
||||
prowler/CHANGELOG.md
|
||||
docs/**
|
||||
permissions/**
|
||||
api/**
|
||||
ui/**
|
||||
dashboard/**
|
||||
mcp_server/**
|
||||
README.md
|
||||
mkdocs.yml
|
||||
.backportrc.json
|
||||
.env
|
||||
docker-compose*
|
||||
examples/**
|
||||
.gitignore
|
||||
contrib/**
|
||||
|
||||
- name: Install Poetry
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: pipx install poetry==2.1.1
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
cache: 'poetry'
|
||||
|
||||
- name: Install dependencies
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry install --no-root
|
||||
poetry run pip list
|
||||
|
||||
- name: Check Poetry lock file
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry check --lock
|
||||
|
||||
- name: Lint with flake8
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run flake8 . --ignore=E266,W503,E203,E501,W605,E128 --exclude contrib,ui,api
|
||||
|
||||
- name: Check format with black
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run black --exclude api ui --check .
|
||||
|
||||
- name: Lint with pylint
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run pylint --disable=W,C,R,E -j 0 -rn -sn prowler/
|
||||
@@ -31,8 +31,7 @@ concurrency:
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
sdk-analyze:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
analyze:
|
||||
name: CodeQL Security Analysis
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
|
||||
@@ -1,217 +0,0 @@
|
||||
name: 'SDK: Container Build and Push'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'v3' # For v3-latest
|
||||
- 'v4.6' # For v4-latest
|
||||
- 'master' # For latest
|
||||
paths-ignore:
|
||||
- '.github/**'
|
||||
- '!.github/workflows/sdk-container-build-push.yml'
|
||||
- 'README.md'
|
||||
- 'docs/**'
|
||||
- 'ui/**'
|
||||
- 'api/**'
|
||||
release:
|
||||
types:
|
||||
- 'published'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
# Container configuration
|
||||
IMAGE_NAME: prowler
|
||||
DOCKERFILE_PATH: ./Dockerfile
|
||||
|
||||
# Python configuration
|
||||
PYTHON_VERSION: '3.12'
|
||||
|
||||
# Tags (dynamically set based on version)
|
||||
LATEST_TAG: latest
|
||||
STABLE_TAG: stable
|
||||
|
||||
# Container registries
|
||||
PROWLERCLOUD_DOCKERHUB_REPOSITORY: prowlercloud
|
||||
PROWLERCLOUD_DOCKERHUB_IMAGE: prowler
|
||||
|
||||
# AWS configuration (for ECR)
|
||||
AWS_REGION: us-east-1
|
||||
|
||||
jobs:
|
||||
container-build-push:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 45
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
outputs:
|
||||
prowler_version: ${{ steps.get-prowler-version.outputs.prowler_version }}
|
||||
prowler_version_major: ${{ steps.get-prowler-version.outputs.prowler_version_major }}
|
||||
env:
|
||||
POETRY_VIRTUALENVS_CREATE: 'false'
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Set up Python ${{ env.PYTHON_VERSION }}
|
||||
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
|
||||
with:
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
|
||||
- name: Install Poetry
|
||||
run: |
|
||||
pipx install poetry==2.1.1
|
||||
pipx inject poetry poetry-bumpversion
|
||||
|
||||
- name: Get Prowler version and set tags
|
||||
id: get-prowler-version
|
||||
run: |
|
||||
PROWLER_VERSION="$(poetry version -s 2>/dev/null)"
|
||||
echo "prowler_version=${PROWLER_VERSION}" >> "${GITHUB_OUTPUT}"
|
||||
echo "PROWLER_VERSION=${PROWLER_VERSION}" >> "${GITHUB_ENV}"
|
||||
|
||||
# Extract major version
|
||||
PROWLER_VERSION_MAJOR="${PROWLER_VERSION%%.*}"
|
||||
echo "prowler_version_major=${PROWLER_VERSION_MAJOR}" >> "${GITHUB_OUTPUT}"
|
||||
echo "PROWLER_VERSION_MAJOR=${PROWLER_VERSION_MAJOR}" >> "${GITHUB_ENV}"
|
||||
|
||||
# Set version-specific tags
|
||||
case ${PROWLER_VERSION_MAJOR} in
|
||||
3)
|
||||
echo "LATEST_TAG=v3-latest" >> "${GITHUB_ENV}"
|
||||
echo "STABLE_TAG=v3-stable" >> "${GITHUB_ENV}"
|
||||
echo "✓ Prowler v3 detected - tags: v3-latest, v3-stable"
|
||||
;;
|
||||
4)
|
||||
echo "LATEST_TAG=v4-latest" >> "${GITHUB_ENV}"
|
||||
echo "STABLE_TAG=v4-stable" >> "${GITHUB_ENV}"
|
||||
echo "✓ Prowler v4 detected - tags: v4-latest, v4-stable"
|
||||
;;
|
||||
5)
|
||||
echo "LATEST_TAG=latest" >> "${GITHUB_ENV}"
|
||||
echo "STABLE_TAG=stable" >> "${GITHUB_ENV}"
|
||||
echo "✓ Prowler v5 detected - tags: latest, stable"
|
||||
;;
|
||||
*)
|
||||
echo "::error::Unsupported Prowler major version: ${PROWLER_VERSION_MAJOR}"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
- name: Login to DockerHub
|
||||
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to Public ECR
|
||||
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
|
||||
with:
|
||||
registry: public.ecr.aws
|
||||
username: ${{ secrets.PUBLIC_ECR_AWS_ACCESS_KEY_ID }}
|
||||
password: ${{ secrets.PUBLIC_ECR_AWS_SECRET_ACCESS_KEY }}
|
||||
env:
|
||||
AWS_REGION: ${{ env.AWS_REGION }}
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
|
||||
- name: Build and push SDK container (latest)
|
||||
if: github.event_name == 'push'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: .
|
||||
file: ${{ env.DOCKERFILE_PATH }}
|
||||
push: true
|
||||
tags: |
|
||||
${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.LATEST_TAG }}
|
||||
${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.LATEST_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Notify container push started
|
||||
if: github.event_name == 'release'
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_PLATFORM_DEPLOYMENTS }}
|
||||
COMPONENT: SDK
|
||||
RELEASE_TAG: ${{ env.PROWLER_VERSION }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-started.json"
|
||||
|
||||
- name: Build and push SDK container (release)
|
||||
if: github.event_name == 'release'
|
||||
id: container-push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: .
|
||||
file: ${{ env.DOCKERFILE_PATH }}
|
||||
push: true
|
||||
tags: |
|
||||
${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.PROWLER_VERSION }}
|
||||
${{ secrets.DOCKER_HUB_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.STABLE_TAG }}
|
||||
${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.PROWLER_VERSION }}
|
||||
${{ secrets.PUBLIC_ECR_REPOSITORY }}/${{ env.IMAGE_NAME }}:${{ env.STABLE_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.PROWLER_VERSION }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Notify container push completed
|
||||
if: github.event_name == 'release' && always()
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_PLATFORM_DEPLOYMENTS }}
|
||||
COMPONENT: SDK
|
||||
RELEASE_TAG: ${{ env.PROWLER_VERSION }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-completed.json"
|
||||
step-outcome: ${{ steps.container-push.outcome }}
|
||||
|
||||
dispatch-v3-deployment:
|
||||
if: needs.container-build-push.outputs.prowler_version_major == '3'
|
||||
needs: container-build-push
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Calculate short SHA
|
||||
id: short-sha
|
||||
run: echo "short_sha=${GITHUB_SHA::7}" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Dispatch v3 deployment (latest)
|
||||
if: github.event_name == 'push'
|
||||
uses: peter-evans/repository-dispatch@5fc4efd1a4797ddb68ffd0714a238564e4cc0e6f # v4.0.0
|
||||
with:
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
repository: ${{ secrets.DISPATCH_OWNER }}/${{ secrets.DISPATCH_REPO }}
|
||||
event-type: dispatch
|
||||
client-payload: '{"version":"v3-latest","tag":"${{ steps.short-sha.outputs.short_sha }}"}'
|
||||
|
||||
- name: Dispatch v3 deployment (release)
|
||||
if: github.event_name == 'release'
|
||||
uses: peter-evans/repository-dispatch@5fc4efd1a4797ddb68ffd0714a238564e4cc0e6f # v4.0.0
|
||||
with:
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
repository: ${{ secrets.DISPATCH_OWNER }}/${{ secrets.DISPATCH_REPO }}
|
||||
event-type: dispatch
|
||||
client-payload: '{"version":"release","tag":"${{ needs.container-build-push.outputs.prowler_version }}"}'
|
||||
@@ -1,103 +0,0 @@
|
||||
name: 'SDK: Container Checks'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
IMAGE_NAME: prowler
|
||||
|
||||
jobs:
|
||||
sdk-dockerfile-lint:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check if Dockerfile changed
|
||||
id: dockerfile-changed
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: Dockerfile
|
||||
|
||||
- name: Lint Dockerfile with Hadolint
|
||||
if: steps.dockerfile-changed.outputs.any_changed == 'true'
|
||||
uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
|
||||
with:
|
||||
dockerfile: Dockerfile
|
||||
ignore: DL3013
|
||||
|
||||
sdk-container-build-and-scan:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
pull-requests: write
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check for SDK changes
|
||||
id: check-changes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: ./**
|
||||
files_ignore: |
|
||||
.github/**
|
||||
prowler/CHANGELOG.md
|
||||
docs/**
|
||||
permissions/**
|
||||
api/**
|
||||
ui/**
|
||||
dashboard/**
|
||||
mcp_server/**
|
||||
README.md
|
||||
mkdocs.yml
|
||||
.backportrc.json
|
||||
.env
|
||||
docker-compose*
|
||||
examples/**
|
||||
.gitignore
|
||||
contrib/**
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
|
||||
- name: Build SDK container
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: .
|
||||
push: false
|
||||
load: true
|
||||
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Scan SDK container with Trivy
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: ./.github/actions/trivy-scan
|
||||
with:
|
||||
image-name: ${{ env.IMAGE_NAME }}
|
||||
image-tag: ${{ github.sha }}
|
||||
fail-on-critical: 'false'
|
||||
severity: 'CRITICAL'
|
||||
@@ -0,0 +1,286 @@
|
||||
name: SDK - Pull Request
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- "master"
|
||||
- "v3"
|
||||
- "v4.*"
|
||||
- "v5.*"
|
||||
pull_request:
|
||||
branches:
|
||||
- "master"
|
||||
- "v3"
|
||||
- "v4.*"
|
||||
- "v5.*"
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
python-version: ["3.9", "3.10", "3.11", "3.12"]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Test if changes are in not ignored paths
|
||||
id: are-non-ignored-files-changed
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: ./**
|
||||
files_ignore: |
|
||||
.github/**
|
||||
docs/**
|
||||
permissions/**
|
||||
api/**
|
||||
ui/**
|
||||
prowler/CHANGELOG.md
|
||||
README.md
|
||||
mkdocs.yml
|
||||
.backportrc.json
|
||||
.env
|
||||
docker-compose*
|
||||
examples/**
|
||||
.gitignore
|
||||
|
||||
- name: Install poetry
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pipx install poetry==2.1.1
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
cache: "poetry"
|
||||
|
||||
- name: Install dependencies
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry install --no-root
|
||||
poetry run pip list
|
||||
VERSION=$(curl --silent "https://api.github.com/repos/hadolint/hadolint/releases/latest" | \
|
||||
grep '"tag_name":' | \
|
||||
sed -E 's/.*"v([^"]+)".*/\1/' \
|
||||
) && curl -L -o /tmp/hadolint "https://github.com/hadolint/hadolint/releases/download/v${VERSION}/hadolint-Linux-x86_64" \
|
||||
&& chmod +x /tmp/hadolint
|
||||
|
||||
- name: Poetry check
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry check --lock
|
||||
|
||||
- name: Lint with flake8
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run flake8 . --ignore=E266,W503,E203,E501,W605,E128 --exclude contrib,ui,api
|
||||
|
||||
- name: Checking format with black
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run black --exclude api ui --check .
|
||||
|
||||
- name: Lint with pylint
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pylint --disable=W,C,R,E -j 0 -rn -sn prowler/
|
||||
|
||||
- name: Bandit
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run bandit -q -lll -x '*_test.py,./contrib/,./api/,./ui' -r .
|
||||
|
||||
- name: Safety
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run safety check --ignore 70612 -r pyproject.toml
|
||||
|
||||
- name: Vulture
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run vulture --exclude "contrib,api,ui" --min-confidence 100 .
|
||||
|
||||
- name: Dockerfile - Check if Dockerfile has changed
|
||||
id: dockerfile-changed-files
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
Dockerfile
|
||||
|
||||
- name: Hadolint
|
||||
if: steps.dockerfile-changed-files.outputs.any_changed == 'true'
|
||||
run: |
|
||||
/tmp/hadolint Dockerfile --ignore=DL3013
|
||||
|
||||
# Test AWS
|
||||
- name: AWS - Check if any file has changed
|
||||
id: aws-changed-files
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/providers/aws/**
|
||||
./tests/providers/aws/**
|
||||
./poetry.lock
|
||||
|
||||
- name: AWS - Test
|
||||
if: steps.aws-changed-files.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/providers/aws --cov-report=xml:aws_coverage.xml tests/providers/aws
|
||||
|
||||
# Test Azure
|
||||
- name: Azure - Check if any file has changed
|
||||
id: azure-changed-files
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/providers/azure/**
|
||||
./tests/providers/azure/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Azure - Test
|
||||
if: steps.azure-changed-files.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/providers/azure --cov-report=xml:azure_coverage.xml tests/providers/azure
|
||||
|
||||
# Test GCP
|
||||
- name: GCP - Check if any file has changed
|
||||
id: gcp-changed-files
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/providers/gcp/**
|
||||
./tests/providers/gcp/**
|
||||
./poetry.lock
|
||||
|
||||
- name: GCP - Test
|
||||
if: steps.gcp-changed-files.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/providers/gcp --cov-report=xml:gcp_coverage.xml tests/providers/gcp
|
||||
|
||||
# Test Kubernetes
|
||||
- name: Kubernetes - Check if any file has changed
|
||||
id: kubernetes-changed-files
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/providers/kubernetes/**
|
||||
./tests/providers/kubernetes/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Kubernetes - Test
|
||||
if: steps.kubernetes-changed-files.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/providers/kubernetes --cov-report=xml:kubernetes_coverage.xml tests/providers/kubernetes
|
||||
|
||||
# Test GitHub
|
||||
- name: GitHub - Check if any file has changed
|
||||
id: github-changed-files
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/providers/github/**
|
||||
./tests/providers/github/**
|
||||
./poetry.lock
|
||||
|
||||
- name: GitHub - Test
|
||||
if: steps.github-changed-files.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/providers/github --cov-report=xml:github_coverage.xml tests/providers/github
|
||||
|
||||
# Test NHN
|
||||
- name: NHN - Check if any file has changed
|
||||
id: nhn-changed-files
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/providers/nhn/**
|
||||
./tests/providers/nhn/**
|
||||
./poetry.lock
|
||||
|
||||
- name: NHN - Test
|
||||
if: steps.nhn-changed-files.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/providers/nhn --cov-report=xml:nhn_coverage.xml tests/providers/nhn
|
||||
|
||||
# Test M365
|
||||
- name: M365 - Check if any file has changed
|
||||
id: m365-changed-files
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/providers/m365/**
|
||||
./tests/providers/m365/**
|
||||
./poetry.lock
|
||||
|
||||
- name: M365 - Test
|
||||
if: steps.m365-changed-files.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/providers/m365 --cov-report=xml:m365_coverage.xml tests/providers/m365
|
||||
|
||||
# Test IaC
|
||||
- name: IaC - Check if any file has changed
|
||||
id: iac-changed-files
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/providers/iac/**
|
||||
./tests/providers/iac/**
|
||||
./poetry.lock
|
||||
|
||||
- name: IaC - Test
|
||||
if: steps.iac-changed-files.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/providers/iac --cov-report=xml:iac_coverage.xml tests/providers/iac
|
||||
|
||||
# Test MongoDB Atlas
|
||||
- name: MongoDB Atlas - Check if any file has changed
|
||||
id: mongodb-atlas-changed-files
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/providers/mongodbatlas/**
|
||||
./tests/providers/mongodbatlas/**
|
||||
.poetry.lock
|
||||
|
||||
- name: MongoDB Atlas - Test
|
||||
if: steps.mongodb-atlas-changed-files.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/providers/mongodbatlas --cov-report=xml:mongodb_atlas_coverage.xml tests/providers/mongodbatlas
|
||||
|
||||
# Test OCI
|
||||
- name: OCI - Check if any file has changed
|
||||
id: oci-changed-files
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/providers/oraclecloud/**
|
||||
./tests/providers/oraclecloud/**
|
||||
./poetry.lock
|
||||
|
||||
- name: OCI - Test
|
||||
if: steps.oci-changed-files.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/providers/oraclecloud --cov-report=xml:oci_coverage.xml tests/providers/oraclecloud
|
||||
|
||||
# Common Tests
|
||||
- name: Lib - Test
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/lib --cov-report=xml:lib_coverage.xml tests/lib
|
||||
|
||||
- name: Config - Test
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
run: |
|
||||
poetry run pytest -n auto --cov=./prowler/config --cov-report=xml:config_coverage.xml tests/config
|
||||
|
||||
# Codecov
|
||||
- name: Upload coverage reports to Codecov
|
||||
if: steps.are-non-ignored-files-changed.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler
|
||||
files: ./aws_coverage.xml,./azure_coverage.xml,./gcp_coverage.xml,./kubernetes_coverage.xml,./github_coverage.xml,./nhn_coverage.xml,./m365_coverage.xml,./oci_coverage.xml,./lib_coverage.xml,./config_coverage.xml
|
||||
@@ -1,119 +1,98 @@
|
||||
name: 'SDK: PyPI Release'
|
||||
name: SDK - PyPI release
|
||||
|
||||
on:
|
||||
release:
|
||||
types:
|
||||
- 'published'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.release.tag_name }}
|
||||
cancel-in-progress: false
|
||||
types: [published]
|
||||
|
||||
env:
|
||||
RELEASE_TAG: ${{ github.event.release.tag_name }}
|
||||
PYTHON_VERSION: '3.12'
|
||||
PYTHON_VERSION: 3.11
|
||||
# CACHE: "poetry"
|
||||
|
||||
jobs:
|
||||
validate-release:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
repository-check:
|
||||
name: Repository check
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
outputs:
|
||||
prowler_version: ${{ steps.parse-version.outputs.version }}
|
||||
major_version: ${{ steps.parse-version.outputs.major }}
|
||||
|
||||
is_repo: ${{ steps.repository_check.outputs.is_repo }}
|
||||
steps:
|
||||
- name: Parse and validate version
|
||||
id: parse-version
|
||||
- name: Repository check
|
||||
id: repository_check
|
||||
working-directory: /tmp
|
||||
run: |
|
||||
if [[ ${{ github.repository }} == "prowler-cloud/prowler" ]]
|
||||
then
|
||||
echo "is_repo=true" >> "${GITHUB_OUTPUT}"
|
||||
else
|
||||
echo "This action only runs for prowler-cloud/prowler"
|
||||
echo "is_repo=false" >> "${GITHUB_OUTPUT}"
|
||||
fi
|
||||
|
||||
release-prowler-job:
|
||||
runs-on: ubuntu-latest
|
||||
needs: repository-check
|
||||
if: needs.repository-check.outputs.is_repo == 'true'
|
||||
env:
|
||||
POETRY_VIRTUALENVS_CREATE: "false"
|
||||
name: Release Prowler to PyPI
|
||||
steps:
|
||||
- name: Repository check
|
||||
working-directory: /tmp
|
||||
run: |
|
||||
if [[ "${{ github.repository }}" != "prowler-cloud/prowler" ]]; then
|
||||
echo "This action only runs for prowler-cloud/prowler"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Get Prowler version
|
||||
run: |
|
||||
PROWLER_VERSION="${{ env.RELEASE_TAG }}"
|
||||
echo "version=${PROWLER_VERSION}" >> "${GITHUB_OUTPUT}"
|
||||
|
||||
# Extract major version
|
||||
MAJOR_VERSION="${PROWLER_VERSION%%.*}"
|
||||
echo "major=${MAJOR_VERSION}" >> "${GITHUB_OUTPUT}"
|
||||
|
||||
# Validate major version
|
||||
case ${MAJOR_VERSION} in
|
||||
3|4|5)
|
||||
echo "✓ Releasing Prowler v${MAJOR_VERSION} with tag ${PROWLER_VERSION}"
|
||||
case ${PROWLER_VERSION%%.*} in
|
||||
3)
|
||||
echo "Releasing Prowler v3 with tag ${PROWLER_VERSION}"
|
||||
;;
|
||||
*)
|
||||
echo "::error::Unsupported Prowler major version: ${MAJOR_VERSION}"
|
||||
4)
|
||||
echo "Releasing Prowler v4 with tag ${PROWLER_VERSION}"
|
||||
;;
|
||||
5)
|
||||
echo "Releasing Prowler v5 with tag ${PROWLER_VERSION}"
|
||||
;;
|
||||
*)
|
||||
echo "Releasing another Prowler major version, aborting..."
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
publish-prowler:
|
||||
needs: validate-release
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
environment:
|
||||
name: pypi-prowler
|
||||
url: https://pypi.org/project/prowler/${{ needs.validate-release.outputs.prowler_version }}/
|
||||
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pipx install poetry==2.1.1
|
||||
|
||||
- name: Install Poetry
|
||||
run: pipx install poetry==2.1.1
|
||||
|
||||
- name: Set up Python ${{ env.PYTHON_VERSION }}
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
|
||||
with:
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
cache: 'poetry'
|
||||
# cache: ${{ env.CACHE }}
|
||||
|
||||
- name: Build Prowler package
|
||||
run: poetry build
|
||||
run: |
|
||||
poetry build
|
||||
|
||||
- name: Publish Prowler package to PyPI
|
||||
uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0
|
||||
with:
|
||||
print-hash: true
|
||||
|
||||
publish-prowler-cloud:
|
||||
needs: validate-release
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
environment:
|
||||
name: pypi-prowler-cloud
|
||||
url: https://pypi.org/project/prowler-cloud/${{ needs.validate-release.outputs.prowler_version }}/
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Install Poetry
|
||||
run: pipx install poetry==2.1.1
|
||||
|
||||
- name: Set up Python ${{ env.PYTHON_VERSION }}
|
||||
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
|
||||
with:
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
cache: 'poetry'
|
||||
|
||||
- name: Install toml package
|
||||
run: pip install toml
|
||||
|
||||
- name: Replicate PyPI package for prowler-cloud
|
||||
run: |
|
||||
rm -rf ./dist ./build prowler.egg-info
|
||||
python util/replicate_pypi_package.py
|
||||
poetry config pypi-token.pypi ${{ secrets.PYPI_API_TOKEN }}
|
||||
poetry publish
|
||||
|
||||
- name: Build prowler-cloud package
|
||||
run: poetry build
|
||||
- name: Replicate PyPI package
|
||||
run: |
|
||||
rm -rf ./dist && rm -rf ./build && rm -rf prowler.egg-info
|
||||
pip install toml
|
||||
python util/replicate_pypi_package.py
|
||||
poetry build
|
||||
|
||||
- name: Publish prowler-cloud package to PyPI
|
||||
uses: pypa/gh-action-pypi-publish@ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0
|
||||
with:
|
||||
print-hash: true
|
||||
run: |
|
||||
poetry config pypi-token.pypi ${{ secrets.PYPI_API_TOKEN }}
|
||||
poetry publish
|
||||
|
||||
@@ -1,90 +1,68 @@
|
||||
name: 'SDK: Refresh AWS Regions'
|
||||
# This is a basic workflow to help you get started with Actions
|
||||
|
||||
name: SDK - Refresh AWS services' regions
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 9 * * 1' # Every Monday at 09:00 UTC
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}
|
||||
cancel-in-progress: false
|
||||
- cron: "0 9 * * 1" # runs at 09:00 UTC every Monday
|
||||
|
||||
env:
|
||||
PYTHON_VERSION: '3.12'
|
||||
AWS_REGION: 'us-east-1'
|
||||
GITHUB_BRANCH: "master"
|
||||
AWS_REGION_DEV: us-east-1
|
||||
|
||||
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
|
||||
jobs:
|
||||
refresh-aws-regions:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
# This workflow contains a single job called "build"
|
||||
build:
|
||||
# The type of runner that the job will run on
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
id-token: write
|
||||
pull-requests: write
|
||||
contents: write
|
||||
|
||||
# Steps represent a sequence of tasks that will be executed as part of the job
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
|
||||
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
with:
|
||||
ref: 'master'
|
||||
ref: ${{ env.GITHUB_BRANCH }}
|
||||
|
||||
- name: Set up Python ${{ env.PYTHON_VERSION }}
|
||||
- name: setup python
|
||||
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
|
||||
with:
|
||||
python-version: ${{ env.PYTHON_VERSION }}
|
||||
cache: 'pip'
|
||||
python-version: 3.9 #install the python needed
|
||||
|
||||
- name: Install dependencies
|
||||
run: pip install boto3
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install boto3
|
||||
|
||||
- name: Configure AWS credentials
|
||||
- name: Configure AWS Credentials -- DEV
|
||||
uses: aws-actions/configure-aws-credentials@a03048d87541d1d9fcf2ecf528a4a65ba9bd7838 # v5.0.0
|
||||
with:
|
||||
aws-region: ${{ env.AWS_REGION }}
|
||||
aws-region: ${{ env.AWS_REGION_DEV }}
|
||||
role-to-assume: ${{ secrets.DEV_IAM_ROLE_ARN }}
|
||||
role-session-name: prowler-refresh-aws-regions
|
||||
role-session-name: refresh-AWS-regions-dev
|
||||
|
||||
- name: Update AWS services regions
|
||||
run: python util/update_aws_services_regions.py
|
||||
# Runs a single command using the runners shell
|
||||
- name: Run a one-line script
|
||||
run: python3 util/update_aws_services_regions.py
|
||||
|
||||
- name: Create pull request
|
||||
id: create-pr
|
||||
# Create pull request
|
||||
- name: Create Pull Request
|
||||
uses: peter-evans/create-pull-request@271a8d0340265f705b14b6d32b9829c1cb33d45e # v7.0.8
|
||||
with:
|
||||
author: prowler-bot <179230569+prowler-bot@users.noreply.github.com>
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
author: 'prowler-bot <179230569+prowler-bot@users.noreply.github.com>'
|
||||
committer: 'github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>'
|
||||
commit-message: 'feat(aws): update regions for AWS services'
|
||||
branch: 'aws-regions-update-${{ github.run_number }}'
|
||||
title: 'feat(aws): Update regions for AWS services'
|
||||
labels: |
|
||||
status/waiting-for-revision
|
||||
severity/low
|
||||
provider/aws
|
||||
no-changelog
|
||||
commit-message: "feat(regions_update): Update regions for AWS services"
|
||||
branch: "aws-services-regions-updated-${{ github.sha }}"
|
||||
labels: "status/waiting-for-revision, severity/low, provider/aws, no-changelog"
|
||||
title: "chore(regions_update): Changes in regions for AWS services"
|
||||
body: |
|
||||
### Description
|
||||
|
||||
Automated update of AWS service regions from the official AWS IP ranges.
|
||||
|
||||
**Trigger:** ${{ github.event_name == 'schedule' && 'Scheduled (weekly)' || github.event_name == 'workflow_dispatch' && 'Manual' || 'Workflow update' }}
|
||||
**Run:** [#${{ github.run_number }}](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
|
||||
|
||||
### Checklist
|
||||
|
||||
- [x] This is an automated update from AWS official sources
|
||||
- [x] No manual review of region data required
|
||||
This PR updates the regions for AWS services.
|
||||
|
||||
### License
|
||||
|
||||
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
|
||||
|
||||
- name: PR creation result
|
||||
run: |
|
||||
if [[ "${{ steps.create-pr.outputs.pull-request-number }}" ]]; then
|
||||
echo "✓ Pull request #${{ steps.create-pr.outputs.pull-request-number }} created successfully"
|
||||
echo "URL: ${{ steps.create-pr.outputs.pull-request-url }}"
|
||||
else
|
||||
echo "✓ No changes detected - AWS regions are up to date"
|
||||
fi
|
||||
|
||||
@@ -1,77 +0,0 @@
|
||||
name: 'SDK: Security'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
sdk-security-scans:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check for SDK changes
|
||||
id: check-changes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: ./**
|
||||
files_ignore: |
|
||||
.github/**
|
||||
prowler/CHANGELOG.md
|
||||
docs/**
|
||||
permissions/**
|
||||
api/**
|
||||
ui/**
|
||||
dashboard/**
|
||||
mcp_server/**
|
||||
README.md
|
||||
mkdocs.yml
|
||||
.backportrc.json
|
||||
.env
|
||||
docker-compose*
|
||||
examples/**
|
||||
.gitignore
|
||||
contrib/**
|
||||
|
||||
- name: Install Poetry
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: pipx install poetry==2.1.1
|
||||
|
||||
- name: Set up Python 3.12
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
|
||||
with:
|
||||
python-version: '3.12'
|
||||
cache: 'poetry'
|
||||
|
||||
- name: Install dependencies
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry install --no-root
|
||||
|
||||
- name: Security scan with Bandit
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run bandit -q -lll -x '*_test.py,./contrib/,./api/,./ui' -r .
|
||||
|
||||
- name: Security scan with Safety
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run safety check --ignore 70612 -r pyproject.toml
|
||||
|
||||
- name: Dead code detection with Vulture
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry run vulture --exclude "contrib,api,ui" --min-confidence 100 .
|
||||
@@ -1,360 +0,0 @@
|
||||
name: 'SDK: Tests'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
sdk-tests:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 120
|
||||
permissions:
|
||||
contents: read
|
||||
strategy:
|
||||
matrix:
|
||||
python-version:
|
||||
- '3.9'
|
||||
- '3.10'
|
||||
- '3.11'
|
||||
- '3.12'
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check for SDK changes
|
||||
id: check-changes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: ./**
|
||||
files_ignore: |
|
||||
.github/**
|
||||
prowler/CHANGELOG.md
|
||||
docs/**
|
||||
permissions/**
|
||||
api/**
|
||||
ui/**
|
||||
dashboard/**
|
||||
mcp_server/**
|
||||
README.md
|
||||
mkdocs.yml
|
||||
.backportrc.json
|
||||
.env
|
||||
docker-compose*
|
||||
examples/**
|
||||
.gitignore
|
||||
contrib/**
|
||||
|
||||
- name: Install Poetry
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: pipx install poetry==2.1.1
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
cache: 'poetry'
|
||||
|
||||
- name: Install dependencies
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: poetry install --no-root
|
||||
|
||||
# AWS Provider
|
||||
- name: Check if AWS files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-aws
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/**/aws/**
|
||||
./tests/**/aws/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run AWS tests
|
||||
if: steps.changed-aws.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/providers/aws --cov-report=xml:aws_coverage.xml tests/providers/aws
|
||||
|
||||
- name: Upload AWS coverage to Codecov
|
||||
if: steps.changed-aws.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-aws
|
||||
files: ./aws_coverage.xml
|
||||
|
||||
# Azure Provider
|
||||
- name: Check if Azure files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-azure
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/**/azure/**
|
||||
./tests/**/azure/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run Azure tests
|
||||
if: steps.changed-azure.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/providers/azure --cov-report=xml:azure_coverage.xml tests/providers/azure
|
||||
|
||||
- name: Upload Azure coverage to Codecov
|
||||
if: steps.changed-azure.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-azure
|
||||
files: ./azure_coverage.xml
|
||||
|
||||
# GCP Provider
|
||||
- name: Check if GCP files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-gcp
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/**/gcp/**
|
||||
./tests/**/gcp/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run GCP tests
|
||||
if: steps.changed-gcp.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/providers/gcp --cov-report=xml:gcp_coverage.xml tests/providers/gcp
|
||||
|
||||
- name: Upload GCP coverage to Codecov
|
||||
if: steps.changed-gcp.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-gcp
|
||||
files: ./gcp_coverage.xml
|
||||
|
||||
# Kubernetes Provider
|
||||
- name: Check if Kubernetes files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-kubernetes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/**/kubernetes/**
|
||||
./tests/**/kubernetes/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run Kubernetes tests
|
||||
if: steps.changed-kubernetes.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/providers/kubernetes --cov-report=xml:kubernetes_coverage.xml tests/providers/kubernetes
|
||||
|
||||
- name: Upload Kubernetes coverage to Codecov
|
||||
if: steps.changed-kubernetes.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-kubernetes
|
||||
files: ./kubernetes_coverage.xml
|
||||
|
||||
# GitHub Provider
|
||||
- name: Check if GitHub files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-github
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/**/github/**
|
||||
./tests/**/github/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run GitHub tests
|
||||
if: steps.changed-github.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/providers/github --cov-report=xml:github_coverage.xml tests/providers/github
|
||||
|
||||
- name: Upload GitHub coverage to Codecov
|
||||
if: steps.changed-github.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-github
|
||||
files: ./github_coverage.xml
|
||||
|
||||
# NHN Provider
|
||||
- name: Check if NHN files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-nhn
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/**/nhn/**
|
||||
./tests/**/nhn/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run NHN tests
|
||||
if: steps.changed-nhn.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/providers/nhn --cov-report=xml:nhn_coverage.xml tests/providers/nhn
|
||||
|
||||
- name: Upload NHN coverage to Codecov
|
||||
if: steps.changed-nhn.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-nhn
|
||||
files: ./nhn_coverage.xml
|
||||
|
||||
# M365 Provider
|
||||
- name: Check if M365 files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-m365
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/**/m365/**
|
||||
./tests/**/m365/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run M365 tests
|
||||
if: steps.changed-m365.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/providers/m365 --cov-report=xml:m365_coverage.xml tests/providers/m365
|
||||
|
||||
- name: Upload M365 coverage to Codecov
|
||||
if: steps.changed-m365.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-m365
|
||||
files: ./m365_coverage.xml
|
||||
|
||||
# IaC Provider
|
||||
- name: Check if IaC files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-iac
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/**/iac/**
|
||||
./tests/**/iac/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run IaC tests
|
||||
if: steps.changed-iac.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/providers/iac --cov-report=xml:iac_coverage.xml tests/providers/iac
|
||||
|
||||
- name: Upload IaC coverage to Codecov
|
||||
if: steps.changed-iac.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-iac
|
||||
files: ./iac_coverage.xml
|
||||
|
||||
# MongoDB Atlas Provider
|
||||
- name: Check if MongoDB Atlas files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-mongodbatlas
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/**/mongodbatlas/**
|
||||
./tests/**/mongodbatlas/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run MongoDB Atlas tests
|
||||
if: steps.changed-mongodbatlas.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/providers/mongodbatlas --cov-report=xml:mongodbatlas_coverage.xml tests/providers/mongodbatlas
|
||||
|
||||
- name: Upload MongoDB Atlas coverage to Codecov
|
||||
if: steps.changed-mongodbatlas.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-mongodbatlas
|
||||
files: ./mongodbatlas_coverage.xml
|
||||
|
||||
# OCI Provider
|
||||
- name: Check if OCI files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-oraclecloud
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/**/oraclecloud/**
|
||||
./tests/**/oraclecloud/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run OCI tests
|
||||
if: steps.changed-oraclecloud.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/providers/oraclecloud --cov-report=xml:oraclecloud_coverage.xml tests/providers/oraclecloud
|
||||
|
||||
- name: Upload OCI coverage to Codecov
|
||||
if: steps.changed-oraclecloud.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-oraclecloud
|
||||
files: ./oraclecloud_coverage.xml
|
||||
|
||||
# Lib
|
||||
- name: Check if Lib files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-lib
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/lib/**
|
||||
./tests/lib/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run Lib tests
|
||||
if: steps.changed-lib.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/lib --cov-report=xml:lib_coverage.xml tests/lib
|
||||
|
||||
- name: Upload Lib coverage to Codecov
|
||||
if: steps.changed-lib.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-lib
|
||||
files: ./lib_coverage.xml
|
||||
|
||||
# Config
|
||||
- name: Check if Config files changed
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
id: changed-config
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
./prowler/config/**
|
||||
./tests/config/**
|
||||
./poetry.lock
|
||||
|
||||
- name: Run Config tests
|
||||
if: steps.changed-config.outputs.any_changed == 'true'
|
||||
run: poetry run pytest -n auto --cov=./prowler/config --cov-report=xml:config_coverage.xml tests/config
|
||||
|
||||
- name: Upload Config coverage to Codecov
|
||||
if: steps.changed-config.outputs.any_changed == 'true'
|
||||
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
|
||||
env:
|
||||
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
|
||||
with:
|
||||
flags: prowler-py${{ matrix.python-version }}-config
|
||||
files: ./config_coverage.xml
|
||||
@@ -0,0 +1,121 @@
|
||||
name: UI - Build and Push containers
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- "master"
|
||||
paths:
|
||||
- "ui/**"
|
||||
- ".github/workflows/ui-build-lint-push-containers.yml"
|
||||
|
||||
# Uncomment the below code to test this action on PRs
|
||||
# pull_request:
|
||||
# branches:
|
||||
# - "master"
|
||||
# paths:
|
||||
# - "ui/**"
|
||||
# - ".github/workflows/ui-build-lint-push-containers.yml"
|
||||
|
||||
release:
|
||||
types: [published]
|
||||
|
||||
env:
|
||||
# Tags
|
||||
LATEST_TAG: latest
|
||||
RELEASE_TAG: ${{ github.event.release.tag_name }}
|
||||
STABLE_TAG: stable
|
||||
|
||||
WORKING_DIRECTORY: ./ui
|
||||
|
||||
# Container Registries
|
||||
PROWLERCLOUD_DOCKERHUB_REPOSITORY: prowlercloud
|
||||
PROWLERCLOUD_DOCKERHUB_IMAGE: prowler-ui
|
||||
NEXT_PUBLIC_API_BASE_URL: http://prowler-api:8080/api/v1
|
||||
|
||||
jobs:
|
||||
repository-check:
|
||||
name: Repository check
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
is_repo: ${{ steps.repository_check.outputs.is_repo }}
|
||||
steps:
|
||||
- name: Repository check
|
||||
id: repository_check
|
||||
working-directory: /tmp
|
||||
run: |
|
||||
if [[ ${{ github.repository }} == "prowler-cloud/prowler" ]]
|
||||
then
|
||||
echo "is_repo=true" >> "${GITHUB_OUTPUT}"
|
||||
else
|
||||
echo "This action only runs for prowler-cloud/prowler"
|
||||
echo "is_repo=false" >> "${GITHUB_OUTPUT}"
|
||||
fi
|
||||
|
||||
# Build Prowler OSS container
|
||||
container-build-push:
|
||||
needs: repository-check
|
||||
if: needs.repository-check.outputs.is_repo == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: ${{ env.WORKING_DIRECTORY }}
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Set short git commit SHA
|
||||
id: vars
|
||||
run: |
|
||||
shortSha=$(git rev-parse --short ${{ github.sha }})
|
||||
echo "SHORT_SHA=${shortSha}" >> $GITHUB_ENV
|
||||
|
||||
- name: Login to DockerHub
|
||||
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
|
||||
- name: Build and push container image (latest)
|
||||
# Comment the following line for testing
|
||||
if: github.event_name == 'push'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.WORKING_DIRECTORY }}
|
||||
build-args: |
|
||||
NEXT_PUBLIC_PROWLER_RELEASE_VERSION=${{ env.SHORT_SHA }}
|
||||
NEXT_PUBLIC_API_BASE_URL=${{ env.NEXT_PUBLIC_API_BASE_URL }}
|
||||
# Set push: false for testing
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.SHORT_SHA }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Build and push container image (release)
|
||||
if: github.event_name == 'release'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.WORKING_DIRECTORY }}
|
||||
build-args: |
|
||||
NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${{ env.RELEASE_TAG }}
|
||||
NEXT_PUBLIC_API_BASE_URL=${{ env.NEXT_PUBLIC_API_BASE_URL }}
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.RELEASE_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Trigger deployment
|
||||
if: github.event_name == 'push'
|
||||
uses: peter-evans/repository-dispatch@5fc4efd1a4797ddb68ffd0714a238564e4cc0e6f # v4.0.0
|
||||
with:
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
repository: ${{ secrets.CLOUD_DISPATCH }}
|
||||
event-type: prowler-ui-deploy
|
||||
client-payload: '{"sha": "${{ github.sha }}", "short_sha": "${{ env.SHORT_SHA }}"}'
|
||||
@@ -27,8 +27,7 @@ concurrency:
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
ui-analyze:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
analyze:
|
||||
name: CodeQL Security Analysis
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
|
||||
@@ -1,143 +0,0 @@
|
||||
name: 'UI: Container Build and Push'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
paths:
|
||||
- 'ui/**'
|
||||
- '.github/workflows/ui-container-build-push.yml'
|
||||
release:
|
||||
types:
|
||||
- 'published'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
env:
|
||||
# Tags
|
||||
LATEST_TAG: latest
|
||||
RELEASE_TAG: ${{ github.event.release.tag_name }}
|
||||
STABLE_TAG: stable
|
||||
WORKING_DIRECTORY: ./ui
|
||||
|
||||
# Container registries
|
||||
PROWLERCLOUD_DOCKERHUB_REPOSITORY: prowlercloud
|
||||
PROWLERCLOUD_DOCKERHUB_IMAGE: prowler-ui
|
||||
|
||||
# Build args
|
||||
NEXT_PUBLIC_API_BASE_URL: http://prowler-api:8080/api/v1
|
||||
|
||||
jobs:
|
||||
setup:
|
||||
if: github.repository == 'prowler-cloud/prowler'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
outputs:
|
||||
short-sha: ${{ steps.set-short-sha.outputs.short-sha }}
|
||||
steps:
|
||||
- name: Calculate short SHA
|
||||
id: set-short-sha
|
||||
run: echo "short-sha=${GITHUB_SHA::7}" >> $GITHUB_OUTPUT
|
||||
|
||||
container-build-push:
|
||||
needs: setup
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Login to DockerHub
|
||||
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
|
||||
- name: Build and push UI container (latest)
|
||||
if: github.event_name == 'push'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.WORKING_DIRECTORY }}
|
||||
build-args: |
|
||||
NEXT_PUBLIC_PROWLER_RELEASE_VERSION=${{ needs.setup.outputs.short-sha }}
|
||||
NEXT_PUBLIC_API_BASE_URL=${{ env.NEXT_PUBLIC_API_BASE_URL }}
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.LATEST_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ needs.setup.outputs.short-sha }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Notify container push started
|
||||
if: github.event_name == 'release'
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_PLATFORM_DEPLOYMENTS }}
|
||||
COMPONENT: UI
|
||||
RELEASE_TAG: ${{ env.RELEASE_TAG }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-started.json"
|
||||
|
||||
- name: Build and push UI container (release)
|
||||
if: github.event_name == 'release'
|
||||
id: container-push
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.WORKING_DIRECTORY }}
|
||||
build-args: |
|
||||
NEXT_PUBLIC_PROWLER_RELEASE_VERSION=v${{ env.RELEASE_TAG }}
|
||||
NEXT_PUBLIC_API_BASE_URL=${{ env.NEXT_PUBLIC_API_BASE_URL }}
|
||||
push: true
|
||||
tags: |
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.RELEASE_TAG }}
|
||||
${{ env.PROWLERCLOUD_DOCKERHUB_REPOSITORY }}/${{ env.PROWLERCLOUD_DOCKERHUB_IMAGE }}:${{ env.STABLE_TAG }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
- name: Notify container push completed
|
||||
if: github.event_name == 'release' && always()
|
||||
uses: ./.github/actions/slack-notification
|
||||
env:
|
||||
SLACK_CHANNEL_ID: ${{ secrets.SLACK_PLATFORM_DEPLOYMENTS }}
|
||||
COMPONENT: UI
|
||||
RELEASE_TAG: ${{ env.RELEASE_TAG }}
|
||||
GITHUB_SERVER_URL: ${{ github.server_url }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
GITHUB_RUN_ID: ${{ github.run_id }}
|
||||
with:
|
||||
slack-bot-token: ${{ secrets.SLACK_BOT_TOKEN }}
|
||||
payload-file-path: "./.github/scripts/slack-messages/container-release-completed.json"
|
||||
step-outcome: ${{ steps.container-push.outcome }}
|
||||
|
||||
trigger-deployment:
|
||||
if: github.event_name == 'push'
|
||||
needs: [setup, container-build-push]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Trigger UI deployment
|
||||
uses: peter-evans/repository-dispatch@5fc4efd1a4797ddb68ffd0714a238564e4cc0e6f # v4.0.0
|
||||
with:
|
||||
token: ${{ secrets.PROWLER_BOT_ACCESS_TOKEN }}
|
||||
repository: ${{ secrets.CLOUD_DISPATCH }}
|
||||
event-type: ui-prowler-deployment
|
||||
client-payload: '{"sha": "${{ github.sha }}", "short_sha": "${{ needs.setup.outputs.short-sha }}"}'
|
||||
@@ -1,91 +0,0 @@
|
||||
name: 'UI: Container Checks'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
UI_WORKING_DIR: ./ui
|
||||
IMAGE_NAME: prowler-ui
|
||||
|
||||
jobs:
|
||||
ui-dockerfile-lint:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check if Dockerfile changed
|
||||
id: dockerfile-changed
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: ui/Dockerfile
|
||||
|
||||
- name: Lint Dockerfile with Hadolint
|
||||
if: steps.dockerfile-changed.outputs.any_changed == 'true'
|
||||
uses: hadolint/hadolint-action@2332a7b74a6de0dda2e2221d575162eba76ba5e5 # v3.3.0
|
||||
with:
|
||||
dockerfile: ui/Dockerfile
|
||||
ignore: DL3018
|
||||
|
||||
ui-container-build-and-scan:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
security-events: write
|
||||
pull-requests: write
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check for UI changes
|
||||
id: check-changes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: ui/**
|
||||
files_ignore: |
|
||||
ui/CHANGELOG.md
|
||||
ui/README.md
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
|
||||
- name: Build UI container
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.UI_WORKING_DIR }}
|
||||
target: prod
|
||||
push: false
|
||||
load: true
|
||||
tags: ${{ env.IMAGE_NAME }}:${{ github.sha }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
build-args: |
|
||||
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=pk_test_51LwpXXXX
|
||||
|
||||
- name: Scan UI container with Trivy
|
||||
if: github.repository == 'prowler-cloud/prowler' && steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: ./.github/actions/trivy-scan
|
||||
with:
|
||||
image-name: ${{ env.IMAGE_NAME }}
|
||||
image-tag: ${{ github.sha }}
|
||||
fail-on-critical: 'false'
|
||||
severity: 'CRITICAL'
|
||||
@@ -18,21 +18,6 @@ jobs:
|
||||
AUTH_TRUST_HOST: true
|
||||
NEXTAUTH_URL: 'http://localhost:3000'
|
||||
NEXT_PUBLIC_API_BASE_URL: 'http://localhost:8080/api/v1'
|
||||
E2E_ADMIN_USER: ${{ secrets.E2E_ADMIN_USER }}
|
||||
E2E_ADMIN_PASSWORD: ${{ secrets.E2E_ADMIN_PASSWORD }}
|
||||
E2E_AWS_PROVIDER_ACCOUNT_ID: ${{ secrets.E2E_AWS_PROVIDER_ACCOUNT_ID }}
|
||||
E2E_AWS_PROVIDER_ACCESS_KEY: ${{ secrets.E2E_AWS_PROVIDER_ACCESS_KEY }}
|
||||
E2E_AWS_PROVIDER_SECRET_KEY: ${{ secrets.E2E_AWS_PROVIDER_SECRET_KEY }}
|
||||
E2E_AWS_PROVIDER_ROLE_ARN: ${{ secrets.E2E_AWS_PROVIDER_ROLE_ARN }}
|
||||
E2E_AZURE_SUBSCRIPTION_ID: ${{ secrets.E2E_AZURE_SUBSCRIPTION_ID }}
|
||||
E2E_AZURE_CLIENT_ID: ${{ secrets.E2E_AZURE_CLIENT_ID }}
|
||||
E2E_AZURE_SECRET_ID: ${{ secrets.E2E_AZURE_SECRET_ID }}
|
||||
E2E_AZURE_TENANT_ID: ${{ secrets.E2E_AZURE_TENANT_ID }}
|
||||
E2E_M365_DOMAIN_ID: ${{ secrets.E2E_M365_DOMAIN_ID }}
|
||||
E2E_M365_CLIENT_ID: ${{ secrets.E2E_M365_CLIENT_ID }}
|
||||
E2E_M365_SECRET_ID: ${{ secrets.E2E_M365_SECRET_ID }}
|
||||
E2E_M365_TENANT_ID: ${{ secrets.E2E_M365_TENANT_ID }}
|
||||
E2E_M365_CERTIFICATE_CONTENT: ${{ secrets.E2E_M365_CERTIFICATE_CONTENT }}
|
||||
E2E_NEW_PASSWORD: ${{ secrets.E2E_NEW_PASSWORD }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
name: UI - Pull Request
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- "master"
|
||||
- "v5.*"
|
||||
paths:
|
||||
- ".github/workflows/ui-pull-request.yml"
|
||||
- "ui/**"
|
||||
pull_request:
|
||||
branches:
|
||||
- master
|
||||
- "v5.*"
|
||||
paths:
|
||||
- 'ui/**'
|
||||
env:
|
||||
UI_WORKING_DIR: ./ui
|
||||
IMAGE_NAME: prowler-ui
|
||||
|
||||
jobs:
|
||||
test-and-coverage:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
matrix:
|
||||
os: [ubuntu-latest]
|
||||
node-version: [20.x]
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
with:
|
||||
persist-credentials: false
|
||||
- name: Setup Node.js ${{ matrix.node-version }}
|
||||
uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5.0.0
|
||||
with:
|
||||
node-version: ${{ matrix.node-version }}
|
||||
cache: 'npm'
|
||||
cache-dependency-path: './ui/package-lock.json'
|
||||
- name: Install dependencies
|
||||
working-directory: ./ui
|
||||
run: npm ci
|
||||
- name: Run Healthcheck
|
||||
working-directory: ./ui
|
||||
run: npm run healthcheck
|
||||
- name: Build the application
|
||||
working-directory: ./ui
|
||||
run: npm run build
|
||||
|
||||
test-container-build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
|
||||
- name: Build Container
|
||||
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83 # v6.18.0
|
||||
with:
|
||||
context: ${{ env.UI_WORKING_DIR }}
|
||||
# Always build using `prod` target
|
||||
target: prod
|
||||
push: false
|
||||
tags: ${{ env.IMAGE_NAME }}:latest
|
||||
outputs: type=docker
|
||||
build-args: |
|
||||
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=pk_test_51LwpXXXX
|
||||
@@ -1,64 +0,0 @@
|
||||
name: 'UI: Tests'
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
pull_request:
|
||||
branches:
|
||||
- 'master'
|
||||
- 'v5.*'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
env:
|
||||
UI_WORKING_DIR: ./ui
|
||||
NODE_VERSION: '20.x'
|
||||
|
||||
jobs:
|
||||
ui-tests:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: read
|
||||
defaults:
|
||||
run:
|
||||
working-directory: ./ui
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
|
||||
|
||||
- name: Check for UI changes
|
||||
id: check-changes
|
||||
uses: tj-actions/changed-files@24d32ffd492484c1d75e0c0b894501ddb9d30d62 # v47.0.0
|
||||
with:
|
||||
files: |
|
||||
ui/**
|
||||
.github/workflows/ui-tests.yml
|
||||
files_ignore: |
|
||||
ui/CHANGELOG.md
|
||||
ui/README.md
|
||||
|
||||
- name: Setup Node.js ${{ env.NODE_VERSION }}
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0
|
||||
with:
|
||||
node-version: ${{ env.NODE_VERSION }}
|
||||
cache: 'npm'
|
||||
cache-dependency-path: './ui/package-lock.json'
|
||||
|
||||
- name: Install dependencies
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: npm ci
|
||||
|
||||
- name: Run healthcheck
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: npm run healthcheck
|
||||
|
||||
- name: Build application
|
||||
if: steps.check-changes.outputs.any_changed == 'true'
|
||||
run: npm run build
|
||||
@@ -39,6 +39,12 @@ secrets-*/
|
||||
# JUnit Reports
|
||||
junit-reports/
|
||||
|
||||
# Test and coverage artifacts
|
||||
*_coverage.xml
|
||||
pytest_*.xml
|
||||
.coverage
|
||||
htmlcov/
|
||||
|
||||
# VSCode files
|
||||
.vscode/
|
||||
|
||||
|
||||
@@ -2,43 +2,10 @@
|
||||
|
||||
All notable changes to the **Prowler API** are documented in this file.
|
||||
|
||||
<<<<<<< HEAD
|
||||
=======
|
||||
## [1.15.0] (Prowler UNRELEASED)
|
||||
|
||||
### Added
|
||||
- IaC (Infrastructure as Code) provider support for remote repositories [(#8751)](https://github.com/prowler-cloud/prowler/pull/8751)
|
||||
- Extend `GET /api/v1/providers` with provider-type filters and optional pagination disable to support the new Overview filters [(#8975)](https://github.com/prowler-cloud/prowler/pull/8975)
|
||||
- New endpoint to retrieve the number of providers grouped by provider type [(#8975)](https://github.com/prowler-cloud/prowler/pull/8975)
|
||||
- Support for configuring multiple LLM providers [(#8772)](https://github.com/prowler-cloud/prowler/pull/8772)
|
||||
- Support C5 compliance framework for Azure provider [(#9081)](https://github.com/prowler-cloud/prowler/pull/9081)
|
||||
- Support for Oracle Cloud Infrastructure (OCI) provider [(#8927)](https://github.com/prowler-cloud/prowler/pull/8927)
|
||||
- Support muting findings based on simple rules with custom reason [(#9051)](https://github.com/prowler-cloud/prowler/pull/9051)
|
||||
- Support C5 compliance framework for the GCP provider [(#9097)](https://github.com/prowler-cloud/prowler/pull/9097)
|
||||
- Support for Amazon Bedrock and OpenAI compatible providers in Lighthouse AI [(#8957)](https://github.com/prowler-cloud/prowler/pull/8957)
|
||||
- Tenant-wide ThreatScore overview aggregation and snapshot persistence with backfill support [(#9148)](https://github.com/prowler-cloud/prowler/pull/9148)
|
||||
- Support for MongoDB Atlas provider [(#9167)](https://github.com/prowler-cloud/prowler/pull/9167)
|
||||
|
||||
### Security
|
||||
- Django updated to the latest 5.1 security release, 5.1.14, due to problems with potential [SQL injection](https://github.com/prowler-cloud/prowler/security/dependabot/113) and [denial-of-service vulnerability](https://github.com/prowler-cloud/prowler/security/dependabot/114) [(#9176)](https://github.com/prowler-cloud/prowler/pull/9176)
|
||||
|
||||
---
|
||||
|
||||
## [1.14.2] (Prowler UNRELEASED)
|
||||
|
||||
### Fixed
|
||||
- Update unique constraint for `Provider` model to exclude soft-deleted entries, resolving duplicate errors when re-deleting providers.[(#9054)](https://github.com/prowler-cloud/prowler/pull/9054)
|
||||
- Remove compliance generation for providers without compliance frameworks [(#9208)](https://github.com/prowler-cloud/prowler/pull/9208)
|
||||
|
||||
>>>>>>> 427dab681 (fix(compliance): handle check_id not in Prowler Checks (#9208))
|
||||
## [1.14.1] (Prowler 5.13.1)
|
||||
|
||||
### Fixed
|
||||
- `/api/v1/overviews/providers` collapses data by provider type so the UI receives a single aggregated record per cloud family even when multiple accounts exist [(#9053)](https://github.com/prowler-cloud/prowler/pull/9053)
|
||||
- Added retry logic to database transactions to handle Aurora read replica connection failures during scale-down events [(#9064)](https://github.com/prowler-cloud/prowler/pull/9064)
|
||||
- Security Hub integrations stop failing when they read relationships via the replica by allowing replica relations and saving updates through the primary [(#9080)](https://github.com/prowler-cloud/prowler/pull/9080)
|
||||
|
||||
---
|
||||
|
||||
## [1.14.0] (Prowler 5.13.0)
|
||||
|
||||
|
||||
Generated
+4
-59
@@ -1,4 +1,4 @@
|
||||
# This file is automatically @generated by Poetry 2.1.3 and should not be changed by hand.
|
||||
# This file is automatically @generated by Poetry 2.2.0 and should not be changed by hand.
|
||||
|
||||
[[package]]
|
||||
name = "about-time"
|
||||
@@ -1164,18 +1164,6 @@ files = [
|
||||
{file = "charset_normalizer-3.4.3.tar.gz", hash = "sha256:6fce4b8500244f6fcb71465d4a4930d132ba9ab8e71a7859e6a5d59851068d14"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "circuitbreaker"
|
||||
version = "2.1.3"
|
||||
description = "Python Circuit Breaker pattern implementation"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "circuitbreaker-2.1.3-py3-none-any.whl", hash = "sha256:87ba6a3ed03fdc7032bc175561c2b04d52ade9d5faf94ca2b035fbdc5e6b1dd1"},
|
||||
{file = "circuitbreaker-2.1.3.tar.gz", hash = "sha256:1a4baee510f7bea3c91b194dcce7c07805fe96c4423ed5594b75af438531d084"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "click"
|
||||
version = "8.2.1"
|
||||
@@ -4058,29 +4046,6 @@ rsa = ["cryptography (>=3.0.0)"]
|
||||
signals = ["blinker (>=1.4.0)"]
|
||||
signedtoken = ["cryptography (>=3.0.0)", "pyjwt (>=2.0.0,<3)"]
|
||||
|
||||
[[package]]
|
||||
name = "oci"
|
||||
version = "2.160.3"
|
||||
description = "Oracle Cloud Infrastructure Python SDK"
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "oci-2.160.3-py3-none-any.whl", hash = "sha256:858bff3e697098bdda44833d2476bfb4632126f0182178e7dbde4dbd156d71f0"},
|
||||
{file = "oci-2.160.3.tar.gz", hash = "sha256:57514889be3b713a8385d86e3ba8a33cf46e3563c2a7e29a93027fb30b8a2537"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
certifi = "*"
|
||||
circuitbreaker = {version = ">=1.3.1,<3.0.0", markers = "python_version >= \"3.7\""}
|
||||
cryptography = ">=3.2.1,<46.0.0"
|
||||
pyOpenSSL = ">=17.5.0,<25.0.0"
|
||||
python-dateutil = ">=2.5.3,<3.0.0"
|
||||
pytz = ">=2016.10"
|
||||
|
||||
[package.extras]
|
||||
adk = ["docstring-parser (>=0.16) ; python_version >= \"3.10\" and python_version < \"4\"", "mcp (>=1.6.0) ; python_version >= \"3.10\" and python_version < \"4\"", "pydantic (>=2.10.6) ; python_version >= \"3.10\" and python_version < \"4\"", "rich (>=13.9.4) ; python_version >= \"3.10\" and python_version < \"4\""]
|
||||
|
||||
[[package]]
|
||||
name = "openai"
|
||||
version = "1.101.0"
|
||||
@@ -4669,7 +4634,6 @@ markdown = "3.9.0"
|
||||
microsoft-kiota-abstractions = "1.9.2"
|
||||
msgraph-sdk = "1.23.0"
|
||||
numpy = "2.0.2"
|
||||
oci = "2.160.3"
|
||||
pandas = "2.2.3"
|
||||
py-iam-expand = "0.1.0"
|
||||
py-ocsf-models = "0.5.0"
|
||||
@@ -4686,8 +4650,8 @@ tzlocal = "5.3.1"
|
||||
[package.source]
|
||||
type = "git"
|
||||
url = "https://github.com/prowler-cloud/prowler.git"
|
||||
reference = "v5.13"
|
||||
resolved_reference = "b1856e42f0143a64e8cc26c7aa3c7643bd1083d3"
|
||||
reference = "master"
|
||||
resolved_reference = "a52697bfdfee83d14a49c11dcbe96888b5cd767e"
|
||||
|
||||
[[package]]
|
||||
name = "psutil"
|
||||
@@ -5172,25 +5136,6 @@ cffi = ">=1.4.1"
|
||||
docs = ["sphinx (>=1.6.5)", "sphinx-rtd-theme"]
|
||||
tests = ["hypothesis (>=3.27.0)", "pytest (>=3.2.1,!=3.3.0)"]
|
||||
|
||||
[[package]]
|
||||
name = "pyopenssl"
|
||||
version = "24.3.0"
|
||||
description = "Python wrapper module around the OpenSSL library"
|
||||
optional = false
|
||||
python-versions = ">=3.7"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "pyOpenSSL-24.3.0-py3-none-any.whl", hash = "sha256:e474f5a473cd7f92221cc04976e48f4d11502804657a08a989fb3be5514c904a"},
|
||||
{file = "pyopenssl-24.3.0.tar.gz", hash = "sha256:49f7a019577d834746bc55c5fce6ecbcec0f2b4ec5ce1cf43a9a173b8138bb36"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
cryptography = ">=41.0.5,<45"
|
||||
|
||||
[package.extras]
|
||||
docs = ["sphinx (!=5.2.0,!=5.2.0.post0,!=7.2.5)", "sphinx_rtd_theme"]
|
||||
test = ["pretend", "pytest (>=3.0.1)", "pytest-rerunfailures"]
|
||||
|
||||
[[package]]
|
||||
name = "pyparsing"
|
||||
version = "3.2.3"
|
||||
@@ -6841,4 +6786,4 @@ type = ["pytest-mypy"]
|
||||
[metadata]
|
||||
lock-version = "2.1"
|
||||
python-versions = ">=3.11,<3.13"
|
||||
content-hash = "8fcb616e55530e7940019d3da33e955b026b9105e1216a3c5f39b411c015b6d7"
|
||||
content-hash = "3c9164d668d37d6373eb5200bbe768232ead934d9312b9c68046b1df922789f3"
|
||||
|
||||
+2
-2
@@ -24,7 +24,7 @@ dependencies = [
|
||||
"drf-spectacular-jsonapi==0.5.1",
|
||||
"gunicorn==23.0.0",
|
||||
"lxml==5.3.2",
|
||||
"prowler @ git+https://github.com/prowler-cloud/prowler.git@v5.13",
|
||||
"prowler @ git+https://github.com/prowler-cloud/prowler.git@master",
|
||||
"psycopg2-binary==2.9.9",
|
||||
"pytest-celery[redis] (>=1.0.1,<2.0.0)",
|
||||
"sentry-sdk[django] (>=2.20.0,<3.0.0)",
|
||||
@@ -43,7 +43,7 @@ name = "prowler-api"
|
||||
package-mode = false
|
||||
# Needed for the SDK compatibility
|
||||
requires-python = ">=3.11,<3.13"
|
||||
version = "1.14.1"
|
||||
version = "1.15.0"
|
||||
|
||||
[project.scripts]
|
||||
celery = "src.backend.config.settings.celery"
|
||||
|
||||
@@ -144,7 +144,6 @@ def generate_scan_compliance(
|
||||
Returns:
|
||||
None: This function modifies the compliance_overview in place.
|
||||
"""
|
||||
|
||||
for compliance_id in PROWLER_CHECKS[provider_type][check_id]:
|
||||
for requirement in compliance_overview[compliance_id]["requirements"].values():
|
||||
if check_id in requirement["checks"]:
|
||||
|
||||
@@ -48,9 +48,8 @@ class MainRouter:
|
||||
return db == self.admin_db
|
||||
|
||||
def allow_relation(self, obj1, obj2, **hints): # noqa: F841
|
||||
# Allow relations when both objects originate from allowed connectors
|
||||
allowed_dbs = {self.default_db, self.admin_db, self.replica_db}
|
||||
if {obj1._state.db, obj2._state.db} <= allowed_dbs:
|
||||
# Allow relations if both objects are in either "default" or "admin" db connectors
|
||||
if {obj1._state.db, obj2._state.db} <= {self.default_db, self.admin_db}:
|
||||
return True
|
||||
return None
|
||||
|
||||
|
||||
@@ -1,35 +1,18 @@
|
||||
import re
|
||||
import secrets
|
||||
import time
|
||||
import uuid
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from celery.utils.log import get_task_logger
|
||||
from config.env import env
|
||||
from django.conf import settings
|
||||
from django.contrib.auth.models import BaseUserManager
|
||||
from django.db import (
|
||||
DEFAULT_DB_ALIAS,
|
||||
OperationalError,
|
||||
connection,
|
||||
connections,
|
||||
models,
|
||||
transaction,
|
||||
)
|
||||
from django.db import DEFAULT_DB_ALIAS, connection, connections, models, transaction
|
||||
from django_celery_beat.models import PeriodicTask
|
||||
from psycopg2 import connect as psycopg2_connect
|
||||
from psycopg2.extensions import AsIs, new_type, register_adapter, register_type
|
||||
from rest_framework_json_api.serializers import ValidationError
|
||||
|
||||
from api.db_router import (
|
||||
READ_REPLICA_ALIAS,
|
||||
get_read_db_alias,
|
||||
reset_read_db_alias,
|
||||
set_read_db_alias,
|
||||
)
|
||||
|
||||
logger = get_task_logger(__name__)
|
||||
from api.db_router import get_read_db_alias, reset_read_db_alias, set_read_db_alias
|
||||
|
||||
DB_USER = settings.DATABASES["default"]["USER"] if not settings.TESTING else "test"
|
||||
DB_PASSWORD = (
|
||||
@@ -45,9 +28,6 @@ TASK_RUNNER_DB_TABLE = "django_celery_results_taskresult"
|
||||
POSTGRES_TENANT_VAR = "api.tenant_id"
|
||||
POSTGRES_USER_VAR = "api.user_id"
|
||||
|
||||
REPLICA_MAX_ATTEMPTS = env.int("POSTGRES_REPLICA_MAX_ATTEMPTS", default=3)
|
||||
REPLICA_RETRY_BASE_DELAY = env.float("POSTGRES_REPLICA_RETRY_BASE_DELAY", default=0.5)
|
||||
|
||||
SET_CONFIG_QUERY = "SELECT set_config(%s, %s::text, TRUE);"
|
||||
|
||||
|
||||
@@ -91,51 +71,24 @@ def rls_transaction(
|
||||
if db_alias not in connections:
|
||||
db_alias = DEFAULT_DB_ALIAS
|
||||
|
||||
alias = db_alias
|
||||
is_replica = READ_REPLICA_ALIAS and alias == READ_REPLICA_ALIAS
|
||||
max_attempts = REPLICA_MAX_ATTEMPTS if is_replica else 1
|
||||
router_token = None
|
||||
try:
|
||||
if db_alias != DEFAULT_DB_ALIAS:
|
||||
router_token = set_read_db_alias(db_alias)
|
||||
|
||||
for attempt in range(1, max_attempts + 1):
|
||||
router_token = None
|
||||
|
||||
# On final attempt, fallback to primary
|
||||
if attempt == max_attempts and is_replica:
|
||||
logger.warning(
|
||||
f"RLS transaction failed after {attempt - 1} attempts on replica, "
|
||||
f"falling back to primary DB"
|
||||
)
|
||||
alias = DEFAULT_DB_ALIAS
|
||||
|
||||
conn = connections[alias]
|
||||
try:
|
||||
if alias != DEFAULT_DB_ALIAS:
|
||||
router_token = set_read_db_alias(alias)
|
||||
|
||||
with transaction.atomic(using=alias):
|
||||
with conn.cursor() as cursor:
|
||||
try:
|
||||
# just in case the value is a UUID object
|
||||
uuid.UUID(str(value))
|
||||
except ValueError:
|
||||
raise ValidationError("Must be a valid UUID")
|
||||
cursor.execute(SET_CONFIG_QUERY, [parameter, value])
|
||||
yield cursor
|
||||
return
|
||||
except OperationalError as e:
|
||||
# If on primary or max attempts reached, raise
|
||||
if not is_replica or attempt == max_attempts:
|
||||
raise
|
||||
|
||||
# Retry with exponential backoff
|
||||
delay = REPLICA_RETRY_BASE_DELAY * (2 ** (attempt - 1))
|
||||
logger.info(
|
||||
f"RLS transaction failed on replica (attempt {attempt}/{max_attempts}), "
|
||||
f"retrying in {delay}s. Error: {e}"
|
||||
)
|
||||
time.sleep(delay)
|
||||
finally:
|
||||
if router_token is not None:
|
||||
reset_read_db_alias(router_token)
|
||||
with transaction.atomic(using=db_alias):
|
||||
conn = connections[db_alias]
|
||||
with conn.cursor() as cursor:
|
||||
try:
|
||||
# just in case the value is a UUID object
|
||||
uuid.UUID(str(value))
|
||||
except ValueError:
|
||||
raise ValidationError("Must be a valid UUID")
|
||||
cursor.execute(SET_CONFIG_QUERY, [parameter, value])
|
||||
yield cursor
|
||||
finally:
|
||||
if router_token is not None:
|
||||
reset_read_db_alias(router_token)
|
||||
|
||||
|
||||
class CustomUserManager(BaseUserManager):
|
||||
|
||||
@@ -27,6 +27,8 @@ from api.models import (
|
||||
Finding,
|
||||
Integration,
|
||||
Invitation,
|
||||
LighthouseProviderConfiguration,
|
||||
LighthouseProviderModels,
|
||||
Membership,
|
||||
OverviewStatusChoices,
|
||||
PermissionChoices,
|
||||
@@ -928,3 +930,45 @@ class TenantApiKeyFilter(FilterSet):
|
||||
"revoked": ["exact"],
|
||||
"name": ["exact", "icontains"],
|
||||
}
|
||||
|
||||
|
||||
class LighthouseProviderConfigFilter(FilterSet):
|
||||
provider_type = ChoiceFilter(
|
||||
choices=LighthouseProviderConfiguration.LLMProviderChoices.choices
|
||||
)
|
||||
provider_type__in = ChoiceInFilter(
|
||||
choices=LighthouseProviderConfiguration.LLMProviderChoices.choices,
|
||||
field_name="provider_type",
|
||||
lookup_expr="in",
|
||||
)
|
||||
is_active = BooleanFilter()
|
||||
|
||||
class Meta:
|
||||
model = LighthouseProviderConfiguration
|
||||
fields = {
|
||||
"provider_type": ["exact", "in"],
|
||||
"is_active": ["exact"],
|
||||
}
|
||||
|
||||
|
||||
class LighthouseProviderModelsFilter(FilterSet):
|
||||
provider_type = ChoiceFilter(
|
||||
choices=LighthouseProviderConfiguration.LLMProviderChoices.choices,
|
||||
field_name="provider_configuration__provider_type",
|
||||
)
|
||||
provider_type__in = ChoiceInFilter(
|
||||
choices=LighthouseProviderConfiguration.LLMProviderChoices.choices,
|
||||
field_name="provider_configuration__provider_type",
|
||||
lookup_expr="in",
|
||||
)
|
||||
|
||||
# Allow filtering by model id
|
||||
model_id = CharFilter(field_name="model_id", lookup_expr="exact")
|
||||
model_id__icontains = CharFilter(field_name="model_id", lookup_expr="icontains")
|
||||
model_id__in = CharInFilter(field_name="model_id", lookup_expr="in")
|
||||
|
||||
class Meta:
|
||||
model = LighthouseProviderModels
|
||||
fields = {
|
||||
"model_id": ["exact", "icontains", "in"],
|
||||
}
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
# Generated by Django 5.1.12 on 2025-10-09 07:50
|
||||
|
||||
import json
|
||||
import logging
|
||||
import uuid
|
||||
|
||||
import django.db.models.deletion
|
||||
from config.custom_logging import BackendLogger
|
||||
from cryptography.fernet import Fernet
|
||||
from django.conf import settings
|
||||
from django.db import migrations, models
|
||||
|
||||
import api.rls
|
||||
from api.db_router import MainRouter
|
||||
|
||||
logger = logging.getLogger(BackendLogger.API)
|
||||
|
||||
|
||||
def migrate_lighthouse_configs_forward(apps, schema_editor):
|
||||
"""
|
||||
Migrate data from old LighthouseConfiguration to new multi-provider models.
|
||||
Old system: one LighthouseConfiguration per tenant (always OpenAI).
|
||||
"""
|
||||
LighthouseConfiguration = apps.get_model("api", "LighthouseConfiguration")
|
||||
LighthouseProviderConfiguration = apps.get_model(
|
||||
"api", "LighthouseProviderConfiguration"
|
||||
)
|
||||
LighthouseTenantConfiguration = apps.get_model(
|
||||
"api", "LighthouseTenantConfiguration"
|
||||
)
|
||||
LighthouseProviderModels = apps.get_model("api", "LighthouseProviderModels")
|
||||
|
||||
fernet = Fernet(settings.SECRETS_ENCRYPTION_KEY.encode())
|
||||
|
||||
# Migrate only tenants that actually have a LighthouseConfiguration
|
||||
for old_config in (
|
||||
LighthouseConfiguration.objects.using(MainRouter.admin_db)
|
||||
.select_related("tenant")
|
||||
.all()
|
||||
):
|
||||
tenant = old_config.tenant
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
try:
|
||||
# Create OpenAI provider configuration for this tenant
|
||||
api_key_decrypted = fernet.decrypt(bytes(old_config.api_key)).decode()
|
||||
credentials_encrypted = fernet.encrypt(
|
||||
json.dumps({"api_key": api_key_decrypted}).encode()
|
||||
)
|
||||
provider_config = LighthouseProviderConfiguration.objects.using(
|
||||
MainRouter.admin_db
|
||||
).create(
|
||||
tenant=tenant,
|
||||
provider_type="openai",
|
||||
credentials=credentials_encrypted,
|
||||
is_active=old_config.is_active,
|
||||
)
|
||||
|
||||
# Create tenant configuration from old values
|
||||
LighthouseTenantConfiguration.objects.using(MainRouter.admin_db).create(
|
||||
tenant=tenant,
|
||||
business_context=old_config.business_context or "",
|
||||
default_provider="openai",
|
||||
default_models={"openai": old_config.model},
|
||||
)
|
||||
|
||||
# Create initial provider model record
|
||||
LighthouseProviderModels.objects.using(MainRouter.admin_db).create(
|
||||
tenant=tenant,
|
||||
provider_configuration=provider_config,
|
||||
model_id=old_config.model,
|
||||
model_name=old_config.model,
|
||||
default_parameters={},
|
||||
)
|
||||
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"Failed to migrate lighthouse config for tenant %s", tenant_id
|
||||
)
|
||||
continue
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
dependencies = [
|
||||
("api", "0049_compliancerequirementoverview_passed_failed_findings"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name="LighthouseProviderConfiguration",
|
||||
fields=[
|
||||
(
|
||||
"id",
|
||||
models.UUIDField(
|
||||
default=uuid.uuid4,
|
||||
editable=False,
|
||||
primary_key=True,
|
||||
serialize=False,
|
||||
),
|
||||
),
|
||||
("inserted_at", models.DateTimeField(auto_now_add=True)),
|
||||
("updated_at", models.DateTimeField(auto_now=True)),
|
||||
(
|
||||
"provider_type",
|
||||
models.CharField(
|
||||
choices=[("openai", "OpenAI")],
|
||||
help_text="LLM provider name",
|
||||
max_length=50,
|
||||
),
|
||||
),
|
||||
("base_url", models.URLField(blank=True, null=True)),
|
||||
(
|
||||
"credentials",
|
||||
models.BinaryField(
|
||||
help_text="Encrypted JSON credentials for the provider"
|
||||
),
|
||||
),
|
||||
("is_active", models.BooleanField(default=True)),
|
||||
],
|
||||
options={
|
||||
"db_table": "lighthouse_provider_configurations",
|
||||
"abstract": False,
|
||||
},
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name="LighthouseProviderModels",
|
||||
fields=[
|
||||
(
|
||||
"id",
|
||||
models.UUIDField(
|
||||
default=uuid.uuid4,
|
||||
editable=False,
|
||||
primary_key=True,
|
||||
serialize=False,
|
||||
),
|
||||
),
|
||||
("inserted_at", models.DateTimeField(auto_now_add=True)),
|
||||
("updated_at", models.DateTimeField(auto_now=True)),
|
||||
("model_id", models.CharField(max_length=100)),
|
||||
("model_name", models.CharField(max_length=100)),
|
||||
("default_parameters", models.JSONField(blank=True, default=dict)),
|
||||
],
|
||||
options={
|
||||
"db_table": "lighthouse_provider_models",
|
||||
"abstract": False,
|
||||
},
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name="LighthouseTenantConfiguration",
|
||||
fields=[
|
||||
(
|
||||
"id",
|
||||
models.UUIDField(
|
||||
default=uuid.uuid4,
|
||||
editable=False,
|
||||
primary_key=True,
|
||||
serialize=False,
|
||||
),
|
||||
),
|
||||
("inserted_at", models.DateTimeField(auto_now_add=True)),
|
||||
("updated_at", models.DateTimeField(auto_now=True)),
|
||||
("business_context", models.TextField(blank=True, default="")),
|
||||
("default_provider", models.CharField(blank=True, max_length=50)),
|
||||
("default_models", models.JSONField(blank=True, default=dict)),
|
||||
],
|
||||
options={
|
||||
"db_table": "lighthouse_tenant_config",
|
||||
"abstract": False,
|
||||
},
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="lighthouseproviderconfiguration",
|
||||
name="tenant",
|
||||
field=models.ForeignKey(
|
||||
on_delete=django.db.models.deletion.CASCADE, to="api.tenant"
|
||||
),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="lighthouseprovidermodels",
|
||||
name="provider_configuration",
|
||||
field=models.ForeignKey(
|
||||
on_delete=django.db.models.deletion.CASCADE,
|
||||
related_name="available_models",
|
||||
to="api.lighthouseproviderconfiguration",
|
||||
),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="lighthouseprovidermodels",
|
||||
name="tenant",
|
||||
field=models.ForeignKey(
|
||||
on_delete=django.db.models.deletion.CASCADE, to="api.tenant"
|
||||
),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="lighthousetenantconfiguration",
|
||||
name="tenant",
|
||||
field=models.ForeignKey(
|
||||
on_delete=django.db.models.deletion.CASCADE, to="api.tenant"
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="lighthouseproviderconfiguration",
|
||||
index=models.Index(
|
||||
fields=["tenant_id", "provider_type"], name="lh_pc_tenant_type_idx"
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="lighthouseproviderconfiguration",
|
||||
constraint=api.rls.RowLevelSecurityConstraint(
|
||||
"tenant_id",
|
||||
name="rls_on_lighthouseproviderconfiguration",
|
||||
statements=["SELECT", "INSERT", "UPDATE", "DELETE"],
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="lighthouseproviderconfiguration",
|
||||
constraint=models.UniqueConstraint(
|
||||
fields=("tenant_id", "provider_type"),
|
||||
name="unique_provider_config_per_tenant",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="lighthouseprovidermodels",
|
||||
index=models.Index(
|
||||
fields=["tenant_id", "provider_configuration"],
|
||||
name="lh_prov_models_cfg_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="lighthouseprovidermodels",
|
||||
constraint=api.rls.RowLevelSecurityConstraint(
|
||||
"tenant_id",
|
||||
name="rls_on_lighthouseprovidermodels",
|
||||
statements=["SELECT", "INSERT", "UPDATE", "DELETE"],
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="lighthouseprovidermodels",
|
||||
constraint=models.UniqueConstraint(
|
||||
fields=("tenant_id", "provider_configuration", "model_id"),
|
||||
name="unique_provider_model_per_configuration",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="lighthousetenantconfiguration",
|
||||
constraint=api.rls.RowLevelSecurityConstraint(
|
||||
"tenant_id",
|
||||
name="rls_on_lighthousetenantconfiguration",
|
||||
statements=["SELECT", "INSERT", "UPDATE", "DELETE"],
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="lighthousetenantconfiguration",
|
||||
constraint=models.UniqueConstraint(
|
||||
fields=("tenant_id",), name="unique_tenant_lighthouse_config"
|
||||
),
|
||||
),
|
||||
# Migrate data from old LighthouseConfiguration to new tables
|
||||
# This runs after all tables, indexes, and constraints are created
|
||||
# The old Lighthouse configuration table is not removed, so reverse_code is noop
|
||||
# During rollbacks, the old Lighthouse configuration remains intact while the new tables are removed
|
||||
migrations.RunPython(
|
||||
migrate_lighthouse_configs_forward,
|
||||
reverse_code=migrations.RunPython.noop,
|
||||
),
|
||||
]
|
||||
+181
-25
@@ -1873,22 +1873,6 @@ class LighthouseConfiguration(RowLevelSecurityProtectedModel):
|
||||
def clean(self):
|
||||
super().clean()
|
||||
|
||||
# Validate temperature
|
||||
if not 0 <= self.temperature <= 1:
|
||||
raise ModelValidationError(
|
||||
detail="Temperature must be between 0 and 1",
|
||||
code="invalid_temperature",
|
||||
pointer="/data/attributes/temperature",
|
||||
)
|
||||
|
||||
# Validate max_tokens
|
||||
if not 500 <= self.max_tokens <= 5000:
|
||||
raise ModelValidationError(
|
||||
detail="Max tokens must be between 500 and 5000",
|
||||
code="invalid_max_tokens",
|
||||
pointer="/data/attributes/max_tokens",
|
||||
)
|
||||
|
||||
@property
|
||||
def api_key_decoded(self):
|
||||
"""Return the decrypted API key, or None if unavailable or invalid."""
|
||||
@@ -1913,15 +1897,6 @@ class LighthouseConfiguration(RowLevelSecurityProtectedModel):
|
||||
code="invalid_api_key",
|
||||
pointer="/data/attributes/api_key",
|
||||
)
|
||||
|
||||
# Validate OpenAI API key format
|
||||
openai_key_pattern = r"^sk-[\w-]+T3BlbkFJ[\w-]+$"
|
||||
if not re.match(openai_key_pattern, value):
|
||||
raise ModelValidationError(
|
||||
detail="Invalid OpenAI API key format.",
|
||||
code="invalid_api_key",
|
||||
pointer="/data/attributes/api_key",
|
||||
)
|
||||
self.api_key = fernet.encrypt(value.encode())
|
||||
|
||||
def save(self, *args, **kwargs):
|
||||
@@ -1984,3 +1959,184 @@ class Processor(RowLevelSecurityProtectedModel):
|
||||
|
||||
class JSONAPIMeta:
|
||||
resource_name = "processors"
|
||||
|
||||
|
||||
class LighthouseProviderConfiguration(RowLevelSecurityProtectedModel):
|
||||
"""
|
||||
Per-tenant configuration for an LLM provider (credentials, base URL, activation).
|
||||
|
||||
One configuration per provider type per tenant.
|
||||
"""
|
||||
|
||||
class LLMProviderChoices(models.TextChoices):
|
||||
OPENAI = "openai", _("OpenAI")
|
||||
|
||||
id = models.UUIDField(primary_key=True, default=uuid4, editable=False)
|
||||
inserted_at = models.DateTimeField(auto_now_add=True, editable=False)
|
||||
updated_at = models.DateTimeField(auto_now=True, editable=False)
|
||||
|
||||
provider_type = models.CharField(
|
||||
max_length=50,
|
||||
choices=LLMProviderChoices.choices,
|
||||
help_text="LLM provider name",
|
||||
)
|
||||
|
||||
# For OpenAI-compatible providers
|
||||
base_url = models.URLField(blank=True, null=True)
|
||||
|
||||
# Encrypted JSON for provider-specific auth
|
||||
credentials = models.BinaryField(
|
||||
blank=False, null=False, help_text="Encrypted JSON credentials for the provider"
|
||||
)
|
||||
|
||||
is_active = models.BooleanField(default=True)
|
||||
|
||||
def __str__(self):
|
||||
return f"{self.get_provider_type_display()} ({self.tenant_id})"
|
||||
|
||||
def clean(self):
|
||||
super().clean()
|
||||
|
||||
@property
|
||||
def credentials_decoded(self):
|
||||
if not self.credentials:
|
||||
return None
|
||||
try:
|
||||
decrypted_data = fernet.decrypt(bytes(self.credentials))
|
||||
return json.loads(decrypted_data.decode())
|
||||
except (InvalidToken, json.JSONDecodeError) as e:
|
||||
logger.warning("Failed to decrypt provider credentials: %s", e)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.exception(
|
||||
"Unexpected error while decrypting provider credentials: %s", e
|
||||
)
|
||||
return None
|
||||
|
||||
@credentials_decoded.setter
|
||||
def credentials_decoded(self, value):
|
||||
"""
|
||||
Set and encrypt credentials (assumes serializer performed validation).
|
||||
"""
|
||||
if not value:
|
||||
raise ModelValidationError(
|
||||
detail="Credentials are required",
|
||||
code="invalid_credentials",
|
||||
pointer="/data/attributes/credentials",
|
||||
)
|
||||
self.credentials = fernet.encrypt(json.dumps(value).encode())
|
||||
|
||||
class Meta(RowLevelSecurityProtectedModel.Meta):
|
||||
db_table = "lighthouse_provider_configurations"
|
||||
|
||||
constraints = [
|
||||
RowLevelSecurityConstraint(
|
||||
field="tenant_id",
|
||||
name="rls_on_%(class)s",
|
||||
statements=["SELECT", "INSERT", "UPDATE", "DELETE"],
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["tenant_id", "provider_type"],
|
||||
name="unique_provider_config_per_tenant",
|
||||
),
|
||||
]
|
||||
|
||||
indexes = [
|
||||
models.Index(
|
||||
fields=["tenant_id", "provider_type"],
|
||||
name="lh_pc_tenant_type_idx",
|
||||
),
|
||||
]
|
||||
|
||||
class JSONAPIMeta:
|
||||
resource_name = "lighthouse-providers"
|
||||
|
||||
|
||||
class LighthouseTenantConfiguration(RowLevelSecurityProtectedModel):
|
||||
"""
|
||||
Tenant-level Lighthouse settings (business context and defaults).
|
||||
One record per tenant.
|
||||
"""
|
||||
|
||||
id = models.UUIDField(primary_key=True, default=uuid4, editable=False)
|
||||
inserted_at = models.DateTimeField(auto_now_add=True, editable=False)
|
||||
updated_at = models.DateTimeField(auto_now=True, editable=False)
|
||||
|
||||
business_context = models.TextField(blank=True, default="")
|
||||
|
||||
# Preferred provider key (e.g., "openai", "bedrock", "openai_compatible")
|
||||
default_provider = models.CharField(max_length=50, blank=True)
|
||||
|
||||
# Mapping of provider -> model id, e.g., {"openai": "gpt-4o", "bedrock": "anthropic.claude-v2"}
|
||||
default_models = models.JSONField(default=dict, blank=True)
|
||||
|
||||
def __str__(self):
|
||||
return f"Lighthouse Tenant Config for {self.tenant_id}"
|
||||
|
||||
def clean(self):
|
||||
super().clean()
|
||||
|
||||
class Meta(RowLevelSecurityProtectedModel.Meta):
|
||||
db_table = "lighthouse_tenant_config"
|
||||
|
||||
constraints = [
|
||||
RowLevelSecurityConstraint(
|
||||
field="tenant_id",
|
||||
name="rls_on_%(class)s",
|
||||
statements=["SELECT", "INSERT", "UPDATE", "DELETE"],
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["tenant_id"], name="unique_tenant_lighthouse_config"
|
||||
),
|
||||
]
|
||||
|
||||
class JSONAPIMeta:
|
||||
resource_name = "lighthouse-config"
|
||||
|
||||
|
||||
class LighthouseProviderModels(RowLevelSecurityProtectedModel):
|
||||
"""
|
||||
Per-tenant, per-provider configuration list of available LLM models.
|
||||
RLS-protected; populated via provider API using tenant-scoped credentials.
|
||||
"""
|
||||
|
||||
id = models.UUIDField(primary_key=True, default=uuid4, editable=False)
|
||||
inserted_at = models.DateTimeField(auto_now_add=True, editable=False)
|
||||
updated_at = models.DateTimeField(auto_now=True, editable=False)
|
||||
|
||||
# Scope to a specific provider configuration within a tenant
|
||||
provider_configuration = models.ForeignKey(
|
||||
LighthouseProviderConfiguration,
|
||||
on_delete=models.CASCADE,
|
||||
related_name="available_models",
|
||||
)
|
||||
model_id = models.CharField(max_length=100)
|
||||
|
||||
# Human-friendly model name
|
||||
model_name = models.CharField(max_length=100)
|
||||
|
||||
# Model-specific default parameters (e.g., temperature, max_tokens)
|
||||
default_parameters = models.JSONField(default=dict, blank=True)
|
||||
|
||||
def __str__(self):
|
||||
return f"{self.provider_configuration.provider_type}:{self.model_id} ({self.tenant_id})"
|
||||
|
||||
class Meta(RowLevelSecurityProtectedModel.Meta):
|
||||
db_table = "lighthouse_provider_models"
|
||||
constraints = [
|
||||
RowLevelSecurityConstraint(
|
||||
field="tenant_id",
|
||||
name="rls_on_%(class)s",
|
||||
statements=["SELECT", "INSERT", "UPDATE", "DELETE"],
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["tenant_id", "provider_configuration", "model_id"],
|
||||
name="unique_provider_model_per_configuration",
|
||||
),
|
||||
]
|
||||
indexes = [
|
||||
models.Index(
|
||||
fields=["tenant_id", "provider_configuration"],
|
||||
name="lh_prov_models_cfg_idx",
|
||||
),
|
||||
]
|
||||
|
||||
@@ -6,7 +6,14 @@ from django.dispatch import receiver
|
||||
from django_celery_results.backends.database import DatabaseBackend
|
||||
|
||||
from api.db_utils import delete_related_daily_task
|
||||
from api.models import Membership, Provider, TenantAPIKey, User
|
||||
from api.models import (
|
||||
LighthouseProviderConfiguration,
|
||||
LighthouseTenantConfiguration,
|
||||
Membership,
|
||||
Provider,
|
||||
TenantAPIKey,
|
||||
User,
|
||||
)
|
||||
|
||||
|
||||
def create_task_result_on_publish(sender=None, headers=None, **kwargs): # noqa: F841
|
||||
@@ -56,3 +63,33 @@ def revoke_membership_api_keys(sender, instance, **kwargs): # noqa: F841
|
||||
TenantAPIKey.objects.filter(
|
||||
entity=instance.user, tenant_id=instance.tenant.id
|
||||
).update(revoked=True)
|
||||
|
||||
|
||||
@receiver(pre_delete, sender=LighthouseProviderConfiguration)
|
||||
def cleanup_lighthouse_defaults_before_delete(sender, instance, **kwargs): # noqa: F841
|
||||
"""
|
||||
Ensure tenant Lighthouse defaults do not reference a soon-to-be-deleted provider.
|
||||
|
||||
This runs for both per-instance deletes and queryset (bulk) deletes.
|
||||
"""
|
||||
try:
|
||||
tenant_cfg = LighthouseTenantConfiguration.objects.get(
|
||||
tenant_id=instance.tenant_id
|
||||
)
|
||||
except LighthouseTenantConfiguration.DoesNotExist:
|
||||
return
|
||||
|
||||
updated = False
|
||||
defaults = tenant_cfg.default_models or {}
|
||||
|
||||
if instance.provider_type in defaults:
|
||||
defaults.pop(instance.provider_type, None)
|
||||
tenant_cfg.default_models = defaults
|
||||
updated = True
|
||||
|
||||
if tenant_cfg.default_provider == instance.provider_type:
|
||||
tenant_cfg.default_provider = ""
|
||||
updated = True
|
||||
|
||||
if updated:
|
||||
tenant_cfg.save()
|
||||
|
||||
+1019
-67
File diff suppressed because it is too large
Load Diff
@@ -1,39 +0,0 @@
|
||||
"""Tests for rls_transaction retry and fallback logic."""
|
||||
|
||||
import pytest
|
||||
from django.db import DEFAULT_DB_ALIAS
|
||||
from rest_framework_json_api.serializers import ValidationError
|
||||
|
||||
from api.db_utils import rls_transaction
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
class TestRLSTransaction:
|
||||
"""Simple integration tests for rls_transaction using real DB."""
|
||||
|
||||
@pytest.fixture
|
||||
def tenant(self, tenants_fixture):
|
||||
return tenants_fixture[0]
|
||||
|
||||
def test_success_on_primary(self, tenant):
|
||||
"""Basic: transaction succeeds on primary database."""
|
||||
with rls_transaction(str(tenant.id), using=DEFAULT_DB_ALIAS) as cursor:
|
||||
cursor.execute("SELECT 1")
|
||||
result = cursor.fetchone()
|
||||
assert result == (1,)
|
||||
|
||||
def test_invalid_uuid_raises_validation_error(self):
|
||||
"""Invalid UUID raises ValidationError before DB operations."""
|
||||
with pytest.raises(ValidationError, match="Must be a valid UUID"):
|
||||
with rls_transaction("not-a-uuid", using=DEFAULT_DB_ALIAS):
|
||||
pass
|
||||
|
||||
def test_custom_parameter_name(self, tenant):
|
||||
"""Test custom RLS parameter name."""
|
||||
custom_param = "api.custom_id"
|
||||
with rls_transaction(
|
||||
str(tenant.id), parameter=custom_param, using=DEFAULT_DB_ALIAS
|
||||
) as cursor:
|
||||
cursor.execute("SELECT current_setting(%s, true)", [custom_param])
|
||||
result = cursor.fetchone()
|
||||
assert result == (str(tenant.id),)
|
||||
@@ -1,15 +1,12 @@
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
from unittest.mock import MagicMock, patch
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
from django.conf import settings
|
||||
from django.db import DEFAULT_DB_ALIAS, OperationalError
|
||||
from freezegun import freeze_time
|
||||
from rest_framework_json_api.serializers import ValidationError
|
||||
|
||||
from api.db_utils import (
|
||||
POSTGRES_TENANT_VAR,
|
||||
_should_create_index_on_partition,
|
||||
batch_delete,
|
||||
create_objects_in_batches,
|
||||
@@ -17,22 +14,11 @@ from api.db_utils import (
|
||||
generate_api_key_prefix,
|
||||
generate_random_token,
|
||||
one_week_from_now,
|
||||
rls_transaction,
|
||||
update_objects_in_batches,
|
||||
)
|
||||
from api.models import Provider
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def enable_read_replica():
|
||||
"""
|
||||
Fixture to enable READ_REPLICA_ALIAS for tests that need replica functionality.
|
||||
This avoids polluting the global test configuration.
|
||||
"""
|
||||
with patch("api.db_utils.READ_REPLICA_ALIAS", "replica"):
|
||||
yield "replica"
|
||||
|
||||
|
||||
class TestEnumToChoices:
|
||||
def test_enum_to_choices_simple(self):
|
||||
class Color(Enum):
|
||||
@@ -353,498 +339,3 @@ class TestGenerateApiKeyPrefix:
|
||||
prefix = generate_api_key_prefix()
|
||||
random_part = prefix[3:] # Strip 'pk_'
|
||||
assert all(char in allowed_chars for char in random_part)
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
class TestRlsTransaction:
|
||||
def test_rls_transaction_valid_uuid_string(self, tenants_fixture):
|
||||
"""Test rls_transaction with valid UUID string."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with rls_transaction(tenant_id) as cursor:
|
||||
assert cursor is not None
|
||||
cursor.execute("SELECT current_setting(%s)", [POSTGRES_TENANT_VAR])
|
||||
result = cursor.fetchone()
|
||||
assert result[0] == tenant_id
|
||||
|
||||
def test_rls_transaction_valid_uuid_object(self, tenants_fixture):
|
||||
"""Test rls_transaction with UUID object."""
|
||||
tenant = tenants_fixture[0]
|
||||
|
||||
with rls_transaction(tenant.id) as cursor:
|
||||
assert cursor is not None
|
||||
cursor.execute("SELECT current_setting(%s)", [POSTGRES_TENANT_VAR])
|
||||
result = cursor.fetchone()
|
||||
assert result[0] == str(tenant.id)
|
||||
|
||||
def test_rls_transaction_invalid_uuid_raises_validation_error(self):
|
||||
"""Test rls_transaction raises ValidationError for invalid UUID."""
|
||||
invalid_uuid = "not-a-valid-uuid"
|
||||
|
||||
with pytest.raises(ValidationError, match="Must be a valid UUID"):
|
||||
with rls_transaction(invalid_uuid):
|
||||
pass
|
||||
|
||||
def test_rls_transaction_uses_default_database_when_no_alias(self, tenants_fixture):
|
||||
"""Test rls_transaction uses DEFAULT_DB_ALIAS when no alias specified."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=None):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
with patch("api.db_utils.transaction.atomic"):
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
mock_connections.__getitem__.assert_called_with(DEFAULT_DB_ALIAS)
|
||||
|
||||
def test_rls_transaction_uses_specified_alias(self, tenants_fixture):
|
||||
"""Test rls_transaction uses specified database alias via using parameter."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
custom_alias = "custom_db"
|
||||
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
with patch("api.db_utils.transaction.atomic"):
|
||||
with patch("api.db_utils.set_read_db_alias") as mock_set_alias:
|
||||
with patch("api.db_utils.reset_read_db_alias") as mock_reset_alias:
|
||||
mock_set_alias.return_value = "test_token"
|
||||
with rls_transaction(tenant_id, using=custom_alias):
|
||||
pass
|
||||
|
||||
mock_connections.__getitem__.assert_called_with(custom_alias)
|
||||
mock_set_alias.assert_called_once_with(custom_alias)
|
||||
mock_reset_alias.assert_called_once_with("test_token")
|
||||
|
||||
def test_rls_transaction_uses_read_replica_from_router(
|
||||
self, tenants_fixture, enable_read_replica
|
||||
):
|
||||
"""Test rls_transaction uses read replica alias from router."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=enable_read_replica):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
with patch("api.db_utils.transaction.atomic"):
|
||||
with patch("api.db_utils.set_read_db_alias") as mock_set_alias:
|
||||
with patch(
|
||||
"api.db_utils.reset_read_db_alias"
|
||||
) as mock_reset_alias:
|
||||
mock_set_alias.return_value = "test_token"
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
mock_connections.__getitem__.assert_called()
|
||||
mock_set_alias.assert_called_once()
|
||||
mock_reset_alias.assert_called_once()
|
||||
|
||||
def test_rls_transaction_fallback_to_default_when_alias_not_in_connections(
|
||||
self, tenants_fixture
|
||||
):
|
||||
"""Test rls_transaction falls back to DEFAULT_DB_ALIAS when alias not in connections."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
invalid_alias = "nonexistent_db"
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=invalid_alias):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
|
||||
def contains_check(alias):
|
||||
return alias == DEFAULT_DB_ALIAS
|
||||
|
||||
mock_connections.__contains__.side_effect = contains_check
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
|
||||
with patch("api.db_utils.transaction.atomic"):
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
mock_connections.__getitem__.assert_called_with(DEFAULT_DB_ALIAS)
|
||||
|
||||
def test_rls_transaction_successful_execution_on_replica_no_retries(
|
||||
self, tenants_fixture, enable_read_replica
|
||||
):
|
||||
"""Test successful execution on replica without retries."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=enable_read_replica):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
with patch("api.db_utils.transaction.atomic"):
|
||||
with patch("api.db_utils.set_read_db_alias", return_value="token"):
|
||||
with patch("api.db_utils.reset_read_db_alias"):
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
assert mock_cursor.execute.call_count == 1
|
||||
|
||||
def test_rls_transaction_retry_with_exponential_backoff_on_operational_error(
|
||||
self, tenants_fixture, enable_read_replica
|
||||
):
|
||||
"""Test retry with exponential backoff on OperationalError on replica."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=enable_read_replica):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
call_count = 0
|
||||
|
||||
def atomic_side_effect(*args, **kwargs):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count < 3:
|
||||
raise OperationalError("Connection error")
|
||||
return MagicMock(
|
||||
__enter__=MagicMock(return_value=None),
|
||||
__exit__=MagicMock(return_value=False),
|
||||
)
|
||||
|
||||
with patch(
|
||||
"api.db_utils.transaction.atomic", side_effect=atomic_side_effect
|
||||
):
|
||||
with patch("api.db_utils.time.sleep") as mock_sleep:
|
||||
with patch(
|
||||
"api.db_utils.set_read_db_alias", return_value="token"
|
||||
):
|
||||
with patch("api.db_utils.reset_read_db_alias"):
|
||||
with patch("api.db_utils.logger") as mock_logger:
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
assert mock_sleep.call_count == 2
|
||||
mock_sleep.assert_any_call(0.5)
|
||||
mock_sleep.assert_any_call(1.0)
|
||||
assert mock_logger.info.call_count == 2
|
||||
|
||||
def test_rls_transaction_max_three_attempts_for_replica(
|
||||
self, tenants_fixture, enable_read_replica
|
||||
):
|
||||
"""Test maximum 3 attempts for replica database."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=enable_read_replica):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
with patch("api.db_utils.transaction.atomic") as mock_atomic:
|
||||
mock_atomic.side_effect = OperationalError("Persistent error")
|
||||
|
||||
with patch("api.db_utils.time.sleep"):
|
||||
with patch(
|
||||
"api.db_utils.set_read_db_alias", return_value="token"
|
||||
):
|
||||
with patch("api.db_utils.reset_read_db_alias"):
|
||||
with pytest.raises(OperationalError):
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
assert mock_atomic.call_count == 3
|
||||
|
||||
def test_rls_transaction_only_one_attempt_for_primary(self, tenants_fixture):
|
||||
"""Test only 1 attempt for primary database."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=None):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
with patch("api.db_utils.transaction.atomic") as mock_atomic:
|
||||
mock_atomic.side_effect = OperationalError("Primary error")
|
||||
|
||||
with pytest.raises(OperationalError):
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
assert mock_atomic.call_count == 1
|
||||
|
||||
def test_rls_transaction_fallback_to_primary_after_max_attempts(
|
||||
self, tenants_fixture, enable_read_replica
|
||||
):
|
||||
"""Test fallback to primary DB after max attempts on replica."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=enable_read_replica):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
call_count = 0
|
||||
|
||||
def atomic_side_effect(*args, **kwargs):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count < 3:
|
||||
raise OperationalError("Replica error")
|
||||
return MagicMock(
|
||||
__enter__=MagicMock(return_value=None),
|
||||
__exit__=MagicMock(return_value=False),
|
||||
)
|
||||
|
||||
with patch(
|
||||
"api.db_utils.transaction.atomic", side_effect=atomic_side_effect
|
||||
):
|
||||
with patch("api.db_utils.time.sleep"):
|
||||
with patch(
|
||||
"api.db_utils.set_read_db_alias", return_value="token"
|
||||
):
|
||||
with patch("api.db_utils.reset_read_db_alias"):
|
||||
with patch("api.db_utils.logger") as mock_logger:
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
mock_logger.warning.assert_called_once()
|
||||
warning_msg = mock_logger.warning.call_args[0][0]
|
||||
assert "falling back to primary DB" in warning_msg
|
||||
|
||||
def test_rls_transaction_logger_warning_on_fallback(
|
||||
self, tenants_fixture, enable_read_replica
|
||||
):
|
||||
"""Test logger warnings are emitted on fallback to primary."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=enable_read_replica):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
call_count = 0
|
||||
|
||||
def atomic_side_effect(*args, **kwargs):
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
if call_count < 3:
|
||||
raise OperationalError("Replica error")
|
||||
return MagicMock(
|
||||
__enter__=MagicMock(return_value=None),
|
||||
__exit__=MagicMock(return_value=False),
|
||||
)
|
||||
|
||||
with patch(
|
||||
"api.db_utils.transaction.atomic", side_effect=atomic_side_effect
|
||||
):
|
||||
with patch("api.db_utils.time.sleep"):
|
||||
with patch(
|
||||
"api.db_utils.set_read_db_alias", return_value="token"
|
||||
):
|
||||
with patch("api.db_utils.reset_read_db_alias"):
|
||||
with patch("api.db_utils.logger") as mock_logger:
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
assert mock_logger.info.call_count == 2
|
||||
assert mock_logger.warning.call_count == 1
|
||||
|
||||
def test_rls_transaction_operational_error_raised_immediately_on_primary(
|
||||
self, tenants_fixture
|
||||
):
|
||||
"""Test OperationalError raised immediately on primary without retry."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=None):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
with patch("api.db_utils.transaction.atomic") as mock_atomic:
|
||||
mock_atomic.side_effect = OperationalError("Primary error")
|
||||
|
||||
with patch("api.db_utils.time.sleep") as mock_sleep:
|
||||
with pytest.raises(OperationalError):
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
mock_sleep.assert_not_called()
|
||||
|
||||
def test_rls_transaction_operational_error_raised_after_max_attempts(
|
||||
self, tenants_fixture, enable_read_replica
|
||||
):
|
||||
"""Test OperationalError raised after max attempts on replica."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=enable_read_replica):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
with patch("api.db_utils.transaction.atomic") as mock_atomic:
|
||||
mock_atomic.side_effect = OperationalError(
|
||||
"Persistent replica error"
|
||||
)
|
||||
|
||||
with patch("api.db_utils.time.sleep"):
|
||||
with patch(
|
||||
"api.db_utils.set_read_db_alias", return_value="token"
|
||||
):
|
||||
with patch("api.db_utils.reset_read_db_alias"):
|
||||
with pytest.raises(OperationalError):
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
def test_rls_transaction_router_token_set_for_non_default_alias(
|
||||
self, tenants_fixture
|
||||
):
|
||||
"""Test router token is set when using non-default alias."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
custom_alias = "custom_db"
|
||||
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
with patch("api.db_utils.transaction.atomic"):
|
||||
with patch("api.db_utils.set_read_db_alias") as mock_set_alias:
|
||||
with patch("api.db_utils.reset_read_db_alias") as mock_reset_alias:
|
||||
mock_set_alias.return_value = "test_token"
|
||||
with rls_transaction(tenant_id, using=custom_alias):
|
||||
pass
|
||||
|
||||
mock_set_alias.assert_called_once_with(custom_alias)
|
||||
mock_reset_alias.assert_called_once_with("test_token")
|
||||
|
||||
def test_rls_transaction_router_token_reset_in_finally_block(self, tenants_fixture):
|
||||
"""Test router token is reset in finally block even on error."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
custom_alias = "custom_db"
|
||||
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
with patch("api.db_utils.transaction.atomic") as mock_atomic:
|
||||
mock_atomic.side_effect = Exception("Unexpected error")
|
||||
|
||||
with patch("api.db_utils.set_read_db_alias", return_value="test_token"):
|
||||
with patch("api.db_utils.reset_read_db_alias") as mock_reset_alias:
|
||||
with pytest.raises(Exception):
|
||||
with rls_transaction(tenant_id, using=custom_alias):
|
||||
pass
|
||||
|
||||
mock_reset_alias.assert_called_once_with("test_token")
|
||||
|
||||
def test_rls_transaction_router_token_not_set_for_default_alias(
|
||||
self, tenants_fixture
|
||||
):
|
||||
"""Test router token is not set when using default alias."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with patch("api.db_utils.get_read_db_alias", return_value=None):
|
||||
with patch("api.db_utils.connections") as mock_connections:
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value.__enter__.return_value = mock_cursor
|
||||
mock_connections.__getitem__.return_value = mock_conn
|
||||
mock_connections.__contains__.return_value = True
|
||||
|
||||
with patch("api.db_utils.transaction.atomic"):
|
||||
with patch("api.db_utils.set_read_db_alias") as mock_set_alias:
|
||||
with patch(
|
||||
"api.db_utils.reset_read_db_alias"
|
||||
) as mock_reset_alias:
|
||||
with rls_transaction(tenant_id):
|
||||
pass
|
||||
|
||||
mock_set_alias.assert_not_called()
|
||||
mock_reset_alias.assert_not_called()
|
||||
|
||||
def test_rls_transaction_set_config_query_executed_with_correct_params(
|
||||
self, tenants_fixture
|
||||
):
|
||||
"""Test SET_CONFIG_QUERY executed with correct parameters."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with rls_transaction(tenant_id) as cursor:
|
||||
cursor.execute("SELECT current_setting(%s)", [POSTGRES_TENANT_VAR])
|
||||
result = cursor.fetchone()
|
||||
assert result[0] == tenant_id
|
||||
|
||||
def test_rls_transaction_custom_parameter(self, tenants_fixture):
|
||||
"""Test rls_transaction with custom parameter name."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
custom_param = "api.user_id"
|
||||
|
||||
with rls_transaction(tenant_id, parameter=custom_param) as cursor:
|
||||
cursor.execute("SELECT current_setting(%s)", [custom_param])
|
||||
result = cursor.fetchone()
|
||||
assert result[0] == tenant_id
|
||||
|
||||
def test_rls_transaction_cursor_yielded_correctly(self, tenants_fixture):
|
||||
"""Test cursor is yielded correctly."""
|
||||
tenant = tenants_fixture[0]
|
||||
tenant_id = str(tenant.id)
|
||||
|
||||
with rls_transaction(tenant_id) as cursor:
|
||||
assert cursor is not None
|
||||
cursor.execute("SELECT 1")
|
||||
result = cursor.fetchone()
|
||||
assert result[0] == 1
|
||||
|
||||
@@ -35,13 +35,15 @@ from api.db_router import MainRouter
|
||||
from api.models import (
|
||||
Integration,
|
||||
Invitation,
|
||||
LighthouseProviderConfiguration,
|
||||
LighthouseProviderModels,
|
||||
LighthouseTenantConfiguration,
|
||||
Membership,
|
||||
Processor,
|
||||
Provider,
|
||||
ProviderGroup,
|
||||
ProviderGroupMembership,
|
||||
ProviderSecret,
|
||||
Resource,
|
||||
Role,
|
||||
RoleProviderGroupRelationship,
|
||||
SAMLConfiguration,
|
||||
@@ -5782,62 +5784,8 @@ class TestOverviewViewSet:
|
||||
assert response.json()["data"][0]["attributes"]["findings"]["pass"] == 2
|
||||
assert response.json()["data"][0]["attributes"]["findings"]["fail"] == 1
|
||||
assert response.json()["data"][0]["attributes"]["findings"]["muted"] == 1
|
||||
# Aggregated resources include all AWS providers present in the tenant
|
||||
assert response.json()["data"][0]["attributes"]["resources"]["total"] == 3
|
||||
|
||||
def test_overview_providers_aggregates_same_provider_type(
|
||||
self,
|
||||
authenticated_client,
|
||||
scan_summaries_fixture,
|
||||
resources_fixture,
|
||||
providers_fixture,
|
||||
tenants_fixture,
|
||||
):
|
||||
tenant = tenants_fixture[0]
|
||||
_provider1, provider2, *_ = providers_fixture
|
||||
|
||||
scan = Scan.objects.create(
|
||||
name="overview scan aws account 2",
|
||||
provider=provider2,
|
||||
trigger=Scan.TriggerChoices.MANUAL,
|
||||
state=StateChoices.COMPLETED,
|
||||
tenant=tenant,
|
||||
)
|
||||
|
||||
ScanSummary.objects.create(
|
||||
tenant=tenant,
|
||||
scan=scan,
|
||||
check_id="check-aws-two",
|
||||
service="service-extra",
|
||||
severity="medium",
|
||||
region="region-extra",
|
||||
_pass=3,
|
||||
fail=2,
|
||||
muted=1,
|
||||
total=6,
|
||||
)
|
||||
|
||||
Resource.objects.create(
|
||||
tenant_id=tenant.id,
|
||||
provider=provider2,
|
||||
uid="arn:aws:ec2:us-west-2:123456789013:instance/i-aggregation",
|
||||
name="Aggregated Instance",
|
||||
region="us-west-2",
|
||||
service="ec2",
|
||||
type="prowler-test",
|
||||
)
|
||||
|
||||
response = authenticated_client.get(reverse("overview-providers"))
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
data = response.json()["data"]
|
||||
assert len(data) == 1
|
||||
attributes = data[0]["attributes"]
|
||||
|
||||
assert attributes["findings"]["total"] == 10
|
||||
assert attributes["findings"]["pass"] == 5
|
||||
assert attributes["findings"]["fail"] == 3
|
||||
assert attributes["findings"]["muted"] == 2
|
||||
assert attributes["resources"]["total"] == 4
|
||||
# Since we rely on completed scans, there are only 2 resources now
|
||||
assert response.json()["data"][0]["attributes"]["resources"]["total"] == 2
|
||||
|
||||
def test_overview_services_list_no_required_filters(
|
||||
self, authenticated_client, scan_summaries_fixture
|
||||
@@ -8758,3 +8706,483 @@ class TestTenantApiKeyViewSet:
|
||||
# Verify error object structure
|
||||
error = response_data["errors"][0]
|
||||
assert "detail" in error or "title" in error
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
class TestLighthouseTenantConfigViewSet:
|
||||
"""Test Lighthouse tenant configuration endpoint (singleton pattern)"""
|
||||
|
||||
def test_lighthouse_tenant_config_create_via_patch(self, authenticated_client):
|
||||
"""Test creating a tenant config successfully via PATCH (upsert)"""
|
||||
payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-config",
|
||||
"attributes": {
|
||||
"business_context": "Test business context for security analysis",
|
||||
"default_provider": "",
|
||||
"default_models": {},
|
||||
},
|
||||
}
|
||||
}
|
||||
response = authenticated_client.patch(
|
||||
reverse("lighthouse-config"),
|
||||
data=payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
data = response.json()["data"]
|
||||
assert (
|
||||
data["attributes"]["business_context"]
|
||||
== "Test business context for security analysis"
|
||||
)
|
||||
assert data["attributes"]["default_provider"] == ""
|
||||
assert data["attributes"]["default_models"] == {}
|
||||
|
||||
def test_lighthouse_tenant_config_upsert_behavior(self, authenticated_client):
|
||||
"""Test that PATCH creates config if not exists and updates if exists (upsert)"""
|
||||
payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-config",
|
||||
"attributes": {
|
||||
"business_context": "First config",
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
# First PATCH creates the config
|
||||
response = authenticated_client.patch(
|
||||
reverse("lighthouse-config"),
|
||||
data=payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
first_data = response.json()["data"]
|
||||
assert first_data["attributes"]["business_context"] == "First config"
|
||||
|
||||
# Second PATCH updates the same config (not creating a duplicate)
|
||||
payload["data"]["attributes"]["business_context"] = "Updated config"
|
||||
response = authenticated_client.patch(
|
||||
reverse("lighthouse-config"),
|
||||
data=payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
second_data = response.json()["data"]
|
||||
assert second_data["attributes"]["business_context"] == "Updated config"
|
||||
# Verify it's the same config (same ID)
|
||||
assert first_data["id"] == second_data["id"]
|
||||
|
||||
@patch("openai.OpenAI")
|
||||
def test_lighthouse_tenant_config_retrieve(
|
||||
self, mock_openai_client, authenticated_client, tenants_fixture
|
||||
):
|
||||
"""Test retrieving the singleton tenant config with proper provider and model validation"""
|
||||
|
||||
# Mock OpenAI client and models response
|
||||
mock_models_response = Mock()
|
||||
mock_models_response.data = [
|
||||
Mock(id="gpt-4o"),
|
||||
Mock(id="gpt-4o-mini"),
|
||||
Mock(id="gpt-5"),
|
||||
]
|
||||
mock_openai_client.return_value.models.list.return_value = mock_models_response
|
||||
|
||||
# Create OpenAI provider configuration
|
||||
provider_config = LighthouseProviderConfiguration.objects.create(
|
||||
tenant_id=tenants_fixture[0].id,
|
||||
provider_type="openai",
|
||||
credentials=b'{"api_key": "sk-test1234567890T3BlbkFJtest1234567890"}',
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
# Create provider models (simulating refresh)
|
||||
LighthouseProviderModels.objects.create(
|
||||
tenant_id=tenants_fixture[0].id,
|
||||
provider_configuration=provider_config,
|
||||
model_id="gpt-4o",
|
||||
default_parameters={},
|
||||
)
|
||||
LighthouseProviderModels.objects.create(
|
||||
tenant_id=tenants_fixture[0].id,
|
||||
provider_configuration=provider_config,
|
||||
model_id="gpt-4o-mini",
|
||||
default_parameters={},
|
||||
)
|
||||
|
||||
# Create tenant configuration with valid provider and model
|
||||
config = LighthouseTenantConfiguration.objects.create(
|
||||
tenant_id=tenants_fixture[0].id,
|
||||
business_context="Test context",
|
||||
default_provider="openai",
|
||||
default_models={"openai": "gpt-4o"},
|
||||
)
|
||||
|
||||
# Retrieve and verify the configuration
|
||||
response = authenticated_client.get(reverse("lighthouse-config"))
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
data = response.json()["data"]
|
||||
assert data["id"] == str(config.id)
|
||||
assert data["attributes"]["business_context"] == "Test context"
|
||||
assert data["attributes"]["default_provider"] == "openai"
|
||||
assert data["attributes"]["default_models"] == {"openai": "gpt-4o"}
|
||||
|
||||
def test_lighthouse_tenant_config_retrieve_not_found(self, authenticated_client):
|
||||
"""Test GET when config doesn't exist returns 404"""
|
||||
response = authenticated_client.get(reverse("lighthouse-config"))
|
||||
assert response.status_code == status.HTTP_404_NOT_FOUND
|
||||
assert "not found" in response.json()["errors"][0]["detail"].lower()
|
||||
|
||||
def test_lighthouse_tenant_config_partial_update(
|
||||
self, authenticated_client, tenants_fixture
|
||||
):
|
||||
"""Test updating tenant config fields"""
|
||||
from api.models import LighthouseTenantConfiguration
|
||||
|
||||
# Create config first
|
||||
config = LighthouseTenantConfiguration.objects.create(
|
||||
tenant_id=tenants_fixture[0].id,
|
||||
business_context="Original context",
|
||||
default_provider="",
|
||||
default_models={},
|
||||
)
|
||||
|
||||
# Update it
|
||||
payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-config",
|
||||
"attributes": {
|
||||
"business_context": "Updated context for cloud security",
|
||||
},
|
||||
}
|
||||
}
|
||||
response = authenticated_client.patch(
|
||||
reverse("lighthouse-config"),
|
||||
data=payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
|
||||
# Verify update
|
||||
config.refresh_from_db()
|
||||
assert config.business_context == "Updated context for cloud security"
|
||||
|
||||
def test_lighthouse_tenant_config_update_invalid_provider(
|
||||
self, authenticated_client, tenants_fixture
|
||||
):
|
||||
"""Test validation fails when default_provider is not configured and active"""
|
||||
from api.models import LighthouseTenantConfiguration
|
||||
|
||||
# Create config first
|
||||
LighthouseTenantConfiguration.objects.create(
|
||||
tenant_id=tenants_fixture[0].id,
|
||||
business_context="Test",
|
||||
)
|
||||
|
||||
# Try to set invalid provider
|
||||
payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-config",
|
||||
"attributes": {
|
||||
"default_provider": "nonexistent-provider",
|
||||
},
|
||||
}
|
||||
}
|
||||
response = authenticated_client.patch(
|
||||
reverse("lighthouse-config"),
|
||||
data=payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert response.status_code == status.HTTP_400_BAD_REQUEST
|
||||
assert "provider" in response.json()["errors"][0]["detail"].lower()
|
||||
|
||||
def test_lighthouse_tenant_config_update_invalid_json_format(
|
||||
self, authenticated_client, tenants_fixture
|
||||
):
|
||||
"""Test that invalid JSON payload is rejected"""
|
||||
from api.models import LighthouseTenantConfiguration
|
||||
|
||||
# Create config first
|
||||
LighthouseTenantConfiguration.objects.create(
|
||||
tenant_id=tenants_fixture[0].id,
|
||||
business_context="Test",
|
||||
)
|
||||
|
||||
# Send invalid JSON
|
||||
response = authenticated_client.patch(
|
||||
reverse("lighthouse-config"),
|
||||
data="invalid json",
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert response.status_code == status.HTTP_400_BAD_REQUEST
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
class TestLighthouseProviderConfigViewSet:
|
||||
"""Tests for LighthouseProviderConfiguration create validations"""
|
||||
|
||||
def test_invalid_provider_type(self, authenticated_client):
|
||||
"""Add invalid provider (testprovider) should error"""
|
||||
payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-providers",
|
||||
"attributes": {
|
||||
"provider_type": "testprovider",
|
||||
"credentials": {"api_key": "sk-testT3BlbkFJkey"},
|
||||
},
|
||||
}
|
||||
}
|
||||
resp = authenticated_client.post(
|
||||
reverse("lighthouse-providers-list"),
|
||||
data=payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert resp.status_code == status.HTTP_400_BAD_REQUEST
|
||||
|
||||
def test_openai_missing_credentials(self, authenticated_client):
|
||||
"""OpenAI provider without credentials should error"""
|
||||
payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-providers",
|
||||
"attributes": {
|
||||
"provider_type": "openai",
|
||||
},
|
||||
}
|
||||
}
|
||||
resp = authenticated_client.post(
|
||||
reverse("lighthouse-providers-list"),
|
||||
data=payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert resp.status_code == status.HTTP_400_BAD_REQUEST
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"credentials",
|
||||
[
|
||||
{}, # empty credentials
|
||||
{"token": "sk-testT3BlbkFJkey"}, # wrong key name
|
||||
{"api_key": "ks-invalid-format"}, # wrong format
|
||||
],
|
||||
)
|
||||
def test_openai_invalid_credentials(self, authenticated_client, credentials):
|
||||
"""OpenAI provider with invalid credentials should error"""
|
||||
payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-providers",
|
||||
"attributes": {
|
||||
"provider_type": "openai",
|
||||
"credentials": credentials,
|
||||
},
|
||||
}
|
||||
}
|
||||
resp = authenticated_client.post(
|
||||
reverse("lighthouse-providers-list"),
|
||||
data=payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert resp.status_code == status.HTTP_400_BAD_REQUEST
|
||||
|
||||
def test_openai_valid_credentials_success(self, authenticated_client):
|
||||
"""OpenAI provider with valid sk-xxx format should succeed"""
|
||||
valid_key = "sk-abc123T3BlbkFJxyz456"
|
||||
payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-providers",
|
||||
"attributes": {
|
||||
"provider_type": "openai",
|
||||
"credentials": {"api_key": valid_key},
|
||||
},
|
||||
}
|
||||
}
|
||||
resp = authenticated_client.post(
|
||||
reverse("lighthouse-providers-list"),
|
||||
data=payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert resp.status_code == status.HTTP_201_CREATED
|
||||
data = resp.json()["data"]
|
||||
|
||||
masked_creds = data["attributes"].get("credentials")
|
||||
assert masked_creds is not None
|
||||
assert "api_key" in masked_creds
|
||||
assert masked_creds["api_key"] == ("*" * len(valid_key))
|
||||
|
||||
def test_openai_provider_duplicate_per_tenant(self, authenticated_client):
|
||||
"""If an OpenAI provider exists for tenant, creating again should error"""
|
||||
valid_key = "sk-dup123T3BlbkFJdup456"
|
||||
payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-providers",
|
||||
"attributes": {
|
||||
"provider_type": "openai",
|
||||
"credentials": {"api_key": valid_key},
|
||||
},
|
||||
}
|
||||
}
|
||||
# First creation succeeds
|
||||
resp1 = authenticated_client.post(
|
||||
reverse("lighthouse-providers-list"),
|
||||
data=payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert resp1.status_code == status.HTTP_201_CREATED
|
||||
|
||||
# Second creation should fail with validation error
|
||||
resp2 = authenticated_client.post(
|
||||
reverse("lighthouse-providers-list"),
|
||||
data=payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert resp2.status_code == status.HTTP_400_BAD_REQUEST
|
||||
assert "already exists" in str(resp2.json()).lower()
|
||||
|
||||
def test_openai_patch_base_url_and_is_active(self, authenticated_client):
|
||||
"""After creating, should be able to patch base_url and is_active"""
|
||||
valid_key = "sk-patch123T3BlbkFJpatch456"
|
||||
create_payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-providers",
|
||||
"attributes": {
|
||||
"provider_type": "openai",
|
||||
"credentials": {"api_key": valid_key},
|
||||
},
|
||||
}
|
||||
}
|
||||
create_resp = authenticated_client.post(
|
||||
reverse("lighthouse-providers-list"),
|
||||
data=create_payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert create_resp.status_code == status.HTTP_201_CREATED
|
||||
provider_id = create_resp.json()["data"]["id"]
|
||||
|
||||
patch_payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-providers",
|
||||
"id": provider_id,
|
||||
"attributes": {
|
||||
"base_url": "https://api.example.com/v1",
|
||||
"is_active": False,
|
||||
},
|
||||
}
|
||||
}
|
||||
patch_resp = authenticated_client.patch(
|
||||
reverse("lighthouse-providers-detail", kwargs={"pk": provider_id}),
|
||||
data=patch_payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert patch_resp.status_code == status.HTTP_200_OK
|
||||
updated = patch_resp.json()["data"]["attributes"]
|
||||
assert updated["base_url"] == "https://api.example.com/v1"
|
||||
assert updated["is_active"] is False
|
||||
|
||||
def test_openai_patch_invalid_credentials(self, authenticated_client):
|
||||
"""PATCH with invalid credentials.api_key should error (400)"""
|
||||
valid_key = "sk-ok123T3BlbkFJok456"
|
||||
create_payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-providers",
|
||||
"attributes": {
|
||||
"provider_type": "openai",
|
||||
"credentials": {"api_key": valid_key},
|
||||
},
|
||||
}
|
||||
}
|
||||
create_resp = authenticated_client.post(
|
||||
reverse("lighthouse-providers-list"),
|
||||
data=create_payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert create_resp.status_code == status.HTTP_201_CREATED
|
||||
provider_id = create_resp.json()["data"]["id"]
|
||||
|
||||
# Try patch with invalid api_key format
|
||||
patch_payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-providers",
|
||||
"id": provider_id,
|
||||
"attributes": {
|
||||
"credentials": {"api_key": "ks-invalid-format"},
|
||||
},
|
||||
}
|
||||
}
|
||||
patch_resp = authenticated_client.patch(
|
||||
reverse("lighthouse-providers-detail", kwargs={"pk": provider_id}),
|
||||
data=patch_payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert patch_resp.status_code == status.HTTP_400_BAD_REQUEST
|
||||
|
||||
def test_openai_get_masking_and_fields_filter(self, authenticated_client):
|
||||
valid_key = "sk-get123T3BlbkFJget456"
|
||||
create_payload = {
|
||||
"data": {
|
||||
"type": "lighthouse-providers",
|
||||
"attributes": {
|
||||
"provider_type": "openai",
|
||||
"credentials": {"api_key": valid_key},
|
||||
},
|
||||
}
|
||||
}
|
||||
create_resp = authenticated_client.post(
|
||||
reverse("lighthouse-providers-list"),
|
||||
data=create_payload,
|
||||
content_type=API_JSON_CONTENT_TYPE,
|
||||
)
|
||||
assert create_resp.status_code == status.HTTP_201_CREATED
|
||||
provider_id = create_resp.json()["data"]["id"]
|
||||
|
||||
# Default GET should return masked credentials
|
||||
get_resp = authenticated_client.get(
|
||||
reverse("lighthouse-providers-detail", kwargs={"pk": provider_id})
|
||||
)
|
||||
assert get_resp.status_code == status.HTTP_200_OK
|
||||
masked = get_resp.json()["data"]["attributes"]["credentials"]["api_key"]
|
||||
assert masked == ("*" * len(valid_key))
|
||||
|
||||
# Fields filter should return decrypted credentials structure
|
||||
get_full = authenticated_client.get(
|
||||
reverse("lighthouse-providers-detail", kwargs={"pk": provider_id})
|
||||
+ "?fields[lighthouse-providers]=credentials"
|
||||
)
|
||||
assert get_full.status_code == status.HTTP_200_OK
|
||||
creds = get_full.json()["data"]["attributes"]["credentials"]
|
||||
assert creds["api_key"] == valid_key
|
||||
|
||||
def test_delete_provider_updates_tenant_defaults(
|
||||
self, authenticated_client, tenants_fixture
|
||||
):
|
||||
"""Deleting a provider config should clear tenant default_provider and its default_model entry."""
|
||||
|
||||
tenant = tenants_fixture[0]
|
||||
|
||||
# Create provider configuration to delete
|
||||
provider = LighthouseProviderConfiguration.objects.create(
|
||||
tenant_id=tenant.id,
|
||||
provider_type="openai",
|
||||
credentials=b'{"api_key":"sk-test123T3BlbkFJ"}',
|
||||
is_active=True,
|
||||
)
|
||||
|
||||
# Seed tenant defaults referencing the provider we will delete
|
||||
cfg = LighthouseTenantConfiguration.objects.create(
|
||||
tenant_id=tenant.id,
|
||||
business_context="Test",
|
||||
default_provider="openai",
|
||||
default_models={"openai": "gpt-4o", "other": "model-x"},
|
||||
)
|
||||
|
||||
# Delete via API and validate response
|
||||
url = reverse("lighthouse-providers-detail", kwargs={"pk": str(provider.id)})
|
||||
resp = authenticated_client.delete(url)
|
||||
assert resp.status_code in (
|
||||
status.HTTP_204_NO_CONTENT,
|
||||
status.HTTP_200_OK,
|
||||
)
|
||||
|
||||
# Tenant defaults should be updated
|
||||
cfg.refresh_from_db()
|
||||
assert cfg.default_provider == ""
|
||||
assert "openai" not in cfg.default_models
|
||||
|
||||
# Unrelated entries should remain untouched
|
||||
assert cfg.default_models.get("other") == "model-x"
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
import re
|
||||
|
||||
from rest_framework_json_api import serializers
|
||||
|
||||
|
||||
class OpenAICredentialsSerializer(serializers.Serializer):
|
||||
api_key = serializers.CharField()
|
||||
|
||||
def validate_api_key(self, value: str) -> str:
|
||||
pattern = r"^sk-[\w-]+$"
|
||||
if not re.match(pattern, value or ""):
|
||||
raise serializers.ValidationError("Invalid OpenAI API key format.")
|
||||
return value
|
||||
@@ -6,8 +6,10 @@ from django.conf import settings
|
||||
from django.contrib.auth import authenticate
|
||||
from django.contrib.auth.models import update_last_login
|
||||
from django.contrib.auth.password_validation import validate_password
|
||||
from django.db import IntegrityError
|
||||
from drf_spectacular.utils import extend_schema_field
|
||||
from jwt.exceptions import InvalidKeyError
|
||||
from rest_framework.reverse import reverse
|
||||
from rest_framework.validators import UniqueTogetherValidator
|
||||
from rest_framework_json_api import serializers
|
||||
from rest_framework_json_api.relations import SerializerMethodResourceRelatedField
|
||||
@@ -25,6 +27,9 @@ from api.models import (
|
||||
Invitation,
|
||||
InvitationRoleRelationship,
|
||||
LighthouseConfiguration,
|
||||
LighthouseProviderConfiguration,
|
||||
LighthouseProviderModels,
|
||||
LighthouseTenantConfiguration,
|
||||
Membership,
|
||||
Processor,
|
||||
Provider,
|
||||
@@ -54,6 +59,7 @@ from api.v1.serializer_utils.integrations import (
|
||||
S3ConfigSerializer,
|
||||
SecurityHubConfigSerializer,
|
||||
)
|
||||
from api.v1.serializer_utils.lighthouse import OpenAICredentialsSerializer
|
||||
from api.v1.serializer_utils.processors import ProcessorConfigField
|
||||
from api.v1.serializer_utils.providers import ProviderSecretField
|
||||
from prowler.lib.mutelist.mutelist import Mutelist
|
||||
@@ -2750,6 +2756,16 @@ class LighthouseConfigCreateSerializer(RLSSerializer, BaseWriteSerializer):
|
||||
"updated_at": {"read_only": True},
|
||||
}
|
||||
|
||||
def validate_temperature(self, value):
|
||||
if not 0 <= value <= 1:
|
||||
raise ValidationError("Temperature must be between 0 and 1.")
|
||||
return value
|
||||
|
||||
def validate_max_tokens(self, value):
|
||||
if not 500 <= value <= 5000:
|
||||
raise ValidationError("Max tokens must be between 500 and 5000.")
|
||||
return value
|
||||
|
||||
def validate(self, attrs):
|
||||
tenant_id = self.context.get("request").tenant_id
|
||||
if LighthouseConfiguration.objects.filter(tenant_id=tenant_id).exists():
|
||||
@@ -2758,6 +2774,11 @@ class LighthouseConfigCreateSerializer(RLSSerializer, BaseWriteSerializer):
|
||||
"tenant_id": "Lighthouse configuration already exists for this tenant."
|
||||
}
|
||||
)
|
||||
api_key = attrs.get("api_key")
|
||||
if api_key is not None:
|
||||
OpenAICredentialsSerializer(data={"api_key": api_key}).is_valid(
|
||||
raise_exception=True
|
||||
)
|
||||
return super().validate(attrs)
|
||||
|
||||
def create(self, validated_data):
|
||||
@@ -2802,6 +2823,24 @@ class LighthouseConfigUpdateSerializer(BaseWriteSerializer):
|
||||
"max_tokens": {"required": False},
|
||||
}
|
||||
|
||||
def validate_temperature(self, value):
|
||||
if not 0 <= value <= 1:
|
||||
raise ValidationError("Temperature must be between 0 and 1.")
|
||||
return value
|
||||
|
||||
def validate_max_tokens(self, value):
|
||||
if not 500 <= value <= 5000:
|
||||
raise ValidationError("Max tokens must be between 500 and 5000.")
|
||||
return value
|
||||
|
||||
def validate(self, attrs):
|
||||
api_key = attrs.get("api_key", None)
|
||||
if api_key is not None:
|
||||
OpenAICredentialsSerializer(data={"api_key": api_key}).is_valid(
|
||||
raise_exception=True
|
||||
)
|
||||
return super().validate(attrs)
|
||||
|
||||
def update(self, instance, validated_data):
|
||||
api_key = validated_data.pop("api_key", None)
|
||||
instance = super().update(instance, validated_data)
|
||||
@@ -2931,3 +2970,352 @@ class TenantApiKeyUpdateSerializer(RLSSerializer, BaseWriteSerializer):
|
||||
):
|
||||
raise ValidationError("An API key with this name already exists.")
|
||||
return value
|
||||
|
||||
|
||||
# Lighthouse: Provider configurations
|
||||
|
||||
|
||||
class LighthouseProviderConfigSerializer(RLSSerializer):
|
||||
"""
|
||||
Read serializer for LighthouseProviderConfiguration.
|
||||
"""
|
||||
|
||||
# Decrypted credentials are only returned in to_representation when requested
|
||||
credentials = serializers.JSONField(required=False, read_only=True)
|
||||
|
||||
class Meta:
|
||||
model = LighthouseProviderConfiguration
|
||||
fields = [
|
||||
"id",
|
||||
"inserted_at",
|
||||
"updated_at",
|
||||
"provider_type",
|
||||
"base_url",
|
||||
"is_active",
|
||||
"credentials",
|
||||
"url",
|
||||
]
|
||||
extra_kwargs = {
|
||||
"id": {"read_only": True},
|
||||
"inserted_at": {"read_only": True},
|
||||
"updated_at": {"read_only": True},
|
||||
"is_active": {"read_only": True},
|
||||
"url": {"read_only": True, "view_name": "lighthouse-providers-detail"},
|
||||
}
|
||||
|
||||
class JSONAPIMeta:
|
||||
resource_name = "lighthouse-providers"
|
||||
|
||||
def to_representation(self, instance):
|
||||
data = super().to_representation(instance)
|
||||
# Support JSON:API fields filter: fields[lighthouse-providers]=credentials,base_url
|
||||
fields_param = self.context.get("request", None) and self.context[
|
||||
"request"
|
||||
].query_params.get("fields[lighthouse-providers]", "")
|
||||
|
||||
creds = instance.credentials_decoded
|
||||
|
||||
requested_fields = (
|
||||
[f.strip() for f in fields_param.split(",")] if fields_param else []
|
||||
)
|
||||
|
||||
if "credentials" in requested_fields:
|
||||
# Return full decrypted credentials JSON
|
||||
data["credentials"] = creds
|
||||
else:
|
||||
# Return masked credentials by default
|
||||
def mask_value(value):
|
||||
if isinstance(value, str):
|
||||
return "*" * len(value)
|
||||
if isinstance(value, dict):
|
||||
return {k: mask_value(v) for k, v in value.items()}
|
||||
if isinstance(value, list):
|
||||
return [mask_value(v) for v in value]
|
||||
return value
|
||||
|
||||
# Always return masked credentials, even if creds is None
|
||||
if creds is not None:
|
||||
data["credentials"] = mask_value(creds)
|
||||
else:
|
||||
# If credentials_decoded returns None, return None for credentials field
|
||||
data["credentials"] = None
|
||||
|
||||
return data
|
||||
|
||||
|
||||
class LighthouseProviderConfigCreateSerializer(RLSSerializer, BaseWriteSerializer):
|
||||
"""
|
||||
Create serializer for LighthouseProviderConfiguration.
|
||||
Accepts credentials as JSON; stored encrypted via credentials_decoded.
|
||||
"""
|
||||
|
||||
credentials = serializers.JSONField(write_only=True, required=True)
|
||||
|
||||
class Meta:
|
||||
model = LighthouseProviderConfiguration
|
||||
fields = [
|
||||
"provider_type",
|
||||
"base_url",
|
||||
"credentials",
|
||||
"is_active",
|
||||
]
|
||||
extra_kwargs = {
|
||||
"is_active": {"required": False},
|
||||
"base_url": {"required": False, "allow_null": True},
|
||||
}
|
||||
|
||||
def create(self, validated_data):
|
||||
credentials = validated_data.pop("credentials")
|
||||
|
||||
instance = LighthouseProviderConfiguration(**validated_data)
|
||||
instance.tenant_id = self.context.get("tenant_id")
|
||||
instance.credentials_decoded = credentials
|
||||
|
||||
try:
|
||||
instance.save()
|
||||
return instance
|
||||
except IntegrityError:
|
||||
raise ValidationError(
|
||||
{
|
||||
"provider_type": "Configuration for this provider already exists for the tenant."
|
||||
}
|
||||
)
|
||||
|
||||
def validate(self, attrs):
|
||||
provider_type = attrs.get("provider_type")
|
||||
credentials = attrs.get("credentials") or {}
|
||||
|
||||
if provider_type == LighthouseProviderConfiguration.LLMProviderChoices.OPENAI:
|
||||
try:
|
||||
OpenAICredentialsSerializer(data=credentials).is_valid(
|
||||
raise_exception=True
|
||||
)
|
||||
except ValidationError as e:
|
||||
details = e.detail.copy()
|
||||
for key, value in details.items():
|
||||
e.detail[f"credentials/{key}"] = value
|
||||
del e.detail[key]
|
||||
raise e
|
||||
|
||||
return super().validate(attrs)
|
||||
|
||||
|
||||
class LighthouseProviderConfigUpdateSerializer(BaseWriteSerializer):
|
||||
"""
|
||||
Update serializer for LighthouseProviderConfiguration.
|
||||
"""
|
||||
|
||||
credentials = serializers.JSONField(write_only=True, required=False)
|
||||
|
||||
class Meta:
|
||||
model = LighthouseProviderConfiguration
|
||||
fields = [
|
||||
"id",
|
||||
"provider_type",
|
||||
"base_url",
|
||||
"credentials",
|
||||
"is_active",
|
||||
]
|
||||
extra_kwargs = {
|
||||
"id": {"read_only": True},
|
||||
"provider_type": {"read_only": True},
|
||||
"base_url": {"required": False, "allow_null": True},
|
||||
"is_active": {"required": False},
|
||||
}
|
||||
|
||||
def update(self, instance, validated_data):
|
||||
credentials = validated_data.pop("credentials", None)
|
||||
|
||||
for attr, value in validated_data.items():
|
||||
setattr(instance, attr, value)
|
||||
|
||||
if credentials is not None:
|
||||
instance.credentials_decoded = credentials
|
||||
|
||||
instance.save()
|
||||
return instance
|
||||
|
||||
def validate(self, attrs):
|
||||
provider_type = getattr(self.instance, "provider_type", None)
|
||||
credentials = attrs.get("credentials", None)
|
||||
|
||||
if (
|
||||
credentials is not None
|
||||
and provider_type
|
||||
== LighthouseProviderConfiguration.LLMProviderChoices.OPENAI
|
||||
):
|
||||
try:
|
||||
OpenAICredentialsSerializer(data=credentials).is_valid(
|
||||
raise_exception=True
|
||||
)
|
||||
except ValidationError as e:
|
||||
details = e.detail.copy()
|
||||
for key, value in details.items():
|
||||
e.detail[f"credentials/{key}"] = value
|
||||
del e.detail[key]
|
||||
raise e
|
||||
|
||||
return super().validate(attrs)
|
||||
|
||||
|
||||
# Lighthouse: Tenant configuration
|
||||
|
||||
|
||||
class LighthouseTenantConfigSerializer(RLSSerializer):
|
||||
"""
|
||||
Read serializer for LighthouseTenantConfiguration.
|
||||
"""
|
||||
|
||||
# Build singleton URL without pk
|
||||
url = serializers.SerializerMethodField()
|
||||
|
||||
def get_url(self, obj):
|
||||
request = self.context.get("request")
|
||||
return reverse("lighthouse-config", request=request)
|
||||
|
||||
class Meta:
|
||||
model = LighthouseTenantConfiguration
|
||||
fields = [
|
||||
"id",
|
||||
"inserted_at",
|
||||
"updated_at",
|
||||
"business_context",
|
||||
"default_provider",
|
||||
"default_models",
|
||||
"url",
|
||||
]
|
||||
extra_kwargs = {
|
||||
"id": {"read_only": True},
|
||||
"inserted_at": {"read_only": True},
|
||||
"updated_at": {"read_only": True},
|
||||
"url": {"read_only": True},
|
||||
}
|
||||
|
||||
|
||||
class LighthouseTenantConfigUpdateSerializer(BaseWriteSerializer):
|
||||
class Meta:
|
||||
model = LighthouseTenantConfiguration
|
||||
fields = [
|
||||
"id",
|
||||
"business_context",
|
||||
"default_provider",
|
||||
"default_models",
|
||||
]
|
||||
extra_kwargs = {
|
||||
"id": {"read_only": True},
|
||||
}
|
||||
|
||||
def validate(self, attrs):
|
||||
request = self.context.get("request")
|
||||
tenant_id = self.context.get("tenant_id") or (
|
||||
getattr(request, "tenant_id", None) if request else None
|
||||
)
|
||||
|
||||
default_provider = attrs.get(
|
||||
"default_provider", getattr(self.instance, "default_provider", "")
|
||||
)
|
||||
default_models = attrs.get(
|
||||
"default_models", getattr(self.instance, "default_models", {})
|
||||
)
|
||||
|
||||
if default_provider:
|
||||
supported = set(LighthouseProviderConfiguration.LLMProviderChoices.values)
|
||||
if default_provider not in supported:
|
||||
raise ValidationError(
|
||||
{"default_provider": f"Unsupported provider '{default_provider}'."}
|
||||
)
|
||||
if not LighthouseProviderConfiguration.objects.filter(
|
||||
tenant_id=tenant_id, provider_type=default_provider, is_active=True
|
||||
).exists():
|
||||
raise ValidationError(
|
||||
{
|
||||
"default_provider": f"No active configuration found for '{default_provider}'."
|
||||
}
|
||||
)
|
||||
|
||||
if default_models is not None and not isinstance(default_models, dict):
|
||||
raise ValidationError(
|
||||
{"default_models": "Must be an object mapping provider -> model_id."}
|
||||
)
|
||||
|
||||
for provider_type, model_id in (default_models or {}).items():
|
||||
provider_cfg = LighthouseProviderConfiguration.objects.filter(
|
||||
tenant_id=tenant_id, provider_type=provider_type, is_active=True
|
||||
).first()
|
||||
if not provider_cfg:
|
||||
raise ValidationError(
|
||||
{
|
||||
"default_models": f"No active configuration for provider '{provider_type}'."
|
||||
}
|
||||
)
|
||||
if not LighthouseProviderModels.objects.filter(
|
||||
tenant_id=tenant_id,
|
||||
provider_configuration=provider_cfg,
|
||||
model_id=model_id,
|
||||
).exists():
|
||||
raise ValidationError(
|
||||
{
|
||||
"default_models": f"Invalid model '{model_id}' for provider '{provider_type}'."
|
||||
}
|
||||
)
|
||||
|
||||
return super().validate(attrs)
|
||||
|
||||
|
||||
# Lighthouse: Provider models
|
||||
|
||||
|
||||
class LighthouseProviderModelsSerializer(RLSSerializer):
|
||||
"""
|
||||
Read serializer for LighthouseProviderModels.
|
||||
"""
|
||||
|
||||
provider_configuration = serializers.ResourceRelatedField(read_only=True)
|
||||
|
||||
class Meta:
|
||||
model = LighthouseProviderModels
|
||||
fields = [
|
||||
"id",
|
||||
"inserted_at",
|
||||
"updated_at",
|
||||
"provider_configuration",
|
||||
"model_id",
|
||||
"model_name",
|
||||
"default_parameters",
|
||||
"url",
|
||||
]
|
||||
extra_kwargs = {
|
||||
"id": {"read_only": True},
|
||||
"inserted_at": {"read_only": True},
|
||||
"updated_at": {"read_only": True},
|
||||
"url": {"read_only": True, "view_name": "lighthouse-models-detail"},
|
||||
}
|
||||
|
||||
|
||||
class LighthouseProviderModelsCreateSerializer(RLSSerializer, BaseWriteSerializer):
|
||||
provider_configuration = serializers.ResourceRelatedField(
|
||||
queryset=LighthouseProviderConfiguration.objects.all()
|
||||
)
|
||||
|
||||
class Meta:
|
||||
model = LighthouseProviderModels
|
||||
fields = [
|
||||
"provider_configuration",
|
||||
"model_id",
|
||||
"default_parameters",
|
||||
]
|
||||
extra_kwargs = {
|
||||
"default_parameters": {"required": False},
|
||||
}
|
||||
|
||||
|
||||
class LighthouseProviderModelsUpdateSerializer(BaseWriteSerializer):
|
||||
class Meta:
|
||||
model = LighthouseProviderModels
|
||||
fields = [
|
||||
"id",
|
||||
"default_parameters",
|
||||
]
|
||||
extra_kwargs = {
|
||||
"id": {"read_only": True},
|
||||
}
|
||||
|
||||
@@ -17,6 +17,9 @@ from api.v1.views import (
|
||||
InvitationAcceptViewSet,
|
||||
InvitationViewSet,
|
||||
LighthouseConfigViewSet,
|
||||
LighthouseProviderConfigViewSet,
|
||||
LighthouseProviderModelsViewSet,
|
||||
LighthouseTenantConfigViewSet,
|
||||
MembershipViewSet,
|
||||
OverviewViewSet,
|
||||
ProcessorViewSet,
|
||||
@@ -34,12 +37,12 @@ from api.v1.views import (
|
||||
ScheduleViewSet,
|
||||
SchemaView,
|
||||
TaskViewSet,
|
||||
TenantApiKeyViewSet,
|
||||
TenantFinishACSView,
|
||||
TenantMembersViewSet,
|
||||
TenantViewSet,
|
||||
UserRoleRelationshipView,
|
||||
UserViewSet,
|
||||
TenantApiKeyViewSet,
|
||||
)
|
||||
|
||||
router = routers.DefaultRouter(trailing_slash=False)
|
||||
@@ -67,6 +70,16 @@ router.register(
|
||||
basename="lighthouseconfiguration",
|
||||
)
|
||||
router.register(r"api-keys", TenantApiKeyViewSet, basename="api-key")
|
||||
router.register(
|
||||
r"lighthouse/providers",
|
||||
LighthouseProviderConfigViewSet,
|
||||
basename="lighthouse-providers",
|
||||
)
|
||||
router.register(
|
||||
r"lighthouse/models",
|
||||
LighthouseProviderModelsViewSet,
|
||||
basename="lighthouse-models",
|
||||
)
|
||||
|
||||
tenants_router = routers.NestedSimpleRouter(router, r"tenants", lookup="tenant")
|
||||
tenants_router.register(
|
||||
@@ -137,6 +150,14 @@ urlpatterns = [
|
||||
),
|
||||
name="provider_group-providers-relationship",
|
||||
),
|
||||
# Lighthouse tenant config as singleton endpoint
|
||||
path(
|
||||
"lighthouse/configuration",
|
||||
LighthouseTenantConfigViewSet.as_view(
|
||||
{"get": "list", "patch": "partial_update"}
|
||||
),
|
||||
name="lighthouse-config",
|
||||
),
|
||||
# API endpoint to start SAML SSO flow
|
||||
path(
|
||||
"auth/saml/initiate/", SAMLInitiateAPIView.as_view(), name="api_saml_initiate"
|
||||
|
||||
+298
-12
@@ -1,5 +1,6 @@
|
||||
import fnmatch
|
||||
import glob
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime, timedelta, timezone
|
||||
@@ -60,11 +61,13 @@ from tasks.tasks import (
|
||||
backfill_scan_resource_summaries_task,
|
||||
check_integration_connection_task,
|
||||
check_lighthouse_connection_task,
|
||||
check_lighthouse_provider_connection_task,
|
||||
check_provider_connection_task,
|
||||
delete_provider_task,
|
||||
delete_tenant_task,
|
||||
jira_integration_task,
|
||||
perform_scan_task,
|
||||
refresh_lighthouse_provider_models_task,
|
||||
)
|
||||
|
||||
from api.base_views import BaseRLSViewSet, BaseTenantViewset, BaseUserViewset
|
||||
@@ -84,6 +87,8 @@ from api.filters import (
|
||||
InvitationFilter,
|
||||
LatestFindingFilter,
|
||||
LatestResourceFilter,
|
||||
LighthouseProviderConfigFilter,
|
||||
LighthouseProviderModelsFilter,
|
||||
MembershipFilter,
|
||||
ProcessorFilter,
|
||||
ProviderFilter,
|
||||
@@ -106,6 +111,9 @@ from api.models import (
|
||||
Integration,
|
||||
Invitation,
|
||||
LighthouseConfiguration,
|
||||
LighthouseProviderConfiguration,
|
||||
LighthouseProviderModels,
|
||||
LighthouseTenantConfiguration,
|
||||
Membership,
|
||||
Processor,
|
||||
Provider,
|
||||
@@ -160,6 +168,12 @@ from api.v1.serializers import (
|
||||
LighthouseConfigCreateSerializer,
|
||||
LighthouseConfigSerializer,
|
||||
LighthouseConfigUpdateSerializer,
|
||||
LighthouseProviderConfigCreateSerializer,
|
||||
LighthouseProviderConfigSerializer,
|
||||
LighthouseProviderConfigUpdateSerializer,
|
||||
LighthouseProviderModelsSerializer,
|
||||
LighthouseTenantConfigSerializer,
|
||||
LighthouseTenantConfigUpdateSerializer,
|
||||
MembershipSerializer,
|
||||
OverviewFindingSerializer,
|
||||
OverviewProviderSerializer,
|
||||
@@ -307,7 +321,7 @@ class SchemaView(SpectacularAPIView):
|
||||
|
||||
def get(self, request, *args, **kwargs):
|
||||
spectacular_settings.TITLE = "Prowler API"
|
||||
spectacular_settings.VERSION = "1.14.1"
|
||||
spectacular_settings.VERSION = "1.15.0"
|
||||
spectacular_settings.DESCRIPTION = (
|
||||
"Prowler API specification.\n\nThis file is auto-generated."
|
||||
)
|
||||
@@ -3776,7 +3790,10 @@ class OverviewViewSet(BaseRLSViewSet):
|
||||
|
||||
findings_aggregated = (
|
||||
queryset.filter(scan_id__in=latest_scan_ids)
|
||||
.values(provider=F("scan__provider__provider"))
|
||||
.values(
|
||||
"scan__provider_id",
|
||||
provider=F("scan__provider__provider"),
|
||||
)
|
||||
.annotate(
|
||||
findings_passed=Coalesce(Sum("_pass"), 0),
|
||||
findings_failed=Coalesce(Sum("fail"), 0),
|
||||
@@ -3785,16 +3802,13 @@ class OverviewViewSet(BaseRLSViewSet):
|
||||
)
|
||||
)
|
||||
|
||||
resources_queryset = Resource.all_objects.filter(tenant_id=tenant_id)
|
||||
if hasattr(self, "allowed_providers"):
|
||||
resources_queryset = resources_queryset.filter(
|
||||
provider__in=self.allowed_providers
|
||||
)
|
||||
resources_aggregated = resources_queryset.values(
|
||||
provider_type=F("provider__provider")
|
||||
).annotate(total_resources=Count("id"))
|
||||
resources_aggregated = (
|
||||
Resource.all_objects.filter(tenant_id=tenant_id)
|
||||
.values("provider_id")
|
||||
.annotate(total_resources=Count("id"))
|
||||
)
|
||||
resource_map = {
|
||||
row["provider_type"]: row["total_resources"] for row in resources_aggregated
|
||||
row["provider_id"]: row["total_resources"] for row in resources_aggregated
|
||||
}
|
||||
|
||||
overview = []
|
||||
@@ -3802,7 +3816,7 @@ class OverviewViewSet(BaseRLSViewSet):
|
||||
overview.append(
|
||||
{
|
||||
"provider": row["provider"],
|
||||
"total_resources": resource_map.get(row["provider"], 0),
|
||||
"total_resources": resource_map.get(row["scan__provider_id"], 0),
|
||||
"total_findings": row["total_findings"],
|
||||
"findings_passed": row["findings_passed"],
|
||||
"findings_failed": row["findings_failed"],
|
||||
@@ -4177,21 +4191,25 @@ class IntegrationJiraViewSet(BaseRLSViewSet):
|
||||
tags=["Lighthouse AI"],
|
||||
summary="List all Lighthouse AI configurations",
|
||||
description="Retrieve a list of all Lighthouse AI configurations.",
|
||||
deprecated=True,
|
||||
),
|
||||
create=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Create a new Lighthouse AI configuration",
|
||||
description="Create a new Lighthouse AI configuration with the specified details.",
|
||||
deprecated=True,
|
||||
),
|
||||
partial_update=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Partially update a Lighthouse AI configuration",
|
||||
description="Update certain fields of an existing Lighthouse AI configuration.",
|
||||
deprecated=True,
|
||||
),
|
||||
destroy=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Delete a Lighthouse AI configuration",
|
||||
description="Remove a Lighthouse AI configuration by its ID.",
|
||||
deprecated=True,
|
||||
),
|
||||
connection=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
@@ -4199,6 +4217,7 @@ class IntegrationJiraViewSet(BaseRLSViewSet):
|
||||
description="Verify the connection to the OpenAI API for a specific Lighthouse AI configuration.",
|
||||
request=None,
|
||||
responses={202: OpenApiResponse(response=TaskSerializer)},
|
||||
deprecated=True,
|
||||
),
|
||||
)
|
||||
class LighthouseConfigViewSet(BaseRLSViewSet):
|
||||
@@ -4249,6 +4268,273 @@ class LighthouseConfigViewSet(BaseRLSViewSet):
|
||||
)
|
||||
|
||||
|
||||
@extend_schema_view(
|
||||
list=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="List all LLM provider configs",
|
||||
description="Retrieve all LLM provider configurations for the current tenant",
|
||||
),
|
||||
retrieve=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Retrieve LLM provider config",
|
||||
description="Get details for a specific provider configuration in the current tenant.",
|
||||
),
|
||||
create=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Create LLM provider config",
|
||||
description="Create a per-tenant configuration for an LLM provider. Only one configuration per provider type is allowed per tenant.",
|
||||
),
|
||||
partial_update=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Update LLM provider config",
|
||||
description="Partially update a provider configuration (e.g., base_url, is_active).",
|
||||
),
|
||||
destroy=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Delete LLM provider config",
|
||||
description="Delete a provider configuration. Any tenant defaults that reference this provider are cleared during deletion.",
|
||||
),
|
||||
)
|
||||
class LighthouseProviderConfigViewSet(BaseRLSViewSet):
|
||||
queryset = LighthouseProviderConfiguration.objects.all()
|
||||
serializer_class = LighthouseProviderConfigSerializer
|
||||
http_method_names = ["get", "post", "patch", "delete"]
|
||||
filterset_class = LighthouseProviderConfigFilter
|
||||
|
||||
def get_queryset(self):
|
||||
if getattr(self, "swagger_fake_view", False):
|
||||
return LighthouseProviderConfiguration.objects.none()
|
||||
return LighthouseProviderConfiguration.objects.filter(
|
||||
tenant_id=self.request.tenant_id
|
||||
)
|
||||
|
||||
def get_serializer_class(self):
|
||||
if self.action == "create":
|
||||
return LighthouseProviderConfigCreateSerializer
|
||||
elif self.action == "partial_update":
|
||||
return LighthouseProviderConfigUpdateSerializer
|
||||
elif self.action in ["connection", "refresh_models"]:
|
||||
return TaskSerializer
|
||||
return super().get_serializer_class()
|
||||
|
||||
def create(self, request, *args, **kwargs):
|
||||
serializer = self.get_serializer(data=request.data)
|
||||
serializer.is_valid(raise_exception=True)
|
||||
instance = serializer.save()
|
||||
|
||||
read_serializer = LighthouseProviderConfigSerializer(
|
||||
instance, context=self.get_serializer_context()
|
||||
)
|
||||
headers = self.get_success_headers(read_serializer.data)
|
||||
return Response(
|
||||
data=read_serializer.data,
|
||||
status=status.HTTP_201_CREATED,
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
def partial_update(self, request, *args, **kwargs):
|
||||
instance = self.get_object()
|
||||
serializer = self.get_serializer(
|
||||
instance,
|
||||
data=request.data,
|
||||
partial=True,
|
||||
context=self.get_serializer_context(),
|
||||
)
|
||||
serializer.is_valid(raise_exception=True)
|
||||
serializer.save()
|
||||
read_serializer = LighthouseProviderConfigSerializer(
|
||||
instance, context=self.get_serializer_context()
|
||||
)
|
||||
return Response(data=read_serializer.data, status=status.HTTP_200_OK)
|
||||
|
||||
@extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Check LLM provider connection",
|
||||
description="Validate provider credentials asynchronously and toggle is_active.",
|
||||
request=None,
|
||||
responses={202: OpenApiResponse(response=TaskSerializer)},
|
||||
)
|
||||
@action(detail=True, methods=["post"], url_name="connection")
|
||||
def connection(self, request, pk=None):
|
||||
instance = self.get_object()
|
||||
if (
|
||||
instance.provider_type
|
||||
!= LighthouseProviderConfiguration.LLMProviderChoices.OPENAI
|
||||
):
|
||||
return Response(
|
||||
data={
|
||||
"errors": [{"detail": "Only 'openai' provider supported in MVP"}]
|
||||
},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
|
||||
with transaction.atomic():
|
||||
task = check_lighthouse_provider_connection_task.delay(
|
||||
provider_config_id=str(instance.id), tenant_id=self.request.tenant_id
|
||||
)
|
||||
|
||||
prowler_task = Task.objects.get(id=task.id)
|
||||
serializer = TaskSerializer(prowler_task)
|
||||
return Response(
|
||||
data=serializer.data,
|
||||
status=status.HTTP_202_ACCEPTED,
|
||||
headers={
|
||||
"Content-Location": reverse(
|
||||
"task-detail", kwargs={"pk": prowler_task.id}
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
@extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Refresh LLM models catalog",
|
||||
description="Fetch available models for this provider configuration and upsert into catalog.",
|
||||
request=None,
|
||||
responses={202: OpenApiResponse(response=TaskSerializer)},
|
||||
)
|
||||
@action(
|
||||
detail=True,
|
||||
methods=["post"],
|
||||
url_path="refresh-models",
|
||||
url_name="refresh-models",
|
||||
)
|
||||
def refresh_models(self, request, pk=None):
|
||||
instance = self.get_object()
|
||||
if (
|
||||
instance.provider_type
|
||||
!= LighthouseProviderConfiguration.LLMProviderChoices.OPENAI
|
||||
):
|
||||
return Response(
|
||||
data={
|
||||
"errors": [{"detail": "Only 'openai' provider supported in MVP"}]
|
||||
},
|
||||
status=status.HTTP_400_BAD_REQUEST,
|
||||
)
|
||||
|
||||
with transaction.atomic():
|
||||
task = refresh_lighthouse_provider_models_task.delay(
|
||||
provider_config_id=str(instance.id), tenant_id=self.request.tenant_id
|
||||
)
|
||||
|
||||
prowler_task = Task.objects.get(id=task.id)
|
||||
serializer = TaskSerializer(prowler_task)
|
||||
return Response(
|
||||
data=serializer.data,
|
||||
status=status.HTTP_202_ACCEPTED,
|
||||
headers={
|
||||
"Content-Location": reverse(
|
||||
"task-detail", kwargs={"pk": prowler_task.id}
|
||||
)
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@extend_schema_view(
|
||||
list=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Get Lighthouse AI Tenant config",
|
||||
description="Retrieve current tenant-level Lighthouse AI settings. Returns a single configuration object.",
|
||||
),
|
||||
partial_update=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Update Lighthouse AI Tenant config",
|
||||
description="Update tenant-level settings. Validates that the default provider is configured and active and that default model IDs exist for the chosen providers. Auto-creates configuration if it doesn't exist.",
|
||||
),
|
||||
)
|
||||
class LighthouseTenantConfigViewSet(BaseRLSViewSet):
|
||||
"""
|
||||
Singleton endpoint for tenant-level Lighthouse AI configuration.
|
||||
|
||||
This viewset implements a true singleton pattern:
|
||||
- GET returns the single configuration object (or 404 if not found)
|
||||
- PATCH updates/creates the configuration (upsert semantics)
|
||||
- No ID is required in the URL
|
||||
"""
|
||||
|
||||
queryset = LighthouseTenantConfiguration.objects.all()
|
||||
serializer_class = LighthouseTenantConfigSerializer
|
||||
http_method_names = ["get", "patch"]
|
||||
|
||||
def get_queryset(self):
|
||||
if getattr(self, "swagger_fake_view", False):
|
||||
return LighthouseTenantConfiguration.objects.none()
|
||||
return LighthouseTenantConfiguration.objects.filter(
|
||||
tenant_id=self.request.tenant_id
|
||||
)
|
||||
|
||||
def get_serializer_class(self):
|
||||
if self.action == "partial_update":
|
||||
return LighthouseTenantConfigUpdateSerializer
|
||||
return super().get_serializer_class()
|
||||
|
||||
def get_object(self):
|
||||
"""Retrieve the singleton instance for the current tenant."""
|
||||
obj = LighthouseTenantConfiguration.objects.filter(
|
||||
tenant_id=self.request.tenant_id
|
||||
).first()
|
||||
if obj is None:
|
||||
raise NotFound("Tenant Lighthouse configuration not found")
|
||||
self.check_object_permissions(self.request, obj)
|
||||
return obj
|
||||
|
||||
def list(self, request, *args, **kwargs):
|
||||
"""GET endpoint for singleton - returns single object, not an array."""
|
||||
instance = self.get_object()
|
||||
serializer = self.get_serializer(instance)
|
||||
return Response(serializer.data)
|
||||
|
||||
def partial_update(self, request, *args, **kwargs):
|
||||
"""PATCH endpoint for singleton - no pk required. Auto-creates if not exists."""
|
||||
# Auto-create tenant config if it doesn't exist (upsert semantics)
|
||||
instance, created = LighthouseTenantConfiguration.objects.get_or_create(
|
||||
tenant_id=self.request.tenant_id,
|
||||
defaults={},
|
||||
)
|
||||
|
||||
# Extract attributes from JSON:API payload
|
||||
try:
|
||||
payload = json.loads(request.body)
|
||||
attributes = payload.get("data", {}).get("attributes", {})
|
||||
except (json.JSONDecodeError, AttributeError):
|
||||
raise ValidationError("Invalid JSON:API payload")
|
||||
|
||||
serializer = self.get_serializer(instance, data=attributes, partial=True)
|
||||
serializer.is_valid(raise_exception=True)
|
||||
serializer.save()
|
||||
read_serializer = LighthouseTenantConfigSerializer(
|
||||
instance, context=self.get_serializer_context()
|
||||
)
|
||||
return Response(read_serializer.data, status=status.HTTP_200_OK)
|
||||
|
||||
|
||||
@extend_schema_view(
|
||||
list=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="List all LLM models",
|
||||
description="List available LLM models per configured provider for the current tenant.",
|
||||
),
|
||||
retrieve=extend_schema(
|
||||
tags=["Lighthouse AI"],
|
||||
summary="Retrieve LLM model details",
|
||||
description="Get details for a specific LLM model.",
|
||||
),
|
||||
)
|
||||
class LighthouseProviderModelsViewSet(BaseRLSViewSet):
|
||||
queryset = LighthouseProviderModels.objects.all()
|
||||
serializer_class = LighthouseProviderModelsSerializer
|
||||
filterset_class = LighthouseProviderModelsFilter
|
||||
# Expose as read-only catalog collection
|
||||
http_method_names = ["get"]
|
||||
|
||||
def get_queryset(self):
|
||||
if getattr(self, "swagger_fake_view", False):
|
||||
return LighthouseProviderModels.objects.none()
|
||||
return LighthouseProviderModels.objects.filter(tenant_id=self.request.tenant_id)
|
||||
|
||||
def get_serializer_class(self):
|
||||
return super().get_serializer_class()
|
||||
|
||||
|
||||
@extend_schema_view(
|
||||
list=extend_schema(
|
||||
tags=["Processor"],
|
||||
|
||||
@@ -5,7 +5,7 @@ from celery.utils.log import get_task_logger
|
||||
from config.django.base import DJANGO_FINDINGS_BATCH_SIZE
|
||||
from tasks.utils import batched
|
||||
|
||||
from api.db_router import READ_REPLICA_ALIAS, MainRouter
|
||||
from api.db_router import READ_REPLICA_ALIAS
|
||||
from api.db_utils import rls_transaction
|
||||
from api.models import Finding, Integration, Provider
|
||||
from api.utils import initialize_prowler_integration, initialize_prowler_provider
|
||||
@@ -179,7 +179,7 @@ def get_security_hub_client_from_integration(
|
||||
if the connection was successful and the SecurityHub client or connection object.
|
||||
"""
|
||||
# Get the provider associated with this integration
|
||||
with rls_transaction(tenant_id, using=READ_REPLICA_ALIAS):
|
||||
with rls_transaction(tenant_id):
|
||||
provider_relationship = integration.integrationproviderrelationship_set.first()
|
||||
if not provider_relationship:
|
||||
return Connection(
|
||||
@@ -208,7 +208,7 @@ def get_security_hub_client_from_integration(
|
||||
regions_status[region] = region in connection.enabled_regions
|
||||
|
||||
# Save regions information in the integration configuration
|
||||
with rls_transaction(tenant_id, using=MainRouter.default_db):
|
||||
with rls_transaction(tenant_id):
|
||||
integration.configuration["regions"] = regions_status
|
||||
integration.save()
|
||||
|
||||
@@ -223,7 +223,7 @@ def get_security_hub_client_from_integration(
|
||||
return True, security_hub
|
||||
else:
|
||||
# Reset regions information if connection fails
|
||||
with rls_transaction(tenant_id, using=MainRouter.default_db):
|
||||
with rls_transaction(tenant_id):
|
||||
integration.configuration["regions"] = {}
|
||||
integration.save()
|
||||
|
||||
@@ -334,11 +334,8 @@ def upload_security_hub_integration(
|
||||
f"Security Hub connection failed for integration {integration.id}: "
|
||||
f"{security_hub.error}"
|
||||
)
|
||||
with rls_transaction(
|
||||
tenant_id, using=MainRouter.default_db
|
||||
):
|
||||
integration.connected = False
|
||||
integration.save()
|
||||
integration.connected = False
|
||||
integration.save()
|
||||
break # Skip this integration
|
||||
|
||||
security_hub_client = security_hub
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
from typing import Dict, Set
|
||||
|
||||
import openai
|
||||
from celery.utils.log import get_task_logger
|
||||
|
||||
from api.models import LighthouseProviderConfiguration, LighthouseProviderModels
|
||||
|
||||
logger = get_task_logger(__name__)
|
||||
|
||||
|
||||
def _extract_openai_api_key(
|
||||
provider_cfg: LighthouseProviderConfiguration,
|
||||
) -> str | None:
|
||||
"""
|
||||
Safely extract the OpenAI API key from a provider configuration.
|
||||
|
||||
Args:
|
||||
provider_cfg (LighthouseProviderConfiguration): The provider configuration instance
|
||||
containing the credentials.
|
||||
|
||||
Returns:
|
||||
str | None: The API key string if present and valid, otherwise None.
|
||||
"""
|
||||
creds = provider_cfg.credentials_decoded
|
||||
if not isinstance(creds, dict):
|
||||
return None
|
||||
api_key = creds.get("api_key")
|
||||
if not isinstance(api_key, str) or not api_key:
|
||||
return None
|
||||
return api_key
|
||||
|
||||
|
||||
def check_lighthouse_provider_connection(provider_config_id: str) -> Dict:
|
||||
"""
|
||||
Validate a Lighthouse provider configuration by calling the provider API and
|
||||
toggle its active state accordingly.
|
||||
|
||||
Currently supports the OpenAI provider by invoking `models.list` to verify that
|
||||
the provided credentials are valid.
|
||||
|
||||
Args:
|
||||
provider_config_id (str): The primary key of the `LighthouseProviderConfiguration`
|
||||
to validate.
|
||||
|
||||
Returns:
|
||||
dict: A result dictionary with the following keys:
|
||||
- "connected" (bool): Whether the provider credentials are valid.
|
||||
- "error" (str | None): The error message when not connected, otherwise None.
|
||||
|
||||
Side Effects:
|
||||
- Updates and persists `is_active` on the `LighthouseProviderConfiguration`.
|
||||
|
||||
Raises:
|
||||
LighthouseProviderConfiguration.DoesNotExist: If no configuration exists with the given ID.
|
||||
"""
|
||||
provider_cfg = LighthouseProviderConfiguration.objects.get(pk=provider_config_id)
|
||||
|
||||
# TODO: Add support for other providers
|
||||
if (
|
||||
provider_cfg.provider_type
|
||||
!= LighthouseProviderConfiguration.LLMProviderChoices.OPENAI
|
||||
):
|
||||
return {"connected": False, "error": "Unsupported provider type"}
|
||||
|
||||
api_key = _extract_openai_api_key(provider_cfg)
|
||||
if not api_key:
|
||||
provider_cfg.is_active = False
|
||||
provider_cfg.save()
|
||||
return {"connected": False, "error": "API key is invalid or missing"}
|
||||
|
||||
try:
|
||||
client = openai.OpenAI(api_key=api_key)
|
||||
_ = client.models.list()
|
||||
provider_cfg.is_active = True
|
||||
provider_cfg.save()
|
||||
return {"connected": True, "error": None}
|
||||
except Exception as e:
|
||||
logger.warning("OpenAI connection check failed: %s", str(e))
|
||||
provider_cfg.is_active = False
|
||||
provider_cfg.save()
|
||||
return {"connected": False, "error": str(e)}
|
||||
|
||||
|
||||
def refresh_lighthouse_provider_models(provider_config_id: str) -> Dict:
|
||||
"""
|
||||
Refresh the catalog of models for a Lighthouse provider configuration.
|
||||
|
||||
For the OpenAI provider, this fetches the current list of models, upserts entries
|
||||
into `LighthouseProviderModels`, and deletes stale entries no longer returned by
|
||||
the provider.
|
||||
|
||||
Args:
|
||||
provider_config_id (str): The primary key of the `LighthouseProviderConfiguration`
|
||||
whose models should be refreshed.
|
||||
|
||||
Returns:
|
||||
dict: A result dictionary with the following keys on success:
|
||||
- "created" (int): Number of new model rows created.
|
||||
- "updated" (int): Number of existing model rows updated.
|
||||
- "deleted" (int): Number of stale model rows removed.
|
||||
If an error occurs, the dictionary will contain an "error" (str) field instead.
|
||||
|
||||
Raises:
|
||||
LighthouseProviderConfiguration.DoesNotExist: If no configuration exists with the given ID.
|
||||
"""
|
||||
provider_cfg = LighthouseProviderConfiguration.objects.get(pk=provider_config_id)
|
||||
|
||||
if (
|
||||
provider_cfg.provider_type
|
||||
!= LighthouseProviderConfiguration.LLMProviderChoices.OPENAI
|
||||
):
|
||||
return {
|
||||
"created": 0,
|
||||
"updated": 0,
|
||||
"deleted": 0,
|
||||
"error": "Unsupported provider type",
|
||||
}
|
||||
|
||||
api_key = _extract_openai_api_key(provider_cfg)
|
||||
if not api_key:
|
||||
return {
|
||||
"created": 0,
|
||||
"updated": 0,
|
||||
"deleted": 0,
|
||||
"error": "API key is invalid or missing",
|
||||
}
|
||||
|
||||
try:
|
||||
client = openai.OpenAI(api_key=api_key)
|
||||
models = client.models.list()
|
||||
fetched_ids: Set[str] = {m.id for m in getattr(models, "data", [])}
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning("OpenAI models refresh failed: %s", str(e))
|
||||
return {"created": 0, "updated": 0, "deleted": 0, "error": str(e)}
|
||||
|
||||
created = 0
|
||||
updated = 0
|
||||
|
||||
for model_id in fetched_ids:
|
||||
obj, was_created = LighthouseProviderModels.objects.update_or_create(
|
||||
tenant_id=provider_cfg.tenant_id,
|
||||
provider_configuration=provider_cfg,
|
||||
model_id=model_id,
|
||||
defaults={
|
||||
"model_name": model_id, # OpenAI doesn't return a separate display name
|
||||
"default_parameters": {},
|
||||
},
|
||||
)
|
||||
if was_created:
|
||||
created += 1
|
||||
else:
|
||||
updated += 1
|
||||
|
||||
# Delete stale models not present anymore
|
||||
deleted, _ = (
|
||||
LighthouseProviderModels.objects.filter(
|
||||
tenant_id=provider_cfg.tenant_id, provider_configuration=provider_cfg
|
||||
)
|
||||
.exclude(model_id__in=fetched_ids)
|
||||
.delete()
|
||||
)
|
||||
|
||||
return {"created": created, "updated": updated, "deleted": deleted}
|
||||
@@ -735,9 +735,9 @@ def create_compliance_requirements(tenant_id: str, scan_id: str):
|
||||
provider_instance = scan_instance.provider
|
||||
prowler_provider = return_prowler_provider(provider_instance)
|
||||
|
||||
compliance_template = PROWLER_COMPLIANCE_OVERVIEW_TEMPLATE.get(
|
||||
provider_instance.provider, {}
|
||||
)
|
||||
compliance_template = PROWLER_COMPLIANCE_OVERVIEW_TEMPLATE[
|
||||
provider_instance.provider
|
||||
]
|
||||
modeled_threatscore_compliance_id = "ProwlerThreatScore-1.0"
|
||||
threatscore_requirements_by_check: dict[str, set[str]] = {}
|
||||
threatscore_framework = compliance_template.get(
|
||||
@@ -809,68 +809,63 @@ def create_compliance_requirements(tenant_id: str, scan_id: str):
|
||||
region: deepcopy(compliance_template) for region in regions
|
||||
}
|
||||
|
||||
compliance_requirement_rows: list[dict[str, Any]] = []
|
||||
|
||||
# Skip if provider has no compliance frameworks
|
||||
if compliance_template:
|
||||
# Apply check statuses to compliance data
|
||||
for region, check_status in check_status_by_region.items():
|
||||
compliance_data = compliance_overview_by_region.setdefault(
|
||||
region, deepcopy(compliance_template)
|
||||
# Apply check statuses to compliance data
|
||||
for region, check_status in check_status_by_region.items():
|
||||
compliance_data = compliance_overview_by_region.setdefault(
|
||||
region, deepcopy(compliance_template)
|
||||
)
|
||||
for check_name, status in check_status.items():
|
||||
generate_scan_compliance(
|
||||
compliance_data,
|
||||
provider_instance.provider,
|
||||
check_name,
|
||||
status,
|
||||
)
|
||||
for check_name, status in check_status.items():
|
||||
generate_scan_compliance(
|
||||
compliance_data,
|
||||
provider_instance.provider,
|
||||
check_name,
|
||||
status,
|
||||
|
||||
# Prepare compliance requirement rows
|
||||
compliance_requirement_rows: list[dict[str, Any]] = []
|
||||
utc_datetime_now = datetime.now(tz=timezone.utc)
|
||||
for region, compliance_data in compliance_overview_by_region.items():
|
||||
for compliance_id, compliance in compliance_data.items():
|
||||
modeled_compliance_id = _normalized_compliance_key(
|
||||
compliance["framework"], compliance["version"]
|
||||
)
|
||||
# Create an overview record for each requirement within each compliance framework
|
||||
for requirement_id, requirement in compliance["requirements"].items():
|
||||
checks_status = requirement["checks_status"]
|
||||
compliance_requirement_rows.append(
|
||||
{
|
||||
"id": uuid.uuid4(),
|
||||
"tenant_id": tenant_id,
|
||||
"inserted_at": utc_datetime_now,
|
||||
"compliance_id": compliance_id,
|
||||
"framework": compliance["framework"],
|
||||
"version": compliance["version"] or "",
|
||||
"description": requirement.get("description") or "",
|
||||
"region": region,
|
||||
"requirement_id": requirement_id,
|
||||
"requirement_status": requirement["status"],
|
||||
"passed_checks": checks_status["pass"],
|
||||
"failed_checks": checks_status["fail"],
|
||||
"total_checks": checks_status["total"],
|
||||
"scan_id": scan_instance.id,
|
||||
"passed_findings": findings_count_by_compliance.get(
|
||||
region, {}
|
||||
)
|
||||
.get(modeled_compliance_id, {})
|
||||
.get(requirement_id, {})
|
||||
.get("pass", 0),
|
||||
"total_findings": findings_count_by_compliance.get(
|
||||
region, {}
|
||||
)
|
||||
.get(modeled_compliance_id, {})
|
||||
.get(requirement_id, {})
|
||||
.get("total", 0),
|
||||
}
|
||||
)
|
||||
|
||||
# Prepare compliance requirement rows
|
||||
utc_datetime_now = datetime.now(tz=timezone.utc)
|
||||
for region, compliance_data in compliance_overview_by_region.items():
|
||||
for compliance_id, compliance in compliance_data.items():
|
||||
modeled_compliance_id = _normalized_compliance_key(
|
||||
compliance["framework"], compliance["version"]
|
||||
)
|
||||
# Create an overview record for each requirement within each compliance framework
|
||||
for requirement_id, requirement in compliance[
|
||||
"requirements"
|
||||
].items():
|
||||
checks_status = requirement["checks_status"]
|
||||
compliance_requirement_rows.append(
|
||||
{
|
||||
"id": uuid.uuid4(),
|
||||
"tenant_id": tenant_id,
|
||||
"inserted_at": utc_datetime_now,
|
||||
"compliance_id": compliance_id,
|
||||
"framework": compliance["framework"],
|
||||
"version": compliance["version"] or "",
|
||||
"description": requirement.get("description") or "",
|
||||
"region": region,
|
||||
"requirement_id": requirement_id,
|
||||
"requirement_status": requirement["status"],
|
||||
"passed_checks": checks_status["pass"],
|
||||
"failed_checks": checks_status["fail"],
|
||||
"total_checks": checks_status["total"],
|
||||
"scan_id": scan_instance.id,
|
||||
"passed_findings": findings_count_by_compliance.get(
|
||||
region, {}
|
||||
)
|
||||
.get(modeled_compliance_id, {})
|
||||
.get(requirement_id, {})
|
||||
.get("pass", 0),
|
||||
"total_findings": findings_count_by_compliance.get(
|
||||
region, {}
|
||||
)
|
||||
.get(modeled_compliance_id, {})
|
||||
.get(requirement_id, {})
|
||||
.get("total", 0),
|
||||
}
|
||||
)
|
||||
|
||||
# Bulk create requirement records using PostgreSQL COPY
|
||||
_persist_compliance_requirement_rows(tenant_id, compliance_requirement_rows)
|
||||
# Bulk create requirement records using PostgreSQL COPY
|
||||
_persist_compliance_requirement_rows(tenant_id, compliance_requirement_rows)
|
||||
|
||||
return {
|
||||
"requirements_created": len(compliance_requirement_rows),
|
||||
|
||||
@@ -27,6 +27,10 @@ from tasks.jobs.integrations import (
|
||||
upload_s3_integration,
|
||||
upload_security_hub_integration,
|
||||
)
|
||||
from tasks.jobs.lighthouse_providers import (
|
||||
check_lighthouse_provider_connection,
|
||||
refresh_lighthouse_provider_models,
|
||||
)
|
||||
from tasks.jobs.report import generate_threatscore_report_job
|
||||
from tasks.jobs.scan import (
|
||||
aggregate_findings,
|
||||
@@ -524,6 +528,24 @@ def check_lighthouse_connection_task(lighthouse_config_id: str, tenant_id: str =
|
||||
return check_lighthouse_connection(lighthouse_config_id=lighthouse_config_id)
|
||||
|
||||
|
||||
@shared_task(base=RLSTask, name="lighthouse-provider-connection-check")
|
||||
@set_tenant
|
||||
def check_lighthouse_provider_connection_task(
|
||||
provider_config_id: str, tenant_id: str | None = None
|
||||
) -> dict:
|
||||
"""Task wrapper to validate provider credentials and set is_active."""
|
||||
return check_lighthouse_provider_connection(provider_config_id=provider_config_id)
|
||||
|
||||
|
||||
@shared_task(base=RLSTask, name="lighthouse-provider-models-refresh")
|
||||
@set_tenant
|
||||
def refresh_lighthouse_provider_models_task(
|
||||
provider_config_id: str, tenant_id: str | None = None
|
||||
) -> dict:
|
||||
"""Task wrapper to refresh provider models catalog for the given configuration."""
|
||||
return refresh_lighthouse_provider_models(provider_config_id=provider_config_id)
|
||||
|
||||
|
||||
@shared_task(name="integration-check")
|
||||
def check_integrations_task(tenant_id: str, provider_id: str, scan_id: str = None):
|
||||
"""
|
||||
|
||||
@@ -9,7 +9,6 @@ from tasks.jobs.integrations import (
|
||||
upload_security_hub_integration,
|
||||
)
|
||||
|
||||
from api.db_router import READ_REPLICA_ALIAS, MainRouter
|
||||
from api.models import Integration
|
||||
from api.utils import prowler_integration_connection_test
|
||||
from prowler.providers.aws.lib.security_hub.security_hub import SecurityHubConnection
|
||||
@@ -881,8 +880,7 @@ class TestSecurityHubIntegrationUploads:
|
||||
# Verify RLS transaction was used correctly
|
||||
# Should be called twice: once for getting provider info, once for resetting regions
|
||||
assert mock_rls.call_count == 2
|
||||
mock_rls.assert_any_call(tenant_id, using=READ_REPLICA_ALIAS)
|
||||
mock_rls.assert_any_call(tenant_id, using=MainRouter.default_db)
|
||||
mock_rls.assert_any_call(tenant_id)
|
||||
|
||||
# Verify test_connection was called with integration credentials (not provider's)
|
||||
mock_test_connection.assert_called_once_with(
|
||||
|
||||
+20
-1
@@ -98,7 +98,7 @@
|
||||
]
|
||||
},
|
||||
"user-guide/tutorials/prowler-app-rbac",
|
||||
"user-guide/providers/prowler-app-api-keys",
|
||||
"user-guide/tutorials/prowler-app-api-keys",
|
||||
"user-guide/tutorials/prowler-app-mute-findings",
|
||||
{
|
||||
"group": "Integrations",
|
||||
@@ -251,6 +251,25 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"tab": "Workshop",
|
||||
"groups": [
|
||||
{
|
||||
"group": "Hands-On Labs",
|
||||
"pages": [
|
||||
"workshop/introduction",
|
||||
"workshop/lab-01-getting-started",
|
||||
"workshop/lab-02-threat-detection",
|
||||
"workshop/lab-03-custom-checks",
|
||||
"workshop/lab-04-azure-multicloud",
|
||||
"workshop/lab-05-gcp-multicloud",
|
||||
"workshop/lab-06-compliance-as-code",
|
||||
"workshop/lab-07-integrations",
|
||||
"workshop/lab-08-prowler-saas"
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"tab": "Developer Guide",
|
||||
"groups": [
|
||||
|
||||
@@ -10,7 +10,7 @@ Configure your MCP client to connect to Prowler MCP Server.
|
||||
**Authentication is optional**: Prowler Hub and Prowler Documentation features work without authentication. An API key is only required for Prowler Cloud and Prowler App (Self-Managed) features.
|
||||
</Note>
|
||||
|
||||
To use Prowler Cloud or Prowler App (Self-Managed) features. To get the API key, please refer to the [API Keys](/user-guide/providers/prowler-app-api-keys) guide.
|
||||
To use Prowler Cloud or Prowler App (Self-Managed) features. To get the API key, please refer to the [API Keys](/user-guide/tutorials/prowler-app-api-keys) guide.
|
||||
|
||||
<Warning>
|
||||
Keep the API key secure. Never share it publicly or commit it to version control.
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
export const VersionBadge = ({ version }) => {
|
||||
return (
|
||||
<code className="version-badge-container">
|
||||
<p className="version-badge">
|
||||
<span className="version-badge-label">Added in:</span>
|
||||
<code className="version-badge-version">{version}</code>
|
||||
</p>
|
||||
</code>
|
||||
|
||||
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,51 @@
|
||||
/* Version Badge Styling */
|
||||
.version-badge-container {
|
||||
display: inline-block;
|
||||
margin: 0 0 1rem 0;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.version-badge {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
margin: 0;
|
||||
padding: 0.375rem 0.75rem;
|
||||
background: linear-gradient(135deg, #1a1a1a 0%, #000000 100%);
|
||||
color: #ffffff;
|
||||
border-radius: 1.25rem;
|
||||
font-weight: 400;
|
||||
font-size: 0.875rem;
|
||||
line-height: 1.25rem;
|
||||
border: 1px solid rgba(0, 0, 0, 0.15);
|
||||
box-shadow: none;
|
||||
}
|
||||
|
||||
.version-badge-label {
|
||||
font-weight: 400;
|
||||
opacity: 1;
|
||||
}
|
||||
|
||||
.version-badge-version {
|
||||
background: rgba(255, 255, 255, 0.12);
|
||||
padding: 0.125rem 0.5rem;
|
||||
border-radius: 0.875rem;
|
||||
font-family: ui-monospace, SFMono-Regular, 'SF Mono', Menlo, Monaco, 'Cascadia Code', 'Roboto Mono', Consolas, 'Courier New', monospace;
|
||||
font-weight: 600;
|
||||
font-size: 0.875rem;
|
||||
color: #ffffff;
|
||||
border: none;
|
||||
}
|
||||
|
||||
|
||||
.dark .version-badge {
|
||||
background: #55B685;
|
||||
color: #000000;
|
||||
border: 2px solid rgba(85, 182, 133, 0.3);
|
||||
box-shadow: none;
|
||||
}
|
||||
|
||||
.dark .version-badge-version {
|
||||
background: rgba(0, 0, 0, 0.1);
|
||||
color: #000000;
|
||||
border: none;
|
||||
}
|
||||
@@ -24,7 +24,7 @@ Standard results will be shown and additionally the framework information as the
|
||||
**If Prowler can't find a resource related with a check from a compliance requirement, this requirement won't appear on the output**
|
||||
</Note>
|
||||
|
||||
## List Available Compliance Frameworks
|
||||
## List Available Compliance Frameworks
|
||||
|
||||
To see which compliance frameworks are covered by Prowler, use the `--list-compliance` option:
|
||||
|
||||
@@ -34,7 +34,7 @@ prowler <provider> --list-compliance
|
||||
|
||||
Or you can visit [Prowler Hub](https://hub.prowler.com/compliance).
|
||||
|
||||
## List Requirements of Compliance Frameworks
|
||||
## List Requirements of Compliance Frameworks
|
||||
To list requirements for a compliance framework, use the `--list-compliance-requirements` option:
|
||||
|
||||
```sh
|
||||
|
||||
@@ -94,7 +94,7 @@ The following list includes all the Azure checks with configurable variables tha
|
||||
|
||||
### Configurable Checks
|
||||
|
||||
## Kubernetes
|
||||
## Kubernetes
|
||||
|
||||
### Configurable Checks
|
||||
The following list includes all the Kubernetes checks with configurable variables that can be changed in the configuration yaml file:
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
title: 'Integrations'
|
||||
---
|
||||
|
||||
## Integration with Slack
|
||||
## Integration with Slack
|
||||
|
||||
Prowler can be integrated with [Slack](https://slack.com/) to send a summary of the execution having configured a Slack APP in your channel with the following command:
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ prowler <provider> -V/-v/--version
|
||||
|
||||
Prowler provides various execution settings.
|
||||
|
||||
### Verbose Execution
|
||||
### Verbose Execution
|
||||
|
||||
To enable verbose mode in Prowler, similar to Version 2, use:
|
||||
|
||||
@@ -54,7 +54,7 @@ To run Prowler without color formatting:
|
||||
prowler <provider> --no-color
|
||||
```
|
||||
|
||||
### Checks in Prowler
|
||||
### Checks in Prowler
|
||||
|
||||
Prowler provides various security checks per cloud provider. Use the following options to list, execute, or exclude specific checks:
|
||||
|
||||
@@ -96,7 +96,7 @@ prowler <provider> -e/--excluded-checks ec2 rds
|
||||
prowler <provider> -C/--checks-file <checks_list>.json
|
||||
```
|
||||
|
||||
## Custom Checks in Prowler
|
||||
## Custom Checks in Prowler
|
||||
|
||||
Prowler supports custom security checks, allowing users to define their own logic.
|
||||
|
||||
|
||||
@@ -27,7 +27,7 @@ If any of the criteria do not match, the check is not muted.
|
||||
Remember that mutelist can be used with regular expressions.
|
||||
|
||||
</Note>
|
||||
## Mutelist Specification
|
||||
## Mutelist Specification
|
||||
|
||||
<Note>
|
||||
- For Azure provider, the Account ID is the Subscription Name and the Region is the Location.
|
||||
@@ -40,10 +40,9 @@ The Mutelist file uses the [YAML](https://en.wikipedia.org/wiki/YAML) format wit
|
||||
```yaml
|
||||
### Account, Check and/or Region can be * to apply for all the cases.
|
||||
### Resources and tags are lists that can have either Regex or Keywords.
|
||||
### Multiple tags in the list are "ANDed" together (ALL must match).
|
||||
### Use regex alternation (|) within a single tag for "OR" logic (e.g., "env=dev|env=stg").
|
||||
### For each check you can use Exceptions to unmute specific Accounts, Regions, Resources and/or Tags.
|
||||
### All conditions (Account, Check, Region, Resource, Tags) are ANDed together.
|
||||
### Tags is an optional list that matches on tuples of 'key=value' and are "ANDed" together.
|
||||
### Use an alternation Regex to match one of multiple tags with "ORed" logic.
|
||||
### For each check you can except Accounts, Regions, Resources and/or Tags.
|
||||
########################### MUTELIST EXAMPLE ###########################
|
||||
Mutelist:
|
||||
Accounts:
|
||||
|
||||
@@ -10,7 +10,7 @@ This can help for really large accounts, but please be aware of AWS API rate lim
|
||||
2. **API Rate Limits**: Most of the rate limits in AWS are applied at the API level. Each API call to an AWS service counts towards the rate limit for that service.
|
||||
3. **Throttling Responses**: When you exceed the rate limit for a service, AWS responds with a throttling error. In AWS SDKs, these are typically represented as `ThrottlingException` or `RateLimitExceeded` errors.
|
||||
|
||||
For information on Prowler's retrier configuration please refer to this [page](https://docs.prowler.com/user-guide/providers/aws/boto3-configuration/).
|
||||
For information on Prowler's retrier configuration please refer to this [page](https://docs.prowler.cloud/en/latest/tutorials/aws/boto3-configuration/).
|
||||
|
||||
<Note>
|
||||
You might need to increase the `--aws-retries-max-attempts` parameter from the default value of 3. The retrier follows an exponential backoff strategy.
|
||||
|
||||
@@ -24,6 +24,6 @@ By default, it extracts resources from all the regions, you could use `-f`/`--fi
|
||||
|
||||

|
||||
|
||||
## Objections
|
||||
## Objections
|
||||
|
||||
The inventorying process is carried out with `resourcegroupstaggingapi` calls, which means that only resources they have or have had tags will appear (except for the IAM and S3 resources which are done with Boto3 API calls).
|
||||
|
||||
@@ -22,7 +22,7 @@ prowler <provider> --output-formats json-asff
|
||||
|
||||
All compliance-related reports are automatically generated when Prowler is executed. These outputs are stored in the `/output/compliance` directory.
|
||||
|
||||
## Custom Output Flags
|
||||
## Custom Output Flags
|
||||
|
||||
By default, Prowler creates a file inside the `output` directory named: `prowler-output-ACCOUNT_NUM-OUTPUT_DATE.format`.
|
||||
|
||||
@@ -53,7 +53,7 @@ Both flags can be used simultaneously to provide a custom directory and filename
|
||||
|
||||
By default, the timestamp format of the output files is ISO 8601. This can be changed with the flag `--unix-timestamp` generating the timestamp fields in pure unix timestamp format.
|
||||
|
||||
## Supported Output Formats
|
||||
## Supported Output Formats
|
||||
|
||||
Prowler natively supports the following reporting output formats:
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ prowler <provider> --scan-unused-services
|
||||
|
||||
## Services Ignored
|
||||
|
||||
### AWS
|
||||
### AWS
|
||||
|
||||
#### ACM (AWS Certificate Manager)
|
||||
|
||||
@@ -22,21 +22,21 @@ Certificates stored in ACM without active usage in AWS resources are excluded. B
|
||||
|
||||
- `acm_certificates_expiration_check`
|
||||
|
||||
#### Athena
|
||||
#### Athena
|
||||
|
||||
Upon AWS account creation, Athena provisions a default primary workgroup for the user. Prowler verifies if this workgroup is enabled and used by checking for queries within the last 45 days. If Athena is unused, findings related to its checks will not appear.
|
||||
|
||||
- `athena_workgroup_encryption`
|
||||
- `athena_workgroup_enforce_configuration`
|
||||
|
||||
#### AWS CloudTrail
|
||||
#### AWS CloudTrail
|
||||
|
||||
AWS CloudTrail should have at least one trail with a data event to record all S3 object-level API operations. Before flagging this issue, Prowler verifies if S3 buckets exist in the account.
|
||||
|
||||
- `cloudtrail_s3_dataevents_read_enabled`
|
||||
- `cloudtrail_s3_dataevents_write_enabled`
|
||||
|
||||
#### AWS Elastic Compute Cloud (EC2)
|
||||
#### AWS Elastic Compute Cloud (EC2)
|
||||
|
||||
If Amazon Elastic Block Store (EBS) default encyption is not enabled, sensitive data at rest will remain unprotected in EC2. However, Prowler will only generate a finding if EBS volumes exist where default encryption could be enforced.
|
||||
|
||||
@@ -56,7 +56,7 @@ Prowler scans only attached security groups to report vulnerabilities in activel
|
||||
|
||||
- `ec2_networkacl_allow_ingress_X_port`
|
||||
|
||||
#### AWS Glue
|
||||
#### AWS Glue
|
||||
|
||||
AWS Glue best practices recommend encrypting metadata and connection passwords in Data Catalogs.
|
||||
|
||||
@@ -71,7 +71,7 @@ Amazon Inspector is a vulnerability discovery service that automates continuous
|
||||
|
||||
- `inspector2_is_enabled`
|
||||
|
||||
#### Amazon Macie
|
||||
#### Amazon Macie
|
||||
|
||||
Amazon Macie leverages machine learning to automatically discover, classify, and protect sensitive data in S3 buckets. Prowler only generates findings if Macie is disabled and there are S3 buckets in the AWS account.
|
||||
|
||||
@@ -83,7 +83,7 @@ A network firewall is essential for monitoring and controlling traffic within a
|
||||
|
||||
- `networkfirewall_in_all_vpc`
|
||||
|
||||
#### Amazon S3
|
||||
#### Amazon S3
|
||||
|
||||
To prevent unintended data exposure:
|
||||
|
||||
@@ -91,7 +91,7 @@ Public Access Block should be enabled at the account level. Prowler only checks
|
||||
|
||||
- `s3_account_level_public_access_blocks`
|
||||
|
||||
#### Virtual Private Cloud (VPC)
|
||||
#### Virtual Private Cloud (VPC)
|
||||
|
||||
VPC settings directly impact network security and availability.
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ The AWS Organizations Bulk Provisioning tool simplifies multi-account onboarding
|
||||
* ProwlerRole (or custom role) deployed across all target accounts
|
||||
* Prowler API key (from Prowler Cloud or self-hosted Prowler App)
|
||||
* For self-hosted Prowler App, remember to [point to your API base URL](./bulk-provider-provisioning#custom-api-endpoints)
|
||||
* Learn how to create API keys: [Prowler App API Keys](../providers/prowler-app-api-keys)
|
||||
* Learn how to create API keys: [Prowler App API Keys](../tutorials/prowler-app-api-keys)
|
||||
|
||||
### Deploying ProwlerRole Across AWS Organizations
|
||||
|
||||
@@ -101,7 +101,7 @@ To create an API key:
|
||||
4. Provide a descriptive name and optionally set an expiration date
|
||||
5. Copy the generated API key (it will only be shown once)
|
||||
|
||||
For detailed instructions, see: [Prowler App API Keys](../providers/prowler-app-api-keys)
|
||||
For detailed instructions, see: [Prowler App API Keys](../tutorials/prowler-app-api-keys)
|
||||
|
||||
## Basic Usage
|
||||
|
||||
|
||||
@@ -28,7 +28,7 @@ The Bulk Provider Provisioning tool automates the creation of cloud providers in
|
||||
* Python 3.7 or higher
|
||||
* Prowler API key (from Prowler Cloud or self-hosted Prowler App)
|
||||
* For self-hosted Prowler App, remember to [point to your API base URL](#custom-api-endpoints)
|
||||
* Learn how to create API keys: [Prowler App API Keys](../providers/prowler-app-api-keys)
|
||||
* Learn how to create API keys: [Prowler App API Keys](../tutorials/prowler-app-api-keys)
|
||||
* Authentication credentials for target cloud providers
|
||||
|
||||
### Installation
|
||||
@@ -57,7 +57,7 @@ To create an API key:
|
||||
4. Provide a descriptive name and optionally set an expiration date
|
||||
5. Copy the generated API key (it will only be shown once)
|
||||
|
||||
For detailed instructions, see: [Prowler App API Keys](../providers/prowler-app-api-keys)
|
||||
For detailed instructions, see: [Prowler App API Keys](../tutorials/prowler-app-api-keys)
|
||||
|
||||
## Configuration File Structure
|
||||
|
||||
|
||||
+4
@@ -2,6 +2,10 @@
|
||||
title: 'API Keys'
|
||||
---
|
||||
|
||||
import { VersionBadge } from "/snippets/version-badge.mdx"
|
||||
|
||||
<VersionBadge version="5.13.0" />
|
||||
|
||||
API key authentication in Prowler App provides an alternative to JWT tokens and empowers automation, CI/CD pipelines, and third-party integrations. This guide explains how to create, manage, and safeguard API keys when working with the Prowler API.
|
||||
|
||||
## API Key Advantages
|
||||
@@ -1,6 +1,9 @@
|
||||
---
|
||||
title: "Jira Integration"
|
||||
---
|
||||
import { VersionBadge } from "/snippets/version-badge.mdx"
|
||||
|
||||
<VersionBadge version="5.12.0" />
|
||||
|
||||
Prowler App enables automatic export of security findings to Jira, providing seamless integration with Atlassian's work item tracking and project management platform. This comprehensive guide demonstrates how to configure and manage Jira integrations to streamline security incident management and enhance team collaboration across security workflows.
|
||||
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
title: 'Prowler Lighthouse AI'
|
||||
---
|
||||
|
||||
import { VersionBadge } from "/snippets/version-badge.mdx"
|
||||
|
||||
<VersionBadge version="5.8.0" />
|
||||
|
||||
Prowler Lighthouse AI is a Cloud Security Analyst chatbot that helps you understand, prioritize, and remediate security findings in your cloud environments. It's designed to provide security expertise for teams without dedicated resources, acting as your 24/7 virtual cloud security analyst.
|
||||
|
||||
<img src="/images/prowler-app/lighthouse-intro.png" alt="Prowler Lighthouse" />
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
---
|
||||
title: 'Mute Findings (Mutelist)'
|
||||
---
|
||||
import { VersionBadge } from "/snippets/version-badge.mdx"
|
||||
|
||||
<VersionBadge version="5.9.0" />
|
||||
|
||||
Prowler App allows users to mute specific findings to focus on the most critical security issues. This comprehensive guide demonstrates how to effectively use the Mutelist feature to manage and prioritize security findings.
|
||||
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
title: 'Managing Users and Role-Based Access Control (RBAC)'
|
||||
---
|
||||
|
||||
import { VersionBadge } from "/snippets/version-badge.mdx"
|
||||
|
||||
<VersionBadge version="5.1.0" />
|
||||
|
||||
**Prowler App** supports multiple users within a single tenant, enabling seamless collaboration by allowing team members to easily share insights and manage security findings.
|
||||
|
||||
[Roles](#roles) help you control user permissions, determining what actions each user can perform and the data they can access within Prowler. By default, each account includes an immutable **admin** role, ensuring that your account always retains administrative access.
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
title: 'Amazon S3 Integration'
|
||||
---
|
||||
|
||||
import { VersionBadge } from "/snippets/version-badge.mdx"
|
||||
|
||||
<VersionBadge version="5.10.0" />
|
||||
|
||||
**Prowler App** allows automatic export of scan results to Amazon S3 buckets, providing seamless integration with existing data workflows and storage infrastructure. This comprehensive guide demonstrates configuration and management of Amazon S3 integrations to streamline security finding management and reporting.
|
||||
|
||||
When enabled and configured, scan results are automatically stored in the configured bucket. Results are provided in `csv`, `html` and `json-ocsf` formats, offering flexibility for custom integrations:
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
---
|
||||
title: "AWS Security Hub Integration"
|
||||
---
|
||||
import { VersionBadge } from "/snippets/version-badge.mdx"
|
||||
|
||||
<VersionBadge version="5.11.0" />
|
||||
|
||||
Prowler App enables automatic export of security findings to AWS Security Hub, providing seamless integration with AWS's native security and compliance service. This comprehensive guide demonstrates how to configure and manage AWS Security Hub integrations to centralize security findings and enhance compliance tracking across AWS environments.
|
||||
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
title: 'Social Login Configuration'
|
||||
---
|
||||
|
||||
import { VersionBadge } from "/snippets/version-badge.mdx"
|
||||
|
||||
<VersionBadge version="5.5.0" />
|
||||
|
||||
**Prowler App** supports social login using Google and GitHub OAuth providers. This document guides you through configuring the required environment variables to enable social authentication.
|
||||
|
||||
<img src="/images/prowler-app/social-login/social_login_buttons.png" alt="Social login buttons" width="700" />
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
title: 'SAML Single Sign-On (SSO)'
|
||||
---
|
||||
|
||||
import { VersionBadge } from "/snippets/version-badge.mdx"
|
||||
|
||||
<VersionBadge version="5.9.0" />
|
||||
|
||||
This guide provides comprehensive instructions to configure SAML-based Single Sign-On (SSO) in Prowler App. This configuration allows users to authenticate using the organization's Identity Provider (IdP).
|
||||
|
||||
This document is divided into two main sections:
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
---
|
||||
title: "Workshop Introduction"
|
||||
description: "Hands-on labs to master Prowler's cloud security capabilities across AWS, Azure, and GCP"
|
||||
---
|
||||
|
||||
# Prowler Workshop
|
||||
|
||||
Welcome to the Prowler Workshop. This hands-on training provides practical experience with Prowler's cloud security monitoring and compliance automation capabilities across multiple cloud platforms.
|
||||
|
||||
## Workshop Overview
|
||||
|
||||
This workshop consists of eight progressive labs designed to guide you through Prowler's core features and advanced capabilities:
|
||||
|
||||
* **Lab 1:** Getting Started with Prowler CLI
|
||||
* **Lab 2:** Threat Detection with Prowler
|
||||
* **Lab 3:** Custom Checks with Prowler
|
||||
* **Lab 4:** Multi-Cloud Security with Prowler (Azure)
|
||||
* **Lab 5:** Multi-Cloud Security with Prowler (GCP)
|
||||
* **Lab 6:** Compliance as Code with Prowler
|
||||
* **Lab 7:** Integrations with Prowler (AWS Security Hub)
|
||||
* **Lab 8:** Prowler SaaS Platform
|
||||
|
||||
## Lab Structure
|
||||
|
||||
Each lab is self-contained and includes:
|
||||
|
||||
* **Prerequisites:** Required cloud accounts, tools, and prior lab dependencies
|
||||
* **Objectives:** Clear learning goals for the lab
|
||||
* **Step-by-step instructions:** Detailed guidance through each task
|
||||
* **Expected outcomes:** What you should achieve by completing the lab
|
||||
* **Verification steps:** How to confirm successful completion
|
||||
|
||||
## Prerequisites Approach
|
||||
|
||||
Each lab specifies its own prerequisites, as different labs require different cloud provider accounts, tools, and access levels. Review the prerequisites section at the beginning of each lab before starting.
|
||||
|
||||
## How to Use This Workshop
|
||||
|
||||
* Labs are designed to be completed sequentially, as later labs may build on concepts from earlier ones
|
||||
* Estimated time to complete varies by lab (typically 30-60 minutes each)
|
||||
* You can pause between labs and resume later
|
||||
* Some labs can be completed independently if you have the necessary prerequisites
|
||||
|
||||
## Getting Help
|
||||
|
||||
If you encounter issues during the workshop:
|
||||
|
||||
* Refer to the [Troubleshooting](/troubleshooting) guide
|
||||
* Join the [Prowler Slack community](https://goto.prowler.com/slack)
|
||||
* Visit the [Prowler GitHub repository](https://github.com/prowler-cloud/prowler) for documentation and issues
|
||||
|
||||
## Ready to Start?
|
||||
|
||||
Begin with [Lab 1: Getting Started with Prowler CLI](/workshop/lab-01-getting-started) to set up your environment and run your first security scan.
|
||||
@@ -0,0 +1,203 @@
|
||||
---
|
||||
title: "Lab 1: Getting Started with Prowler CLI"
|
||||
description: "Install Prowler CLI and run your first cloud security assessment on AWS"
|
||||
---
|
||||
|
||||
<Note>
|
||||
**Tags:** `workshop` `aws` `getting-started` `beginner` `cli`
|
||||
</Note>
|
||||
|
||||
# Lab 1: Getting Started with Prowler CLI
|
||||
|
||||
Learn to install Prowler CLI and perform your first cloud security assessment on AWS.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* AWS account with active resources
|
||||
* AWS CLI installed and configured
|
||||
* IAM credentials with appropriate permissions (see [AWS Authentication](/user-guide/providers/aws/authentication))
|
||||
* Python 3.9 or higher
|
||||
* Basic command-line experience
|
||||
|
||||
**Estimated Time:** 30 minutes
|
||||
|
||||
## Lab Objectives
|
||||
|
||||
By completing this lab, you will:
|
||||
|
||||
* Install Prowler CLI using pip
|
||||
* Configure AWS credentials for Prowler
|
||||
* Execute your first security scan
|
||||
* Understand Prowler's output formats
|
||||
* Review security findings
|
||||
|
||||
## Step 1: Install Prowler CLI
|
||||
|
||||
Install Prowler using pip:
|
||||
|
||||
```bash
|
||||
pip install prowler
|
||||
```
|
||||
|
||||
Verify the installation:
|
||||
|
||||
```bash
|
||||
prowler -v
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
Prowler X.X.X
|
||||
```
|
||||
|
||||
<Tip>
|
||||
For alternative installation methods (Docker, from source), see [Prowler CLI Installation](/getting-started/installation/prowler-cli).
|
||||
</Tip>
|
||||
|
||||
## Step 2: Configure AWS Credentials
|
||||
|
||||
Ensure AWS credentials are configured. Prowler uses the same credential chain as AWS CLI.
|
||||
|
||||
Verify credentials:
|
||||
|
||||
```bash
|
||||
aws sts get-caller-identity
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```json
|
||||
{
|
||||
"UserId": "AIDACKCEVSQ6C2EXAMPLE",
|
||||
"Account": "123456789012",
|
||||
"Arn": "arn:aws:iam::123456789012:user/username"
|
||||
}
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 8 showing AWS credential verification - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 3: Run Your First Scan
|
||||
|
||||
Execute a basic Prowler scan:
|
||||
|
||||
```bash
|
||||
prowler aws
|
||||
```
|
||||
|
||||
This command:
|
||||
* Scans all enabled AWS regions
|
||||
* Runs all available security checks
|
||||
* Generates output in the current directory
|
||||
|
||||
<Note>
|
||||
The scan may take 5-15 minutes depending on the number of resources in your AWS account.
|
||||
</Note>
|
||||
|
||||
## Step 4: Understanding the Output
|
||||
|
||||
Prowler generates multiple output formats in the `output` directory:
|
||||
|
||||
* **CSV:** Detailed findings (`prowler-output-*.csv`)
|
||||
* **JSON:** Machine-readable format (`prowler-output-*.json`)
|
||||
* **HTML:** Human-readable report (`prowler-output-*.html`)
|
||||
|
||||
Review the HTML report:
|
||||
|
||||
```bash
|
||||
open output/prowler-output-*.html
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 10 showing HTML report - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 5: Analyze Security Findings
|
||||
|
||||
Examine the findings structure in the HTML report:
|
||||
|
||||
* **Status:** PASS, FAIL, or MANUAL
|
||||
* **Severity:** critical, high, medium, low, informational
|
||||
* **Service:** AWS service affected (e.g., S3, IAM, EC2)
|
||||
* **Check ID:** Unique identifier for each check
|
||||
* **Region:** AWS region where the resource exists
|
||||
* **Resource:** Specific resource ARN or identifier
|
||||
|
||||
Example finding structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"Status": "FAIL",
|
||||
"Severity": "high",
|
||||
"Service": "s3",
|
||||
"CheckID": "s3_bucket_public_access",
|
||||
"Region": "us-east-1",
|
||||
"Resource": "arn:aws:s3:::my-bucket"
|
||||
}
|
||||
```
|
||||
|
||||
## Step 6: Filter Scan by Service
|
||||
|
||||
Run a targeted scan for specific AWS services:
|
||||
|
||||
```bash
|
||||
prowler aws --services s3 iam
|
||||
```
|
||||
|
||||
This scans only S3 and IAM services, reducing execution time.
|
||||
|
||||
## Step 7: Run Checks by Severity
|
||||
|
||||
Scan for critical and high-severity findings only:
|
||||
|
||||
```bash
|
||||
prowler aws --severity critical high
|
||||
```
|
||||
|
||||
This focuses on the most important security issues.
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 13 showing severity filtering - to be added]
|
||||
</Note>
|
||||
|
||||
## Verification Steps
|
||||
|
||||
Confirm successful lab completion:
|
||||
|
||||
1. Prowler CLI installed and version verified
|
||||
2. AWS credentials properly configured
|
||||
3. First scan completed successfully
|
||||
4. Output files generated in the `output` directory
|
||||
5. HTML report reviewed and findings understood
|
||||
6. Filtered scans executed by service and severity
|
||||
|
||||
## Expected Outcomes
|
||||
|
||||
After completing this lab, you should have:
|
||||
|
||||
* Working Prowler CLI installation
|
||||
* Understanding of basic Prowler commands
|
||||
* Knowledge of output formats
|
||||
* Ability to run targeted scans
|
||||
* Familiarity with finding severity levels
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Issue:** `prowler: command not found`
|
||||
* **Solution:** Ensure Python's bin directory is in your PATH, or use `python3 -m prowler`
|
||||
|
||||
**Issue:** AWS credentials error
|
||||
* **Solution:** Run `aws configure` to set up credentials, or use environment variables
|
||||
|
||||
**Issue:** Scan takes too long
|
||||
* **Solution:** Use `--services` to scan specific services or `--regions` to limit regions
|
||||
|
||||
## Next Steps
|
||||
|
||||
Continue to [Lab 2: Threat Detection with Prowler](/workshop/lab-02-threat-detection) to learn about identifying security threats in your AWS environment.
|
||||
|
||||
## Additional Resources
|
||||
|
||||
* [Prowler CLI Documentation](/getting-started/basic-usage/prowler-cli)
|
||||
* [AWS Authentication Methods](/user-guide/providers/aws/authentication)
|
||||
* [Output Formats](/user-guide/cli/tutorials/reporting)
|
||||
@@ -0,0 +1,263 @@
|
||||
---
|
||||
title: "Lab 2: Threat Detection with Prowler"
|
||||
description: "Identify and analyze security threats in AWS environments using Prowler's threat detection capabilities"
|
||||
---
|
||||
|
||||
<Note>
|
||||
**Tags:** `workshop` `aws` `threat-detection` `intermediate` `security`
|
||||
</Note>
|
||||
|
||||
# Lab 2: Threat Detection with Prowler
|
||||
|
||||
Learn to identify security threats, exposed resources, and potential attack vectors in AWS environments using Prowler's threat detection features.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* Completion of [Lab 1: Getting Started with Prowler CLI](/workshop/lab-01-getting-started)
|
||||
* AWS account with resources (EC2 instances, S3 buckets, security groups)
|
||||
* Prowler CLI installed and configured
|
||||
* Basic understanding of AWS security concepts
|
||||
|
||||
**Estimated Time:** 45 minutes
|
||||
|
||||
## Lab Objectives
|
||||
|
||||
By completing this lab, you will:
|
||||
|
||||
* Understand Prowler's threat detection capabilities
|
||||
* Identify publicly exposed resources
|
||||
* Detect insecure configurations
|
||||
* Analyze CloudTrail events for suspicious activity
|
||||
* Prioritize security findings by risk
|
||||
|
||||
## Step 1: Understanding Threat Detection Checks
|
||||
|
||||
Prowler includes checks that identify:
|
||||
|
||||
* Public exposure (S3 buckets, EC2 instances, RDS databases)
|
||||
* Insecure network configurations (security groups, NACLs)
|
||||
* Weak encryption settings
|
||||
* Suspicious IAM permissions
|
||||
* CloudTrail anomalies
|
||||
|
||||
List threat detection checks:
|
||||
|
||||
```bash
|
||||
prowler aws --list-checks | grep -i "public\|exposed\|open"
|
||||
```
|
||||
|
||||
## Step 2: Scan for Publicly Exposed Resources
|
||||
|
||||
Run a scan focusing on public exposure:
|
||||
|
||||
```bash
|
||||
prowler aws --checks s3_bucket_public_access ec2_instance_public_ip rds_instance_publicly_accessible
|
||||
```
|
||||
|
||||
This identifies:
|
||||
* S3 buckets with public access
|
||||
* EC2 instances with public IPs
|
||||
* RDS databases accessible from the internet
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 17 showing public exposure findings - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 3: Analyze Security Group Misconfigurations
|
||||
|
||||
Security groups control network access. Scan for insecure rules:
|
||||
|
||||
```bash
|
||||
prowler aws --services ec2 --checks ec2_securitygroup*
|
||||
```
|
||||
|
||||
Look for findings related to:
|
||||
* `0.0.0.0/0` ingress rules (any IP can connect)
|
||||
* Open high-risk ports (22, 3389, 3306, 5432)
|
||||
* Overly permissive egress rules
|
||||
|
||||
Example vulnerable security group:
|
||||
```
|
||||
Port 22 (SSH) open to 0.0.0.0/0
|
||||
Port 3389 (RDP) open to 0.0.0.0/0
|
||||
```
|
||||
|
||||
<Warning>
|
||||
Security groups with `0.0.0.0/0` on sensitive ports expose resources to the entire internet and should be restricted immediately.
|
||||
</Warning>
|
||||
|
||||
## Step 4: Check for Unencrypted Data
|
||||
|
||||
Scan for unencrypted storage and data transmission:
|
||||
|
||||
```bash
|
||||
prowler aws --checks s3_bucket_default_encryption ebs_volume_encryption rds_instance_storage_encrypted
|
||||
```
|
||||
|
||||
Key checks:
|
||||
* S3 bucket default encryption disabled
|
||||
* EBS volumes without encryption
|
||||
* RDS instances with unencrypted storage
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 20 showing encryption findings - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 5: CloudTrail Threat Detection
|
||||
|
||||
Enable CloudTrail event analysis to detect suspicious activity:
|
||||
|
||||
```bash
|
||||
prowler aws --services cloudtrail
|
||||
```
|
||||
|
||||
Prowler checks for:
|
||||
* CloudTrail disabled in regions
|
||||
* Log file validation disabled
|
||||
* S3 bucket not encrypted
|
||||
* CloudWatch logging not configured
|
||||
|
||||
<Tip>
|
||||
CloudTrail provides audit logs of API calls. Proper configuration is essential for threat detection and incident response.
|
||||
</Tip>
|
||||
|
||||
## Step 6: Analyze IAM Security Risks
|
||||
|
||||
Identify IAM misconfigurations that could lead to privilege escalation:
|
||||
|
||||
```bash
|
||||
prowler aws --services iam --severity critical high
|
||||
```
|
||||
|
||||
Look for:
|
||||
* Root account usage
|
||||
* IAM users without MFA
|
||||
* Overly permissive IAM policies (e.g., `*:*`)
|
||||
* Inactive credentials not rotated
|
||||
|
||||
Example critical finding:
|
||||
```
|
||||
IAM user with administrative privileges without MFA enabled
|
||||
```
|
||||
|
||||
## Step 7: Generate a Threat-Focused Report
|
||||
|
||||
Create a filtered report with only security threats:
|
||||
|
||||
```bash
|
||||
prowler aws --severity critical high --status FAIL -o html json
|
||||
```
|
||||
|
||||
This generates reports containing only:
|
||||
* Critical and high-severity findings
|
||||
* Failed checks (PASS checks excluded)
|
||||
|
||||
Review the HTML report:
|
||||
|
||||
```bash
|
||||
open output/prowler-output-*.html
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 25 showing threat-focused report - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 8: Prioritize Findings
|
||||
|
||||
Categorize findings by risk level:
|
||||
|
||||
**Critical Priority (Address Immediately):**
|
||||
* S3 buckets with public write access
|
||||
* Root account without MFA
|
||||
* Database instances publicly accessible
|
||||
* Security groups open to `0.0.0.0/0` on sensitive ports
|
||||
|
||||
**High Priority (Address Soon):**
|
||||
* Unencrypted storage volumes
|
||||
* CloudTrail logging disabled
|
||||
* IAM users without MFA
|
||||
* Overly permissive IAM policies
|
||||
|
||||
**Medium Priority (Address as Resources Allow):**
|
||||
* Old access keys not rotated
|
||||
* S3 bucket logging disabled
|
||||
* VPC flow logs not enabled
|
||||
|
||||
## Step 9: Export Findings for Remediation
|
||||
|
||||
Export findings to CSV for tracking:
|
||||
|
||||
```bash
|
||||
prowler aws --severity critical high --status FAIL -o csv
|
||||
```
|
||||
|
||||
Share the CSV with your security team for remediation tracking.
|
||||
|
||||
## Verification Steps
|
||||
|
||||
Confirm successful lab completion:
|
||||
|
||||
1. Identified publicly exposed resources
|
||||
2. Detected insecure security group configurations
|
||||
3. Found unencrypted data storage
|
||||
4. Reviewed CloudTrail security settings
|
||||
5. Analyzed IAM security risks
|
||||
6. Generated threat-focused reports
|
||||
7. Prioritized findings by risk level
|
||||
|
||||
## Expected Outcomes
|
||||
|
||||
After completing this lab, you should:
|
||||
|
||||
* Understand common AWS security threats
|
||||
* Know how to identify exposed resources
|
||||
* Be able to prioritize security findings
|
||||
* Have generated threat detection reports
|
||||
|
||||
## Remediation Examples
|
||||
|
||||
**Example 1: Remove public access from S3 bucket**
|
||||
```bash
|
||||
aws s3api put-public-access-block \
|
||||
--bucket my-bucket \
|
||||
--public-access-block-configuration \
|
||||
"BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
|
||||
```
|
||||
|
||||
**Example 2: Restrict security group rule**
|
||||
```bash
|
||||
aws ec2 revoke-security-group-ingress \
|
||||
--group-id sg-12345678 \
|
||||
--protocol tcp \
|
||||
--port 22 \
|
||||
--cidr 0.0.0.0/0
|
||||
```
|
||||
|
||||
**Example 3: Enable S3 bucket encryption**
|
||||
```bash
|
||||
aws s3api put-bucket-encryption \
|
||||
--bucket my-bucket \
|
||||
--server-side-encryption-configuration \
|
||||
'{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Issue:** Too many findings to review
|
||||
* **Solution:** Use `--severity critical high` to focus on the most important issues first
|
||||
|
||||
**Issue:** Don't understand a finding
|
||||
* **Solution:** Use `--describe-check <check-id>` to get detailed information
|
||||
|
||||
**Issue:** Need to share findings with team
|
||||
* **Solution:** Export to CSV or JSON and use collaboration tools
|
||||
|
||||
## Next Steps
|
||||
|
||||
Continue to [Lab 3: Custom Checks with Prowler](/workshop/lab-03-custom-checks) to learn how to create organization-specific security checks.
|
||||
|
||||
## Additional Resources
|
||||
|
||||
* [AWS Threat Detection Guide](/user-guide/providers/aws/threat-detection)
|
||||
* [Security Best Practices](/user-guide/providers/aws/getting-started-aws)
|
||||
* [Prowler Check Reference](https://hub.prowler.com)
|
||||
@@ -0,0 +1,359 @@
|
||||
---
|
||||
title: "Lab 3: Custom Checks with Prowler"
|
||||
description: "Create organization-specific security checks and customize Prowler for your security requirements"
|
||||
---
|
||||
|
||||
<Note>
|
||||
**Tags:** `workshop` `aws` `custom-checks` `advanced` `development`
|
||||
</Note>
|
||||
|
||||
# Lab 3: Custom Checks with Prowler
|
||||
|
||||
Learn to create custom security checks tailored to your organization's specific security policies and compliance requirements.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* Completion of [Lab 1: Getting Started with Prowler CLI](/workshop/lab-01-getting-started)
|
||||
* Prowler CLI installed from source (for custom check development)
|
||||
* Python 3.9 or higher
|
||||
* Basic Python programming knowledge
|
||||
* Understanding of AWS SDK (boto3)
|
||||
* Text editor or IDE (VS Code, PyCharm)
|
||||
|
||||
**Estimated Time:** 60 minutes
|
||||
|
||||
## Lab Objectives
|
||||
|
||||
By completing this lab, you will:
|
||||
|
||||
* Understand Prowler's check structure
|
||||
* Create a custom security check
|
||||
* Test and validate custom checks
|
||||
* Use custom check metadata
|
||||
* Integrate custom checks into scans
|
||||
|
||||
## Step 1: Install Prowler from Source
|
||||
|
||||
To develop custom checks, install Prowler from source:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/prowler-cloud/prowler
|
||||
cd prowler
|
||||
pip install poetry
|
||||
poetry install
|
||||
```
|
||||
|
||||
Activate the virtual environment:
|
||||
|
||||
```bash
|
||||
poetry shell
|
||||
```
|
||||
|
||||
Verify installation:
|
||||
|
||||
```bash
|
||||
prowler -v
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 29 showing source installation - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 2: Understand Check Structure
|
||||
|
||||
Prowler checks are Python files located in:
|
||||
```
|
||||
prowler/providers/<provider>/services/<service>/
|
||||
```
|
||||
|
||||
Example check structure:
|
||||
```
|
||||
prowler/providers/aws/services/s3/s3_bucket_custom_check/
|
||||
├── s3_bucket_custom_check.py # Check logic
|
||||
└── s3_bucket_custom_check.metadata.json # Check metadata
|
||||
```
|
||||
|
||||
## Step 3: Create a Custom Check Directory
|
||||
|
||||
Create a custom check to verify S3 buckets have specific naming conventions:
|
||||
|
||||
```bash
|
||||
mkdir -p prowler/providers/aws/services/s3/s3_bucket_naming_convention
|
||||
cd prowler/providers/aws/services/s3/s3_bucket_naming_convention
|
||||
```
|
||||
|
||||
## Step 4: Write the Check Logic
|
||||
|
||||
Create `s3_bucket_naming_convention.py`:
|
||||
|
||||
```python
|
||||
from prowler.lib.check.models import Check, Check_Report_AWS
|
||||
from prowler.providers.aws.services.s3.s3_client import s3_client
|
||||
|
||||
class s3_bucket_naming_convention(Check):
|
||||
def execute(self):
|
||||
findings = []
|
||||
# Define your organization's naming pattern
|
||||
naming_pattern = "company-"
|
||||
|
||||
for bucket in s3_client.buckets:
|
||||
report = Check_Report_AWS(self.metadata())
|
||||
report.region = bucket.region
|
||||
report.resource_id = bucket.name
|
||||
report.resource_arn = bucket.arn
|
||||
report.resource_tags = bucket.tags
|
||||
|
||||
# Check if bucket name follows naming convention
|
||||
if bucket.name.startswith(naming_pattern):
|
||||
report.status = "PASS"
|
||||
report.status_extended = f"S3 bucket {bucket.name} follows naming convention."
|
||||
else:
|
||||
report.status = "FAIL"
|
||||
report.status_extended = f"S3 bucket {bucket.name} does not follow naming convention (should start with '{naming_pattern}')."
|
||||
|
||||
findings.append(report)
|
||||
|
||||
return findings
|
||||
```
|
||||
|
||||
<Tip>
|
||||
Customize the `naming_pattern` variable to match your organization's requirements (e.g., "prod-", "dev-", "projectname-").
|
||||
</Tip>
|
||||
|
||||
## Step 5: Create Check Metadata
|
||||
|
||||
Create `s3_bucket_naming_convention.metadata.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"Provider": "aws",
|
||||
"CheckID": "s3_bucket_naming_convention",
|
||||
"CheckTitle": "Check if S3 buckets follow naming convention",
|
||||
"CheckType": ["Software and Configuration Checks"],
|
||||
"ServiceName": "s3",
|
||||
"SubServiceName": "",
|
||||
"ResourceIdTemplate": "arn:aws:s3:::bucket_name",
|
||||
"Severity": "low",
|
||||
"ResourceType": "AwsS3Bucket",
|
||||
"Description": "Ensure S3 buckets follow the organization's naming convention for consistency and management.",
|
||||
"Risk": "S3 buckets not following naming conventions may lead to management difficulties and confusion.",
|
||||
"RelatedUrl": "https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html",
|
||||
"Remediation": {
|
||||
"Code": {
|
||||
"CLI": "",
|
||||
"NativeIaC": "",
|
||||
"Other": "Rename the S3 bucket to follow the organization's naming convention or update bucket policies.",
|
||||
"Terraform": ""
|
||||
},
|
||||
"Recommendation": {
|
||||
"Text": "Ensure all S3 buckets follow the defined naming convention for your organization.",
|
||||
"Url": "https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html"
|
||||
}
|
||||
},
|
||||
"Categories": [
|
||||
"forensics-ready"
|
||||
],
|
||||
"DependsOn": [],
|
||||
"RelatedTo": [],
|
||||
"Notes": "This is a custom check created for organization-specific requirements."
|
||||
}
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 33 showing metadata structure - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 6: Test the Custom Check
|
||||
|
||||
Run only your custom check:
|
||||
|
||||
```bash
|
||||
prowler aws --checks s3_bucket_naming_convention
|
||||
```
|
||||
|
||||
Review the output to verify:
|
||||
* Check executes without errors
|
||||
* Findings are generated for each S3 bucket
|
||||
* Status is correct (PASS/FAIL) based on naming convention
|
||||
|
||||
## Step 7: Create a Custom Check for EC2 Instance Tags
|
||||
|
||||
Create another custom check to enforce EC2 tagging policies:
|
||||
|
||||
```bash
|
||||
mkdir -p prowler/providers/aws/services/ec2/ec2_instance_required_tags
|
||||
cd prowler/providers/aws/services/ec2/ec2_instance_required_tags
|
||||
```
|
||||
|
||||
Create `ec2_instance_required_tags.py`:
|
||||
|
||||
```python
|
||||
from prowler.lib.check.models import Check, Check_Report_AWS
|
||||
from prowler.providers.aws.services.ec2.ec2_client import ec2_client
|
||||
|
||||
class ec2_instance_required_tags(Check):
|
||||
def execute(self):
|
||||
findings = []
|
||||
# Define required tags
|
||||
required_tags = ["Environment", "Owner", "CostCenter"]
|
||||
|
||||
for instance in ec2_client.instances:
|
||||
report = Check_Report_AWS(self.metadata())
|
||||
report.region = instance.region
|
||||
report.resource_id = instance.id
|
||||
report.resource_arn = instance.arn
|
||||
report.resource_tags = instance.tags
|
||||
|
||||
# Get instance tag keys
|
||||
instance_tag_keys = [tag["Key"] for tag in instance.tags] if instance.tags else []
|
||||
|
||||
# Check if all required tags are present
|
||||
missing_tags = [tag for tag in required_tags if tag not in instance_tag_keys]
|
||||
|
||||
if not missing_tags:
|
||||
report.status = "PASS"
|
||||
report.status_extended = f"EC2 instance {instance.id} has all required tags."
|
||||
else:
|
||||
report.status = "FAIL"
|
||||
report.status_extended = f"EC2 instance {instance.id} is missing required tags: {', '.join(missing_tags)}."
|
||||
|
||||
findings.append(report)
|
||||
|
||||
return findings
|
||||
```
|
||||
|
||||
Create `ec2_instance_required_tags.metadata.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"Provider": "aws",
|
||||
"CheckID": "ec2_instance_required_tags",
|
||||
"CheckTitle": "Check if EC2 instances have required tags",
|
||||
"CheckType": ["Software and Configuration Checks"],
|
||||
"ServiceName": "ec2",
|
||||
"SubServiceName": "",
|
||||
"ResourceIdTemplate": "arn:aws:ec2:region:account-id:instance/instance-id",
|
||||
"Severity": "medium",
|
||||
"ResourceType": "AwsEc2Instance",
|
||||
"Description": "Ensure EC2 instances have required tags for proper resource management and cost allocation.",
|
||||
"Risk": "EC2 instances without required tags may lead to difficulties in cost tracking, ownership identification, and resource management.",
|
||||
"RelatedUrl": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html",
|
||||
"Remediation": {
|
||||
"Code": {
|
||||
"CLI": "aws ec2 create-tags --resources <instance-id> --tags Key=Environment,Value=<value> Key=Owner,Value=<value> Key=CostCenter,Value=<value>",
|
||||
"NativeIaC": "",
|
||||
"Other": "",
|
||||
"Terraform": "resource \"aws_ec2_tag\" \"example\" {\n resource_id = aws_instance.example.id\n key = \"Environment\"\n value = \"Production\"\n}"
|
||||
},
|
||||
"Recommendation": {
|
||||
"Text": "Add the required tags (Environment, Owner, CostCenter) to all EC2 instances.",
|
||||
"Url": "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html"
|
||||
}
|
||||
},
|
||||
"Categories": [
|
||||
"tagging"
|
||||
],
|
||||
"DependsOn": [],
|
||||
"RelatedTo": [],
|
||||
"Notes": "Customize the required_tags list in the check code to match your organization's tagging policy."
|
||||
}
|
||||
```
|
||||
|
||||
## Step 8: Test Multiple Custom Checks
|
||||
|
||||
Run both custom checks together:
|
||||
|
||||
```bash
|
||||
prowler aws --checks s3_bucket_naming_convention ec2_instance_required_tags
|
||||
```
|
||||
|
||||
## Step 9: Create a Custom Checks Group
|
||||
|
||||
Create a file to group your custom checks:
|
||||
|
||||
Create `prowler/config/custom_checks.yaml`:
|
||||
|
||||
```yaml
|
||||
custom-checks:
|
||||
- s3_bucket_naming_convention
|
||||
- ec2_instance_required_tags
|
||||
```
|
||||
|
||||
Run all custom checks:
|
||||
|
||||
```bash
|
||||
prowler aws --checks-file prowler/config/custom_checks.yaml
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 38 showing custom checks output - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 10: Validate Check Metadata
|
||||
|
||||
Prowler includes metadata validation. Ensure your metadata follows guidelines:
|
||||
|
||||
```bash
|
||||
python -m prowler.lib.check.check_metadata_validator
|
||||
```
|
||||
|
||||
This validates:
|
||||
* Required metadata fields are present
|
||||
* Severity values are valid
|
||||
* URLs are properly formatted
|
||||
* JSON structure is correct
|
||||
|
||||
## Verification Steps
|
||||
|
||||
Confirm successful lab completion:
|
||||
|
||||
1. Prowler installed from source
|
||||
2. Custom S3 naming convention check created
|
||||
3. Custom EC2 tagging check created
|
||||
4. Both checks execute successfully
|
||||
5. Metadata files are properly formatted
|
||||
6. Custom checks grouped for easy execution
|
||||
|
||||
## Expected Outcomes
|
||||
|
||||
After completing this lab, you should:
|
||||
|
||||
* Understand Prowler's check architecture
|
||||
* Be able to create custom security checks
|
||||
* Know how to write check metadata
|
||||
* Be capable of testing and validating checks
|
||||
* Have created reusable custom security policies
|
||||
|
||||
## Best Practices for Custom Checks
|
||||
|
||||
1. **Follow naming conventions:** Use descriptive check IDs (e.g., `service_resource_requirement`)
|
||||
2. **Set appropriate severity:** Match severity to the security impact
|
||||
3. **Provide clear descriptions:** Help users understand what the check validates
|
||||
4. **Include remediation guidance:** Provide actionable steps to fix findings
|
||||
5. **Test thoroughly:** Verify checks work across different AWS regions and account configurations
|
||||
6. **Document assumptions:** Note any specific requirements or limitations
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Issue:** Check not found when running
|
||||
* **Solution:** Ensure the check directory and files follow the correct naming convention and location
|
||||
|
||||
**Issue:** Import errors in check code
|
||||
* **Solution:** Verify you're using the Poetry virtual environment (`poetry shell`)
|
||||
|
||||
**Issue:** Metadata validation fails
|
||||
* **Solution:** Review the metadata format against Prowler's schema requirements
|
||||
|
||||
**Issue:** Check returns no findings
|
||||
* **Solution:** Add print statements or use a debugger to verify the service client has data
|
||||
|
||||
## Next Steps
|
||||
|
||||
Continue to [Lab 4: Multi-Cloud Security with Prowler (Azure)](/workshop/lab-04-azure-multicloud) to extend security monitoring to Azure environments.
|
||||
|
||||
## Additional Resources
|
||||
|
||||
* [Custom Checks Development Guide](/developer-guide/checks)
|
||||
* [Check Metadata Guidelines](/developer-guide/check-metadata-guidelines)
|
||||
* [Prowler Development Documentation](/developer-guide/introduction)
|
||||
* [Prowler Check Kreator](/user-guide/cli/tutorials/prowler-check-kreator)
|
||||
@@ -0,0 +1,346 @@
|
||||
---
|
||||
title: "Lab 4: Multi-Cloud Security with Prowler (Azure)"
|
||||
description: "Extend security monitoring to Azure environments using Prowler's multi-cloud capabilities"
|
||||
---
|
||||
|
||||
<Note>
|
||||
**Tags:** `workshop` `azure` `multi-cloud` `intermediate` `authentication`
|
||||
</Note>
|
||||
|
||||
# Lab 4: Multi-Cloud Security with Prowler (Azure)
|
||||
|
||||
Learn to secure Azure environments using Prowler's multi-cloud security assessment capabilities.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* Prowler CLI installed ([Lab 1](/workshop/lab-01-getting-started))
|
||||
* Active Azure subscription
|
||||
* Azure CLI installed
|
||||
* Azure account with appropriate permissions (Reader role minimum)
|
||||
* Basic understanding of Azure services
|
||||
|
||||
**Estimated Time:** 45 minutes
|
||||
|
||||
## Lab Objectives
|
||||
|
||||
By completing this lab, you will:
|
||||
|
||||
* Configure Azure authentication for Prowler
|
||||
* Run security assessments on Azure subscriptions
|
||||
* Understand Azure-specific security checks
|
||||
* Compare security findings across cloud providers
|
||||
* Implement multi-cloud security strategies
|
||||
|
||||
## Step 1: Install Azure CLI
|
||||
|
||||
Install Azure CLI if not already present:
|
||||
|
||||
**macOS:**
|
||||
```bash
|
||||
brew install azure-cli
|
||||
```
|
||||
|
||||
**Linux:**
|
||||
```bash
|
||||
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
|
||||
```
|
||||
|
||||
**Windows:**
|
||||
```powershell
|
||||
winget install Microsoft.AzureCLI
|
||||
```
|
||||
|
||||
Verify installation:
|
||||
```bash
|
||||
az --version
|
||||
```
|
||||
|
||||
## Step 2: Authenticate to Azure
|
||||
|
||||
Sign in to Azure:
|
||||
|
||||
```bash
|
||||
az login
|
||||
```
|
||||
|
||||
This opens a browser window for authentication.
|
||||
|
||||
Verify authentication:
|
||||
```bash
|
||||
az account show
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```json
|
||||
{
|
||||
"id": "12345678-1234-1234-1234-123456789012",
|
||||
"name": "My Subscription",
|
||||
"tenantId": "87654321-4321-4321-4321-210987654321",
|
||||
"state": "Enabled"
|
||||
}
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 43 showing Azure authentication - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 3: List Azure Subscriptions
|
||||
|
||||
If you have multiple subscriptions, list them:
|
||||
|
||||
```bash
|
||||
az account list --output table
|
||||
```
|
||||
|
||||
Set the active subscription:
|
||||
```bash
|
||||
az account set --subscription "subscription-id"
|
||||
```
|
||||
|
||||
## Step 4: Configure Azure Service Principal (Optional)
|
||||
|
||||
For automated scans, create a service principal:
|
||||
|
||||
```bash
|
||||
az ad sp create-for-rbac --name "prowler-scanner" --role Reader --scopes /subscriptions/{subscription-id}
|
||||
```
|
||||
|
||||
This returns:
|
||||
```json
|
||||
{
|
||||
"appId": "app-id",
|
||||
"displayName": "prowler-scanner",
|
||||
"password": "password",
|
||||
"tenant": "tenant-id"
|
||||
}
|
||||
```
|
||||
|
||||
<Warning>
|
||||
Store service principal credentials securely. These provide programmatic access to your Azure subscription.
|
||||
</Warning>
|
||||
|
||||
Export credentials as environment variables:
|
||||
```bash
|
||||
export AZURE_CLIENT_ID="app-id"
|
||||
export AZURE_CLIENT_SECRET="password"
|
||||
export AZURE_TENANT_ID="tenant-id"
|
||||
export AZURE_SUBSCRIPTION_ID="subscription-id"
|
||||
```
|
||||
|
||||
## Step 5: Run Your First Azure Scan
|
||||
|
||||
Execute Prowler against Azure:
|
||||
|
||||
```bash
|
||||
prowler azure
|
||||
```
|
||||
|
||||
This command:
|
||||
* Uses Azure CLI credentials (or service principal if configured)
|
||||
* Scans the active subscription
|
||||
* Runs all Azure security checks
|
||||
* Generates output in multiple formats
|
||||
|
||||
<Note>
|
||||
Azure scans typically take 5-10 minutes depending on resource count.
|
||||
</Note>
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 47 showing Azure scan execution - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 6: Scan Specific Azure Services
|
||||
|
||||
Run targeted scans for specific services:
|
||||
|
||||
```bash
|
||||
prowler azure --services storage network
|
||||
```
|
||||
|
||||
This focuses on:
|
||||
* Azure Storage accounts
|
||||
* Virtual networks
|
||||
* Network security groups
|
||||
|
||||
## Step 7: Analyze Azure Security Findings
|
||||
|
||||
Review Azure-specific security checks:
|
||||
|
||||
**Storage Account Security:**
|
||||
* Public blob access disabled
|
||||
* Secure transfer required (HTTPS)
|
||||
* Storage encryption enabled
|
||||
* Soft delete enabled
|
||||
|
||||
**Network Security:**
|
||||
* Network security groups properly configured
|
||||
* No overly permissive rules
|
||||
* DDoS protection enabled
|
||||
* Network watcher enabled
|
||||
|
||||
**Identity and Access:**
|
||||
* Multi-factor authentication enabled
|
||||
* Conditional access policies configured
|
||||
* Privileged identity management enabled
|
||||
|
||||
Open the HTML report:
|
||||
```bash
|
||||
open output/prowler-output-azure-*.html
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 50 showing Azure findings report - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 8: Compare AWS and Azure Security Posture
|
||||
|
||||
If you completed Lab 1, compare security findings:
|
||||
|
||||
**AWS findings:**
|
||||
```bash
|
||||
cat output/prowler-output-aws-*.csv | wc -l
|
||||
```
|
||||
|
||||
**Azure findings:**
|
||||
```bash
|
||||
cat output/prowler-output-azure-*.csv | wc -l
|
||||
```
|
||||
|
||||
Key comparison metrics:
|
||||
* Total findings by severity
|
||||
* Service coverage
|
||||
* Compliance status
|
||||
* Resource exposure
|
||||
|
||||
## Step 9: Multi-Cloud Security Dashboard
|
||||
|
||||
Generate a combined security view:
|
||||
|
||||
Create a directory for multi-cloud reports:
|
||||
```bash
|
||||
mkdir -p multi-cloud-reports
|
||||
cp output/prowler-output-aws-*.json multi-cloud-reports/
|
||||
cp output/prowler-output-azure-*.json multi-cloud-reports/
|
||||
```
|
||||
|
||||
<Tip>
|
||||
Use Prowler Cloud or custom dashboards to visualize multi-cloud security posture in a unified interface.
|
||||
</Tip>
|
||||
|
||||
## Step 10: Azure-Specific Remediation
|
||||
|
||||
Example remediations for common Azure findings:
|
||||
|
||||
**Enable secure transfer for storage account:**
|
||||
```bash
|
||||
az storage account update \
|
||||
--name mystorageaccount \
|
||||
--resource-group myresourcegroup \
|
||||
--https-only true
|
||||
```
|
||||
|
||||
**Enable storage encryption:**
|
||||
```bash
|
||||
az storage account update \
|
||||
--name mystorageaccount \
|
||||
--resource-group myresourcegroup \
|
||||
--encryption-services blob
|
||||
```
|
||||
|
||||
**Disable public blob access:**
|
||||
```bash
|
||||
az storage account update \
|
||||
--name mystorageaccount \
|
||||
--resource-group myresourcegroup \
|
||||
--allow-blob-public-access false
|
||||
```
|
||||
|
||||
**Update network security group rule:**
|
||||
```bash
|
||||
az network nsg rule update \
|
||||
--resource-group myresourcegroup \
|
||||
--nsg-name mynsg \
|
||||
--name mynsgrule \
|
||||
--source-address-prefixes 10.0.0.0/16
|
||||
```
|
||||
|
||||
## Step 11: Scan Multiple Azure Subscriptions
|
||||
|
||||
Scan all subscriptions in your tenant:
|
||||
|
||||
```bash
|
||||
prowler azure --subscription-ids subscription-id-1 subscription-id-2
|
||||
```
|
||||
|
||||
Or scan all accessible subscriptions:
|
||||
```bash
|
||||
prowler azure --az-cli-auth
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 56 showing multi-subscription scan - to be added]
|
||||
</Note>
|
||||
|
||||
## Verification Steps
|
||||
|
||||
Confirm successful lab completion:
|
||||
|
||||
1. Azure CLI installed and authenticated
|
||||
2. First Azure scan completed successfully
|
||||
3. Azure security findings reviewed
|
||||
4. Service-specific scans executed
|
||||
5. Multi-cloud comparison performed
|
||||
6. Azure-specific remediations understood
|
||||
|
||||
## Expected Outcomes
|
||||
|
||||
After completing this lab, you should:
|
||||
|
||||
* Be able to authenticate Prowler with Azure
|
||||
* Understand Azure security checks
|
||||
* Know how to scan multiple subscriptions
|
||||
* Have compared security posture across AWS and Azure
|
||||
* Be familiar with Azure-specific remediation commands
|
||||
|
||||
## Common Azure Security Findings
|
||||
|
||||
**Storage Accounts:**
|
||||
* Public blob access enabled
|
||||
* Secure transfer (HTTPS) not required
|
||||
* Storage encryption disabled
|
||||
* Logging not configured
|
||||
|
||||
**Virtual Networks:**
|
||||
* Network security groups allow 0.0.0.0/0 access
|
||||
* DDoS protection not enabled
|
||||
* Network watcher not configured
|
||||
|
||||
**Identity:**
|
||||
* MFA not enabled for all users
|
||||
* Guest users have excessive permissions
|
||||
* Password policies are weak
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Issue:** Azure authentication fails
|
||||
* **Solution:** Run `az login` and ensure you have the correct subscription selected
|
||||
|
||||
**Issue:** Permission errors during scan
|
||||
* **Solution:** Ensure your account or service principal has Reader role at subscription level
|
||||
|
||||
**Issue:** Subscription not found
|
||||
* **Solution:** Verify subscription ID with `az account list` and check it's enabled
|
||||
|
||||
**Issue:** Slow scan performance
|
||||
* **Solution:** Use `--services` flag to scan specific services instead of all
|
||||
|
||||
## Next Steps
|
||||
|
||||
Continue to [Lab 5: Multi-Cloud Security with Prowler (GCP)](/workshop/lab-05-gcp-multicloud) to add Google Cloud Platform to your multi-cloud security monitoring.
|
||||
|
||||
## Additional Resources
|
||||
|
||||
* [Azure Getting Started Guide](/user-guide/providers/azure/getting-started-azure)
|
||||
* [Azure Authentication Methods](/user-guide/providers/azure/authentication)
|
||||
* [Create Prowler Service Principal](/user-guide/providers/azure/create-prowler-service-principal)
|
||||
* [Azure Subscriptions Management](/user-guide/providers/azure/subscriptions)
|
||||
@@ -0,0 +1,377 @@
|
||||
---
|
||||
title: "Lab 5: Multi-Cloud Security with Prowler (GCP)"
|
||||
description: "Complete your multi-cloud security coverage by adding Google Cloud Platform assessments"
|
||||
---
|
||||
|
||||
<Note>
|
||||
**Tags:** `workshop` `gcp` `multi-cloud` `intermediate` `authentication`
|
||||
</Note>
|
||||
|
||||
# Lab 5: Multi-Cloud Security with Prowler (GCP)
|
||||
|
||||
Learn to secure Google Cloud Platform environments and achieve comprehensive multi-cloud security coverage with Prowler.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* Prowler CLI installed ([Lab 1](/workshop/lab-01-getting-started))
|
||||
* Active GCP project
|
||||
* Google Cloud SDK (gcloud) installed
|
||||
* GCP account with appropriate permissions (Viewer role minimum)
|
||||
* Basic understanding of GCP services
|
||||
|
||||
**Estimated Time:** 45 minutes
|
||||
|
||||
## Lab Objectives
|
||||
|
||||
By completing this lab, you will:
|
||||
|
||||
* Configure GCP authentication for Prowler
|
||||
* Run security assessments on GCP projects
|
||||
* Understand GCP-specific security checks
|
||||
* Achieve comprehensive multi-cloud security coverage (AWS, Azure, GCP)
|
||||
* Implement unified security policies across cloud providers
|
||||
|
||||
## Step 1: Install Google Cloud SDK
|
||||
|
||||
Install gcloud CLI if not already present:
|
||||
|
||||
**macOS:**
|
||||
```bash
|
||||
brew install google-cloud-sdk
|
||||
```
|
||||
|
||||
**Linux:**
|
||||
```bash
|
||||
curl https://sdk.cloud.google.com | bash
|
||||
exec -l $SHELL
|
||||
```
|
||||
|
||||
**Windows:**
|
||||
Download and install from: https://cloud.google.com/sdk/docs/install
|
||||
|
||||
Verify installation:
|
||||
```bash
|
||||
gcloud --version
|
||||
```
|
||||
|
||||
## Step 2: Authenticate to GCP
|
||||
|
||||
Initialize gcloud and authenticate:
|
||||
|
||||
```bash
|
||||
gcloud init
|
||||
```
|
||||
|
||||
This prompts you to:
|
||||
1. Log in to your Google account
|
||||
2. Select or create a GCP project
|
||||
3. Configure default region/zone (optional)
|
||||
|
||||
Verify authentication:
|
||||
```bash
|
||||
gcloud auth list
|
||||
```
|
||||
|
||||
Display active project:
|
||||
```bash
|
||||
gcloud config get-value project
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 60 showing GCP authentication - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 3: Configure Application Default Credentials
|
||||
|
||||
Prowler uses Application Default Credentials (ADC):
|
||||
|
||||
```bash
|
||||
gcloud auth application-default login
|
||||
```
|
||||
|
||||
This creates credentials file at:
|
||||
* **Linux/macOS:** `~/.config/gcloud/application_default_credentials.json`
|
||||
* **Windows:** `%APPDATA%\gcloud\application_default_credentials.json`
|
||||
|
||||
## Step 4: Set Up Service Account (Optional)
|
||||
|
||||
For automated scans, create a service account:
|
||||
|
||||
```bash
|
||||
# Create service account
|
||||
gcloud iam service-accounts create prowler-scanner \
|
||||
--display-name="Prowler Security Scanner"
|
||||
|
||||
# Get project ID
|
||||
PROJECT_ID=$(gcloud config get-value project)
|
||||
|
||||
# Grant Viewer role
|
||||
gcloud projects add-iam-policy-binding $PROJECT_ID \
|
||||
--member="serviceAccount:prowler-scanner@${PROJECT_ID}.iam.gserviceaccount.com" \
|
||||
--role="roles/viewer"
|
||||
|
||||
# Generate key file
|
||||
gcloud iam service-accounts keys create ~/prowler-credentials.json \
|
||||
--iam-account=prowler-scanner@${PROJECT_ID}.iam.gserviceaccount.com
|
||||
```
|
||||
|
||||
<Warning>
|
||||
Store service account key files securely. These provide programmatic access to your GCP project.
|
||||
</Warning>
|
||||
|
||||
Use service account credentials:
|
||||
```bash
|
||||
export GOOGLE_APPLICATION_CREDENTIALS=~/prowler-credentials.json
|
||||
```
|
||||
|
||||
## Step 5: Run Your First GCP Scan
|
||||
|
||||
Execute Prowler against GCP:
|
||||
|
||||
```bash
|
||||
prowler gcp
|
||||
```
|
||||
|
||||
This command:
|
||||
* Uses Application Default Credentials (or service account)
|
||||
* Scans the active project
|
||||
* Runs all GCP security checks
|
||||
* Generates output in multiple formats
|
||||
|
||||
<Note>
|
||||
GCP scans typically take 5-10 minutes depending on resource count.
|
||||
</Note>
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 65 showing GCP scan execution - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 6: Scan Specific GCP Projects
|
||||
|
||||
Scan a specific project:
|
||||
|
||||
```bash
|
||||
prowler gcp --project-id my-project-id
|
||||
```
|
||||
|
||||
Scan multiple projects:
|
||||
```bash
|
||||
prowler gcp --project-id project-1 project-2 project-3
|
||||
```
|
||||
|
||||
## Step 7: Scan Specific GCP Services
|
||||
|
||||
Run targeted scans for specific services:
|
||||
|
||||
```bash
|
||||
prowler gcp --services storage compute iam
|
||||
```
|
||||
|
||||
This focuses on:
|
||||
* Cloud Storage buckets
|
||||
* Compute Engine instances
|
||||
* IAM policies and permissions
|
||||
|
||||
## Step 8: Analyze GCP Security Findings
|
||||
|
||||
Review GCP-specific security checks:
|
||||
|
||||
**Cloud Storage Security:**
|
||||
* Buckets not publicly accessible
|
||||
* Uniform bucket-level access enabled
|
||||
* Encryption at rest enabled
|
||||
* Versioning enabled
|
||||
|
||||
**Compute Engine Security:**
|
||||
* OS Login enabled
|
||||
* Serial port access disabled
|
||||
* Shielded VMs enabled
|
||||
* IP forwarding disabled
|
||||
|
||||
**IAM Security:**
|
||||
* Service accounts with minimal permissions
|
||||
* No primitive roles (Owner, Editor, Viewer) assigned to users
|
||||
* Service account keys rotated regularly
|
||||
* Cloud Identity-Aware Proxy (IAP) enabled
|
||||
|
||||
Open the HTML report:
|
||||
```bash
|
||||
open output/prowler-output-gcp-*.html
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 69 showing GCP findings report - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 9: Multi-Cloud Security Overview
|
||||
|
||||
You now have security coverage across three major cloud providers:
|
||||
|
||||
Create a comprehensive multi-cloud report directory:
|
||||
|
||||
```bash
|
||||
mkdir -p multi-cloud-security-reports
|
||||
cp output/prowler-output-aws-*.json multi-cloud-security-reports/
|
||||
cp output/prowler-output-azure-*.json multi-cloud-security-reports/
|
||||
cp output/prowler-output-gcp-*.json multi-cloud-security-reports/
|
||||
```
|
||||
|
||||
Compare security posture metrics:
|
||||
|
||||
```bash
|
||||
# Count findings by provider
|
||||
echo "AWS findings:"
|
||||
jq '.findings | length' multi-cloud-security-reports/prowler-output-aws-*.json
|
||||
|
||||
echo "Azure findings:"
|
||||
jq '.findings | length' multi-cloud-security-reports/prowler-output-azure-*.json
|
||||
|
||||
echo "GCP findings:"
|
||||
jq '.findings | length' multi-cloud-security-reports/prowler-output-gcp-*.json
|
||||
```
|
||||
|
||||
## Step 10: GCP-Specific Remediation
|
||||
|
||||
Example remediations for common GCP findings:
|
||||
|
||||
**Enable uniform bucket-level access:**
|
||||
```bash
|
||||
gsutil uniformbucketlevelaccess set on gs://bucket-name
|
||||
```
|
||||
|
||||
**Disable public access to bucket:**
|
||||
```bash
|
||||
gsutil iam ch -d allUsers gs://bucket-name
|
||||
gsutil iam ch -d allAuthenticatedUsers gs://bucket-name
|
||||
```
|
||||
|
||||
**Enable OS Login on project:**
|
||||
```bash
|
||||
gcloud compute project-info add-metadata \
|
||||
--metadata enable-oslogin=TRUE
|
||||
```
|
||||
|
||||
**Disable serial port access:**
|
||||
```bash
|
||||
gcloud compute instances add-metadata instance-name \
|
||||
--metadata serial-port-enable=FALSE
|
||||
```
|
||||
|
||||
**Remove primitive role binding:**
|
||||
```bash
|
||||
gcloud projects remove-iam-policy-binding PROJECT_ID \
|
||||
--member='user:email@example.com' \
|
||||
--role='roles/editor'
|
||||
```
|
||||
|
||||
## Step 11: Scan GCP Organization
|
||||
|
||||
If you have organization-level access:
|
||||
|
||||
```bash
|
||||
prowler gcp --organization-id org-id
|
||||
```
|
||||
|
||||
This scans all projects within the organization.
|
||||
|
||||
<Tip>
|
||||
Organization-level scanning requires `resourcemanager.organizations.get` permission at the organization level.
|
||||
</Tip>
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 74 showing organization scan - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 12: Multi-Cloud Security Strategy
|
||||
|
||||
Establish consistent security controls across clouds:
|
||||
|
||||
**Identity and Access:**
|
||||
* Enforce MFA across all providers
|
||||
* Implement least privilege access
|
||||
* Regular access reviews
|
||||
* Centralized identity management
|
||||
|
||||
**Data Protection:**
|
||||
* Encryption at rest and in transit
|
||||
* Regular backups
|
||||
* Data retention policies
|
||||
* Access logging enabled
|
||||
|
||||
**Network Security:**
|
||||
* Zero-trust network architecture
|
||||
* Network segmentation
|
||||
* DDoS protection
|
||||
* Traffic inspection
|
||||
|
||||
**Monitoring and Compliance:**
|
||||
* Centralized logging
|
||||
* Security information and event management (SIEM)
|
||||
* Regular compliance scans
|
||||
* Automated remediation where possible
|
||||
|
||||
## Verification Steps
|
||||
|
||||
Confirm successful lab completion:
|
||||
|
||||
1. Google Cloud SDK installed and authenticated
|
||||
2. First GCP scan completed successfully
|
||||
3. GCP security findings reviewed
|
||||
4. Service-specific scans executed
|
||||
5. Multi-cloud reports collected (AWS, Azure, GCP)
|
||||
6. GCP-specific remediations understood
|
||||
|
||||
## Expected Outcomes
|
||||
|
||||
After completing this lab, you should:
|
||||
|
||||
* Be able to authenticate Prowler with GCP
|
||||
* Understand GCP security checks
|
||||
* Know how to scan multiple projects and organizations
|
||||
* Have achieved multi-cloud security coverage
|
||||
* Be familiar with GCP-specific remediation commands
|
||||
|
||||
## Common GCP Security Findings
|
||||
|
||||
**Cloud Storage:**
|
||||
* Buckets with public access
|
||||
* Uniform bucket-level access not enabled
|
||||
* Versioning disabled
|
||||
* Logging not configured
|
||||
|
||||
**Compute Engine:**
|
||||
* OS Login not enabled
|
||||
* Legacy metadata endpoints enabled
|
||||
* Serial port access enabled
|
||||
* IP forwarding enabled on instances
|
||||
|
||||
**IAM:**
|
||||
* Primitive roles assigned to users
|
||||
* Service account keys not rotated
|
||||
* Over-permissive service accounts
|
||||
* No organization policies enforced
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Issue:** GCP authentication fails
|
||||
* **Solution:** Run `gcloud auth application-default login` and ensure project is set
|
||||
|
||||
**Issue:** Permission errors during scan
|
||||
* **Solution:** Ensure account has Viewer role at project or organization level
|
||||
|
||||
**Issue:** Project not found
|
||||
* **Solution:** Verify project ID with `gcloud projects list` and check it's active
|
||||
|
||||
**Issue:** API not enabled errors
|
||||
* **Solution:** Enable required APIs: `gcloud services enable cloudresourcemanager.googleapis.com`
|
||||
|
||||
## Next Steps
|
||||
|
||||
Continue to [Lab 6: Compliance as Code with Prowler](/workshop/lab-06-compliance-as-code) to learn how to automate compliance reporting across all cloud providers.
|
||||
|
||||
## Additional Resources
|
||||
|
||||
* [GCP Getting Started Guide](/user-guide/providers/gcp/getting-started-gcp)
|
||||
* [GCP Authentication Methods](/user-guide/providers/gcp/authentication)
|
||||
* [GCP Projects Management](/user-guide/providers/gcp/projects)
|
||||
* [GCP Organization Scanning](/user-guide/providers/gcp/organization)
|
||||
@@ -0,0 +1,465 @@
|
||||
---
|
||||
title: "Lab 6: Compliance as Code with Prowler"
|
||||
description: "Automate compliance reporting and validation against industry standards and regulatory frameworks"
|
||||
---
|
||||
|
||||
<Note>
|
||||
**Tags:** `workshop` `aws` `compliance` `intermediate` `automation` `frameworks`
|
||||
</Note>
|
||||
|
||||
# Lab 6: Compliance as Code with Prowler
|
||||
|
||||
Learn to automate compliance validation and reporting against industry standards such as CIS, PCI-DSS, HIPAA, and custom compliance frameworks.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* Completion of [Lab 1: Getting Started with Prowler CLI](/workshop/lab-01-getting-started)
|
||||
* AWS account with resources
|
||||
* Prowler CLI installed and configured
|
||||
* Understanding of compliance frameworks (CIS, PCI-DSS, HIPAA)
|
||||
|
||||
**Estimated Time:** 50 minutes
|
||||
|
||||
## Lab Objectives
|
||||
|
||||
By completing this lab, you will:
|
||||
|
||||
* Understand compliance frameworks in Prowler
|
||||
* Generate compliance reports for industry standards
|
||||
* Validate compliance status programmatically
|
||||
* Create custom compliance frameworks
|
||||
* Automate compliance reporting in CI/CD pipelines
|
||||
|
||||
## Step 1: List Available Compliance Frameworks
|
||||
|
||||
View all supported compliance frameworks:
|
||||
|
||||
```bash
|
||||
prowler aws --list-compliance
|
||||
```
|
||||
|
||||
This displays frameworks such as:
|
||||
* CIS AWS Foundations Benchmark (multiple versions)
|
||||
* PCI-DSS v4.0
|
||||
* HIPAA
|
||||
* SOC2
|
||||
* GDPR
|
||||
* ISO 27001
|
||||
* NIST 800-53
|
||||
* AWS Foundational Security Best Practices
|
||||
* Custom frameworks
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 78 showing compliance frameworks list - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 2: Run CIS Benchmark Compliance Scan
|
||||
|
||||
Execute a CIS AWS Foundations Benchmark scan:
|
||||
|
||||
```bash
|
||||
prowler aws --compliance cis_2.0_aws
|
||||
```
|
||||
|
||||
This command:
|
||||
* Runs only checks mapped to CIS Benchmark v2.0
|
||||
* Generates a compliance report
|
||||
* Shows compliance percentage
|
||||
* Identifies non-compliant controls
|
||||
|
||||
Review compliance summary:
|
||||
```bash
|
||||
open output/compliance/prowler-compliance-cis_2.0_aws-*.html
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 80 showing CIS compliance report - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 3: Analyze Compliance Requirements
|
||||
|
||||
Understanding compliance report structure:
|
||||
|
||||
**Requirement ID:** Control identifier (e.g., 1.1, 1.2)
|
||||
**Requirement Description:** What the control validates
|
||||
**Status:** PASS or FAIL
|
||||
**Related Checks:** Prowler checks that map to this requirement
|
||||
**Resources Affected:** Specific resources that failed
|
||||
|
||||
Example CIS requirement:
|
||||
|
||||
```
|
||||
ID: 1.4
|
||||
Description: Ensure no root account access key exists
|
||||
Status: FAIL
|
||||
Checks: iam_root_user_no_access_keys
|
||||
Resources: Root account has 1 active access key
|
||||
```
|
||||
|
||||
## Step 4: Generate Multiple Compliance Reports
|
||||
|
||||
Run scans for multiple frameworks:
|
||||
|
||||
```bash
|
||||
prowler aws --compliance cis_2.0_aws pci_dss_v4.0_aws hipaa_aws
|
||||
```
|
||||
|
||||
This generates three separate compliance reports:
|
||||
* `prowler-compliance-cis_2.0_aws-*.html`
|
||||
* `prowler-compliance-pci_dss_v4.0_aws-*.html`
|
||||
* `prowler-compliance-hipaa_aws-*.html`
|
||||
|
||||
Compare compliance posture across frameworks:
|
||||
```bash
|
||||
grep "Compliance Status" output/compliance/*.html
|
||||
```
|
||||
|
||||
## Step 5: Export Compliance Data
|
||||
|
||||
Export compliance results to JSON for automation:
|
||||
|
||||
```bash
|
||||
prowler aws --compliance cis_2.0_aws -o json-ocsf
|
||||
```
|
||||
|
||||
The JSON output includes:
|
||||
* Compliance score (percentage)
|
||||
* Passed requirements
|
||||
* Failed requirements
|
||||
* Resource-level details
|
||||
* Remediation guidance
|
||||
|
||||
Query compliance status programmatically:
|
||||
```bash
|
||||
jq '.compliance.cis_2.0_aws.score' output/prowler-output-*.json-ocsf
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 84 showing JSON compliance output - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 6: Create a Custom Compliance Framework
|
||||
|
||||
Create a custom framework for organization-specific requirements:
|
||||
|
||||
Create `custom_compliance.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"Framework": "custom_security_baseline",
|
||||
"Version": "1.0",
|
||||
"Provider": "aws",
|
||||
"Description": "Organization Security Baseline Requirements",
|
||||
"Requirements": [
|
||||
{
|
||||
"Id": "1.1",
|
||||
"Description": "S3 buckets must have encryption enabled",
|
||||
"Attributes": [
|
||||
{
|
||||
"Section": "Data Protection",
|
||||
"SubSection": "Encryption at Rest",
|
||||
"Type": "automated",
|
||||
"Service": "s3"
|
||||
}
|
||||
],
|
||||
"Checks": [
|
||||
"s3_bucket_default_encryption",
|
||||
"s3_bucket_secure_transport_policy"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Id": "1.2",
|
||||
"Description": "CloudTrail must be enabled in all regions",
|
||||
"Attributes": [
|
||||
{
|
||||
"Section": "Logging and Monitoring",
|
||||
"SubSection": "Audit Logging",
|
||||
"Type": "automated",
|
||||
"Service": "cloudtrail"
|
||||
}
|
||||
],
|
||||
"Checks": [
|
||||
"cloudtrail_multi_region_enabled",
|
||||
"cloudtrail_log_file_validation_enabled"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Id": "2.1",
|
||||
"Description": "IAM users must have MFA enabled",
|
||||
"Attributes": [
|
||||
{
|
||||
"Section": "Identity and Access Management",
|
||||
"SubSection": "Multi-Factor Authentication",
|
||||
"Type": "automated",
|
||||
"Service": "iam"
|
||||
}
|
||||
],
|
||||
"Checks": [
|
||||
"iam_user_mfa_enabled_console_access",
|
||||
"iam_root_mfa_enabled"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Id": "3.1",
|
||||
"Description": "Security groups must not allow unrestricted access",
|
||||
"Attributes": [
|
||||
{
|
||||
"Section": "Network Security",
|
||||
"SubSection": "Firewall Rules",
|
||||
"Type": "automated",
|
||||
"Service": "ec2"
|
||||
}
|
||||
],
|
||||
"Checks": [
|
||||
"ec2_securitygroup_allow_ingress_from_internet_to_tcp_port_22",
|
||||
"ec2_securitygroup_allow_ingress_from_internet_to_tcp_port_3389"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Save to `prowler/compliance/aws/`:
|
||||
```bash
|
||||
cp custom_compliance.json ~/.prowler/compliance/aws/
|
||||
```
|
||||
|
||||
## Step 7: Run Custom Compliance Framework
|
||||
|
||||
Execute scan against custom framework:
|
||||
|
||||
```bash
|
||||
prowler aws --compliance-framework custom_compliance.json
|
||||
```
|
||||
|
||||
Or if placed in Prowler's compliance directory:
|
||||
```bash
|
||||
prowler aws --compliance custom_security_baseline
|
||||
```
|
||||
|
||||
Review custom compliance report:
|
||||
```bash
|
||||
open output/compliance/prowler-compliance-custom_security_baseline-*.html
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 88 showing custom compliance report - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 8: Compliance Reporting for Audits
|
||||
|
||||
Generate audit-ready compliance reports:
|
||||
|
||||
```bash
|
||||
prowler aws \
|
||||
--compliance cis_2.0_aws \
|
||||
-o html csv json \
|
||||
--output-directory ./audit-reports-$(date +%Y%m%d)
|
||||
```
|
||||
|
||||
This creates:
|
||||
* HTML report for human review
|
||||
* CSV for spreadsheet analysis
|
||||
* JSON for programmatic processing
|
||||
|
||||
Package for auditors:
|
||||
```bash
|
||||
tar -czf compliance-audit-$(date +%Y%m%d).tar.gz audit-reports-*
|
||||
```
|
||||
|
||||
## Step 9: Automate Compliance Validation
|
||||
|
||||
Create a compliance validation script:
|
||||
|
||||
Create `compliance-check.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
# Configuration
|
||||
COMPLIANCE_FRAMEWORK="cis_2.0_aws"
|
||||
REQUIRED_SCORE=85
|
||||
OUTPUT_DIR="./compliance-reports"
|
||||
|
||||
# Run Prowler
|
||||
prowler aws \
|
||||
--compliance $COMPLIANCE_FRAMEWORK \
|
||||
-o json \
|
||||
--output-directory $OUTPUT_DIR
|
||||
|
||||
# Extract compliance score
|
||||
SCORE=$(jq -r ".compliance.${COMPLIANCE_FRAMEWORK}.score" \
|
||||
$OUTPUT_DIR/prowler-output-*.json | head -1)
|
||||
|
||||
echo "Compliance Score: ${SCORE}%"
|
||||
|
||||
# Validate compliance threshold
|
||||
if (( $(echo "$SCORE >= $REQUIRED_SCORE" | bc -l) )); then
|
||||
echo "✓ Compliance check PASSED (score: ${SCORE}% >= ${REQUIRED_SCORE}%)"
|
||||
exit 0
|
||||
else
|
||||
echo "✗ Compliance check FAILED (score: ${SCORE}% < ${REQUIRED_SCORE}%)"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
Make executable:
|
||||
```bash
|
||||
chmod +x compliance-check.sh
|
||||
```
|
||||
|
||||
Run validation:
|
||||
```bash
|
||||
./compliance-check.sh
|
||||
```
|
||||
|
||||
## Step 10: Integrate with CI/CD Pipeline
|
||||
|
||||
Example GitHub Actions workflow:
|
||||
|
||||
Create `.github/workflows/compliance-check.yml`:
|
||||
|
||||
```yaml
|
||||
name: Compliance Validation
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 0 * * *' # Daily at midnight
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
prowler-compliance:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v3
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.11'
|
||||
|
||||
- name: Install Prowler
|
||||
run: pip install prowler
|
||||
|
||||
- name: Configure AWS credentials
|
||||
uses: aws-actions/configure-aws-credentials@v2
|
||||
with:
|
||||
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
|
||||
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
aws-region: us-east-1
|
||||
|
||||
- name: Run compliance scan
|
||||
run: |
|
||||
prowler aws \
|
||||
--compliance cis_2.0_aws \
|
||||
-o html json \
|
||||
--output-directory ./reports
|
||||
|
||||
- name: Upload compliance reports
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: compliance-reports
|
||||
path: ./reports/
|
||||
|
||||
- name: Check compliance threshold
|
||||
run: |
|
||||
SCORE=$(jq -r '.compliance.cis_2.0_aws.score' reports/prowler-output-*.json)
|
||||
if (( $(echo "$SCORE < 85" | bc -l) )); then
|
||||
echo "Compliance score ${SCORE}% is below threshold"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 92 showing CI/CD integration - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 11: Continuous Compliance Monitoring
|
||||
|
||||
Implement continuous compliance monitoring:
|
||||
|
||||
**Daily Scans:**
|
||||
* Schedule automated scans
|
||||
* Track compliance trends over time
|
||||
* Alert on compliance score drops
|
||||
|
||||
**Drift Detection:**
|
||||
* Compare current state vs. baseline
|
||||
* Identify new non-compliant resources
|
||||
* Generate remediation tickets automatically
|
||||
|
||||
**Compliance Dashboard:**
|
||||
* Visualize compliance status
|
||||
* Track remediation progress
|
||||
* Generate executive reports
|
||||
|
||||
## Verification Steps
|
||||
|
||||
Confirm successful lab completion:
|
||||
|
||||
1. Listed available compliance frameworks
|
||||
2. Generated CIS compliance report
|
||||
3. Created multiple framework reports
|
||||
4. Built custom compliance framework
|
||||
5. Automated compliance validation
|
||||
6. Integrated compliance checks in CI/CD
|
||||
|
||||
## Expected Outcomes
|
||||
|
||||
After completing this lab, you should:
|
||||
|
||||
* Understand Prowler compliance capabilities
|
||||
* Be able to generate compliance reports
|
||||
* Know how to create custom frameworks
|
||||
* Have automated compliance validation
|
||||
* Be ready for audit processes
|
||||
|
||||
## Compliance Framework Mapping
|
||||
|
||||
Common frameworks supported:
|
||||
|
||||
**AWS:**
|
||||
* CIS AWS Foundations Benchmark v1.4, v1.5, v2.0, v3.0
|
||||
* AWS Foundational Security Best Practices
|
||||
* PCI-DSS v4.0
|
||||
* HIPAA
|
||||
* SOC2
|
||||
* GDPR
|
||||
* ISO 27001
|
||||
* NIST 800-53
|
||||
* FedRAMP
|
||||
* ENS (Spanish National Security Scheme)
|
||||
|
||||
**Azure:**
|
||||
* CIS Microsoft Azure Foundations Benchmark
|
||||
* Azure Security Benchmark
|
||||
|
||||
**GCP:**
|
||||
* CIS Google Cloud Platform Foundation Benchmark
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Issue:** Compliance framework not found
|
||||
* **Solution:** Use `--list-compliance` to see exact framework names
|
||||
|
||||
**Issue:** Low compliance score
|
||||
* **Solution:** Review failed checks and prioritize remediation by severity
|
||||
|
||||
**Issue:** Missing compliance report
|
||||
* **Solution:** Check `output/compliance/` directory for framework-specific reports
|
||||
|
||||
**Issue:** Custom framework not loading
|
||||
* **Solution:** Validate JSON syntax and ensure file is in correct directory
|
||||
|
||||
## Next Steps
|
||||
|
||||
Continue to [Lab 7: Integrations with Prowler](/workshop/lab-07-integrations) to learn how to integrate Prowler with AWS Security Hub and other security tools.
|
||||
|
||||
## Additional Resources
|
||||
|
||||
* [Compliance Reporting Guide](/user-guide/cli/tutorials/compliance)
|
||||
* [Compliance Frameworks Documentation](/user-guide/cli/tutorials/compliance)
|
||||
* [Custom Compliance Framework Guide](/developer-guide/security-compliance-framework)
|
||||
* [Prowler Hub Compliance Frameworks](https://hub.prowler.com/compliance)
|
||||
@@ -0,0 +1,425 @@
|
||||
---
|
||||
title: "Lab 7: Integrations with Prowler"
|
||||
description: "Integrate Prowler findings with AWS Security Hub and other security tools for centralized security management"
|
||||
---
|
||||
|
||||
<Note>
|
||||
**Tags:** `workshop` `aws` `integrations` `intermediate` `security-hub` `automation`
|
||||
</Note>
|
||||
|
||||
# Lab 7: Integrations with Prowler
|
||||
|
||||
Learn to integrate Prowler with AWS Security Hub and other security tools to centralize security findings and automate remediation workflows.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* Completion of [Lab 1: Getting Started with Prowler CLI](/workshop/lab-01-getting-started)
|
||||
* AWS account with Security Hub enabled
|
||||
* IAM permissions for Security Hub operations
|
||||
* Prowler CLI installed and configured
|
||||
* Basic understanding of AWS Security Hub
|
||||
|
||||
**Estimated Time:** 45 minutes
|
||||
|
||||
## Lab Objectives
|
||||
|
||||
By completing this lab, you will:
|
||||
|
||||
* Enable AWS Security Hub integration
|
||||
* Send Prowler findings to Security Hub
|
||||
* Understand finding formats and mapping
|
||||
* Configure automated finding synchronization
|
||||
* Integrate with third-party security tools
|
||||
* Implement centralized security dashboards
|
||||
|
||||
## Step 1: Enable AWS Security Hub
|
||||
|
||||
Enable Security Hub in your AWS account:
|
||||
|
||||
**Via AWS Console:**
|
||||
1. Navigate to AWS Security Hub
|
||||
2. Click "Go to Security Hub"
|
||||
3. Click "Enable Security Hub"
|
||||
|
||||
**Via AWS CLI:**
|
||||
```bash
|
||||
aws securityhub enable-security-hub
|
||||
```
|
||||
|
||||
Verify Security Hub is enabled:
|
||||
```bash
|
||||
aws securityhub get-enabled-standards
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 96 showing Security Hub enablement - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 2: Configure IAM Permissions
|
||||
|
||||
Ensure your IAM role/user has Security Hub permissions:
|
||||
|
||||
Required permissions:
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"securityhub:BatchImportFindings",
|
||||
"securityhub:GetFindings"
|
||||
],
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Create and attach policy:
|
||||
```bash
|
||||
aws iam create-policy \
|
||||
--policy-name ProwlerSecurityHubIntegration \
|
||||
--policy-document file://securityhub-policy.json
|
||||
|
||||
aws iam attach-user-policy \
|
||||
--user-name prowler-user \
|
||||
--policy-arn arn:aws:iam::ACCOUNT_ID:policy/ProwlerSecurityHubIntegration
|
||||
```
|
||||
|
||||
## Step 3: Run Prowler with Security Hub Integration
|
||||
|
||||
Execute Prowler and send findings to Security Hub:
|
||||
|
||||
```bash
|
||||
prowler aws --security-hub
|
||||
```
|
||||
|
||||
This command:
|
||||
* Runs all security checks
|
||||
* Transforms findings to AWS Security Finding Format (ASFF)
|
||||
* Sends findings to Security Hub via `BatchImportFindings` API
|
||||
* Generates local reports
|
||||
|
||||
<Warning>
|
||||
Security Hub has API rate limits. For large environments, findings are sent in batches automatically.
|
||||
</Warning>
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 99 showing Prowler sending findings to Security Hub - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 4: View Findings in Security Hub
|
||||
|
||||
Navigate to AWS Security Hub console and review Prowler findings:
|
||||
|
||||
**Filter by Product:**
|
||||
1. Go to "Findings" in Security Hub
|
||||
2. Add filter: `Product name is Prowler`
|
||||
3. Review findings by severity
|
||||
|
||||
**View Finding Details:**
|
||||
* Severity (CRITICAL, HIGH, MEDIUM, LOW, INFORMATIONAL)
|
||||
* Affected resource
|
||||
* Compliance framework mapping
|
||||
* Remediation guidance
|
||||
* Workflow status
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 101 showing Security Hub findings view - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 5: Understanding ASFF Mapping
|
||||
|
||||
Prowler findings are mapped to AWS Security Finding Format:
|
||||
|
||||
**Prowler Status → Security Hub Compliance Status:**
|
||||
* PASS → PASSED
|
||||
* FAIL → FAILED
|
||||
* MANUAL → NOT_AVAILABLE
|
||||
|
||||
**Prowler Severity → Security Hub Severity:**
|
||||
* critical → CRITICAL (90-100)
|
||||
* high → HIGH (70-89)
|
||||
* medium → MEDIUM (40-69)
|
||||
* low → LOW (1-39)
|
||||
* informational → INFORMATIONAL (0)
|
||||
|
||||
Example ASFF finding structure:
|
||||
```json
|
||||
{
|
||||
"SchemaVersion": "2018-10-08",
|
||||
"Id": "prowler-aws/account/region/check/resource",
|
||||
"ProductArn": "arn:aws:securityhub:region::product/prowler/prowler",
|
||||
"GeneratorId": "prowler-check-id",
|
||||
"AwsAccountId": "123456789012",
|
||||
"Types": ["Software and Configuration Checks"],
|
||||
"CreatedAt": "2024-01-01T00:00:00.000Z",
|
||||
"UpdatedAt": "2024-01-01T00:00:00.000Z",
|
||||
"Severity": {
|
||||
"Label": "HIGH"
|
||||
},
|
||||
"Title": "Check title",
|
||||
"Description": "Check description",
|
||||
"Resources": [
|
||||
{
|
||||
"Type": "AwsS3Bucket",
|
||||
"Id": "arn:aws:s3:::bucket-name"
|
||||
}
|
||||
],
|
||||
"Compliance": {
|
||||
"Status": "FAILED"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Step 6: Update Existing Findings
|
||||
|
||||
Run subsequent scans to update Security Hub findings:
|
||||
|
||||
```bash
|
||||
prowler aws --security-hub
|
||||
```
|
||||
|
||||
Prowler automatically:
|
||||
* Updates existing findings (same resource, same check)
|
||||
* Marks remediated issues as PASSED
|
||||
* Creates new findings for new resources
|
||||
* Archives findings for deleted resources
|
||||
|
||||
## Step 7: Regional Security Hub Integration
|
||||
|
||||
Send findings to Security Hub in specific regions:
|
||||
|
||||
```bash
|
||||
prowler aws --security-hub --region us-east-1 us-west-2
|
||||
```
|
||||
|
||||
Or enable aggregation in a single region:
|
||||
|
||||
```bash
|
||||
# Configure finding aggregator in Security Hub
|
||||
aws securityhub create-finding-aggregator \
|
||||
--region-linking-mode ALL_REGIONS
|
||||
```
|
||||
|
||||
<Tip>
|
||||
Use Security Hub finding aggregation to centralize findings from multiple regions in a single dashboard.
|
||||
</Tip>
|
||||
|
||||
## Step 8: Filter Findings Sent to Security Hub
|
||||
|
||||
Send only critical and high-severity findings:
|
||||
|
||||
```bash
|
||||
prowler aws --security-hub --severity critical high
|
||||
```
|
||||
|
||||
Send findings for specific compliance frameworks:
|
||||
|
||||
```bash
|
||||
prowler aws --security-hub --compliance cis_2.0_aws
|
||||
```
|
||||
|
||||
## Step 9: Integrate with S3 for Long-Term Storage
|
||||
|
||||
Store Prowler reports in S3 alongside Security Hub integration:
|
||||
|
||||
```bash
|
||||
prowler aws \
|
||||
--security-hub \
|
||||
-o html json csv \
|
||||
--output-bucket-no-assume s3://my-security-reports-bucket
|
||||
```
|
||||
|
||||
This enables:
|
||||
* Long-term retention of historical reports
|
||||
* Compliance audit trails
|
||||
* Trend analysis over time
|
||||
* Cost-effective storage
|
||||
|
||||
Configure S3 bucket lifecycle policies:
|
||||
```bash
|
||||
aws s3api put-bucket-lifecycle-configuration \
|
||||
--bucket my-security-reports-bucket \
|
||||
--lifecycle-configuration file://lifecycle.json
|
||||
```
|
||||
|
||||
`lifecycle.json`:
|
||||
```json
|
||||
{
|
||||
"Rules": [
|
||||
{
|
||||
"Id": "ArchiveOldReports",
|
||||
"Status": "Enabled",
|
||||
"Transitions": [
|
||||
{
|
||||
"Days": 90,
|
||||
"StorageClass": "GLACIER"
|
||||
}
|
||||
],
|
||||
"Expiration": {
|
||||
"Days": 365
|
||||
},
|
||||
"Filter": {
|
||||
"Prefix": "prowler-reports/"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 107 showing S3 integration - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 10: Integrate with Third-Party Tools
|
||||
|
||||
**Send to Slack:**
|
||||
```bash
|
||||
prowler aws --security-hub | \
|
||||
jq -r '.findings[] | select(.status=="FAIL" and .severity=="critical")' | \
|
||||
curl -X POST -H 'Content-type: application/json' \
|
||||
--data @- https://hooks.slack.com/services/YOUR/WEBHOOK/URL
|
||||
```
|
||||
|
||||
**Send to Jira:**
|
||||
Create Jira tickets for critical findings using Jira API:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
JIRA_URL="https://your-domain.atlassian.net"
|
||||
JIRA_API_TOKEN="your-api-token"
|
||||
JIRA_PROJECT="SEC"
|
||||
|
||||
# Extract critical findings
|
||||
FINDINGS=$(prowler aws -o json-ocsf | \
|
||||
jq '.findings[] | select(.status=="FAIL" and .severity=="critical")')
|
||||
|
||||
# Create Jira tickets
|
||||
echo "$FINDINGS" | jq -c '.' | while read finding; do
|
||||
TITLE=$(echo $finding | jq -r '.check_title')
|
||||
DESCRIPTION=$(echo $finding | jq -r '.status_extended')
|
||||
|
||||
curl -X POST "$JIRA_URL/rest/api/2/issue" \
|
||||
-H "Authorization: Bearer $JIRA_API_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{
|
||||
\"fields\": {
|
||||
\"project\": {\"key\": \"$JIRA_PROJECT\"},
|
||||
\"summary\": \"$TITLE\",
|
||||
\"description\": \"$DESCRIPTION\",
|
||||
\"issuetype\": {\"name\": \"Task\"}
|
||||
}
|
||||
}"
|
||||
done
|
||||
```
|
||||
|
||||
**Send to Splunk:**
|
||||
```bash
|
||||
prowler aws -o json-ocsf | \
|
||||
curl -k https://splunk-server:8088/services/collector/event \
|
||||
-H "Authorization: Splunk YOUR-HEC-TOKEN" \
|
||||
-d @-
|
||||
```
|
||||
|
||||
## Step 11: Automate Security Hub Updates
|
||||
|
||||
Create a Lambda function to run Prowler periodically:
|
||||
|
||||
**Lambda Function (Python):**
|
||||
```python
|
||||
import subprocess
|
||||
import boto3
|
||||
|
||||
def lambda_handler(event, context):
|
||||
# Run Prowler with Security Hub integration
|
||||
result = subprocess.run(
|
||||
['prowler', 'aws', '--security-hub'],
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
|
||||
return {
|
||||
'statusCode': 200,
|
||||
'body': f'Prowler scan completed. Output: {result.stdout}'
|
||||
}
|
||||
```
|
||||
|
||||
**Schedule with EventBridge:**
|
||||
```bash
|
||||
aws events put-rule \
|
||||
--name DailyProwlerScan \
|
||||
--schedule-expression "cron(0 2 * * ? *)"
|
||||
|
||||
aws events put-targets \
|
||||
--rule DailyProwlerScan \
|
||||
--targets "Id"="1","Arn"="arn:aws:lambda:region:account:function:ProwlerScanFunction"
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 111 showing automated integration - to be added]
|
||||
</Note>
|
||||
|
||||
## Verification Steps
|
||||
|
||||
Confirm successful lab completion:
|
||||
|
||||
1. AWS Security Hub enabled
|
||||
2. Prowler findings sent to Security Hub
|
||||
3. Findings visible in Security Hub console
|
||||
4. Subsequent scans update existing findings
|
||||
5. S3 integration configured for report storage
|
||||
6. Third-party integration examples tested
|
||||
|
||||
## Expected Outcomes
|
||||
|
||||
After completing this lab, you should:
|
||||
|
||||
* Understand Security Hub integration
|
||||
* Know how to send findings to Security Hub
|
||||
* Be able to configure automated synchronization
|
||||
* Have integrated with S3 for storage
|
||||
* Be familiar with third-party tool integrations
|
||||
|
||||
## Security Hub Benefits
|
||||
|
||||
**Centralized Security:**
|
||||
* Aggregate findings from multiple tools
|
||||
* Unified view across AWS accounts and regions
|
||||
* Compliance dashboard
|
||||
|
||||
**Automated Workflows:**
|
||||
* Trigger remediation workflows
|
||||
* Create incidents automatically
|
||||
* Integrate with SIEM tools
|
||||
|
||||
**Prioritization:**
|
||||
* Filter by severity and compliance status
|
||||
* Track remediation progress
|
||||
* Generate executive reports
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Issue:** Security Hub not enabled
|
||||
* **Solution:** Run `aws securityhub enable-security-hub` to enable
|
||||
|
||||
**Issue:** Permission denied sending findings
|
||||
* **Solution:** Ensure IAM role has `securityhub:BatchImportFindings` permission
|
||||
|
||||
**Issue:** Findings not appearing in Security Hub
|
||||
* **Solution:** Check Prowler output for errors, verify region configuration
|
||||
|
||||
**Issue:** Rate limit errors
|
||||
* **Solution:** Prowler batches findings automatically; retry if transient failures occur
|
||||
|
||||
## Next Steps
|
||||
|
||||
Continue to [Lab 8: Prowler SaaS Platform](/workshop/lab-08-prowler-saas) to explore the managed Prowler Cloud platform with advanced features.
|
||||
|
||||
## Additional Resources
|
||||
|
||||
* [Security Hub Integration Guide](/user-guide/providers/aws/securityhub)
|
||||
* [S3 Integration Guide](/user-guide/providers/aws/s3)
|
||||
* [Integrations Documentation](/user-guide/cli/tutorials/integrations)
|
||||
* [AWS Security Hub Documentation](https://docs.aws.amazon.com/securityhub/)
|
||||
@@ -0,0 +1,440 @@
|
||||
---
|
||||
title: "Lab 8: Prowler SaaS Platform"
|
||||
description: "Explore Prowler Cloud's managed platform with advanced features, team collaboration, and continuous monitoring"
|
||||
---
|
||||
|
||||
<Note>
|
||||
**Tags:** `workshop` `prowler-cloud` `saas` `intermediate` `platform` `collaboration`
|
||||
</Note>
|
||||
|
||||
# Lab 8: Prowler SaaS Platform
|
||||
|
||||
Learn to use Prowler Cloud, the managed SaaS platform that provides advanced security monitoring, team collaboration, compliance dashboards, and AI-powered security insights.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* Completion of previous labs (recommended but not required)
|
||||
* Prowler Cloud account (free trial available)
|
||||
* Cloud provider accounts (AWS, Azure, or GCP)
|
||||
* Basic understanding of Prowler concepts
|
||||
|
||||
**Estimated Time:** 60 minutes
|
||||
|
||||
## Lab Objectives
|
||||
|
||||
By completing this lab, you will:
|
||||
|
||||
* Set up Prowler Cloud account
|
||||
* Connect cloud providers to Prowler Cloud
|
||||
* Navigate the Prowler Cloud interface
|
||||
* Use team collaboration features
|
||||
* Leverage AI-powered security insights
|
||||
* Configure continuous monitoring and alerts
|
||||
* Generate executive compliance reports
|
||||
|
||||
## Step 1: Create Prowler Cloud Account
|
||||
|
||||
Sign up for Prowler Cloud:
|
||||
|
||||
1. Visit [https://cloud.prowler.com](https://cloud.prowler.com)
|
||||
2. Click "Start Free Trial"
|
||||
3. Choose authentication method:
|
||||
* Email/password
|
||||
* Google authentication
|
||||
* GitHub authentication
|
||||
* SSO (for enterprise plans)
|
||||
4. Verify email address
|
||||
5. Complete onboarding wizard
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 115 showing Prowler Cloud signup - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 2: Connect Your First Cloud Provider
|
||||
|
||||
**Connect AWS Account:**
|
||||
|
||||
1. Navigate to "Providers" in Prowler Cloud
|
||||
2. Click "Add Provider"
|
||||
3. Select "AWS"
|
||||
4. Choose connection method:
|
||||
* **CloudFormation Stack** (recommended)
|
||||
* **Manual IAM Role**
|
||||
5. Deploy CloudFormation template
|
||||
6. Copy Role ARN and External ID
|
||||
7. Test connection
|
||||
8. Click "Save"
|
||||
|
||||
**CloudFormation Stack Deployment:**
|
||||
```bash
|
||||
aws cloudformation create-stack \
|
||||
--stack-name prowler-integration \
|
||||
--template-url https://prowler-public.s3.amazonaws.com/prowler-role.yaml \
|
||||
--parameters ParameterKey=ExternalId,ParameterValue=<your-external-id> \
|
||||
--capabilities CAPABILITY_NAMED_IAM
|
||||
```
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 118 showing provider connection - to be added]
|
||||
</Note>
|
||||
|
||||
<Tip>
|
||||
The CloudFormation template creates a read-only IAM role with the minimum permissions required for Prowler scans.
|
||||
</Tip>
|
||||
|
||||
## Step 3: Run Your First Cloud Scan
|
||||
|
||||
Initiate a security scan:
|
||||
|
||||
1. Go to "Scans" page
|
||||
2. Click "New Scan"
|
||||
3. Select provider(s) to scan
|
||||
4. Choose scan type:
|
||||
* **Quick Scan:** Essential security checks
|
||||
* **Full Scan:** Comprehensive assessment
|
||||
* **Compliance Scan:** Framework-specific validation
|
||||
5. Click "Start Scan"
|
||||
|
||||
Monitor scan progress:
|
||||
* Real-time progress indicator
|
||||
* Checks completed
|
||||
* Resources discovered
|
||||
* Findings identified
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 121 showing scan execution - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 4: Explore the Findings Dashboard
|
||||
|
||||
Navigate findings dashboard:
|
||||
|
||||
**Overview Statistics:**
|
||||
* Total findings by severity
|
||||
* Compliance score
|
||||
* Trend over time
|
||||
* Top affected services
|
||||
|
||||
**Filtering Options:**
|
||||
* Severity (Critical, High, Medium, Low)
|
||||
* Status (Open, In Progress, Resolved)
|
||||
* Cloud provider
|
||||
* Service
|
||||
* Compliance framework
|
||||
* Resource tags
|
||||
|
||||
**Finding Details:**
|
||||
* Detailed description
|
||||
* Affected resources
|
||||
* Risk assessment
|
||||
* Remediation steps
|
||||
* Related compliance requirements
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 124 showing findings dashboard - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 5: Use AI-Powered Security Insights
|
||||
|
||||
Leverage Prowler Lighthouse AI features:
|
||||
|
||||
**AI Security Assistant:**
|
||||
1. Click "Lighthouse" in navigation
|
||||
2. Ask questions about your security posture:
|
||||
* "What are my critical security risks?"
|
||||
* "Show me publicly exposed resources"
|
||||
* "How can I improve my compliance score?"
|
||||
* "What encryption issues exist?"
|
||||
|
||||
**AI Remediation Guidance:**
|
||||
* Select any finding
|
||||
* Click "AI Remediation"
|
||||
* Review generated remediation steps
|
||||
* Get customized code/CLI commands
|
||||
* Apply fixes with confidence
|
||||
|
||||
**AI Threat Analysis:**
|
||||
* Identifies attack patterns
|
||||
* Correlates related findings
|
||||
* Suggests priority order for remediation
|
||||
* Explains security impact
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 127 showing Lighthouse AI - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 6: Configure Team Collaboration
|
||||
|
||||
Set up team access and workflows:
|
||||
|
||||
**Invite Team Members:**
|
||||
1. Go to "Settings" → "Team"
|
||||
2. Click "Invite Member"
|
||||
3. Enter email address
|
||||
4. Assign role:
|
||||
* **Admin:** Full access
|
||||
* **Editor:** Scan and remediate
|
||||
* **Viewer:** Read-only access
|
||||
5. Send invitation
|
||||
|
||||
**Assign Findings:**
|
||||
1. Select findings
|
||||
2. Click "Assign"
|
||||
3. Choose team member
|
||||
4. Add due date
|
||||
5. Add comments/notes
|
||||
|
||||
**Workflow States:**
|
||||
* Open → New finding
|
||||
* In Progress → Being investigated/fixed
|
||||
* Resolved → Remediated
|
||||
* False Positive → Not applicable
|
||||
* Risk Accepted → Acknowledged but not fixed
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 130 showing team collaboration - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 7: Configure Continuous Monitoring
|
||||
|
||||
Set up automated scanning:
|
||||
|
||||
**Scheduled Scans:**
|
||||
1. Go to "Scans" → "Schedules"
|
||||
2. Click "Create Schedule"
|
||||
3. Configure:
|
||||
* Name: "Daily Security Scan"
|
||||
* Frequency: Daily, Weekly, or Custom cron
|
||||
* Time: 2:00 AM UTC
|
||||
* Providers: Select all
|
||||
* Notification preferences
|
||||
4. Save schedule
|
||||
|
||||
**Real-Time Monitoring:**
|
||||
* Enable CloudTrail integration
|
||||
* Receive alerts for security events
|
||||
* Detect configuration drift
|
||||
* Identify new resources
|
||||
|
||||
<Tip>
|
||||
Schedule scans during off-peak hours to minimize performance impact on your cloud APIs.
|
||||
</Tip>
|
||||
|
||||
## Step 8: Configure Alerts and Notifications
|
||||
|
||||
Set up security alerts:
|
||||
|
||||
**Alert Rules:**
|
||||
1. Navigate to "Alerts"
|
||||
2. Click "Create Alert Rule"
|
||||
3. Define conditions:
|
||||
* Finding severity ≥ High
|
||||
* Compliance score drops below 80%
|
||||
* New critical findings discovered
|
||||
* Public exposure detected
|
||||
4. Choose notification channels:
|
||||
* Email
|
||||
* Slack
|
||||
* Microsoft Teams
|
||||
* PagerDuty
|
||||
* Webhooks
|
||||
5. Save rule
|
||||
|
||||
**Slack Integration:**
|
||||
1. Go to "Integrations" → "Slack"
|
||||
2. Click "Connect to Slack"
|
||||
3. Authorize Prowler app
|
||||
4. Select channel for notifications
|
||||
5. Configure alert preferences
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 134 showing alert configuration - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 9: Generate Compliance Reports
|
||||
|
||||
Create compliance reports for auditors:
|
||||
|
||||
**Compliance Dashboard:**
|
||||
1. Navigate to "Compliance"
|
||||
2. View compliance scores by framework:
|
||||
* CIS Benchmarks
|
||||
* PCI-DSS
|
||||
* HIPAA
|
||||
* SOC2
|
||||
* ISO 27001
|
||||
3. Drill down into requirements
|
||||
4. View evidence for each control
|
||||
|
||||
**Export Reports:**
|
||||
1. Select compliance framework
|
||||
2. Click "Generate Report"
|
||||
3. Choose format:
|
||||
* PDF (executive summary)
|
||||
* Excel (detailed findings)
|
||||
* CSV (raw data)
|
||||
4. Schedule recurring reports:
|
||||
* Weekly status updates
|
||||
* Monthly compliance reports
|
||||
* Quarterly audit packages
|
||||
|
||||
**Report Customization:**
|
||||
* Add company logo
|
||||
* Include executive summary
|
||||
* Filter by business unit
|
||||
* Show remediation progress
|
||||
* Include trend analysis
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 137 showing compliance reports - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 10: Multi-Account and Multi-Cloud Management
|
||||
|
||||
Manage multiple cloud environments:
|
||||
|
||||
**Add Multiple Providers:**
|
||||
1. Connect AWS accounts (dev, staging, production)
|
||||
2. Connect Azure subscriptions
|
||||
3. Connect GCP projects
|
||||
4. Organize with tags/labels
|
||||
|
||||
**Provider Groups:**
|
||||
1. Create provider groups:
|
||||
* Production environments
|
||||
* Development environments
|
||||
* By business unit
|
||||
* By geographic region
|
||||
2. Run group-wide scans
|
||||
3. Generate consolidated reports
|
||||
|
||||
**Cross-Cloud Insights:**
|
||||
* Compare security posture across providers
|
||||
* Identify configuration inconsistencies
|
||||
* Standardize security policies
|
||||
* Track multi-cloud compliance
|
||||
|
||||
<Note>
|
||||
[Note: Screenshot of slide 140 showing multi-cloud management - to be added]
|
||||
</Note>
|
||||
|
||||
## Step 11: Advanced Features
|
||||
|
||||
Explore advanced Prowler Cloud capabilities:
|
||||
|
||||
**Custom Checks:**
|
||||
* Create organization-specific security policies
|
||||
* Define custom compliance requirements
|
||||
* Share with team
|
||||
|
||||
**API Access:**
|
||||
* Programmatic access to findings
|
||||
* Integrate with internal tools
|
||||
* Automate workflows
|
||||
|
||||
**RBAC (Role-Based Access Control):**
|
||||
* Fine-grained permissions
|
||||
* Provider-level access control
|
||||
* Audit logging
|
||||
|
||||
**Security Integrations:**
|
||||
* AWS Security Hub
|
||||
* Jira
|
||||
* ServiceNow
|
||||
* Splunk
|
||||
* Custom webhooks
|
||||
|
||||
## Verification Steps
|
||||
|
||||
Confirm successful lab completion:
|
||||
|
||||
1. Prowler Cloud account created
|
||||
2. Cloud provider(s) connected
|
||||
3. Security scan completed
|
||||
4. Findings dashboard explored
|
||||
5. AI insights leveraged
|
||||
6. Team collaboration configured
|
||||
7. Continuous monitoring set up
|
||||
8. Compliance reports generated
|
||||
|
||||
## Expected Outcomes
|
||||
|
||||
After completing this lab, you should:
|
||||
|
||||
* Understand Prowler Cloud platform capabilities
|
||||
* Be able to connect and scan cloud providers
|
||||
* Know how to use AI-powered insights
|
||||
* Have configured team collaboration
|
||||
* Be able to generate compliance reports
|
||||
* Have set up continuous monitoring
|
||||
|
||||
## Prowler Cloud vs. Prowler CLI
|
||||
|
||||
**Prowler Cloud Advantages:**
|
||||
* Managed infrastructure (no installation)
|
||||
* Web-based interface
|
||||
* Team collaboration features
|
||||
* AI-powered insights (Lighthouse)
|
||||
* Continuous monitoring
|
||||
* Historical trend analysis
|
||||
* Executive dashboards
|
||||
* Built-in integrations
|
||||
* Scheduled scans
|
||||
* Role-based access control
|
||||
|
||||
**Prowler CLI Advantages:**
|
||||
* Self-hosted (on-premises)
|
||||
* No data leaves your environment
|
||||
* Scriptable and automatable
|
||||
* Free and open source
|
||||
* Custom integrations
|
||||
* Offline scanning
|
||||
|
||||
<Tip>
|
||||
Many organizations use both: Prowler CLI for automated CI/CD pipelines and Prowler Cloud for centralized visibility and team collaboration.
|
||||
</Tip>
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Issue:** Cannot connect cloud provider
|
||||
* **Solution:** Verify IAM role permissions and trust relationship, check External ID
|
||||
|
||||
**Issue:** Scan fails or times out
|
||||
* **Solution:** Check provider credentials are valid, ensure APIs are not rate-limited
|
||||
|
||||
**Issue:** No findings appearing
|
||||
* **Solution:** Verify scan completed successfully, check filtering settings
|
||||
|
||||
**Issue:** Alert notifications not received
|
||||
* **Solution:** Verify integration configuration, check notification channel settings
|
||||
|
||||
## Workshop Completion
|
||||
|
||||
Congratulations on completing the Prowler Workshop! You have learned:
|
||||
|
||||
* Prowler CLI installation and basic usage
|
||||
* Threat detection techniques
|
||||
* Custom check development
|
||||
* Multi-cloud security (AWS, Azure, GCP)
|
||||
* Compliance automation
|
||||
* Security tool integrations
|
||||
* Prowler Cloud platform capabilities
|
||||
|
||||
## Next Steps
|
||||
|
||||
Continue your Prowler journey:
|
||||
|
||||
* Join the [Prowler Community](https://goto.prowler.com/slack)
|
||||
* Contribute to [Prowler Open Source](https://github.com/prowler-cloud/prowler)
|
||||
* Explore [Prowler Hub](https://hub.prowler.com) for checks and frameworks
|
||||
* Read the [Prowler Documentation](https://docs.prowler.com)
|
||||
* Follow [Prowler on Twitter](https://twitter.com/prowlercloud)
|
||||
* Subscribe to [Prowler YouTube](https://www.youtube.com/@prowlercloud)
|
||||
|
||||
## Additional Resources
|
||||
|
||||
* [Prowler Cloud Documentation](/getting-started/products/prowler-cloud)
|
||||
* [Prowler Cloud Pricing](/getting-started/products/prowler-cloud-pricing)
|
||||
* [AWS Marketplace Listing](/getting-started/products/prowler-cloud-aws-marketplace)
|
||||
* [Prowler API Reference](/getting-started/goto/prowler-api-reference)
|
||||
* [Prowler Lighthouse AI](/user-guide/tutorials/prowler-app-lighthouse)
|
||||
@@ -0,0 +1,366 @@
|
||||
# Prowler SDK Agent Guide
|
||||
|
||||
**Complete guide for AI agents and developers working on the Prowler SDK - the core Python security scanning engine.**
|
||||
|
||||
## Project Overview
|
||||
|
||||
The Prowler SDK is the core Python engine that powers Prowler's cloud security assessment capabilities. It provides:
|
||||
|
||||
- **Multi-cloud Security Scanning**: AWS, Azure, GCP, Kubernetes, GitHub, M365, Oracle Cloud, MongoDB Atlas, and more
|
||||
- **Compliance Frameworks**: 30+ frameworks including CIS, NIST, PCI-DSS, SOC2, GDPR
|
||||
- **1000+ Security Checks**: Comprehensive coverage across all supported providers
|
||||
- **Multiple Output Formats**: JSON, CSV, HTML, ASFF, OCSF, and compliance-specific formats
|
||||
|
||||
## Mission & Scope
|
||||
|
||||
- Maintain and enhance the core Prowler SDK functionality with security and stability as top priorities
|
||||
- Follow best practices for Python patterns, code style, security, and comprehensive testing
|
||||
- To get more information about development guidelines, please refer to the Prowler Developer Guide in `docs/developer-guide/`
|
||||
|
||||
---
|
||||
|
||||
## Architecture Rules
|
||||
|
||||
### 1. Provider Architecture Pattern
|
||||
|
||||
All Prowler providers MUST follow the established pattern:
|
||||
|
||||
```
|
||||
prowler/providers/{provider}/
|
||||
├── {provider}_provider.py # Main provider class
|
||||
├── models.py # Provider-specific models
|
||||
├── config.py # Provider configuration
|
||||
├── exceptions/ # Provider-specific exceptions
|
||||
├── lib/ # Provider libraries (as minimun it should have implemented the next folders: service, arguments, mutelist)
|
||||
│ ├── service/ # Provider-specific service class to be inherited by all services of the provider
|
||||
│ ├── arguments/ # Provider-specific CLI arguments parser
|
||||
│ └── mutelist/ # Provider-specific mutelist functionality
|
||||
└── services/ # All provider services to be audited
|
||||
└── {service}/ # Individual service
|
||||
├── {service}_service.py # Class to fetch the needed resources from the API and store them to be used by the checks
|
||||
├── {service}_client.py # Python instance of the service class to be used by the checks
|
||||
└── {check_name}/ # Individual check folder
|
||||
├── {check_name}.py # Python class to implement the check logic
|
||||
└── {check_name}.metadata.json # JSON file to store the check metadata
|
||||
└── {check_name_2}/ # Other checks can be added to the same service folder
|
||||
├── {check_name_2}.py
|
||||
└── {check_name_2}.metadata.json
|
||||
...
|
||||
└── {service_2}/ # Other services can be added to the same provider folder
|
||||
...
|
||||
```
|
||||
|
||||
### 2. Check Implementation Standards
|
||||
|
||||
Every security check MUST implement:
|
||||
|
||||
```python
|
||||
from prowler.lib.check.models import Check, CheckReport<Provider>
|
||||
from prowler.providers.<provider>.services.<service>.<service>_client import <service>_client
|
||||
|
||||
class check_name(Check):
|
||||
"""Ensure that <resource> meets <security_requirement>."""
|
||||
def execute(self) -> list[CheckReport<Provider>]:
|
||||
"""Execute the check logic.
|
||||
|
||||
Returns:
|
||||
A list of reports containing the result of the check.
|
||||
"""
|
||||
findings = []
|
||||
# Check implementation here
|
||||
for resource in <service>_client.<resources>:
|
||||
# Security validation logic
|
||||
report = CheckReport<Provider>(metadata=self.metadata(), resource=resource)
|
||||
report.status = "PASS" | "FAIL"
|
||||
report.status_extended = "Detailed explanation"
|
||||
findings.append(report) # Add the report to the list of findings
|
||||
return findings
|
||||
```
|
||||
|
||||
### 3. Compliance Framework Integration
|
||||
|
||||
All compliance frameworks must be defined in:
|
||||
- `prowler/compliance/{provider}/{framework}.json`
|
||||
- Follow the established Compliance model structure
|
||||
- Include proper requirement mappings and metadata
|
||||
|
||||
---
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Language**: Python 3.9+
|
||||
- **Dependency Management**: Poetry 2+
|
||||
- **CLI Framework**: Custom argument parser with provider-specific subcommands
|
||||
- **Testing**: Pytest with extensive unit and integration tests
|
||||
- **Code Quality**: Pre-commit hooks for Black, Flake8, Pylint, Bandit for security scanning
|
||||
|
||||
## Commands
|
||||
|
||||
### Development Environment
|
||||
|
||||
```bash
|
||||
# Core development setup
|
||||
poetry install --with dev # Install all dependencies
|
||||
poetry run pre-commit install # Install pre-commit hooks
|
||||
|
||||
# Code quality
|
||||
poetry run pre-commit run --all-files
|
||||
|
||||
# Run tests
|
||||
poetry run pytest -n auto -vvv -s -x tests/
|
||||
```
|
||||
|
||||
### Running Prowler CLI
|
||||
|
||||
```bash
|
||||
# Run Prowler
|
||||
poetry run python prowler-cli.py --help
|
||||
|
||||
# Run Prowler with a specific provider
|
||||
poetry run python prowler-cli.py <provider>
|
||||
|
||||
# Run Prowler with error logging
|
||||
poetry run python prowler-cli.py <provider> --log-level ERROR --verbose
|
||||
|
||||
# Run specific checks
|
||||
poetry run python prowler-cli.py <provider> --checks <check_name_1> <check_name_2>
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
prowler/
|
||||
├── __main__.py # Main CLI entry point
|
||||
├── config/ # Global configuration
|
||||
│ ├── config.py # Core configuration settings
|
||||
│ └── __init__.py
|
||||
├── lib/ # Core library functions
|
||||
│ ├── check/ # Check execution engine
|
||||
│ │ ├── check.py # Check execution logic
|
||||
│ │ ├── checks_loader.py # Dynamic check loading
|
||||
│ │ ├── compliance.py # Compliance framework handling
|
||||
│ │ └── models.py # Check and report models
|
||||
│ ├── cli/ # Command-line interface
|
||||
│ │ └── parser.py # Argument parsing
|
||||
│ ├── outputs/ # Output format handlers
|
||||
│ │ ├── csv/ # CSV output
|
||||
│ │ ├── html/ # HTML reports
|
||||
│ │ ├── json/ # JSON formats
|
||||
│ │ └── compliance/ # Compliance reports
|
||||
│ ├── scan/ # Scan orchestration
|
||||
│ ├── utils/ # Utility functions
|
||||
│ └── mutelist/ # Mute list functionality
|
||||
├── providers/ # Cloud provider implementations
|
||||
│ ├── aws/ # AWS provider
|
||||
│ ├── azure/ # Azure provider
|
||||
│ ├── gcp/ # Google Cloud provider
|
||||
│ ├── kubernetes/ # Kubernetes provider
|
||||
│ ├── github/ # GitHub provider
|
||||
│ ├── m365/ # Microsoft 365 provider
|
||||
│ ├── mongodbatlas/ # MongoDB Atlas provider
|
||||
│ ├── oci/ # Oracle Cloud provider
|
||||
│ ├── ...
|
||||
│ └── common/ # Shared provider utilities
|
||||
├── compliance/ # Compliance framework definitions
|
||||
│ ├── aws/ # AWS compliance frameworks
|
||||
│ ├── azure/ # Azure compliance frameworks
|
||||
│ ├── gcp/ # GCP compliance frameworks
|
||||
│ ├── ...
|
||||
└── exceptions/ # Global exception definitions
|
||||
```
|
||||
|
||||
## Key Components
|
||||
|
||||
### 1. Provider System
|
||||
|
||||
Each cloud provider implements:
|
||||
|
||||
```python
|
||||
class Provider:
|
||||
"""Base provider class"""
|
||||
|
||||
def __init__(self, arguments):
|
||||
self.session = self._setup_session(arguments)
|
||||
self.regions = self._get_regions()
|
||||
# Initialize all services
|
||||
|
||||
def _setup_session(self, arguments):
|
||||
"""Provider-specific authentication"""
|
||||
pass
|
||||
|
||||
def _get_regions(self):
|
||||
"""Get available regions for provider"""
|
||||
pass
|
||||
```
|
||||
|
||||
### 2. Check Engine
|
||||
|
||||
The check execution system:
|
||||
|
||||
- **Dynamic Loading**: Automatically discovers and loads checks
|
||||
- **Parallel Execution**: Runs checks in parallel for performance
|
||||
- **Error Isolation**: Individual check failures don't affect others
|
||||
- **Comprehensive Reporting**: Detailed findings with remediation guidance
|
||||
|
||||
### 3. Compliance Framework Engine
|
||||
|
||||
Compliance frameworks are defined as JSON files mapping checks to requirements:
|
||||
|
||||
```json
|
||||
{
|
||||
"Framework": "CIS",
|
||||
"Name": "CIS Amazon Web Services Foundations Benchmark v2.0.0",
|
||||
"Version": "2.0",
|
||||
"Provider": "AWS",
|
||||
"Description": "The CIS Amazon Web Services Foundations Benchmark provides prescriptive guidance for configuring security options for a subset of Amazon Web Services with an emphasis on foundational, testable, and architecture agnostic settings.",
|
||||
"Requirements": [
|
||||
{
|
||||
"Id": "1.1",
|
||||
"Description": "Maintain current contact details",
|
||||
"Checks": ["account_contact_details_configured"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Output System
|
||||
|
||||
Multiple output formats supported:
|
||||
|
||||
- **JSON**: Machine-readable findings
|
||||
- **CSV**: Spreadsheet-compatible format
|
||||
- **HTML**: Interactive web reports
|
||||
- **ASFF**: AWS Security Finding Format
|
||||
- **OCSF**: Open Cybersecurity Schema Framework
|
||||
|
||||
## Development Patterns
|
||||
|
||||
### Adding New Cloud Providers
|
||||
|
||||
1. **Create Provider Structure**:
|
||||
```bash
|
||||
mkdir -p prowler/providers/{provider}
|
||||
mkdir -p prowler/providers/{provider}/services
|
||||
mkdir -p prowler/providers/{provider}/lib/{service,arguments,mutelist}
|
||||
mkdir -p prowler/providers/{provider}/exceptions
|
||||
```
|
||||
|
||||
2. **Implement Provider Class**:
|
||||
```python
|
||||
from prowler.providers.common.provider import Provider
|
||||
|
||||
class NewProvider(Provider):
|
||||
def __init__(self, arguments):
|
||||
super().__init__(arguments)
|
||||
# Provider-specific initialization
|
||||
```
|
||||
|
||||
3. **Add Provider to CLI**:
|
||||
Update `prowler/lib/cli/parser.py` to include new provider arguments.
|
||||
|
||||
### Adding New Security Checks
|
||||
|
||||
The most common high level steps to create a new check are:
|
||||
|
||||
1. Prerequisites:
|
||||
- Verify the check does not already exist by searching in the same service folder as `prowler/providers/<provider>/services/<service>/<check_name_want_to_implement>/`.
|
||||
- Ensure required provider and service exist. If not, you will need to create them first.
|
||||
- Confirm the service has implemented all required methods and attributes for the check (in most cases, you will need to add or modify some methods in the service to get the data you need for the check).
|
||||
2. Navigate to the service directory. The path should be as follows: `prowler/providers/<provider>/services/<service>`.
|
||||
3. Create a check-specific folder. The path should follow this pattern: `prowler/providers/<provider>/services/<service>/<check_name_want_to_implement>`. Adhere to the [Naming Format for Checks](/developer-guide/checks#naming-format-for-checks).
|
||||
4. Create the check files, you can use next commands:
|
||||
```bash
|
||||
mkdir -p prowler/providers/<provider>/services/<service>/<check_name_want_to_implement>
|
||||
touch prowler/providers/<provider>/services/<service>/<check_name_want_to_implement>/__init__.py
|
||||
touch prowler/providers/<provider>/services/<service>/<check_name_want_to_implement>/<check_name_want_to_implement>.py
|
||||
touch prowler/providers/<provider>/services/<service>/<check_name_want_to_implement>/<check_name_want_to_implement>.metadata.json
|
||||
```
|
||||
5. Run the check locally to ensure it works as expected. For checking you can use the CLI in the next way:
|
||||
- To ensure the check has been detected by Prowler: `poetry run python prowler-cli.py <provider> --list-checks | grep <check_name>`.
|
||||
- To run the check, to find possible issues: `poetry run python prowler-cli.py <provider> --log-level ERROR --verbose --check <check_name>`.
|
||||
6. Create comprehensive tests for the check that cover multiple scenarios including both PASS (compliant) and FAIL (non-compliant) cases. For detailed information about test structure and implementation guidelines, refer to the [Testing](/developer-guide/unit-testing) documentation.
|
||||
7. If the check and its corresponding tests are working as expected, you can submit a PR to Prowler.
|
||||
|
||||
### Adding Compliance Frameworks
|
||||
|
||||
1. **Create Framework File**:
|
||||
```bash
|
||||
# Create prowler/compliance/{provider}/{framework}.json
|
||||
```
|
||||
|
||||
2. **Define Requirements**:
|
||||
Map framework requirements to existing checks.
|
||||
|
||||
3. **Test Compliance**:
|
||||
```bash
|
||||
poetry run python -m prowler {provider} --compliance {framework}
|
||||
```
|
||||
|
||||
## Code Quality Standards
|
||||
|
||||
### 1. Python Style
|
||||
|
||||
- **PEP 8 Compliance**: Enforced by black and flake8
|
||||
- **Type Hints**: Required for all public functions
|
||||
- **Docstrings**: Required for all classes and methods
|
||||
- **Import Organization**: Use isort for consistent import ordering
|
||||
|
||||
```python
|
||||
import standard_library
|
||||
|
||||
from third_party import library
|
||||
|
||||
from prowler.lib import internal_module
|
||||
|
||||
class ExampleClass:
|
||||
"""Class docstring."""
|
||||
|
||||
def method(self, param: str) -> dict | list | None:
|
||||
"""Method docstring.
|
||||
|
||||
Args:
|
||||
param: Description of parameter
|
||||
|
||||
Returns:
|
||||
Description of return value
|
||||
"""
|
||||
return None
|
||||
```
|
||||
|
||||
### 2. Error Handling
|
||||
|
||||
```python
|
||||
from prowler.lib.logger import logger
|
||||
|
||||
try:
|
||||
# Risky operation
|
||||
result = api_call()
|
||||
except ProviderSpecificException as e:
|
||||
logger.error(f"Provider error: {e}")
|
||||
# Graceful handling
|
||||
except Exception as e:
|
||||
logger.error(f"Unexpected error: {e}")
|
||||
# Never let checks crash the entire scan
|
||||
```
|
||||
|
||||
### 3. Security Practices
|
||||
|
||||
- **No Hardcoded Secrets**: Use environment variables or secure credential management
|
||||
- **Input Validation**: Validate all external inputs
|
||||
- **Principle of Least Privilege**: Request minimal necessary permissions
|
||||
- **Secure Defaults**: Default to secure configurations
|
||||
|
||||
## Testing Guidelines
|
||||
|
||||
### Unit Tests
|
||||
|
||||
- **100% Coverage Goal**: Aim for complete test coverage
|
||||
- **Mock External Services**: Use mock objects to simulate the external services
|
||||
- **Test Edge Cases**: Include error conditions and boundary cases
|
||||
|
||||
## References
|
||||
|
||||
- **Root Project Guide**: `../AGENTS.md` (takes priority for cross-component guidance)
|
||||
- **Provider Examples**: Reference existing providers for implementation patterns
|
||||
- **Check Examples**: Study existing checks for proper implementation patterns
|
||||
- **Compliance Framework Examples**: Review existing frameworks for structure
|
||||
+8
-14
@@ -2,25 +2,19 @@
|
||||
|
||||
All notable changes to the **Prowler SDK** are documented in this file.
|
||||
|
||||
## [v5.13.2] (Prowler UNRELEASED)
|
||||
## [v5.14.0] (Prowler UNRELEASED)
|
||||
|
||||
### Fixed
|
||||
- Check `check_name` has no `resource_name` error for GCP provider [(#9169)](https://github.com/prowler-cloud/prowler/pull/9169)
|
||||
- Depth Truncation and parsing error in PowerShell queries [(#9181)](https://github.com/prowler-cloud/prowler/pull/9181)
|
||||
- False negative in `iam_role_cross_service_confused_deputy_prevention` check [(#9213)](https://github.com/prowler-cloud/prowler/pull/9213)
|
||||
- Fix M365 Teams `--sp-env-auth` connection error and enhanced timeout logging [(#9191)](https://github.com/prowler-cloud/prowler/pull/9191)
|
||||
- Rename `get_oci_assessment_summary` to `get_oraclecloud_assessment_summary` in HTML output [(#9200)](https://github.com/prowler-cloud/prowler/pull/9200)
|
||||
- Fix Validation and other errors in Azure provider [(#8915)](https://github.com/prowler-cloud/prowler/pull/8915)
|
||||
- Update documentation URLs from docs.prowler.cloud to docs.prowler.com [(#9240)](https://github.com/prowler-cloud/prowler/pull/9240)
|
||||
- Fix file name parsing for checks on Windows [(#9268)](https://github.com/prowler-cloud/prowler/pull/9268)
|
||||
### Added
|
||||
- GitHub provider check `organization_default_repository_permission_strict` [(#8785)](https://github.com/prowler-cloud/prowler/pull/8785)
|
||||
- Update AWS Direct Connect service metadata to new format [(#8855)](https://github.com/prowler-cloud/prowler/pull/8855)
|
||||
|
||||
## [v5.13.1] (Prowler v5.13.1)
|
||||
---
|
||||
|
||||
## [v5.13.1] (Prowler UNRELEASED)
|
||||
|
||||
### Fixed
|
||||
- Add `resource_name` for checks under `logging` for the GCP provider [(#9023)](https://github.com/prowler-cloud/prowler/pull/9023)
|
||||
- Fix `ec2_instance_with_outdated_ami` check to handle None AMIs [(#9046)](https://github.com/prowler-cloud/prowler/pull/9046)
|
||||
- Handle timestamp when transforming compliance findings in CCC [(#9042)](https://github.com/prowler-cloud/prowler/pull/9042)
|
||||
- Update `resource_id` for admincenter service and avoid unnecessary msgraph requests [(#9019)](https://github.com/prowler-cloud/prowler/pull/9019)
|
||||
|
||||
|
||||
---
|
||||
|
||||
|
||||
+1
-1
@@ -422,7 +422,7 @@ def prowler():
|
||||
else:
|
||||
# Refactor(CLI)
|
||||
logger.critical(
|
||||
"Slack integration needs SLACK_API_TOKEN and SLACK_CHANNEL_NAME environment variables (see more in https://docs.prowler.com/user-guide/cli/tutorials/integrations#configuration-of-the-integration-with-slack)."
|
||||
"Slack integration needs SLACK_API_TOKEN and SLACK_CHANNEL_NAME environment variables (see more in https://docs.prowler.cloud/en/latest/tutorials/integrations/#slack)."
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
@@ -753,7 +753,9 @@
|
||||
{
|
||||
"Id": "1.3.8",
|
||||
"Description": "Base permissions define the permission level automatically granted to all organization members. Define strict base access permissions for all of the repositories in the organization, including new ones.",
|
||||
"Checks": [],
|
||||
"Checks": [
|
||||
"organization_default_repository_permission_strict"
|
||||
],
|
||||
"Attributes": [
|
||||
{
|
||||
"Section": "1 Source Code",
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user