mirror of
https://github.com/prowler-cloud/prowler.git
synced 2026-07-04 19:21:51 +00:00
Daniel Barranquero
120 lines
6.8 KiB
Plaintext
120 lines
6.8 KiB
Plaintext
---
|
|
title: 'Secret-Scanning Checks'
|
|
---
|
|
|
|
import { VersionBadge } from "/snippets/version-badge.mdx"
|
|
|
|
<VersionBadge version="5.32.0" />
|
|
|
|
Prowler scans audited resources for plaintext secrets using [Kingfisher](https://github.com/mongodb/kingfisher), an open-source secret-scanning engine that Prowler invokes as a subprocess. This guide explains the structure every secret-scanning check must follow to keep scanning correct and efficient on large accounts.
|
|
|
|
<Note>
|
|
Since Prowler 5.32.0 the secret-scanning checks scan with Kingfisher. Earlier versions used the `detect-secrets` library.
|
|
</Note>
|
|
|
|
## Overview
|
|
|
|
Secret detection runs through a single helper in `prowler/lib/utils/utils.py`:
|
|
|
|
- **`detect_secrets_scan_batch(payloads, excluded_secrets=..., validate=...)`** scans many payloads in chunked subprocess invocations and returns a `{key: [findings]}` dictionary. To scan a single payload, pass a one-entry mapping (for example, `{0: data}`).
|
|
|
|
Every Kingfisher invocation carries a fixed process-startup cost (around 100 ms). Scanning once per resource would spawn thousands of subprocesses on large accounts (for example, thousands of CloudWatch log groups). `detect_secrets_scan_batch` amortizes that cost: it writes each payload to a temporary file as it consumes them, runs one subprocess per chunk (500 payloads by default), and maps the findings back to each payload by key.
|
|
|
|
## The Batched Structure
|
|
|
|
Every secret-scanning check follows three phases.
|
|
|
|
### Phase 1: Collect
|
|
|
|
Define a generator that yields `(key, payload)` for each scannable unit. The generator builds payload strings only — it does not call Kingfisher. Lazy yielding keeps memory and temporary-disk usage bounded to a single chunk, which matters when an account holds thousands of resources.
|
|
|
|
### Phase 2: Batch
|
|
|
|
Call `detect_secrets_scan_batch` once with the generator. The helper consumes it in chunks, runs Kingfisher per chunk, and returns the keys that produced findings mapped to their finding lists.
|
|
|
|
### Phase 3: Report
|
|
|
|
Iterate the resources, look up the findings by key, and build one report per resource. Emit a finding for **every** iterated resource — never drop one silently. When a resource's payload cannot be prepared for scanning (for example, user data that fails to base64-decode or decompress), report it as `MANUAL` with a status explaining the scan could not inspect it, rather than omitting it or claiming `PASS`.
|
|
|
|
```python
|
|
from prowler.lib.check.models import Check, Check_Report_AWS
|
|
from prowler.lib.utils.utils import (
|
|
annotate_verified_secrets,
|
|
detect_secrets_scan_batch,
|
|
)
|
|
from prowler.providers.aws.services.example.example_client import example_client
|
|
|
|
|
|
class example_resource_no_secrets(Check):
|
|
def execute(self):
|
|
findings = []
|
|
excluded = example_client.audit_config.get("secrets_ignore_patterns", [])
|
|
validate = example_client.audit_config.get("secrets_validate", False)
|
|
resources = list(example_client.resources)
|
|
|
|
# Phase 1: collect — builds strings only, no scan.
|
|
def payloads():
|
|
for index, resource in enumerate(resources):
|
|
if resource.scannable_data:
|
|
yield index, serialize(resource)
|
|
|
|
# Phase 2: batch — one call, chunked subprocesses.
|
|
batch_results = detect_secrets_scan_batch(
|
|
payloads(), excluded_secrets=excluded, validate=validate
|
|
)
|
|
|
|
# Phase 3: report — look up findings by key.
|
|
for index, resource in enumerate(resources):
|
|
report = Check_Report_AWS(metadata=self.metadata(), resource=resource)
|
|
report.status = "PASS"
|
|
report.status_extended = f"No secrets found in {resource.name}."
|
|
detect_secrets_output = batch_results.get(index)
|
|
if detect_secrets_output:
|
|
report.status = "FAIL"
|
|
report.status_extended = (
|
|
f"Potential secret found in {resource.name} -> ..."
|
|
)
|
|
annotate_verified_secrets(report, detect_secrets_output)
|
|
findings.append(report)
|
|
|
|
return findings
|
|
```
|
|
|
|
## Choosing the Key
|
|
|
|
The key maps each finding back to its source. Two shapes cover every check:
|
|
|
|
- **One payload per resource:** use the resource index. This fits checks that serialize a single payload per resource, such as launch configurations, CloudFormation outputs, SSM documents, Step Functions definitions, and OpenStack metadata.
|
|
- **Several payloads per resource:** use a `(resource_index, fragment)` tuple, where the fragment identifies the variable, log stream, container, file, or version. Phase 3 groups the per-fragment findings to build the resource report. This fits CloudWatch log streams, ECS containers, CodeBuild variables, Glue arguments, and Lambda code files.
|
|
|
|
Derive the indices from the same `list(...)` of resources in both Phase 1 and Phase 3 so the order stays stable and the keys align.
|
|
|
|
## Preserving Per-Payload Results
|
|
|
|
`detect_secrets_scan_batch` runs Kingfisher with `--no-dedup`, so a secret that appears in more than one payload is reported for each one. This reproduces the result of scanning each payload individually. Build payload strings exactly as a single scan would: serialize the same data and keep line ordering, because messages often map a finding's `line_number` back to a variable name or metadata key.
|
|
|
|
## Validation and Severity
|
|
|
|
`detect_secrets_scan_batch` accepts `validate`, read from `secrets_validate` in the provider configuration or the `--scan-secrets-validate` flag. When enabled, Kingfisher confirms whether each secret is live, and confirmed secrets carry `is_verified: True`.
|
|
|
|
After marking a report as `FAIL`, pass the findings to `annotate_verified_secrets(report, findings)`. When any secret is verified, the helper escalates the finding to critical severity and appends a note that the secret was confirmed live. Validation stays off by default because it sends the discovered secret to the provider API.
|
|
|
|
## Excluded Secrets
|
|
|
|
`detect_secrets_scan_batch` applies `secrets_ignore_patterns` — regular expressions from the provider configuration — against each finding's source line and drops the matches, mirroring single-scan behavior.
|
|
|
|
## Testing
|
|
|
|
To assert on the verified-secret path, mock `detect_secrets_scan_batch` in the check module and return the keyed dictionary. For a single resource scanned at index `0`:
|
|
|
|
```python
|
|
mock.patch(
|
|
"prowler.providers.aws.services.example.example_resource_no_secrets.example_resource_no_secrets.detect_secrets_scan_batch",
|
|
return_value={
|
|
0: [{"type": "...", "line_number": 1, "is_verified": True}]
|
|
},
|
|
)
|
|
```
|
|
|
|
Most tests need no mock at all: they seed resources that contain example secrets and assert on the `FAIL` status and message, which exercises the real batched path. Refer to the [Testing](/developer-guide/unit-testing) documentation for the general structure.
|