ben.carrasco/prowler
1
0
Fork 0
mirror of https://github.com/prowler-cloud/prowler.git synced 2026-03-21 18:58:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore(ingestions): rename flag, update docs (#10236)

Browse Source
This commit is contained in:
Pepe Fagoaga
2026-03-03 14:04:34 +00:00
committed by GitHub
parent e96ea54f3b
commit 71ee4213b3
8 changed files with 86 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
docs/user-guide/tutorials/prowler-app-import-findings.mdx
View File

@@ -132,69 +132,77 @@ Only **Detection Finding** (`class_uid: 2004`) records are accepted. Other OCSF
## Required permissions
The **Manage Ingestions** RBAC permission controls access to the ingestion endpoints. Without this permission, findings cannot be submitted via the API or `--export-ocsf`.
The **Manage Ingestions** RBAC permission controls access to the ingestion endpoints. Without this permission, findings cannot be submitted via the API or `--push-to-cloud`.
For more information about RBAC permissions, refer to the [Prowler App RBAC documentation](/user-guide/tutorials/prowler-app-rbac).
## Using the CLI
The `--export-ocsf` flag uploads scan results directly to Prowler Cloud after a scan completes. This approach automates the ingestion process without manual file uploads.
The `--push-to-cloud` flag uploads scan results directly to Prowler Cloud after a scan completes. This approach automates the ingestion process without manual file uploads.
### Prerequisites
- A valid Prowler Cloud API key (see [API Keys](/user-guide/tutorials/prowler-app-api-keys))
- The `PROWLER_API_KEY` environment variable configured
- The `PROWLER_CLOUD_API_KEY` environment variable configured
### Basic usage
```bash
export PROWLER_API_KEY="pk_your_api_key_here"
export PROWLER_CLOUD_API_KEY="pk_your_api_key_here"
prowler aws --export-ocsf
prowler aws --push-to-cloud
```
### Combining with output formats
When using `--export-ocsf` with custom output formats that exclude OCSF, Prowler generates a temporary OCSF file for upload:
When using `--push-to-cloud` with custom output formats that exclude OCSF, Prowler generates a temporary OCSF file for upload:
The temporary OCSF file is saved in the system temporary directory and not in the output path passed with `-o`.
```bash
prowler aws --services accessanalyzer -M csv --export-ocsf -o /tmp/scan-output
prowler aws --services accessanalyzer -M csv --push-to-cloud -o /tmp/scan-output
```
When default output formats include OCSF, Prowler reuses the existing file. Default output formats include JSON-OCSF:
```bash
prowler aws --services accessanalyzer --export-ocsf -o /tmp/scan-output
prowler aws --services accessanalyzer --push-to-cloud -o /tmp/scan-output
```
### CLI output examples
**Successful upload:**
```
Exporting OCSF to Prowler Cloud, please wait...
Pushing findings to Prowler Cloud, please wait...
OCSF export accepted. Ingestion job: fa8bc8c5-4925-46a0-9fe0-f6575905e094
Findings successfully pushed to Prowler Cloud. Ingestion job: fa8bc8c5-4925-46a0-9fe0-f6575905e094
See more details here: https://cloud.prowler.com/scans
```
**Missing API key:**
```
WARNING: OCSF export skipped: no API key configured. Set the PROWLER_API_KEY
Push to Prowler Cloud skipped: no API key configured. Set the PROWLER_CLOUD_API_KEY
environment variable to enable it. Scan results were saved to
/tmp/scan-output/prowler-output-123456789012-20260217131755.ocsf.json
```
**API unreachable:**
```
WARNING: OCSF export skipped: could not reach the Prowler Cloud API at
Push to Prowler Cloud failed: could not reach the Prowler Cloud API at
https://api.prowler.com. Check the URL and your network connection. Scan results
were saved to /tmp/scan-output/prowler-output-123456789012-20260217131755.ocsf.json
```
**No subscription:**
```
Push to Prowler Cloud failed: this feature is only available with a Prowler Cloud
subscription. Scan results were saved to
/tmp/scan-output/prowler-output-123456789012-20260217131755.ocsf.json
```
**Invalid API key:**
```
WARNING: OCSF export failed: the API returned HTTP 401. Verify your API key is
Push to Prowler Cloud failed: the API returned HTTP 401. Verify your API key is
valid and has the right permissions. Scan results were saved to
/tmp/scan-output/prowler-output-123456789012-20260217131755.ocsf.json
```
@@ -212,10 +220,10 @@ The Ingestion API provides endpoints for submitting OCSF files and monitoring jo
Include the API key in the `Authorization` header:
```bash
export PROWLER_API_KEY="pk_your_api_key_here"
export PROWLER_CLOUD_API_KEY="pk_your_api_key_here"
curl -X POST \
-H "Authorization: Api-Key ${PROWLER_API_KEY}" \
-H "Authorization: Api-Key ${PROWLER_CLOUD_API_KEY}" \
-F "file=@/path/to/findings.ocsf.json" \
https://api.prowler.com/api/v1/ingestions
```
@@ -229,7 +237,7 @@ Upload a `.ocsf.json` file containing a JSON array of OCSF Detection Finding rec
**Request:**
```bash
curl -X POST \
-H "Authorization: Api-Key ${PROWLER_API_KEY}" \
-H "Authorization: Api-Key ${PROWLER_CLOUD_API_KEY}" \
-F "file=@scan-results.ocsf.json" \
https://api.prowler.com/api/v1/ingestions
```
@@ -267,7 +275,7 @@ Monitor the progress of an ingestion job.
**Request:**
```bash
curl -X GET \
-H "Authorization: Api-Key ${PROWLER_API_KEY}" \
-H "Authorization: Api-Key ${PROWLER_CLOUD_API_KEY}" \
-H "Accept: application/vnd.api+json" \
https://api.prowler.com/api/v1/ingestions/3650fef9-8e5f-4808-a95f-74f0afae8499
```
@@ -319,7 +327,7 @@ Retrieve a list of ingestion jobs for the tenant.
**Request:**
```bash
curl -X GET \
-H "Authorization: Api-Key ${PROWLER_API_KEY}" \
-H "Authorization: Api-Key ${PROWLER_CLOUD_API_KEY}" \
-H "Accept: application/vnd.api+json" \
"https://api.prowler.com/api/v1/ingestions?filter[status]=completed&page[size]=10"
```
@@ -333,7 +341,7 @@ Retrieve error details for a specific ingestion job.
**Request:**
```bash
curl -X GET \
-H "Authorization: Api-Key ${PROWLER_API_KEY}" \
-H "Authorization: Api-Key ${PROWLER_CLOUD_API_KEY}" \
-H "Accept: application/vnd.api+json" \
https://api.prowler.com/api/v1/ingestions/3650fef9-8e5f-4808-a95f-74f0afae8499/errors
```
@@ -363,9 +371,9 @@ Prowler must be installed in the CI/CD environment before running scans. Refer t
- name: Run Prowler and upload to Cloud
env:
PROWLER_API_KEY: ${{ secrets.PROWLER_API_KEY }}
PROWLER_CLOUD_API_KEY: ${{ secrets.PROWLER_CLOUD_API_KEY }}
run: |
prowler aws --services s3,iam --export-ocsf
prowler aws --services s3,iam --push-to-cloud
```
### GitLab CI
@@ -374,9 +382,9 @@ Prowler must be installed in the CI/CD environment before running scans. Refer t
prowler_scan:
script:
- pip install prowler
- prowler aws --services s3,iam --export-ocsf
- prowler aws --services s3,iam --push-to-cloud
variables:
PROWLER_API_KEY: $PROWLER_API_KEY
PROWLER_CLOUD_API_KEY: $PROWLER_CLOUD_API_KEY
```
## Billing impact
@@ -392,6 +400,23 @@ For pricing details, see [Prowler Cloud Pricing](https://prowler.com/pricing).
## Troubleshooting
### "Push to Prowler Cloud skipped: no API key configured"
- Set the `PROWLER_CLOUD_API_KEY` environment variable before running the scan
- Verify the variable is exported and not empty
### "Push to Prowler Cloud failed: could not reach the Prowler Cloud API"
- Verify network connectivity to `api.prowler.com`
- Check firewall rules allow outbound HTTPS traffic
- Confirm the API endpoint is not blocked by proxy settings
- If using a custom base URL via `PROWLER_CLOUD_API_BASE_URL`, verify it is correct
### "Push to Prowler Cloud failed: this feature is only available with a Prowler Cloud subscription"
- The API returned HTTP 402, meaning your tenant does not have an active subscription
- Visit [Prowler Cloud Pricing](https://prowler.com/pricing) to review available plans
### HTTP 401 Unauthorized
- Verify the API key is valid and not revoked
@@ -408,9 +433,3 @@ For pricing details, see [Prowler Cloud Pricing](https://prowler.com/pricing).
- Check the `/api/v1/ingestions/{id}/errors` endpoint for details
- Verify the OCSF file format is valid
- Ensure the file contains Detection Finding records
### CLI reports "could not reach the Prowler Cloud API"
- Verify network connectivity to `api.prowler.com`
- Check firewall rules allow outbound HTTPS traffic
- Confirm the API endpoint is not blocked by proxy settings

2
docs/user-guide/tutorials/prowler-app-rbac.mdx
View File

@@ -238,6 +238,6 @@ To grant all administrative permissions, select the **Grant all admin permission
The following permissions are available exclusively in **Prowler Cloud**:
**Manage Ingestions:** Submit and manage findings ingestion jobs via the API. Required to upload OCSF scan results using the `--export-ocsf` CLI flag or the ingestion endpoints. See [Import Findings](/user-guide/tutorials/prowler-app-import-findings) for details.
**Manage Ingestions:** Submit and manage findings ingestion jobs via the API. Required to upload OCSF scan results using the `--push-to-cloud` CLI flag or the ingestion endpoints. See [Import Findings](/user-guide/tutorials/prowler-app-import-findings) for details.
**Manage Billing:** Access and manage billing settings, subscription plans, and payment methods.

21
prowler/__main__.py
View File

@@ -529,7 +529,7 @@ def prowler():
provider=global_provider, stats=stats
)
if getattr(args, "export_ocsf", False):
if getattr(args, "push_to_cloud", False):
if not ocsf_output or not getattr(ocsf_output, "file_path", None):
tmp_ocsf = tempfile.NamedTemporaryFile(
suffix=json_ocsf_file_suffix, delete=False
@@ -541,49 +541,50 @@ def prowler():
tmp_ocsf.close()
ocsf_output.batch_write_data_to_file()
print(
f"{Style.BRIGHT}\nExporting OCSF to Prowler Cloud, please wait...{Style.RESET_ALL}"
f"{Style.BRIGHT}\nPushing findings to Prowler Cloud, please wait...{Style.RESET_ALL}"
)
try:
response = send_ocsf_to_api(ocsf_output.file_path)
except ValueError:
print(
f"{Style.BRIGHT}{Fore.YELLOW}\nOCSF export skipped: no API key configured. "
"Set the PROWLER_API_KEY environment variable to enable it. "
f"{Style.BRIGHT}{Fore.YELLOW}\nPush to Prowler Cloud skipped: no API key configured. "
"Set the PROWLER_CLOUD_API_KEY environment variable to enable it. "
f"Scan results were saved to {ocsf_output.file_path}{Style.RESET_ALL}"
)
except requests.ConnectionError:
print(
f"{Style.BRIGHT}{Fore.RED}\nOCSF export failed: could not reach the Prowler Cloud API at "
f"{Style.BRIGHT}{Fore.RED}\nPush to Prowler Cloud failed: could not reach the Prowler Cloud API at "
f"{cloud_api_base_url}. Check the URL and your network connection. "
f"Scan results were saved to {ocsf_output.file_path}{Style.RESET_ALL}"
)
except requests.HTTPError as http_err:
if http_err.response.status_code == 402:
print(
f"{Style.BRIGHT}{Fore.RED}\nOCSF export failed: "
f"{Style.BRIGHT}{Fore.RED}\nPush to Prowler Cloud failed: "
"this feature is only available with a Prowler Cloud subscription. "
f"Scan results were saved to {ocsf_output.file_path}{Style.RESET_ALL}"
)
else:
print(
f"{Style.BRIGHT}{Fore.RED}\nOCSF export failed: the API returned HTTP {http_err.response.status_code}. "
f"{Style.BRIGHT}{Fore.RED}\nPush to Prowler Cloud failed: the API returned HTTP {http_err.response.status_code}. "
"Verify your API key is valid and has the right permissions. "
f"Scan results were saved to {ocsf_output.file_path}{Style.RESET_ALL}"
)
except Exception as error:
print(
f"{Style.BRIGHT}{Fore.RED}\nOCSF export failed unexpectedly: {error}. "
f"{Style.BRIGHT}{Fore.RED}\nPush to Prowler Cloud failed unexpectedly: {error}. "
f"Scan results were saved to {ocsf_output.file_path}{Style.RESET_ALL}"
)
else:
job_id = response.get("data", {}).get("id") if response else None
if job_id:
print(
f"{Style.BRIGHT}{Fore.GREEN}\nOCSF export accepted. Ingestion job: {job_id}{Style.RESET_ALL}"
f"{Style.BRIGHT}{Fore.GREEN}\nFindings successfully pushed to Prowler Cloud. Ingestion job: {job_id}"
f"\nSee more details here: https://cloud.prowler.com/scans{Style.RESET_ALL}"
)
else:
logger.warning(
"OCSF export: unexpected API response (missing ingestion job ID). "
"Push to Prowler Cloud: unexpected API response (missing ingestion job ID). "
f"Scan results were saved to {ocsf_output.file_path}"
)

4
prowler/config/config.py
View File

@@ -122,8 +122,8 @@ encoding_format_utf_8 = "utf-8"
available_output_formats = ["csv", "json-asff", "json-ocsf", "html"]
# Prowler Cloud API settings
cloud_api_base_url = os.getenv("PROWLER_CLOUD_API_BASE", "https://api.prowler.com")
cloud_api_key = os.getenv("PROWLER_API_KEY", "")
cloud_api_base_url = os.getenv("PROWLER_CLOUD_API_BASE_URL", "https://api.prowler.com")
cloud_api_key = os.getenv("PROWLER_CLOUD_API_KEY", "")
cloud_api_ingestion_path = "/api/v1/ingestions"

9
prowler/lib/cli/parser.py
View File

@@ -217,12 +217,13 @@ Detailed documentation at https://docs.prowler.com
help="Set the output timestamp format as unix timestamps instead of iso format timestamps (default mode).",
)
common_outputs_parser.add_argument(
"--export-ocsf",
"--push-to-cloud",
action="store_true",
help=(
"Send OCSF output to Prowler Cloud ingestion endpoint. "
"Requires PROWLER_API_KEY environment variable. "
"For the IaC provider, --provider-uid is also required."
"Send findings in OCSF format to Prowler Cloud. "
"Requires PROWLER_CLOUD_API_KEY environment variable. "
"For the IaC provider, --provider-uid is also required. "
"More details here: https://goto.prowler.com/import-findings"
),
)

8
prowler/lib/outputs/ocsf/ingestion.py
View File

@@ -21,9 +21,9 @@ def send_ocsf_to_api(
Args:
file_path: Path to the OCSF JSON file to upload.
base_url: API base URL. Falls back to PROWLER_CLOUD_API_BASE env var,
base_url: API base URL. Falls back to PROWLER_CLOUD_API_BASE_URL env var,
then to https://api.prowler.com.
api_key: API key. Falls back to PROWLER_API_KEY env var.
api_key: API key. Falls back to PROWLER_CLOUD_API_KEY env var.
timeout: Request timeout in seconds.
Returns:
@@ -42,7 +42,9 @@ def send_ocsf_to_api(
api_key = api_key or cloud_api_key
if not api_key:
raise ValueError("Missing API key. Set PROWLER_API_KEY environment variable.")
raise ValueError(
"Missing API key. Set PROWLER_CLOUD_API_KEY environment variable."
)
base_url = base_url or cloud_api_base_url
base_url = base_url.rstrip("/")

8
prowler/providers/iac/lib/arguments/arguments.py
View File

@@ -74,7 +74,7 @@ def init_parser(self):
"--provider-uid",
dest="provider_uid",
default=None,
help="Unique identifier for the IaC provider. Required when using --export-ocsf.",
help="Unique identifier for the IaC provider. Required when using --push-to-cloud.",
)
@@ -88,12 +88,12 @@ def validate_arguments(arguments):
False,
"--scan-path (-P) and --scan-repository-url (-R) are mutually exclusive. Please specify only one.",
)
export_ocsf = getattr(arguments, "export_ocsf", False)
push_to_cloud = getattr(arguments, "push_to_cloud", False)
provider_uid = getattr(arguments, "provider_uid", None)
if export_ocsf and not provider_uid:
if push_to_cloud and not provider_uid:
return (
False,
"--provider-uid is required when using --export-ocsf with the IAC provider.",
"--provider-uid is required when using --push-to-cloud with the IAC provider.",
)
if provider_uid and not re.match(
r"^(https?://|git@|ssh://)[^\s/]+[^\s]*\.git$|^(https?://)[^\s/]+[^\s]*$",

20
tests/providers/iac/lib/arguments/iac_arguments_test.py
View File

@@ -33,12 +33,12 @@ def test_validate_arguments_mutual_exclusion():
assert msg == ""
def test_validate_arguments_export_ocsf_requires_provider_uid():
# --export-ocsf without provider_uid should fail
def test_validate_arguments_push_to_cloud_requires_provider_uid():
# --push-to-cloud without provider_uid should fail
args = Args(
scan_path=".",
scan_repository_url=None,
export_ocsf=True,
push_to_cloud=True,
provider_uid=None,
)
valid, msg = iac_arguments.validate_arguments(args)
@@ -46,12 +46,12 @@ def test_validate_arguments_export_ocsf_requires_provider_uid():
assert "--provider-uid is required" in msg
def test_validate_arguments_export_ocsf_with_provider_uid_passes():
# --export-ocsf with valid provider_uid should pass
def test_validate_arguments_push_to_cloud_with_provider_uid_passes():
# --push-to-cloud with valid provider_uid should pass
args = Args(
scan_path=".",
scan_repository_url=None,
export_ocsf=True,
push_to_cloud=True,
provider_uid="https://github.com/user/repo.git",
)
valid, msg = iac_arguments.validate_arguments(args)
@@ -59,19 +59,19 @@ def test_validate_arguments_export_ocsf_with_provider_uid_passes():
assert msg == ""
def test_validate_arguments_no_export_ocsf_without_provider_uid_passes():
# No --export-ocsf, no provider_uid — should pass
def test_validate_arguments_no_push_to_cloud_without_provider_uid_passes():
# No --push-to-cloud, no provider_uid — should pass
args = Args(
scan_path=".",
scan_repository_url=None,
export_ocsf=False,
push_to_cloud=False,
provider_uid=None,
)
valid, msg = iac_arguments.validate_arguments(args)
assert valid
assert msg == ""
# No export_ocsf attr at all — should pass
# No push_to_cloud attr at all — should pass
args = Args(scan_path=".", scan_repository_url=None)
valid, msg = iac_arguments.validate_arguments(args)
assert valid