mirror of
https://github.com/prowler-cloud/prowler.git
synced 2026-03-22 03:08:23 +00:00
Compare commits
2 Commits
PROWLER-12
...
copilot/su
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b818dae347 | ||
|
|
651b7cf4ac |
@@ -132,69 +132,70 @@ Only **Detection Finding** (`class_uid: 2004`) records are accepted. Other OCSF
|
||||
|
||||
## Required permissions
|
||||
|
||||
The **Manage Ingestions** RBAC permission controls access to the ingestion endpoints. Without this permission, findings cannot be submitted via the API or `--export-ocsf`.
|
||||
The **Manage Ingestions** RBAC permission controls access to the ingestion endpoints. Without this permission, findings cannot be submitted via the API or `--push-to-cloud`.
|
||||
|
||||
For more information about RBAC permissions, refer to the [Prowler App RBAC documentation](/user-guide/tutorials/prowler-app-rbac).
|
||||
|
||||
## Using the CLI
|
||||
|
||||
The `--export-ocsf` flag uploads scan results directly to Prowler Cloud after a scan completes. This approach automates the ingestion process without manual file uploads.
|
||||
The `--push-to-cloud` flag uploads scan results directly to Prowler Cloud after a scan completes. This approach automates the ingestion process without manual file uploads.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- A valid Prowler Cloud API key (see [API Keys](/user-guide/tutorials/prowler-app-api-keys))
|
||||
- The `PROWLER_API_KEY` environment variable configured
|
||||
- The `PROWLER_CLOUD_API_KEY` environment variable configured
|
||||
|
||||
### Basic usage
|
||||
|
||||
```bash
|
||||
export PROWLER_API_KEY="pk_your_api_key_here"
|
||||
export PROWLER_CLOUD_API_KEY="pk_your_api_key_here"
|
||||
|
||||
prowler aws --export-ocsf
|
||||
prowler aws --push-to-cloud
|
||||
```
|
||||
|
||||
### Combining with output formats
|
||||
|
||||
When using `--export-ocsf` with custom output formats that exclude OCSF, Prowler generates a temporary OCSF file for upload:
|
||||
When using `--push-to-cloud` with custom output formats that exclude OCSF, Prowler generates a temporary OCSF file for upload:
|
||||
|
||||
The temporary OCSF file is saved in the system temporary directory and not in the output path passed with `-o`.
|
||||
|
||||
```bash
|
||||
prowler aws --services accessanalyzer -M csv --export-ocsf -o /tmp/scan-output
|
||||
prowler aws --services accessanalyzer -M csv --push-to-cloud -o /tmp/scan-output
|
||||
```
|
||||
|
||||
When default output formats include OCSF, Prowler reuses the existing file. Default output formats include JSON-OCSF:
|
||||
|
||||
```bash
|
||||
prowler aws --services accessanalyzer --export-ocsf -o /tmp/scan-output
|
||||
prowler aws --services accessanalyzer --push-to-cloud -o /tmp/scan-output
|
||||
```
|
||||
|
||||
### CLI output examples
|
||||
|
||||
**Successful upload:**
|
||||
```
|
||||
Exporting OCSF to Prowler Cloud, please wait...
|
||||
Pushing findings to Prowler Cloud, please wait...
|
||||
|
||||
OCSF export accepted. Ingestion job: fa8bc8c5-4925-46a0-9fe0-f6575905e094
|
||||
Findings successfully pushed to Prowler Cloud. Ingestion job: fa8bc8c5-4925-46a0-9fe0-f6575905e094
|
||||
See more details here: https://cloud.prowler.com/scans
|
||||
```
|
||||
|
||||
**Missing API key:**
|
||||
```
|
||||
WARNING: OCSF export skipped: no API key configured. Set the PROWLER_API_KEY
|
||||
WARNING: Push to cloud skipped: no API key configured. Set the PROWLER_CLOUD_API_KEY
|
||||
environment variable to enable it. Scan results were saved to
|
||||
/tmp/scan-output/prowler-output-123456789012-20260217131755.ocsf.json
|
||||
```
|
||||
|
||||
**API unreachable:**
|
||||
```
|
||||
WARNING: OCSF export skipped: could not reach the Prowler Cloud API at
|
||||
WARNING: Push to cloud failed: could not reach the Prowler Cloud API at
|
||||
https://api.prowler.com. Check the URL and your network connection. Scan results
|
||||
were saved to /tmp/scan-output/prowler-output-123456789012-20260217131755.ocsf.json
|
||||
```
|
||||
|
||||
**Invalid API key:**
|
||||
```
|
||||
WARNING: OCSF export failed: the API returned HTTP 401. Verify your API key is
|
||||
WARNING: Push to cloud failed: the API returned HTTP 401. Verify your API key is
|
||||
valid and has the right permissions. Scan results were saved to
|
||||
/tmp/scan-output/prowler-output-123456789012-20260217131755.ocsf.json
|
||||
```
|
||||
@@ -212,10 +213,10 @@ The Ingestion API provides endpoints for submitting OCSF files and monitoring jo
|
||||
Include the API key in the `Authorization` header:
|
||||
|
||||
```bash
|
||||
export PROWLER_API_KEY="pk_your_api_key_here"
|
||||
export PROWLER_CLOUD_API_KEY="pk_your_api_key_here"
|
||||
|
||||
curl -X POST \
|
||||
-H "Authorization: Api-Key ${PROWLER_API_KEY}" \
|
||||
-H "Authorization: Api-Key ${PROWLER_CLOUD_API_KEY}" \
|
||||
-F "file=@/path/to/findings.ocsf.json" \
|
||||
https://api.prowler.com/api/v1/ingestions
|
||||
```
|
||||
@@ -229,7 +230,7 @@ Upload a `.ocsf.json` file containing a JSON array of OCSF Detection Finding rec
|
||||
**Request:**
|
||||
```bash
|
||||
curl -X POST \
|
||||
-H "Authorization: Api-Key ${PROWLER_API_KEY}" \
|
||||
-H "Authorization: Api-Key ${PROWLER_CLOUD_API_KEY}" \
|
||||
-F "file=@scan-results.ocsf.json" \
|
||||
https://api.prowler.com/api/v1/ingestions
|
||||
```
|
||||
@@ -267,7 +268,7 @@ Monitor the progress of an ingestion job.
|
||||
**Request:**
|
||||
```bash
|
||||
curl -X GET \
|
||||
-H "Authorization: Api-Key ${PROWLER_API_KEY}" \
|
||||
-H "Authorization: Api-Key ${PROWLER_CLOUD_API_KEY}" \
|
||||
-H "Accept: application/vnd.api+json" \
|
||||
https://api.prowler.com/api/v1/ingestions/3650fef9-8e5f-4808-a95f-74f0afae8499
|
||||
```
|
||||
@@ -319,7 +320,7 @@ Retrieve a list of ingestion jobs for the tenant.
|
||||
**Request:**
|
||||
```bash
|
||||
curl -X GET \
|
||||
-H "Authorization: Api-Key ${PROWLER_API_KEY}" \
|
||||
-H "Authorization: Api-Key ${PROWLER_CLOUD_API_KEY}" \
|
||||
-H "Accept: application/vnd.api+json" \
|
||||
"https://api.prowler.com/api/v1/ingestions?filter[status]=completed&page[size]=10"
|
||||
```
|
||||
@@ -333,7 +334,7 @@ Retrieve error details for a specific ingestion job.
|
||||
**Request:**
|
||||
```bash
|
||||
curl -X GET \
|
||||
-H "Authorization: Api-Key ${PROWLER_API_KEY}" \
|
||||
-H "Authorization: Api-Key ${PROWLER_CLOUD_API_KEY}" \
|
||||
-H "Accept: application/vnd.api+json" \
|
||||
https://api.prowler.com/api/v1/ingestions/3650fef9-8e5f-4808-a95f-74f0afae8499/errors
|
||||
```
|
||||
@@ -363,9 +364,9 @@ Prowler must be installed in the CI/CD environment before running scans. Refer t
|
||||
|
||||
- name: Run Prowler and upload to Cloud
|
||||
env:
|
||||
PROWLER_API_KEY: ${{ secrets.PROWLER_API_KEY }}
|
||||
PROWLER_CLOUD_API_KEY: ${{ secrets.PROWLER_CLOUD_API_KEY }}
|
||||
run: |
|
||||
prowler aws --services s3,iam --export-ocsf
|
||||
prowler aws --services s3,iam --push-to-cloud
|
||||
```
|
||||
|
||||
### GitLab CI
|
||||
@@ -374,9 +375,9 @@ Prowler must be installed in the CI/CD environment before running scans. Refer t
|
||||
prowler_scan:
|
||||
script:
|
||||
- pip install prowler
|
||||
- prowler aws --services s3,iam --export-ocsf
|
||||
- prowler aws --services s3,iam --push-to-cloud
|
||||
variables:
|
||||
PROWLER_API_KEY: $PROWLER_API_KEY
|
||||
PROWLER_CLOUD_API_KEY: $PROWLER_CLOUD_API_KEY
|
||||
```
|
||||
|
||||
## Billing impact
|
||||
|
||||
@@ -238,6 +238,6 @@ To grant all administrative permissions, select the **Grant all admin permission
|
||||
|
||||
The following permissions are available exclusively in **Prowler Cloud**:
|
||||
|
||||
**Manage Ingestions:** Submit and manage findings ingestion jobs via the API. Required to upload OCSF scan results using the `--export-ocsf` CLI flag or the ingestion endpoints. See [Import Findings](/user-guide/tutorials/prowler-app-import-findings) for details.
|
||||
**Manage Ingestions:** Submit and manage findings ingestion jobs via the API. Required to upload OCSF scan results using the `--push-to-cloud` CLI flag or the ingestion endpoints. See [Import Findings](/user-guide/tutorials/prowler-app-import-findings) for details.
|
||||
|
||||
**Manage Billing:** Access and manage billing settings, subscription plans, and payment methods.
|
||||
|
||||
@@ -529,7 +529,7 @@ def prowler():
|
||||
provider=global_provider, stats=stats
|
||||
)
|
||||
|
||||
if getattr(args, "export_ocsf", False):
|
||||
if getattr(args, "push_to_cloud", False):
|
||||
if not ocsf_output or not getattr(ocsf_output, "file_path", None):
|
||||
tmp_ocsf = tempfile.NamedTemporaryFile(
|
||||
suffix=json_ocsf_file_suffix, delete=False
|
||||
@@ -541,49 +541,50 @@ def prowler():
|
||||
tmp_ocsf.close()
|
||||
ocsf_output.batch_write_data_to_file()
|
||||
print(
|
||||
f"{Style.BRIGHT}\nExporting OCSF to Prowler Cloud, please wait...{Style.RESET_ALL}"
|
||||
f"{Style.BRIGHT}\nPushing findings to Prowler Cloud, please wait...{Style.RESET_ALL}"
|
||||
)
|
||||
try:
|
||||
response = send_ocsf_to_api(ocsf_output.file_path)
|
||||
except ValueError:
|
||||
print(
|
||||
f"{Style.BRIGHT}{Fore.YELLOW}\nOCSF export skipped: no API key configured. "
|
||||
"Set the PROWLER_API_KEY environment variable to enable it. "
|
||||
f"{Style.BRIGHT}{Fore.YELLOW}\nPush to Prowler Cloud skipped: no API key configured. "
|
||||
"Set the PROWLER_CLOUD_API_KEY environment variable to enable it. "
|
||||
f"Scan results were saved to {ocsf_output.file_path}{Style.RESET_ALL}"
|
||||
)
|
||||
except requests.ConnectionError:
|
||||
print(
|
||||
f"{Style.BRIGHT}{Fore.RED}\nOCSF export failed: could not reach the Prowler Cloud API at "
|
||||
f"{Style.BRIGHT}{Fore.RED}\nPush to Prowler Cloud failed: could not reach the Prowler Cloud API at "
|
||||
f"{cloud_api_base_url}. Check the URL and your network connection. "
|
||||
f"Scan results were saved to {ocsf_output.file_path}{Style.RESET_ALL}"
|
||||
)
|
||||
except requests.HTTPError as http_err:
|
||||
if http_err.response.status_code == 402:
|
||||
print(
|
||||
f"{Style.BRIGHT}{Fore.RED}\nOCSF export failed: "
|
||||
f"{Style.BRIGHT}{Fore.RED}\nPush to Prowler Cloud failed: "
|
||||
"this feature is only available with a Prowler Cloud subscription. "
|
||||
f"Scan results were saved to {ocsf_output.file_path}{Style.RESET_ALL}"
|
||||
)
|
||||
else:
|
||||
print(
|
||||
f"{Style.BRIGHT}{Fore.RED}\nOCSF export failed: the API returned HTTP {http_err.response.status_code}. "
|
||||
f"{Style.BRIGHT}{Fore.RED}\nPush to Prowler Cloud failed: the API returned HTTP {http_err.response.status_code}. "
|
||||
"Verify your API key is valid and has the right permissions. "
|
||||
f"Scan results were saved to {ocsf_output.file_path}{Style.RESET_ALL}"
|
||||
)
|
||||
except Exception as error:
|
||||
print(
|
||||
f"{Style.BRIGHT}{Fore.RED}\nOCSF export failed unexpectedly: {error}. "
|
||||
f"{Style.BRIGHT}{Fore.RED}\nPush to Prowler Cloud failed unexpectedly: {error}. "
|
||||
f"Scan results were saved to {ocsf_output.file_path}{Style.RESET_ALL}"
|
||||
)
|
||||
else:
|
||||
job_id = response.get("data", {}).get("id") if response else None
|
||||
if job_id:
|
||||
print(
|
||||
f"{Style.BRIGHT}{Fore.GREEN}\nOCSF export accepted. Ingestion job: {job_id}{Style.RESET_ALL}"
|
||||
f"{Style.BRIGHT}{Fore.GREEN}\nFindings successfully pushed to Prowler Cloud. Ingestion job: {job_id}"
|
||||
f"\nSee more details here: https://cloud.prowler.com/scans{Style.RESET_ALL}"
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"OCSF export: unexpected API response (missing ingestion job ID). "
|
||||
"Push to Prowler Cloud: unexpected API response (missing ingestion job ID). "
|
||||
f"Scan results were saved to {ocsf_output.file_path}"
|
||||
)
|
||||
|
||||
|
||||
@@ -122,8 +122,8 @@ encoding_format_utf_8 = "utf-8"
|
||||
available_output_formats = ["csv", "json-asff", "json-ocsf", "html"]
|
||||
|
||||
# Prowler Cloud API settings
|
||||
cloud_api_base_url = os.getenv("PROWLER_CLOUD_API_BASE", "https://api.prowler.com")
|
||||
cloud_api_key = os.getenv("PROWLER_API_KEY", "")
|
||||
cloud_api_base_url = os.getenv("PROWLER_CLOUD_API_BASE_URL", "https://api.prowler.com")
|
||||
cloud_api_key = os.getenv("PROWLER_CLOUD_API_KEY", "")
|
||||
cloud_api_ingestion_path = "/api/v1/ingestions"
|
||||
|
||||
|
||||
|
||||
@@ -217,12 +217,13 @@ Detailed documentation at https://docs.prowler.com
|
||||
help="Set the output timestamp format as unix timestamps instead of iso format timestamps (default mode).",
|
||||
)
|
||||
common_outputs_parser.add_argument(
|
||||
"--export-ocsf",
|
||||
"--push-to-cloud",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Send OCSF output to Prowler Cloud ingestion endpoint. "
|
||||
"Requires PROWLER_API_KEY environment variable. "
|
||||
"For the IaC provider, --provider-uid is also required."
|
||||
"Send findings in OCSF format to Prowler Cloud. "
|
||||
"Requires PROWLER_CLOUD_API_KEY environment variable. "
|
||||
"For the IaC provider, --provider-uid is also required. "
|
||||
"More details here: https://goto.prowler.com/import-findings"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@@ -21,9 +21,9 @@ def send_ocsf_to_api(
|
||||
|
||||
Args:
|
||||
file_path: Path to the OCSF JSON file to upload.
|
||||
base_url: API base URL. Falls back to PROWLER_CLOUD_API_BASE env var,
|
||||
base_url: API base URL. Falls back to PROWLER_CLOUD_API_BASE_URL env var,
|
||||
then to https://api.prowler.com.
|
||||
api_key: API key. Falls back to PROWLER_API_KEY env var.
|
||||
api_key: API key. Falls back to PROWLER_CLOUD_API_KEY env var.
|
||||
timeout: Request timeout in seconds.
|
||||
|
||||
Returns:
|
||||
@@ -42,7 +42,9 @@ def send_ocsf_to_api(
|
||||
|
||||
api_key = api_key or cloud_api_key
|
||||
if not api_key:
|
||||
raise ValueError("Missing API key. Set PROWLER_API_KEY environment variable.")
|
||||
raise ValueError(
|
||||
"Missing API key. Set PROWLER_CLOUD_API_KEY environment variable."
|
||||
)
|
||||
|
||||
base_url = base_url or cloud_api_base_url
|
||||
base_url = base_url.rstrip("/")
|
||||
|
||||
@@ -74,7 +74,7 @@ def init_parser(self):
|
||||
"--provider-uid",
|
||||
dest="provider_uid",
|
||||
default=None,
|
||||
help="Unique identifier for the IaC provider. Required when using --export-ocsf.",
|
||||
help="Unique identifier for the IaC provider. Required when using --push-to-cloud.",
|
||||
)
|
||||
|
||||
|
||||
@@ -88,12 +88,12 @@ def validate_arguments(arguments):
|
||||
False,
|
||||
"--scan-path (-P) and --scan-repository-url (-R) are mutually exclusive. Please specify only one.",
|
||||
)
|
||||
export_ocsf = getattr(arguments, "export_ocsf", False)
|
||||
push_to_cloud = getattr(arguments, "push_to_cloud", False)
|
||||
provider_uid = getattr(arguments, "provider_uid", None)
|
||||
if export_ocsf and not provider_uid:
|
||||
if push_to_cloud and not provider_uid:
|
||||
return (
|
||||
False,
|
||||
"--provider-uid is required when using --export-ocsf with the IAC provider.",
|
||||
"--provider-uid is required when using --push-to-cloud with the IAC provider.",
|
||||
)
|
||||
if provider_uid and not re.match(
|
||||
r"^(https?://|git@|ssh://)[^\s/]+[^\s]*\.git$|^(https?://)[^\s/]+[^\s]*$",
|
||||
|
||||
@@ -33,12 +33,12 @@ def test_validate_arguments_mutual_exclusion():
|
||||
assert msg == ""
|
||||
|
||||
|
||||
def test_validate_arguments_export_ocsf_requires_provider_uid():
|
||||
# --export-ocsf without provider_uid should fail
|
||||
def test_validate_arguments_push_to_cloud_requires_provider_uid():
|
||||
# --push-to-cloud without provider_uid should fail
|
||||
args = Args(
|
||||
scan_path=".",
|
||||
scan_repository_url=None,
|
||||
export_ocsf=True,
|
||||
push_to_cloud=True,
|
||||
provider_uid=None,
|
||||
)
|
||||
valid, msg = iac_arguments.validate_arguments(args)
|
||||
@@ -46,12 +46,12 @@ def test_validate_arguments_export_ocsf_requires_provider_uid():
|
||||
assert "--provider-uid is required" in msg
|
||||
|
||||
|
||||
def test_validate_arguments_export_ocsf_with_provider_uid_passes():
|
||||
# --export-ocsf with valid provider_uid should pass
|
||||
def test_validate_arguments_push_to_cloud_with_provider_uid_passes():
|
||||
# --push-to-cloud with valid provider_uid should pass
|
||||
args = Args(
|
||||
scan_path=".",
|
||||
scan_repository_url=None,
|
||||
export_ocsf=True,
|
||||
push_to_cloud=True,
|
||||
provider_uid="https://github.com/user/repo.git",
|
||||
)
|
||||
valid, msg = iac_arguments.validate_arguments(args)
|
||||
@@ -59,19 +59,19 @@ def test_validate_arguments_export_ocsf_with_provider_uid_passes():
|
||||
assert msg == ""
|
||||
|
||||
|
||||
def test_validate_arguments_no_export_ocsf_without_provider_uid_passes():
|
||||
# No --export-ocsf, no provider_uid — should pass
|
||||
def test_validate_arguments_no_push_to_cloud_without_provider_uid_passes():
|
||||
# No --push-to-cloud, no provider_uid — should pass
|
||||
args = Args(
|
||||
scan_path=".",
|
||||
scan_repository_url=None,
|
||||
export_ocsf=False,
|
||||
push_to_cloud=False,
|
||||
provider_uid=None,
|
||||
)
|
||||
valid, msg = iac_arguments.validate_arguments(args)
|
||||
assert valid
|
||||
assert msg == ""
|
||||
|
||||
# No export_ocsf attr at all — should pass
|
||||
# No push_to_cloud attr at all — should pass
|
||||
args = Args(scan_path=".", scan_repository_url=None)
|
||||
valid, msg = iac_arguments.validate_arguments(args)
|
||||
assert valid
|
||||
|
||||
Reference in New Issue
Block a user