A Python script to bulk-provision cloud providers in Prowler Cloud/App via REST API. This tool streamlines the process of adding multiple cloud providers to Prowler by reading configuration from YAML and making API calls with concurrency and retry support.

Supported Providers

AWS (Amazon Web Services)
Azure (Microsoft Azure)
GCP (Google Cloud Platform)
Kubernetes
M365 (Microsoft 365)
GitHub

Features

Concurrent Processing: Configurable concurrency for faster bulk operations
Retry Logic: Built-in retry mechanism for handling temporary API failures
Dry-Run Mode: Test configuration without making actual API calls
Flexible Authentication: Supports various authentication methods per provider
Error Handling: Comprehensive error reporting and validation
Connection Testing: Built-in provider connection verification
AWS Organizations Support: Automated YAML generation for all accounts in an AWS Organization

How It Works

The script uses a two-step process to provision providers in Prowler:

Provider Creation: Creates the provider with basic information (provider type, UID, alias)
Secret Creation: Creates and links authentication credentials as a separate secret resource

This two-step approach follows the Prowler API design where providers and their credentials are managed as separate but linked resources, providing better security and flexibility.

Installation

Requirements

Python 3.7 or higher
Required packages (install via requirements.txt)

Setup

Clone or navigate to the Prowler repository:

cd contrib/other-contrib/provider-bulk-importer

Install dependencies:
```
pip install -r requirements.txt
```
Get your Prowler API key:
- Prowler Cloud: Create an API key at https://api.prowler.com
- Self-hosted Prowler App: Create an API key in your local instance
- Click Profile → Account → Create API Key

export PROWLER_API_KEY="pk_example-api-key"

For detailed instructions on creating API keys, see: https://docs.prowler.com/user-guide/tutorials/prowler-app-api-keys

AWS Organizations Integration

For organizations with many AWS accounts, use the included aws_org_generator.py script to automatically generate configuration for all accounts in your AWS Organization.

📖 Full Guide: See AWS Organizations Bulk Provisioning Tutorial for complete documentation, examples, and troubleshooting.

Prerequisites

Before using the AWS Organizations generator, deploy the ProwlerRole across all accounts using CloudFormation StackSets:

Documentation: Deploying Prowler IAM Roles Across AWS Organizations

Quick Start

Install additional dependencies:

pip install -r requirements-aws-org.txt

Generate YAML configuration for all organization accounts:

python aws_org_generator.py -o aws-accounts.yaml --external-id example-external-id

Run bulk provisioning:

python prowler_bulk_provisioning.py aws-accounts.yaml

AWS Organizations Generator Options

python aws_org_generator.py -o aws-accounts.yaml \
  --role-name ProwlerRole \
  --external-id my-external-id \
  --exclude 123456789012 \
  --profile org-management

Option	Description	Default
`-o, --output`	Output YAML file path	`aws-org-accounts.yaml`
`--role-name`	IAM role name across accounts	`ProwlerRole`
`--external-id`	External ID for role assumption	None (recommended)
`--session-name`	Session name for role assumption	None
`--duration-seconds`	Session duration in seconds	None
`--alias-format`	Alias template: `{name}`, `{id}`, `{email}`	`{name}`
`--exclude`	Comma-separated account IDs to exclude	None
`--include`	Comma-separated account IDs to include	None
`--profile`	AWS CLI profile name	Default credentials
`--region`	AWS region	`us-east-1`
`--dry-run`	Print to stdout without writing	`False`

Examples

Generate config for all accounts with custom external ID:

python aws_org_generator.py -o aws-accounts.yaml --external-id prowler-2024-abc123

Exclude management account:

python aws_org_generator.py -o aws-accounts.yaml \
  --external-id prowler-ext-id \
  --exclude 123456789012

Use specific AWS profile:

python aws_org_generator.py -o aws-accounts.yaml \
  --profile org-admin \
  --external-id prowler-ext-id

Custom alias format:

python aws_org_generator.py -o aws-accounts.yaml \
  --alias-format "{name}-{id}" \
  --external-id prowler-ext-id

Configuration

Environment Variables

export PROWLER_API_KEY="pk_example-api-key"
export PROWLER_API_BASE="https://api.prowler.com/api/v1"  # Optional, defaults to Prowler Cloud

Provider Configuration Files

Create a configuration file (YAML recommended) listing the providers to add:

YAML Format (Recommended)

# providers.yaml
- provider: aws
  uid: "123456789012"              # AWS Account ID
  alias: "prod-root"
  auth_method: role                # role | credentials
  credentials:
    role_arn: "arn:aws:iam::123456789012:role/ProwlerScan"
    external_id: "ext-abc123"      # optional
    session_name: "prowler-bulk"   # optional
    duration_seconds: 3600         # optional

- provider: aws
  uid: "210987654321"
  alias: "dev"
  auth_method: credentials         # long/short-lived keys
  credentials:
    access_key_id: "AKIA..."
    secret_access_key: "..."
    session_token: "..."           # optional

- provider: azure
  uid: "00000000-1111-2222-3333-444444444444" # Subscription ID
  alias: "sub-eastus"
  auth_method: service_principal
  credentials:
    tenant_id: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
    client_id: "ffffffff-1111-2222-3333-444444444444"
    client_secret: "..."

- provider: gcp
  uid: "my-gcp-project-id"         # Project ID
  alias: "gcp-prod"
  auth_method: service_account     # Service Account authentication
  credentials:
    service_account_key_json_path: "./gcp-key.json"

- provider: kubernetes
  uid: "my-eks-context"            # kubeconfig context name
  alias: "eks-prod"
  auth_method: kubeconfig
  credentials:
    kubeconfig_path: "~/.kube/config"

- provider: m365
  uid: "contoso.onmicrosoft.com"   # Domain ID
  alias: "contoso"
  auth_method: service_principal
  credentials:
    tenant_id: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
    client_id: "ffffffff-1111-2222-3333-444444444444"
    client_secret: "..."

- provider: github
  uid: "my-org"                    # organization or username
  alias: "gh-org"
  auth_method: personal_access_token  # oauth_app_token | github_app
  credentials:
    token: "ghp_..."

Usage

Basic Usage

python prowler_bulk_provisioning.py providers.yaml

Advanced Usage

python prowler_bulk_provisioning.py providers.yaml \
  --base-url https://api.prowler.com/api/v1 \
  --providers-endpoint /providers \
  --concurrency 6 \
  --timeout 120

Command Line Options

Option	Description	Default
`input_file`	YAML file with provider entries	Required
`--base-url`	API base URL	`https://api.prowler.com/api/v1`
`--api-key`	Prowler API key	`PROWLER_API_KEY` env var
`--providers-endpoint`	Providers API endpoint	`/providers`
`--concurrency`	Number of concurrent requests	`5`
`--timeout`	Per-request timeout in seconds	`60`
`--insecure`	Disable TLS verification	`False`
`--dry-run`	Print payloads without sending	`False`
`--test-provider`	Test connection after creating each provider (true/false)	`true` (enabled by default)
`--test-provider-only`	Only test connections for existing providers (skip creation)	`False`

Self-hosted Prowler App

For self-hosted installations:

python prowler_bulk_provisioning.py providers.yaml \
  --base-url http://localhost:8080/api/v1

Provider-Specific Configuration

AWS Authentication Methods

IAM Role (Recommended)

- provider: aws
  uid: "123456789012"
  alias: "prod"
  auth_method: role
  credentials:
    role_arn: "arn:aws:iam::123456789012:role/ProwlerScan"
    external_id: "optional-external-id"

Access Keys

- provider: aws
  uid: "123456789012"
  alias: "dev"
  auth_method: credentials
  credentials:
    access_key_id: "AKIA..."
    secret_access_key: "..."
    session_token: "..."  # optional for temporary credentials

Azure Authentication

- provider: azure
  uid: "subscription-uuid"
  alias: "azure-prod"
  auth_method: service_principal
  credentials:
    tenant_id: "tenant-uuid"
    client_id: "client-uuid"
    client_secret: "client-secret"

GCP Authentication

The Prowler API supports the following authentication methods for GCP:

Method 1: Service Account JSON (Recommended)

- provider: gcp
  uid: "project-id"
  alias: "gcp-prod"
  auth_method: service_account  # or 'service_account_json'
  credentials:
    service_account_key_json_path: "/path/to/key.json"
    # OR inline:
    # inline_json:
    #   type: "service_account"
    #   project_id: "example-project"
    #   private_key_id: "example-key-id"
    #   private_key: "-----BEGIN PRIVATE KEY-----\n..."
    #   client_email: "service-account@project.iam.gserviceaccount.com"
    #   client_id: "1234567890"
    #   auth_uri: "https://accounts.google.com/o/oauth2/auth"
    #   token_uri: "https://oauth2.googleapis.com/token"
    #   auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs"
    #   client_x509_cert_url: "https://www.googleapis.com/robot/v1/metadata/x509/..."

Method 2: OAuth2 Credentials

- provider: gcp
  uid: "project-id"
  alias: "gcp-prod"
  auth_method: oauth2  # or 'adc' for Application Default Credentials
  credentials:
    client_id: "123456789012345678901.apps.googleusercontent.com"
    client_secret: "GOCSPX-xxxxxxxxxxxxxxxxx"
    refresh_token: "1//0exxxxxxxxxxxxxxxxx"

Kubernetes Authentication

- provider: kubernetes
  uid: "context-name"
  alias: "k8s-prod"
  auth_method: kubeconfig
  credentials:
    kubeconfig_path: "~/.kube/config"
    # OR
    # kubeconfig_inline: |
    #   apiVersion: v1
    #   clusters: ...

Microsoft 365 Authentication

- provider: m365
  uid: "domain.onmicrosoft.com"
  alias: "m365-tenant"
  auth_method: service_principal
  credentials:
    tenant_id: "tenant-uuid"
    client_id: "client-uuid"
    client_secret: "client-secret"

GitHub Authentication

Personal Access Token

- provider: github
  uid: "organization-name"
  alias: "gh-org"
  auth_method: personal_access_token
  credentials:
    token: "ghp_..."

GitHub App

- provider: github
  uid: "organization-name"
  alias: "gh-org"
  auth_method: github_app
  credentials:
    app_id: "123456"
    private_key_path: "/path/to/private-key.pem"
    # OR
    # private_key_inline: "-----BEGIN RSA PRIVATE KEY-----\n..."

Connection Testing

The script includes built-in connection testing to verify that providers can successfully authenticate with their respective cloud services.

By default, the script tests connections immediately after creating providers:

python prowler_bulk_provisioning.py providers.yaml

This will:

Create the provider
Add credentials
Test the connection
Report connection status

To skip connection testing, use:

python prowler_bulk_provisioning.py providers.yaml --test-provider false

Test Existing Providers

Test connections for already existing providers without creating new ones:

python prowler_bulk_provisioning.py providers.yaml --test-provider-only

This is useful for:

Verifying existing provider configurations
Debugging authentication issues
Regular connection health checks
Testing after credential updates

Example Output

[1] ✅ Created provider (id=db9a8985-f9ec-4dd8-b5a0-e05ab3880bed)
[1] ✅ Created secret (id=466f76c6-5878-4602-a4bc-13f9522c1fd2)
[1] ✅ Connection test: Connected

[2] ✅ Created provider (id=7a99f789-0cf5-4329-8279-2d443a962676)
[2] ✅ Created secret (id=c5702180-f7c4-40fd-be0e-f6433479b126)
[2] ❌ Connection test: Not connected

Advanced Features

Dry Run Mode

Test your configuration without making API calls:

python prowler_bulk_provisioning.py providers.yaml --dry-run

Troubleshooting

Common Issues

Invalid API Key

Error: 401 Unauthorized
Solution: Check your PROWLER_API_KEY environment variable or --api-key parameter

Network Timeouts

Error: Request timeout
Solution: Increase --timeout value or check network connectivity

Invalid Provider Configuration

Error: Each item must include 'provider' and 'uid'
Solution: Verify all required fields are present in your config file

File Not Found Errors

Error: No such file or directory
Solution: Check file paths for credentials files (JSON keys, kubeconfig, etc.)

Examples

See the examples/ directory for sample configuration files:

examples/simple-providers.yaml - Basic example with minimal configuration

Support

For issues and questions:

Check the Prowler documentation
Review the API documentation
Open an issue in the Prowler repository

License

This tool is part of the Prowler project and follows the same licensing terms.