mirror of
https://github.com/prowler-cloud/prowler.git
synced 2026-07-24 13:01:56 +00:00
Daniel Barranquero
96 lines
2.7 KiB
Markdown
96 lines
2.7 KiB
Markdown
# GCP Retry Configuration in Prowler
|
|
|
|
Prowler's GCP Provider uses Google Cloud Python SDK's integrated retry mechanism to automatically retry API calls when encountering rate limiting errors (HTTP 429).
|
|
|
|
## Quick Configuration
|
|
|
|
### Using Command Line Flag (Recommended)
|
|
```bash
|
|
prowler gcp --gcp-retries-max-attempts 5
|
|
```
|
|
|
|
### Using Configuration File
|
|
Modify `prowler/providers/gcp/config.py`:
|
|
```python
|
|
DEFAULT_RETRY_ATTEMPTS = 5 # Default: 3
|
|
```
|
|
|
|
## How It Works
|
|
|
|
- **Automatic Detection**: Handles HTTP 429 and quota exceeded errors
|
|
- **Exponential Backoff**: Each retry uses randomized exponential backoff
|
|
- **Centralized Config**: All GCP services use the same retry configuration
|
|
- **Transparent**: No additional code needed in services
|
|
|
|
## Error Examples Handled
|
|
|
|
```
|
|
HttpError 429 when requesting https://cloudresourcemanager.googleapis.com/v1/projects/vms-uat-eiger:getIamPolicy?alt=json returned "Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute'"
|
|
```
|
|
|
|
## Implementation
|
|
|
|
### Client-Level Configuration
|
|
```python
|
|
from prowler.providers.gcp.config import DEFAULT_RETRY_ATTEMPTS
|
|
|
|
client = discovery.build(
|
|
service, version, credentials=credentials,
|
|
num_retries=DEFAULT_RETRY_ATTEMPTS
|
|
)
|
|
```
|
|
|
|
### Request-Level Configuration
|
|
```python
|
|
response = request.execute(num_retries=DEFAULT_RETRY_ATTEMPTS)
|
|
```
|
|
|
|
## Services with Retry Support
|
|
|
|
All major GCP services are covered:
|
|
- Cloud Resource Manager, Compute Engine, IAM
|
|
- BigQuery, KMS, Cloud Storage, Monitoring
|
|
- DNS, Logging, Cloud SQL, GKE, API Keys, DataProc
|
|
|
|
## Validation
|
|
|
|
### Debug Logging
|
|
```bash
|
|
prowler gcp --log-level DEBUG --log-file debuglogs.txt --project-id your-project-id
|
|
```
|
|
|
|
### Check for Retry Messages
|
|
```bash
|
|
grep -i "sleeping\|retry\|quota exceeded" debuglogs.txt
|
|
```
|
|
|
|
### Expected Output
|
|
```
|
|
"Sleeping 1.52 seconds before retry 1 of 3"
|
|
"Sleeping 3.23 seconds before retry 2 of 3"
|
|
```
|
|
|
|
## Testing in Real Environment
|
|
|
|
1. **Reduce API Quotas** in GCP Console:
|
|
- APIs & Services > Quotas
|
|
- Reduce "Read requests per minute" for Compute Engine API
|
|
- Reduce "Policy Read Requests per minute" for IAM API
|
|
|
|
2. **Run Prowler** with debug logging
|
|
3. **Monitor logs** for retry messages
|
|
|
|
## Troubleshooting
|
|
|
|
If experiencing rate limiting:
|
|
1. Use `--gcp-retries-max-attempts` flag to increase attempts
|
|
2. Request quota increases from Google Cloud support
|
|
3. Optimize scanning to reduce simultaneous API calls
|
|
4. Verify retry functionality with debug logging
|
|
|
|
## Official References
|
|
|
|
- [Google Cloud Python Client Libraries](https://cloud.google.com/python/docs)
|
|
- [Google Cloud Quotas](https://cloud.google.com/docs/quotas)
|
|
- [Google API Core Retry](https://googleapis.dev/python/google-api-core/latest/retry.html)
|