ben.carrasco/prowler
1
0
Fork 0
mirror of https://github.com/prowler-cloud/prowler.git synced 2026-07-24 13:01:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(api): add Silk profiling and improve test data guidance

Browse Source
This commit is contained in:
Alan Buscaglia
2025-12-16 15:19:06 +01:00
parent 47d66c9c4c
commit acee366e82
1 changed files with 133 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
api/docs/query-performance-guide.md
+133 -40
View File
@@ -4,24 +4,81 @@
This guide explains how to validate query performance when developing new endpoints or modifying existing ones. **This is part of the development process**, not a separate task—just like writing unit tests.
The goal is simple: ensure PostgreSQL uses indexes correctly. You don't need millions of records or complex stress testing setups. With 2-3 tenants and 2-3 scans, the query planner has enough information to decide whether to use indexes.
The goal is simple: ensure PostgreSQL uses indexes correctly for the queries your code generates.
## When to Validate
You **must** run `EXPLAIN ANALYZE` when:
You **must** validate query performance when:
- Creating a new endpoint that queries the database
- Modifying an existing query (adding filters, joins, or sorting)
- Adding new indexes
- Working on performance-critical endpoints (overviews, findings, resources)
## How to Run EXPLAIN ANALYZE
## Profiling with Django Silk (Recommended)
[Django Silk](https://github.com/jazzband/django-silk) is the recommended way to profile queries because it captures the actual SQL generated by your code during real HTTP requests. This gives you the most accurate picture of what happens in production.
### Enabling Silk
Silk is installed as a dev dependency but disabled by default. To enable it temporarily for profiling:
#### 1. Add Silk to your local settings
In `api/src/backend/config/django/devel.py`, add at the end of the file:
```python
# Silk profiler (temporary - remove after profiling)
INSTALLED_APPS += ["silk"] # noqa: F405
MIDDLEWARE += ["silk.middleware.SilkyMiddleware"] # noqa: F405
```
#### 2. Add Silk URLs
In `api/src/backend/api/v1/urls.py`, add at the end:
```python
from django.conf import settings
if settings.DEBUG:
urlpatterns += [path("silk/", include("silk.urls", namespace="silk"))]
```
#### 3. Run Silk migrations
```bash
cd api/src/backend
poetry run python manage.py migrate --database admin
```
#### 4. Access Silk
Start the development server and navigate to `http://localhost:8000/api/v1/silk/`
### Using Silk
1. Make requests to the endpoint you want to profile
2. Open Silk UI and find your request
3. Click on the request to see all SQL queries executed
4. For each query, you can see:
- Execution time
- Number of similar queries (N+1 detection)
- The actual SQL with parameters
- **EXPLAIN output** (click on a query to see it)
### Disabling Silk
After profiling, **remove the changes** you made to `devel.py` and `urls.py`. Don't commit Silk configuration to the repository.
## Manual Query Analysis with EXPLAIN ANALYZE
For quick checks or when you need more control, you can run `EXPLAIN ANALYZE` directly.
### 1. Get Your Query
#### Option A: Using Django Shell with RLS
This is the recommended approach since it mirrors how queries run in production with Row Level Security enabled:
This approach mirrors how queries run in production with Row Level Security enabled:
```bash
cd api/src/backend
@@ -59,7 +116,7 @@ print(queryset.query)
#### Option C: Enable SQL Logging
Set `DJANGO_LOGGING_LEVEL=DEBUG` in your environment. The `django.db` logger will output all SQL queries to the console.
Set `DJANGO_LOGGING_LEVEL=DEBUG` in your environment:
```bash
DJANGO_LOGGING_LEVEL=DEBUG poetry run python manage.py runserver
@@ -251,32 +308,56 @@ class Meta:
]
```
## Test Data Requirements
The amount of test data you need depends on what you're testing. PostgreSQL's query planner considers table statistics, index definitions, and data distribution when choosing execution plans.
### Important Considerations
1. **Small datasets may not use indexes**: PostgreSQL may choose a sequential scan over an index scan if the table is small enough that scanning it directly is faster. This is expected behavior.
2. **Data must exist in the tables you're querying**: If your endpoint queries `findings`, `resources`, `scans`, or other tables, ensure those tables have data. Use the `findings` management command to generate test data:
```bash
cd api/src/backend
poetry run python manage.py findings \
--tenant <TENANT_ID> \
--findings 1000 \
--resources 500 \
--batch 500
```
3. **Update table statistics**: After inserting test data, run `ANALYZE` to update PostgreSQL's statistics:
```sql
ANALYZE findings;
ANALYZE resources;
ANALYZE scans;
-- Add other tables as needed
```
4. **Test with realistic data distribution**: If your query filters by a specific value (e.g., `status='FAIL'`), ensure your test data includes a realistic mix of values.
### When Index Usage Matters Most
Focus on validating index usage when:
- The table will have thousands or millions of rows in production
- The query is called frequently (list endpoints, dashboards)
- The query has multiple filters or joins
For small lookup tables or infrequently-called endpoints, sequential scans may be acceptable.
## Performance Checklist for PRs
Before submitting a PR that adds or modifies database queries:
- [ ] Ran `EXPLAIN ANALYZE` on new/modified queries
- [ ] Verified indexes are being used (no unexpected `Seq Scan`)
- [ ] Profiled queries with Silk or `EXPLAIN ANALYZE`
- [ ] Verified indexes are being used (no unexpected `Seq Scan` on large tables)
- [ ] Checked `Rows Removed by Filter` is reasonable
- [ ] Tested with RLS enabled
- [ ] For critical endpoints: documented the query plan in the PR
## Minimum Test Data
You don't need millions of records. The query planner makes decisions based on:
- Table statistics (run `ANALYZE` if needed)
- Index definitions
- Query structure
**Minimum setup for validation:**
- 2-3 tenants
- 2-3 providers per tenant
- 2-3 scans per provider
- A few hundred findings/resources
This is enough for PostgreSQL to decide whether to use indexes.
## Useful Commands
### Update Table Statistics
@@ -322,31 +403,42 @@ ORDER BY pg_total_relation_size(relid) DESC;
## Working with Partitioned Tables
The `findings` table is partitioned. When adding indexes, use the helper functions from `api.db_utils`:
The `findings` and `resource_finding_mappings` tables are partitioned. When adding indexes, use the helper functions from `api.db_utils`:
### Adding Indexes to Partitions
```python
# In a migration file
from functools import partial
from django.db import migrations
from api.db_utils import create_index_on_partitions, drop_index_on_partitions
def forwards(apps, schema_editor):
create_index_on_partitions(
apps,
schema_editor,
parent_table="findings",
index_name="my_new_idx",
columns="tenant_id, status, severity",
all_partitions=False, # Only current/future partitions
)
def backwards(apps, schema_editor):
drop_index_on_partitions(
apps,
schema_editor,
parent_table="findings",
index_name="my_new_idx",
)
class Migration(migrations.Migration):
atomic = False # Required for CONCURRENTLY
dependencies = [
("api", "XXXX_previous_migration"),
]
operations = [
migrations.RunPython(
partial(
create_index_on_partitions,
parent_table="findings",
index_name="my_new_idx",
columns="tenant_id, status, severity",
all_partitions=False, # Only current/future partitions
),
reverse_code=partial(
drop_index_on_partitions,
parent_table="findings",
index_name="my_new_idx",
),
),
]
```
### Parameters
@@ -358,6 +450,7 @@ See [Partitions Documentation](./partitions.md) for more details on partitioning
## Further Reading
- [Django Silk Documentation](https://github.com/jazzband/django-silk)
- [PostgreSQL EXPLAIN Documentation](https://www.postgresql.org/docs/current/sql-explain.html)
- [Using EXPLAIN](https://www.postgresql.org/docs/current/using-explain.html)
- [Index Types in PostgreSQL](https://www.postgresql.org/docs/current/indexes-types.html)