diff --git a/api/docs/query-performance-guide.md b/api/docs/query-performance-guide.md index d99f23bba2..5346f75617 100644 --- a/api/docs/query-performance-guide.md +++ b/api/docs/query-performance-guide.md @@ -4,24 +4,81 @@ This guide explains how to validate query performance when developing new endpoints or modifying existing ones. **This is part of the development process**, not a separate task—just like writing unit tests. -The goal is simple: ensure PostgreSQL uses indexes correctly. You don't need millions of records or complex stress testing setups. With 2-3 tenants and 2-3 scans, the query planner has enough information to decide whether to use indexes. +The goal is simple: ensure PostgreSQL uses indexes correctly for the queries your code generates. ## When to Validate -You **must** run `EXPLAIN ANALYZE` when: +You **must** validate query performance when: - Creating a new endpoint that queries the database - Modifying an existing query (adding filters, joins, or sorting) - Adding new indexes - Working on performance-critical endpoints (overviews, findings, resources) -## How to Run EXPLAIN ANALYZE +## Profiling with Django Silk (Recommended) + +[Django Silk](https://github.com/jazzband/django-silk) is the recommended way to profile queries because it captures the actual SQL generated by your code during real HTTP requests. This gives you the most accurate picture of what happens in production. + +### Enabling Silk + +Silk is installed as a dev dependency but disabled by default. To enable it temporarily for profiling: + +#### 1. Add Silk to your local settings + +In `api/src/backend/config/django/devel.py`, add at the end of the file: + +```python +# Silk profiler (temporary - remove after profiling) +INSTALLED_APPS += ["silk"] # noqa: F405 +MIDDLEWARE += ["silk.middleware.SilkyMiddleware"] # noqa: F405 +``` + +#### 2. Add Silk URLs + +In `api/src/backend/api/v1/urls.py`, add at the end: + +```python +from django.conf import settings + +if settings.DEBUG: + urlpatterns += [path("silk/", include("silk.urls", namespace="silk"))] +``` + +#### 3. Run Silk migrations + +```bash +cd api/src/backend +poetry run python manage.py migrate --database admin +``` + +#### 4. Access Silk + +Start the development server and navigate to `http://localhost:8000/api/v1/silk/` + +### Using Silk + +1. Make requests to the endpoint you want to profile +2. Open Silk UI and find your request +3. Click on the request to see all SQL queries executed +4. For each query, you can see: + - Execution time + - Number of similar queries (N+1 detection) + - The actual SQL with parameters + - **EXPLAIN output** (click on a query to see it) + +### Disabling Silk + +After profiling, **remove the changes** you made to `devel.py` and `urls.py`. Don't commit Silk configuration to the repository. + +## Manual Query Analysis with EXPLAIN ANALYZE + +For quick checks or when you need more control, you can run `EXPLAIN ANALYZE` directly. ### 1. Get Your Query #### Option A: Using Django Shell with RLS -This is the recommended approach since it mirrors how queries run in production with Row Level Security enabled: +This approach mirrors how queries run in production with Row Level Security enabled: ```bash cd api/src/backend @@ -59,7 +116,7 @@ print(queryset.query) #### Option C: Enable SQL Logging -Set `DJANGO_LOGGING_LEVEL=DEBUG` in your environment. The `django.db` logger will output all SQL queries to the console. +Set `DJANGO_LOGGING_LEVEL=DEBUG` in your environment: ```bash DJANGO_LOGGING_LEVEL=DEBUG poetry run python manage.py runserver @@ -251,32 +308,56 @@ class Meta: ] ``` +## Test Data Requirements + +The amount of test data you need depends on what you're testing. PostgreSQL's query planner considers table statistics, index definitions, and data distribution when choosing execution plans. + +### Important Considerations + +1. **Small datasets may not use indexes**: PostgreSQL may choose a sequential scan over an index scan if the table is small enough that scanning it directly is faster. This is expected behavior. + +2. **Data must exist in the tables you're querying**: If your endpoint queries `findings`, `resources`, `scans`, or other tables, ensure those tables have data. Use the `findings` management command to generate test data: + + ```bash + cd api/src/backend + poetry run python manage.py findings \ + --tenant \ + --findings 1000 \ + --resources 500 \ + --batch 500 + ``` + +3. **Update table statistics**: After inserting test data, run `ANALYZE` to update PostgreSQL's statistics: + + ```sql + ANALYZE findings; + ANALYZE resources; + ANALYZE scans; + -- Add other tables as needed + ``` + +4. **Test with realistic data distribution**: If your query filters by a specific value (e.g., `status='FAIL'`), ensure your test data includes a realistic mix of values. + +### When Index Usage Matters Most + +Focus on validating index usage when: + +- The table will have thousands or millions of rows in production +- The query is called frequently (list endpoints, dashboards) +- The query has multiple filters or joins + +For small lookup tables or infrequently-called endpoints, sequential scans may be acceptable. + ## Performance Checklist for PRs Before submitting a PR that adds or modifies database queries: -- [ ] Ran `EXPLAIN ANALYZE` on new/modified queries -- [ ] Verified indexes are being used (no unexpected `Seq Scan`) +- [ ] Profiled queries with Silk or `EXPLAIN ANALYZE` +- [ ] Verified indexes are being used (no unexpected `Seq Scan` on large tables) - [ ] Checked `Rows Removed by Filter` is reasonable - [ ] Tested with RLS enabled - [ ] For critical endpoints: documented the query plan in the PR -## Minimum Test Data - -You don't need millions of records. The query planner makes decisions based on: - -- Table statistics (run `ANALYZE` if needed) -- Index definitions -- Query structure - -**Minimum setup for validation:** -- 2-3 tenants -- 2-3 providers per tenant -- 2-3 scans per provider -- A few hundred findings/resources - -This is enough for PostgreSQL to decide whether to use indexes. - ## Useful Commands ### Update Table Statistics @@ -322,31 +403,42 @@ ORDER BY pg_total_relation_size(relid) DESC; ## Working with Partitioned Tables -The `findings` table is partitioned. When adding indexes, use the helper functions from `api.db_utils`: +The `findings` and `resource_finding_mappings` tables are partitioned. When adding indexes, use the helper functions from `api.db_utils`: ### Adding Indexes to Partitions ```python # In a migration file +from functools import partial + +from django.db import migrations + from api.db_utils import create_index_on_partitions, drop_index_on_partitions -def forwards(apps, schema_editor): - create_index_on_partitions( - apps, - schema_editor, - parent_table="findings", - index_name="my_new_idx", - columns="tenant_id, status, severity", - all_partitions=False, # Only current/future partitions - ) -def backwards(apps, schema_editor): - drop_index_on_partitions( - apps, - schema_editor, - parent_table="findings", - index_name="my_new_idx", - ) +class Migration(migrations.Migration): + atomic = False # Required for CONCURRENTLY + + dependencies = [ + ("api", "XXXX_previous_migration"), + ] + + operations = [ + migrations.RunPython( + partial( + create_index_on_partitions, + parent_table="findings", + index_name="my_new_idx", + columns="tenant_id, status, severity", + all_partitions=False, # Only current/future partitions + ), + reverse_code=partial( + drop_index_on_partitions, + parent_table="findings", + index_name="my_new_idx", + ), + ), + ] ``` ### Parameters @@ -358,6 +450,7 @@ See [Partitions Documentation](./partitions.md) for more details on partitioning ## Further Reading +- [Django Silk Documentation](https://github.com/jazzband/django-silk) - [PostgreSQL EXPLAIN Documentation](https://www.postgresql.org/docs/current/sql-explain.html) - [Using EXPLAIN](https://www.postgresql.org/docs/current/using-explain.html) - [Index Types in PostgreSQL](https://www.postgresql.org/docs/current/indexes-types.html)