mirror of
https://github.com/prowler-cloud/prowler.git
synced 2026-10-04 02:04:06 +00:00
test(mcp): add test foundation for the MCP server (#12291)
This commit is contained in:
Rubén De la Torre Vico
@@ -72,10 +72,12 @@ Use `@mcp.tool()` decorator directly—no BaseTool or models required.
|
||||
- [ ] Error handling returns `{"error": str, "status": "failed"}`
|
||||
- [ ] Parameters use `Field()` with descriptions
|
||||
- [ ] No hardcoded secrets
|
||||
- [ ] Tests added under `mcp_server/tests/`
|
||||
|
||||
---
|
||||
|
||||
## Resources
|
||||
|
||||
- **Full Guide**: [docs/developer-guide/mcp-server.mdx](../../../docs/developer-guide/mcp-server.mdx)
|
||||
- **Full Guide**: [docs/developer-guide/mcp-server.mdx](../../docs/developer-guide/mcp-server.mdx)
|
||||
- **Templates**: See [assets/](assets/) for tool and model templates
|
||||
- **Testing**: See [prowler-test-mcp](../prowler-test-mcp/SKILL.md) for fixtures and test patterns
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
---
|
||||
name: prowler-test-mcp
|
||||
description: >
|
||||
Testing patterns for the Prowler MCP Server: in-memory FastMCP clients, the
|
||||
ProwlerAPIClient singleton, JSON:API model builders and mocked httpx transports.
|
||||
Trigger: When writing tests under mcp_server/tests/ (tools, models, api_client, auth, sub-servers).
|
||||
license: Apache-2.0
|
||||
metadata:
|
||||
author: prowler-cloud
|
||||
version: "1.0.0"
|
||||
scope: [root, mcp_server]
|
||||
auto_invoke:
|
||||
- "Writing Prowler MCP server tests"
|
||||
- "Testing MCP tools or models"
|
||||
allowed-tools: Read, Edit, Write, Glob, Grep, Bash, WebFetch, WebSearch, Task
|
||||
---
|
||||
|
||||
## Critical Rules
|
||||
|
||||
- ALWAYS drive tools through an in-memory client: `async with Client(mcp_root_server)`.
|
||||
Tool parameters use pydantic `Field(default=...)`, and only FastMCP's wrapper
|
||||
resolves those defaults. Calling a tool method directly with an argument omitted
|
||||
leaves it as a raw `FieldInfo` — which is truthy, so `if email:` silently builds
|
||||
a filter out of the `FieldInfo` repr. Direct calls MUST pass every argument.
|
||||
- NEVER open a `fastmcp.Client` inside a fixture. FastMCP warns this causes
|
||||
hard-to-diagnose event-loop issues; open it inline in the test.
|
||||
- ALWAYS use the `mock_api_client` fixture; NEVER construct a `ProwlerAPIClient`.
|
||||
Tool instances captured the singleton by reference at import time, so only an
|
||||
in-place patch of `.client` reaches them.
|
||||
- NEVER clear `SingletonMeta._instances`. It orphans every registered tool on an
|
||||
instance holding a real `httpx.AsyncClient`. Use `isolated_api_client` if you
|
||||
genuinely need a fresh instance.
|
||||
- NEVER strip `PROWLER_API_KEY`. Tools are built at import time and a construction
|
||||
failure is swallowed, so the whole `prowler_*` namespace silently drops to zero
|
||||
tools. It is pinned in `[tool.pytest_env]`.
|
||||
- For `ProwlerAppAuth`, pass `mode=` / `base_url=` explicitly. Those are resolved in
|
||||
default arguments, evaluated once at module import, so `monkeypatch.setenv` has
|
||||
no effect on them.
|
||||
- NEVER assert an exact tool count — every future branch would have to bump it.
|
||||
- Assert on `result.data` (structured output), not `result.content[0].text`.
|
||||
- Tests are `test_*.py` (prefix), like the API — not the SDK's `*_test.py` suffix.
|
||||
- `__init__.py` IS required in every `tests/` subdirectory here (unlike the SDK's
|
||||
repo-root `tests/`), or same-named modules collide under pytest's import mode.
|
||||
- Async tests need no marker (`asyncio_mode = "auto"`). Do not use `@pytest.mark.anyio`.
|
||||
- Use only obviously-fake credentials from `tests.helpers.tokens` (TruffleHog).
|
||||
- One behaviour per test; keep tests self-contained and order-independent.
|
||||
|
||||
---
|
||||
|
||||
## 1. Layout
|
||||
|
||||
Mirror the source tree *below the package root* — drop the `prowler_mcp_server/`
|
||||
level, exactly as the SDK maps `prowler/providers/...` to `tests/providers/...`.
|
||||
So `prowler_mcp_server/prowler_app/tools/` is tested in `tests/prowler_app/tools/`.
|
||||
|
||||
```text
|
||||
mcp_server/tests/
|
||||
├── conftest.py # all shared fixtures
|
||||
├── helpers/ # jsonapi.py, http.py, assertions.py, tokens.py
|
||||
├── test_server.py # mounted-server contract
|
||||
├── test_health.py
|
||||
├── prowler_app/{models,tools,utils}/
|
||||
├── prowler_hub/
|
||||
└── prowler_documentation/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Fixtures
|
||||
|
||||
| Fixture | Autouse | What it gives you |
|
||||
|---------|---------|-------------------|
|
||||
| `_pinned_environment` | yes | Deterministic env; blocks a developer's `.env` from leaking |
|
||||
| `_no_real_network` | yes | Any real socket connect raises `RuntimeError` |
|
||||
| `_singleton_registry_guard` | yes | Snapshots/restores `SingletonMeta._instances` |
|
||||
| `mock_router` | no | Route registry + request recorder |
|
||||
| `api_client` | no | The live `ProwlerAPIClient` singleton |
|
||||
| `mock_api_client` | no | **The workhorse** — singleton with a mocked transport |
|
||||
| `isolated_api_client` | no | Evicts the singleton, for construction/identity tests |
|
||||
| `mcp_root_server` | no | The mounted root server (session-scoped) |
|
||||
| `health_client` | no | Starlette `TestClient` for `/health` |
|
||||
| `http_request_headers` | no | Injects headers for HTTP-mode auth |
|
||||
| `hub_router` | no | Mocks the Hub sub-server's two sync clients |
|
||||
| `docs_router` | no | Mocks the docs search engine's two sync clients |
|
||||
|
||||
### `MockRouter`
|
||||
|
||||
```python
|
||||
mock_router.add("GET", "/api/v1/users", json=jsonapi_collection([...]))
|
||||
mock_router.add("GET", "/api/v1/tasks/t1", json=task_document("t1", "completed"))
|
||||
|
||||
mock_router.request_for("GET", "/api/v1/users") # last request, for header asserts
|
||||
mock_router.query_params("GET", "/api/v1/users") # decoded query string
|
||||
mock_router.paths() # everything requested so far
|
||||
```
|
||||
|
||||
Register a route more than once to return a sequence — the last response repeats.
|
||||
That is how you drive `poll_task_until_complete` (`executing`, `executing`,
|
||||
`completed`). An unregistered request raises, listing what *was* registered.
|
||||
|
||||
---
|
||||
|
||||
## 3. Patterns
|
||||
|
||||
**Tool test** — see `assets/mcp_tool_test.py`. Register routes, call through the
|
||||
in-memory client, assert on `result.data` *and* on the recorded request. When a
|
||||
tool chooses between endpoints, assert `mock_router.paths()` — a wrong choice is
|
||||
invisible in the response body.
|
||||
|
||||
**Model test** — see `assets/mcp_model_test.py`. Build the document with the
|
||||
`jsonapi` helpers, run `from_api_response()`, assert on both the model and
|
||||
`model_dump()`. `MinimalSerializerMixin` makes those differ, and an absent
|
||||
relationship (`None`) must never be conflated with an empty one (`[]`).
|
||||
|
||||
**Contract test** — see `assets/mcp_contract_test.py`. Namespacing and
|
||||
description coverage across every registered tool.
|
||||
|
||||
The worked example in the repo is `findings`, covered across both layers in
|
||||
`tests/prowler_app/{models,tools}/test_findings.py`. Read those first — they
|
||||
exercise every foundation capability in one feature.
|
||||
|
||||
### Reading coverage
|
||||
|
||||
Coverage has a meaningless high floor. Model modules are almost entirely class-body
|
||||
`Field(...)` declarations that execute at import, and `prowler_app/server.py` imports
|
||||
every model module at import time. **Importing the package with zero tests already
|
||||
reports 36% overall**, and individual model modules 54–84%.
|
||||
|
||||
So a model module at ~68% with no tests has none of its logic covered — the missing
|
||||
ranges are the `from_api_response()` bodies, which is the only part worth testing.
|
||||
Compare against the import-only floor, never against zero, and do not set a Codecov
|
||||
target from the raw total.
|
||||
|
||||
### Where fixture data lives
|
||||
|
||||
`tests/helpers/` is feature-agnostic and must stay that way: it holds the JSON:API
|
||||
*shape*, not any feature's data. Per-feature attribute dictionaries
|
||||
(`FINDING_ATTRIBUTES`, `CHECK_METADATA`, …) belong as module-level constants in
|
||||
the test module that uses them. Do not add feature fixtures to `helpers/`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Commands
|
||||
|
||||
From `mcp_server/`:
|
||||
|
||||
```bash
|
||||
cd mcp_server
|
||||
|
||||
uv run pytest # whole suite
|
||||
uv run pytest tests/prowler_app/models # one area
|
||||
uv run pytest --cov=./prowler_mcp_server # with coverage
|
||||
```
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
make test-mcp # runs the MCP suite exactly as CI does
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Reference
|
||||
|
||||
- Fixtures and the reasoning behind them: `mcp_server/tests/conftest.py`
|
||||
- Testing section of `docs/developer-guide/mcp-server.mdx`
|
||||
- Official FastMCP testing guide: <https://gofastmcp.com/development/tests>
|
||||
@@ -0,0 +1,49 @@
|
||||
# Example: Prowler MCP Server contract test patterns
|
||||
# Source: mcp_server/tests/test_server.py
|
||||
|
||||
from fastmcp import Client
|
||||
from tests.helpers.assertions import (
|
||||
assert_namespaced,
|
||||
assert_tool_contract,
|
||||
tools_in_namespace,
|
||||
)
|
||||
|
||||
|
||||
async def test_every_sub_server_contributes_tools(mcp_root_server):
|
||||
"""Guard against a silent startup failure.
|
||||
|
||||
`setup_main_server()` wraps each mount in try/except and `load_all_tools`
|
||||
swallows per-tool construction errors, so a sub-server that registers nothing
|
||||
is still logged as "successfully mounted". Assert each namespace is non-empty
|
||||
-- never assert an exact count, which every future branch would have to bump.
|
||||
"""
|
||||
async with Client(mcp_root_server) as client:
|
||||
tools = await client.list_tools()
|
||||
|
||||
assert tools_in_namespace(tools, "prowler_hub_"), "Prowler Hub registered no tools"
|
||||
assert tools_in_namespace(tools, "prowler_docs_"), (
|
||||
"Prowler Docs registered no tools"
|
||||
)
|
||||
assert tools_in_namespace(tools, "prowler_"), "Prowler App registered no tools"
|
||||
|
||||
|
||||
async def test_every_tool_is_namespaced(mcp_root_server):
|
||||
"""Tool names are a published interface; nothing may escape the namespaces."""
|
||||
async with Client(mcp_root_server) as client:
|
||||
tools = await client.list_tools()
|
||||
|
||||
for tool in tools:
|
||||
assert_namespaced(tool)
|
||||
|
||||
|
||||
async def test_every_tool_and_parameter_is_described(mcp_root_server):
|
||||
"""Descriptions are the contract a model reads before calling a tool.
|
||||
|
||||
A tool or parameter with no description is registered but effectively
|
||||
invisible, so this is a correctness check rather than a style one.
|
||||
"""
|
||||
async with Client(mcp_root_server) as client:
|
||||
tools = await client.list_tools()
|
||||
|
||||
for tool in tools:
|
||||
assert_tool_contract(tool)
|
||||
@@ -0,0 +1,116 @@
|
||||
# Example: Prowler MCP Server model test patterns
|
||||
# Source: mcp_server/tests/prowler_app/models/test_findings.py
|
||||
|
||||
from prowler_mcp_server.prowler_app.models.findings import (
|
||||
DetailedFinding,
|
||||
FindingsListResponse,
|
||||
SimplifiedFinding,
|
||||
)
|
||||
|
||||
from tests.helpers.jsonapi import (
|
||||
jsonapi_collection,
|
||||
jsonapi_relationship_many,
|
||||
jsonapi_relationship_one,
|
||||
jsonapi_resource,
|
||||
)
|
||||
|
||||
CHECK_METADATA = {
|
||||
"checkid": "s3_bucket_public_access",
|
||||
"checktitle": "Ensure S3 buckets block public access",
|
||||
"description": "Checks whether the bucket blocks public access.",
|
||||
"provider": "aws",
|
||||
"servicename": "s3",
|
||||
"resourcetype": "AwsS3Bucket",
|
||||
"risk": "Public buckets expose data to the internet.",
|
||||
"additionalurls": [],
|
||||
"categories": ["internet-exposed"],
|
||||
}
|
||||
|
||||
FINDING_ATTRIBUTES = {
|
||||
"uid": "prowler-aws-s3_bucket_public_access-123456789012-us-east-1-my-bucket",
|
||||
"status": "FAIL",
|
||||
"severity": "high",
|
||||
"status_extended": "S3 bucket my-bucket is publicly accessible.",
|
||||
"delta": "new",
|
||||
"muted": False,
|
||||
"muted_reason": None,
|
||||
"check_metadata": CHECK_METADATA,
|
||||
}
|
||||
|
||||
DETAILED_ATTRIBUTES = {
|
||||
**FINDING_ATTRIBUTES,
|
||||
"inserted_at": "2025-01-15T10:00:00Z",
|
||||
"updated_at": "2025-01-15T10:00:00Z",
|
||||
}
|
||||
|
||||
|
||||
def test_nested_attributes_are_flattened_onto_the_model():
|
||||
"""Assert on the fields the model derives, not the ones it copies verbatim."""
|
||||
finding = SimplifiedFinding.from_api_response(
|
||||
jsonapi_resource("findings", "f1", FINDING_ATTRIBUTES)
|
||||
)
|
||||
|
||||
assert finding.check_id == "s3_bucket_public_access"
|
||||
|
||||
|
||||
def test_empty_fields_are_dropped_from_the_serialized_payload():
|
||||
"""Assert on `model_dump()` too -- MinimalSerializerMixin drops empty values.
|
||||
|
||||
A model may override that for fields whose empty form carries meaning, and
|
||||
that override is exactly the kind of thing a refactor breaks silently.
|
||||
"""
|
||||
finding = SimplifiedFinding.from_api_response(
|
||||
jsonapi_resource("findings", "f1", FINDING_ATTRIBUTES)
|
||||
)
|
||||
|
||||
assert "muted_reason" not in finding.model_dump()
|
||||
|
||||
|
||||
def test_both_relationship_shapes_are_parsed():
|
||||
"""To-one reduces to a single id, to-many to a list of ids."""
|
||||
resource = jsonapi_resource(
|
||||
"findings",
|
||||
"f1",
|
||||
attributes=DETAILED_ATTRIBUTES,
|
||||
relationships={
|
||||
"scan": jsonapi_relationship_one("scans", "s1"),
|
||||
"resources": jsonapi_relationship_many("resources", "r1", "r2"),
|
||||
},
|
||||
)
|
||||
|
||||
finding = DetailedFinding.from_api_response(resource)
|
||||
|
||||
assert finding.scan_id == "s1"
|
||||
assert finding.resource_ids == ["r1", "r2"]
|
||||
|
||||
|
||||
def test_missing_relationships_are_tolerated():
|
||||
"""Omit `relationships=` entirely to express absence.
|
||||
|
||||
Pass an empty `jsonapi_relationship_many(...)` instead to express "present and
|
||||
empty" -- some models must distinguish the two.
|
||||
"""
|
||||
finding = DetailedFinding.from_api_response(
|
||||
jsonapi_resource("findings", "f1", DETAILED_ATTRIBUTES)
|
||||
)
|
||||
|
||||
assert finding.scan_id is None
|
||||
assert finding.resource_ids == []
|
||||
|
||||
|
||||
def test_list_response_carries_pagination_metadata():
|
||||
"""`jsonapi_collection` emits meta.pagination exactly as *ListResponse reads it."""
|
||||
response = jsonapi_collection(
|
||||
[jsonapi_resource("findings", "f1", FINDING_ATTRIBUTES)],
|
||||
page=2,
|
||||
pages=7,
|
||||
count=312,
|
||||
)
|
||||
|
||||
result = FindingsListResponse.from_api_response(response)
|
||||
|
||||
assert (result.current_page, result.total_num_pages, result.total_num_finding) == (
|
||||
2,
|
||||
7,
|
||||
312,
|
||||
)
|
||||
@@ -0,0 +1,125 @@
|
||||
# Example: Prowler MCP Server tool test patterns
|
||||
# Source: mcp_server/tests/prowler_app/tools/test_findings.py
|
||||
|
||||
import pytest
|
||||
from fastmcp import Client
|
||||
|
||||
from tests.helpers.jsonapi import jsonapi_collection, jsonapi_error, jsonapi_resource
|
||||
|
||||
LATEST = "/api/v1/findings/latest"
|
||||
HISTORICAL = "/api/v1/findings"
|
||||
|
||||
FINDING_ATTRIBUTES = {
|
||||
"uid": "prowler-aws-s3_bucket_public_access-123456789012-us-east-1-my-bucket",
|
||||
"status": "FAIL",
|
||||
"severity": "high",
|
||||
"status_extended": "S3 bucket my-bucket is publicly accessible.",
|
||||
"delta": "new",
|
||||
"muted": False,
|
||||
"muted_reason": None,
|
||||
"check_metadata": {"checkid": "s3_bucket_public_access"},
|
||||
}
|
||||
|
||||
|
||||
async def test_tool_returns_a_simplified_payload(
|
||||
mcp_root_server, mock_api_client, mock_router
|
||||
):
|
||||
"""Drive the tool through the protocol; assert on the structured result.
|
||||
|
||||
This is the default pattern. Going through the in-memory client is what
|
||||
resolves the pydantic `Field(default=...)` declarations on the tool's
|
||||
parameters -- calling the method directly leaves omitted arguments as raw
|
||||
`FieldInfo` objects, which are truthy and build nonsense filters.
|
||||
"""
|
||||
mock_router.add(
|
||||
"GET",
|
||||
LATEST,
|
||||
json=jsonapi_collection(
|
||||
[jsonapi_resource("findings", "f1", FINDING_ATTRIBUTES)]
|
||||
),
|
||||
)
|
||||
|
||||
async with Client(mcp_root_server) as client:
|
||||
result = await client.call_tool("prowler_search_security_findings", {})
|
||||
|
||||
assert result.data["findings"][0]["check_id"] == "s3_bucket_public_access"
|
||||
|
||||
|
||||
async def test_tool_arguments_become_api_query_parameters(
|
||||
mcp_root_server, mock_api_client, mock_router
|
||||
):
|
||||
"""Assert on the recorded request, not only the returned payload.
|
||||
|
||||
The request is where filter translation, pagination and field selection live,
|
||||
and it is what breaks silently when an API contract shifts.
|
||||
"""
|
||||
mock_router.add("GET", LATEST, json=jsonapi_collection([]))
|
||||
|
||||
async with Client(mcp_root_server) as client:
|
||||
await client.call_tool(
|
||||
"prowler_search_security_findings", {"severity": ["critical", "high"]}
|
||||
)
|
||||
|
||||
params = mock_router.query_params("GET", LATEST)
|
||||
assert params["filter[severity__in]"] == "critical,high" # lists become CSV
|
||||
assert params["filter[status__in]"] == "FAIL" # the tool's default
|
||||
|
||||
|
||||
async def test_tool_picks_the_right_endpoint(
|
||||
mcp_root_server, mock_api_client, mock_router
|
||||
):
|
||||
"""Assert which endpoint was called when the tool chooses between several.
|
||||
|
||||
A wrong choice here is a performance regression the response body alone would
|
||||
never reveal, so `paths()` is the assertion that catches it.
|
||||
"""
|
||||
mock_router.add("GET", HISTORICAL, json=jsonapi_collection([]))
|
||||
|
||||
async with Client(mcp_root_server) as client:
|
||||
await client.call_tool(
|
||||
"prowler_search_security_findings", {"date_from": "2025-01-15"}
|
||||
)
|
||||
|
||||
assert mock_router.paths() == [f"GET {HISTORICAL}"]
|
||||
|
||||
|
||||
async def test_tool_validates_input_before_calling_the_api(
|
||||
mcp_root_server, mock_api_client, mock_router
|
||||
):
|
||||
"""Local validation must reject before any request goes out."""
|
||||
async with Client(mcp_root_server) as client:
|
||||
with pytest.raises(Exception, match="Must be between 1 and 1000"):
|
||||
await client.call_tool(
|
||||
"prowler_search_security_findings", {"page_size": 5000}
|
||||
)
|
||||
|
||||
assert mock_router.requests == []
|
||||
|
||||
|
||||
async def test_tool_surfaces_the_api_error_detail(
|
||||
mcp_root_server, mock_api_client, mock_router
|
||||
):
|
||||
"""Error text reaches the model, so assert on it rather than on the type alone."""
|
||||
mock_router.add(
|
||||
"GET", f"{HISTORICAL}/nope", status=404, json=jsonapi_error(404, "Not found.")
|
||||
)
|
||||
|
||||
async with Client(mcp_root_server) as client:
|
||||
with pytest.raises(Exception, match="Not found."):
|
||||
await client.call_tool(
|
||||
"prowler_get_finding_details", {"finding_id": "nope"}
|
||||
)
|
||||
|
||||
|
||||
async def test_polling_tool_waits_for_a_terminal_task_state(
|
||||
mock_api_client, mock_router
|
||||
):
|
||||
"""Register a route repeatedly to return a sequence; the last entry repeats."""
|
||||
from tests.helpers.jsonapi import task_document
|
||||
|
||||
mock_router.add("GET", "/api/v1/tasks/t1", json=task_document("t1", "executing"))
|
||||
mock_router.add("GET", "/api/v1/tasks/t1", json=task_document("t1", "completed"))
|
||||
|
||||
result = await mock_api_client.poll_task_until_complete("t1", poll_interval=0)
|
||||
|
||||
assert result["data"]["attributes"]["state"] == "completed"
|
||||
Reference in New Issue
Block a user