From a626e41162ce272fc188ff0c6191611e2c8dcd3a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rub=C3=A9n=20De=20la=20Torre=20Vico?= Date: Wed, 18 Jun 2025 15:20:33 +0200 Subject: [PATCH] docs: add provider-specific developer guide sections (#7996) Co-authored-by: Andoni Alonso <14891798+andoniaf@users.noreply.github.com> --- docs/developer-guide/aws-details.md | 122 +++++++++++++++++++ docs/developer-guide/azure-details.md | 121 +++++++++++++++++++ docs/developer-guide/gcp-details.md | 133 +++++++++++++++++++++ docs/developer-guide/github-details.md | 116 ++++++++++++++++++ docs/developer-guide/introduction.md | 2 +- docs/developer-guide/kubernetes-details.md | 117 ++++++++++++++++++ docs/developer-guide/m365-details.md | 131 ++++++++++++++++++++ mkdocs.yml | 29 +++-- 8 files changed, 760 insertions(+), 11 deletions(-) create mode 100644 docs/developer-guide/aws-details.md create mode 100644 docs/developer-guide/azure-details.md create mode 100644 docs/developer-guide/gcp-details.md create mode 100644 docs/developer-guide/github-details.md create mode 100644 docs/developer-guide/kubernetes-details.md create mode 100644 docs/developer-guide/m365-details.md diff --git a/docs/developer-guide/aws-details.md b/docs/developer-guide/aws-details.md new file mode 100644 index 0000000000..819f04fa64 --- /dev/null +++ b/docs/developer-guide/aws-details.md @@ -0,0 +1,122 @@ +# AWS Provider + +In this page you can find all the details about [Amazon Web Services (AWS)](https://aws.amazon.com/) provider implementation in Prowler. + +By default, Prowler will audit just one account and organization settings per scan. To configure it, follow the [getting started](../index.md#aws) page. + +## AWS Provider Classes Architecture + +The AWS provider implementation follows the general [Provider structure](./provider.md). This section focuses on the AWS-specific implementation, highlighting how the generic provider concepts are realized for AWS in Prowler. For a full overview of the provider pattern, base classes, and extension guidelines, see [Provider documentation](./provider.md). In next subsection you can find a list of the main classes of the AWS provider. + +### `AwsProvider` (Main Class) + +- **Location:** [`prowler/providers/aws/aws_provider.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/aws/aws_provider.py) +- **Base Class:** Inherits from `Provider` (see [base class details](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/common/provider.py)). +- **Purpose:** Central orchestrator for AWS-specific logic, session management, credential validation, role assumption, region and organization discovery, and configuration. +- **Key AWS Responsibilities:** + - Initializes and manages AWS sessions (with or without role assumption, MFA, etc.). + - Validates credentials and sets up the AWS identity context. + - Loads and manages configuration, mutelist, and fixer settings. + - Discovers enabled AWS regions and organization metadata. + - Provides properties and methods for downstream AWS service classes to access session, identity, and configuration data. + +### Data Models + +- **Location:** [`prowler/providers/aws/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/aws/models.py) +- **Purpose:** Define structured data for AWS identity, session, credentials, organization info, and more. +- **Key AWS Models:** + - `AWSOrganizationsInfo`: Holds AWS Organizations metadata, to be used by the checks. + - `AWSCredentials`, `AWSAssumeRoleInfo`, `AWSAssumeRoleConfiguration`: Used for role assumption and session management. + - `AWSIdentityInfo`: Stores account, user, partition, and region context for the scan. + - `AWSSession`: Wraps the current and original [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) sessions and config. + +### `AWSService` (Service Base Class) + +- **Location:** [`prowler/providers/aws/lib/service/service.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/aws/lib/service/service.py) +- **Purpose:** Abstract base class that all AWS service-specific classes inherit from. This implements the generic service pattern (described in [service page](./services.md#service-base-class)) specifically for AWS. +- **Key AWS Responsibilities:** + - Receives an `AwsProvider` instance to access session, identity, and configuration. + - Manages clients for all services by regions. + - Provides `__threading_call__` method to make boto3 calls in parallel. By default, this calls are made by region, but it can be overridden with the first parameter of the method and use by resource. + - Exposes common audit context (`audited_account`, `audited_account_arn`, `audited_partition`, `audited_resources`) to subclasses. + +### Exception Handling + +- **Location:** [`prowler/providers/aws/exceptions/exceptions.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/aws/exceptions/exceptions.py) +- **Purpose:** Custom exception classes for AWS-specific error handling, such as credential and role errors. + +### Session and Utility Helpers + +- **Location:** [`prowler/providers/aws/lib/`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/aws/lib/) +- **Purpose:** Helpers for session setup, ARN parsing, mutelist management, and other cross-cutting concerns. + +## Specific Patterns in AWS Services + +The generic service pattern is described in [service page](./services.md#service-structure-and-initialisation). You can find all the right now implemented services in the following locations: + +- Directly in the code, in location [`prowler/providers/aws/services/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/aws/services) +- In the [Prowler Hub](https://hub.prowler.com/). For a more human-readable view. + +The best reference to understand how to implement a new service is following the [service implementation documentation](./services.md#adding-a-new-service) and taking other services already implemented as reference. In next subsection you can find a list of common patterns that are used accross all AWS services. + +### AWS Service Common Patterns + +- Services communicate with AWS using boto3, you can find the documentation with all the services [here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/index.html). +- Every AWS service class inherits from `AWSService`, ensuring access to session, identity, configuration, and threading utilities. +- The constructor (`__init__`) always calls `super().__init__` with the service name and provider (e.g. `super().__init__(__class__.__name__, provider))`). Ensure that the service name in boto3 is the same that you use in the constructor. Usually is used the `__class__.__name__` to get the service name because it is the same as the class name. +- Resource containers **must** be initialized in the constructor. They should be dictionaries, with the key being the resource ARN or equivalent unique identifier and the value being the resource object. +- Resource discovery and attribute collection are parallelized using `self.__threading_call__`, typically by region or resource, for performance. The first parameter of the method is the iterator, if not provided, it will be the region; but if present indicate an array of the resources to be processed. +- Resource filtering is consistently enforced using `self.audit_resources` attribute and `is_resource_filtered` function, it is used to see if user has provided some resource that is not in the audit scope, so we can skip it in the service logic. Normally it is used befor storing the resource in the service container as follows: `if not self.audit_resources or (is_resource_filtered(resource["arn"], self.audit_resources)):`. +- All AWS resources are represented as Pydantic `BaseModel` classes, providing type safety and structured access to resource attributes. +- AWS API calls are wrapped in try/except blocks, with specific handling for `ClientError` and generic exceptions, always logging errors. +- If ARN is not present for some resource, it can be constructed using string interpolation, always including partition, service, region, account, and resource ID. +- Tags and additional attributes that cannot be retrieved from the default call, should be collected and stored for each resource using dedicated methods and threading using the resource object list as iterator. + +## Specific Patterns in AWS Checks + +The AWS checks pattern is described in [checks page](./checks.md). You can find all the right now implemented checks: + +- Directly in the code, within each service folder, each check has its own folder named after the name of the check. (e.g. [`prowler/providers/aws/services/s3/s3_bucket_acl_prohibited/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/aws/services/s3/s3_bucket_acl_prohibited)) +- In the [Prowler Hub](https://hub.prowler.com/). For a more human-readable view. + +The best reference to understand how to implement a new check is following the [check creation documentation](./checks.md#creating-a-check) and taking other similar checks as reference. + +### Check Report Class + +The `Check_Report_AWS` class models a single finding for an AWS resource in a check report. It is defined in [`prowler/lib/check/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/lib/check/models.py) and inherits from the generic `Check_Report` base class. + +#### Purpose + +`Check_Report_AWS` extends the base report structure with AWS-specific fields, enabling detailed tracking of the resource, ARN, and region associated with each finding. + +#### Constructor and Attribute Population + +When you instantiate `Check_Report_AWS`, you must provide the check metadata and a resource object. The class will attempt to automatically populate its AWS-specific attributes from the resource, using the following logic (in order of precedence): + +- **`resource_id`**: + - Uses `resource.id` if present. + - Otherwise, uses `resource.name` if present. + - Defaults to an empty string if none are available. + +- **`resource_arn`**: + - Uses `resource.arn` if present. + - Defaults to an empty string if ARN is not present in the resource object. + +- **`region`**: + - Uses `resource.region` if present. + - Defaults to an empty string if region is not present in the resource object. + +If the resource object does not contain the required attributes, you must set them manually in the check logic. + +Other attributes are inherited from the `Check_Report` class, from that ones you **always** have to set the `status` and `status_extended` attributes in the check logic. + +#### Example Usage + +```python +report = Check_Report_AWS( + metadata=check_metadata, + resource=resource_object +) +report.status = "PASS" +report.status_extended = "Resource is compliant." +``` diff --git a/docs/developer-guide/azure-details.md b/docs/developer-guide/azure-details.md new file mode 100644 index 0000000000..9d21706acd --- /dev/null +++ b/docs/developer-guide/azure-details.md @@ -0,0 +1,121 @@ +# Azure Provider + +In this page you can find all the details about [Microsoft Azure](https://azure.microsoft.com/) provider implementation in Prowler. + +By default, Prowler will audit all the subscriptions that it is able to list in the Microsoft Entra tenant, and tenant Entra ID service. To configure it, follow the [getting started](../index.md#azure) page. + +## Azure Provider Classes Architecture + +The Azure provider implementation follows the general [Provider structure](./provider.md). This section focuses on the Azure-specific implementation, highlighting how the generic provider concepts are realized for Azure in Prowler. For a full overview of the provider pattern, base classes, and extension guidelines, see [Provider documentation](./provider.md). In next subsection you can find a list of the main classes of the Azure provider. + +### `AzureProvider` (Main Class) + +- **Location:** [`prowler/providers/azure/azure_provider.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/azure/azure_provider.py) +- **Base Class:** Inherits from `Provider` (see [base class details](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/common/provider.py)). +- **Purpose:** Central orchestrator for Azure-specific logic, session management, credential validation, and configuration. +- **Key Azure Responsibilities:** + - Initializes and manages Azure sessions (supports Service Principal, CLI, Browser, and Managed Identity authentication). + - Validates credentials and sets up the Azure identity context. + - Loads and manages configuration, mutelist, and fixer settings. + - Retrieves subscription(s) metadata. + - Provides properties and methods for downstream Azure service classes to access session, identity, and configuration data. + +### Data Models + +- **Location:** [`prowler/providers/azure/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/azure/models.py) +- **Purpose:** Define structured data for Azure identity, session, region configuration, and subscription info. +- **Key Azure Models:** + - `AzureIdentityInfo`: Holds Azure identity metadata, including tenant ID, domain, subscription names and IDs, and locations. + - `AzureRegionConfig`: Stores the specific region that will be audited. That can be: Global, US Government or China. + - `AzureSubscription`: Represents a subscription with ID, display name, and state. + +### `AzureService` (Service Base Class) + +- **Location:** [`prowler/providers/azure/lib/service/service.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/azure/lib/service/service.py) +- **Purpose:** Abstract base class that all Azure service-specific classes inherit from. This implements the generic service pattern (described in [service page](./services.md#service-base-class)) specifically for Azure. +- **Key Azure Responsibilities:** + - Receives an `AzureProvider` instance to access session, identity, and configuration. + - Manages clients for all services by subscription. + - Exposes common audit context (`subscriptions`, `locations`, `audit_config`, `fixer_config`) to subclasses. + +### Exception Handling + +- **Location:** [`prowler/providers/azure/exceptions/exceptions.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/azure/exceptions/exceptions.py) +- **Purpose:** Custom exception classes for Azure-specific error handling, such as credential, region, and session errors. + +### Session and Utility Helpers + +- **Location:** [`prowler/providers/azure/lib/`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/azure/lib/) +- **Purpose:** Helpers for argument parsing, region setup, mutelist management, and other cross-cutting concerns. + +## Specific Patterns in Azure Services + +The generic service pattern is described in [service page](./services.md#service-structure-and-initialisation). You can find all the currently implemented services in the following locations: + +- Directly in the code, in location [`prowler/providers/azure/services/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/azure/services) +- In the [Prowler Hub](https://hub.prowler.com/) for a more human-readable view. + +The best reference to understand how to implement a new service is following the [service implementation documentation](./services.md#adding-a-new-service) and taking other services already implemented as reference. In next subsection you can find a list of common patterns that are used accross all Azure services. + +### Azure Service Common Patterns + +- Services communicate with Azure using the Azure Python SDK, mainly using the Azure Management Client (except for the Microsoft Entra ID service, that is using the Microsoft Graph API), you can find the documentation with all the management services [here](https://learn.microsoft.com/en-us/python/api/overview/azure/?view=azure-python). +- Every Azure service class inherits from `AzureService`, ensuring access to session, identity, configuration, and client utilities. +- The constructor (`__init__`) always calls `super().__init__` with the service Azure Management Client and Prowler provider object (e.g `super().__init__(WebSiteManagementClient, provider)`). +- Resource containers **must** be initialized in the constructor, and they should be dictionaries, with the key being the subscription ID, the value being a dictionary with the resource ID as key and the resource object as value. +- All Azure resources are represented as Pydantic `BaseModel` classes, providing type safety and structured access to resource attributes. Some are represented as dataclasses due to legacy reasons, but new resources should be represented as Pydantic `BaseModel` classes. +- Azure SDK functions are wrapped in try/except blocks, with specific handling for errors, always logging errors. It is a best practice to create a custom function for every Azure SDK call, in that way we can handle the errors in a more specific way. + +## Specific Patterns in Azure Checks + +The Azure checks pattern is described in [checks page](./checks.md). You can find all the currently implemented checks: + +- Directly in the code, within each service folder, each check has its own folder named after the name of the check. (e.g. [`prowler/providers/azure/services/storage/storage_blob_public_access_level_is_disabled/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/azure/services/storage/storage_blob_public_access_level_is_disabled)) +- In the [Prowler Hub](https://hub.prowler.com/) for a more human-readable view. + +The best reference to understand how to implement a new check is the [Azure check implementation documentation](./checks.md#creating-a-check) and taking other similar checks as reference. + +### Check Report Class + +The `Check_Report_Azure` class models a single finding for an Azure resource in a check report. It is defined in [`prowler/lib/check/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/lib/check/models.py) and inherits from the generic `Check_Report` base class. + +#### Purpose + +`Check_Report_Azure` extends the base report structure with Azure-specific fields, enabling detailed tracking of the resource, resource ID, name, subscription, and location associated with each finding. + +#### Constructor and Attribute Population + +When you instantiate `Check_Report_Azure`, you must provide the check metadata and a resource object. The class will attempt to automatically populate its Azure-specific attributes from the resource, using the following logic (in order of precedence): + +- **`resource_id`**: + - Uses `resource.id` if present. + - Otherwise, uses `resource.resource_id` if present. + - Defaults to an empty string if not available. + +- **`resource_name`**: + - Uses `resource.name` if present. + - Otherwise, uses `resource.resource_name` if present. + - Defaults to an empty string if not available. + +- **`subscription`**: + - Defaults to an empty string, it **must** be set in the check logic. + +- **`location`**: + - Uses `resource.location` if present. + - Defaults to an empty string if not available. + +If the resource object does not contain the required attributes, you must set them manually in the check logic. + +Other attributes are inherited from the `Check_Report` class, from which you **always** have to set the `status` and `status_extended` attributes in the check logic. + +#### Example Usage + +```python +report = Check_Report_Azure( + metadata=check_metadata, + resource=resource_object +) +report.subscription = subscription_id +report.status = "PASS" +report.status_extended = "Resource is compliant." +``` diff --git a/docs/developer-guide/gcp-details.md b/docs/developer-guide/gcp-details.md new file mode 100644 index 0000000000..f9a7ecbd01 --- /dev/null +++ b/docs/developer-guide/gcp-details.md @@ -0,0 +1,133 @@ +# Google Cloud Provider + +This page details the [Google Cloud Platform (GCP)](https://cloud.google.com/) provider implementation in Prowler. + +By default, Prowler will audit all the GCP projects that the authenticated identity can access. To configure it, follow the [getting started](../index.md#google-cloud) page. + +## GCP Provider Classes Architecture + +The GCP provider implementation follows the general [Provider structure](./provider.md). This section focuses on the GCP-specific implementation, highlighting how the generic provider concepts are realized for GCP in Prowler. For a full overview of the provider pattern, base classes, and extension guidelines, see [Provider documentation](./provider.md). + +### Main Class + +- **Location:** [`prowler/providers/gcp/gcp_provider.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/gcp/gcp_provider.py) +- **Base Class:** Inherits from `Provider` (see [base class details](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/common/provider.py)). +- **Purpose:** Central orchestrator for GCP-specific logic, session management, credential validation, project and organization discovery, and configuration. +- **Key GCP Responsibilities:** + - Initializes and manages GCP sessions (supports Application Default Credentials, Service Account, OAuth, and impersonation). + - Validates credentials and sets up the GCP identity context. + - Loads and manages configuration, mutelist, and fixer settings. + - Discovers accessible GCP projects and organization metadata. + - Provides properties and methods for downstream GCP service classes to access session, identity, and configuration data. + +### Data Models + +- **Location:** [`prowler/providers/gcp/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/gcp/models.py) +- **Purpose:** Define structured data for GCP identity, project, and organization info. +- **Key GCP Models:** + - `GCPIdentityInfo`: Holds GCP identity metadata, such as the profile name. + - `GCPOrganization`: Represents a GCP organization with ID, name, and display name. + - `GCPProject`: Represents a GCP project with number, ID, name, organization, labels, and lifecycle state. + +### `GCPService` (Service Base Class) + +- **Location:** [`prowler/providers/gcp/lib/service/service.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/gcp/lib/service/service.py) +- **Purpose:** Abstract base class that all GCP service-specific classes inherit from. This implements the generic service pattern (described in [service page](./services.md#service-base-class)) specifically for GCP. +- **Key GCP Responsibilities:** + - Receives a `GcpProvider` instance to access session, identity, and configuration. + - Manages clients for all services by project. + - Filters projects to only those with the relevant API enabled. + - Provides `__threading_call__` method to make API calls in parallel by project or resource. + - Exposes common audit context (`project_ids`, `projects`, `default_project_id`, `audit_config`, `fixer_config`) to subclasses. + +### Exception Handling + +- **Location:** [`prowler/providers/gcp/exceptions/exceptions.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/gcp/exceptions/exceptions.py) +- **Purpose:** Custom exception classes for GCP-specific error handling, such as credential, session, and project access errors. + +### Session and Utility Helpers + +- **Location:** [`prowler/providers/gcp/lib/`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/gcp/lib/) +- **Purpose:** Helpers for argument parsing, mutelist management, and other cross-cutting concerns. + +## Specific Patterns in GCP Services + +The generic service pattern is described in [service page](./services.md#service-structure-and-initialisation). You can find all the currently implemented services in the following locations: + +- Directly in the code, in location [`prowler/providers/gcp/services/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/gcp/services) +- In the [Prowler Hub](https://hub.prowler.com/) for a more human-readable view. + +The best reference to understand how to implement a new service is following the [service implementation documentation](./services.md#adding-a-new-service) and taking other services already implemented as reference. In next subsection you can find a list of common patterns that are used accross all GCP services. + +### GCP Service Common Patterns + +- Services communicate with GCP using the Google Cloud Python SDK, you can find the documentation with all the services [here](https://cloud.google.com/python/docs/reference). +- Every GCP service class inherits from `GCPService`, ensuring access to session, identity, configuration, and client utilities. +- The constructor (`__init__`) always calls `super().__init__` with the service name, provider, region (default "global"), and API version (default "v1"). Usually, the service name is the class name in lowercase, so it is called like `super().__init__(__class__.__name__, provider)`. +- Resource containers **must** be initialized in the constructor, typically as dictionaries keyed by resource ID and the value is the resource object. +- Only projects with the API enabled are included in the audit scope. +- Resource discovery and attribute collection can be parallelized using `self.__threading_call__`, typically by region/zone or resource. +- All GCP resources are represented as Pydantic `BaseModel` classes, providing type safety and structured access to resource attributes. +- Each GCP API calls are wrapped in try/except blocks, always logging errors. +- Tags and additional attributes that cannot be retrieved from the default call should be collected and stored for each resource using dedicated methods and threading. + +## Specific Patterns in GCP Checks + +The GCP checks pattern is described in [checks page](./checks.md). You can find all the currently implemented checks: + +- Directly in the code, within each service folder, each check has its own folder named after the name of the check. (e.g. [`prowler/providers/gcp/services/iam/iam_sa_user_managed_key_unused/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/gcp/services/iam/iam_sa_user_managed_key_unused)) +- In the [Prowler Hub](https://hub.prowler.com/) for a more human-readable view. + +The best reference to understand how to implement a new check is following the [GCP check implementation documentation](./checks.md#creating-a-check) and taking other similar checks as reference. + +### Check Report Class + +The `Check_Report_GCP` class models a single finding for a GCP resource in a check report. It is defined in [`prowler/lib/check/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/lib/check/models.py) and inherits from the generic `Check_Report` base class. + +#### Purpose + +`Check_Report_GCP` extends the base report structure with GCP-specific fields, enabling detailed tracking of the resource, project, and location associated with each finding. + +#### Constructor and Attribute Population + +When you instantiate `Check_Report_GCP`, you must provide the check metadata and a resource object. The class will attempt to automatically populate its GCP-specific attributes from the resource, using the following logic (in order of precedence): + +- **`resource_id`**: + - Uses the explicit `resource_id` argument if provided. + - Otherwise, uses `resource.id` if present. + - Otherwise, uses `resource.name` if present. + - Defaults to an empty string if none are available. + +- **`resource_name`**: + - Uses the explicit `resource_name` argument if provided. + - Otherwise, uses `resource.name` if present. + - Defaults to an empty string. + +- **`project_id`**: + - Uses the explicit `project_id` argument if provided. + - Otherwise, uses `resource.project_id` if present. + - Defaults to an empty string. + +- **`location`**: + - Uses the explicit `location` argument if provided. + - Otherwise, uses `resource.location` if present. + - Otherwise, uses `resource.region` if present. + - Defaults to "global" if none are available. + +All these attributes can be overridden by passing the corresponding argument to the constructor. If the resource object does not contain the required attributes, you must set them manually. +Others attributes are inherited from the `Check_Report` class, from that ones you **always** have to set the `status` and `status_extended` attributes in the check logic. + +#### Example Usage + +```python +report = Check_Report_GCP( + metadata=check_metadata, + resource=resource_object, + resource_id="custom-id", # Optional override + resource_name="custom-name", # Optional override + project_id="my-gcp-project", # Optional override + location="us-central1" # Optional override +) +report.status = "PASS" +report.status_extended = "Resource is compliant." +``` diff --git a/docs/developer-guide/github-details.md b/docs/developer-guide/github-details.md new file mode 100644 index 0000000000..0dc3c5460d --- /dev/null +++ b/docs/developer-guide/github-details.md @@ -0,0 +1,116 @@ +# GitHub Provider + +This page details the [GitHub](https://github.com/) provider implementation in Prowler. + +By default, Prowler will audit the GitHub account - scanning all repositories, organizations, and applications that your configured credentials can access. To configure it, follow the [getting started](../index.md#github) page. + +## GitHub Provider Classes Architecture + +The GitHub provider implementation follows the general [Provider structure](./provider.md). This section focuses on the GitHub-specific implementation, highlighting how the generic provider concepts are realized for GitHub in Prowler. For a full overview of the provider pattern, base classes, and extension guidelines, see [Provider documentation](./provider.md). + +### `GithubProvider` (Main Class) + +- **Location:** [`prowler/providers/github/github_provider.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/github/github_provider.py) +- **Base Class:** Inherits from `Provider` (see [base class details](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/common/provider.py)). +- **Purpose:** Central orchestrator for GitHub-specific logic, session management, credential validation, and configuration. +- **Key GitHub Responsibilities:** + - Initializes and manages GitHub sessions (supports Personal Access Token, OAuth App, and GitHub App authentication). + - Validates credentials and sets up the GitHub identity context. + - Loads and manages configuration, mutelist, and fixer settings. + - Provides properties and methods for downstream GitHub service classes to access session, identity, and configuration data. + +### Data Models + +- **Location:** [`prowler/providers/github/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/github/models.py) +- **Purpose:** Define structured data for GitHub identity, session, and output options. +- **Key GitHub Models:** + - `GithubSession`: Holds authentication tokens and keys for the session. + - `GithubIdentityInfo`, `GithubAppIdentityInfo`: Store account or app identity metadata. + +### `GithubService` (Service Base Class) + +- **Location:** [`prowler/providers/github/lib/service/service.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/github/lib/service/service.py) +- **Purpose:** Abstract base class for all GitHub service-specific classes. +- **Key GitHub Responsibilities:** + - Receives a `GithubProvider` instance to access session, identity, and configuration. + - Manages GitHub API clients for the authenticated user or app. + - Exposes common audit context (`audit_config`, `fixer_config`) to subclasses. + +### Exception Handling + +- **Location:** [`prowler/providers/github/exceptions/exceptions.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/github/exceptions/exceptions.py) +- **Purpose:** Custom exception classes for GitHub-specific error handling, such as credential and session errors. + +### Session and Utility Helpers + +- **Location:** [`prowler/providers/github/lib/`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/github/lib/) +- **Purpose:** Helpers for argument parsing, mutelist management, and other cross-cutting concerns. + +## Specific Patterns in GitHub Services + +The generic service pattern is described in [service page](./services.md#service-structure-and-initialisation). You can find all the currently implemented services in the following locations: + +- Directly in the code, in location [`prowler/providers/github/services/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/github/services) +- In the [Prowler Hub](https://hub.prowler.com/) for a more human-readable view. + +The best reference to understand how to implement a new service is following the [service implementation documentation](./services.md#adding-a-new-service) and by taking other already implemented services as reference. + +### GitHub Service Common Patterns + +- Services communicate with GitHub using the PyGithub Python SDK. See the [official documentation](https://pygithub.readthedocs.io/). +- Every GitHub service class inherits from `GithubService`, ensuring access to session, identity, configuration, and client utilities. +- The constructor (`__init__`) always calls `super().__init__` with the service name and provider (e.g. `super().__init__(__class__.__name__, provider))`). Ensure that the service name in PyGithub is the same that you use in the constructor. Usually is used the `__class__.__name__` to get the service name because it is the same as the class name. +- Resource containers **must** be initialized in the constructor, typically as dictionaries keyed by resource ID or name. +- All GitHub resources are represented as Pydantic `BaseModel` classes, providing type safety and structured access to resource attributes. +- GitHub API calls are wrapped in try/except blocks, always logging errors. + +## Specific Patterns in GitHub Checks + +The GitHub checks pattern is described in [checks page](./checks.md). You can find all the currently implemented checks in: + +- Directly in the code, within each service folder, each check has its own folder named after the name of the check. (e.g. [`prowler/providers/github/services/repository/repository_secret_scanning_enabled/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/github/services/repository/repository_secret_scanning_enabled)) +- In the [Prowler Hub](https://hub.prowler.com/) for a more human-readable view. + +The best reference to understand how to implement a new check is the [GitHub check implementation documentation](./checks.md#creating-a-check) and by taking other checks as reference. + +### Check Report Class + +The `CheckReportGithub` class models a single finding for a GitHub resource in a check report. It is defined in [`prowler/lib/check/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/lib/check/models.py) and inherits from the generic `Check_Report` base class. + +#### Purpose + +`CheckReportGithub` extends the base report structure with GitHub-specific fields, enabling detailed tracking of the resource, name, and owner associated with each finding. + +#### Constructor and Attribute Population + +When you instantiate `CheckReportGithub`, you must provide the check metadata and a resource object. The class will attempt to automatically populate its GitHub-specific attributes from the resource, using the following logic (in order of precedence): + +- **`resource_id`**: + - Uses the explicit `resource_id` argument if provided. + - Otherwise, uses `resource.id` if present. + - Defaults to an empty string if not available. + +- **`resource_name`**: + - Uses the explicit `resource_name` argument if provided. + - Otherwise, uses `resource.name` if present. + - Defaults to an empty string if not available. + +- **`owner`**: + - Uses the explicit `owner` argument if provided. + - Otherwise, uses `resource.owner` for repositories and `resource.name` for organizations. + - Defaults to an empty string if not available. + +If the resource object does not contain the required attributes, you must set them manually in the check logic. + +Other attributes are inherited from the `Check_Report` class, from which you **always** have to set the `status` and `status_extended` attributes in the check logic. + +#### Example Usage + +```python +report = CheckReportGithub( + metadata=check_metadata, + resource=resource_object +) +report.status = "PASS" +report.status_extended = "Resource is compliant." +``` diff --git a/docs/developer-guide/introduction.md b/docs/developer-guide/introduction.md index 597cb7a979..dc18fd4bbc 100644 --- a/docs/developer-guide/introduction.md +++ b/docs/developer-guide/introduction.md @@ -1,4 +1,4 @@ -# Developer Guide +# Introduction to developing in Prowler Extending Prowler diff --git a/docs/developer-guide/kubernetes-details.md b/docs/developer-guide/kubernetes-details.md new file mode 100644 index 0000000000..8b08b51e64 --- /dev/null +++ b/docs/developer-guide/kubernetes-details.md @@ -0,0 +1,117 @@ +# Kubernetes Provider + +This page details the [Kubernetes](https://kubernetes.io/) provider implementation in Prowler. + +By default, Prowler will audit all namespaces in the Kubernetes cluster accessible by the configured context. To configure it, follow the [getting started](../index.md#kubernetes) page. + +## Kubernetes Provider Classes Architecture + +The Kubernetes provider implementation follows the general [Provider structure](./provider.md). This section focuses on the Kubernetes-specific implementation, highlighting how the generic provider concepts are realized for Kubernetes in Prowler. For a full overview of the provider pattern, base classes, and extension guidelines, see [Provider documentation](./provider.md). + +### `KubernetesProvider` (Main Class) + +- **Location:** [`prowler/providers/kubernetes/kubernetes_provider.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/kubernetes/kubernetes_provider.py) +- **Base Class:** Inherits from `Provider` (see [base class details](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/common/provider.py)). +- **Purpose:** Central orchestrator for Kubernetes-specific logic, session management, context and namespace discovery, credential validation, and configuration. +- **Key Kubernetes Responsibilities:** + - Initializes and manages Kubernetes sessions (supports kubeconfig file or content, context selection, and namespace scoping). + - Validates credentials and sets up the Kubernetes identity context. + - Loads and manages configuration, mutelist, and fixer settings. + - Discovers accessible namespaces and cluster metadata. + - Provides properties and methods for downstream Kubernetes service classes to access session, identity, and configuration data. + +### Data Models + +- **Location:** [`prowler/providers/kubernetes/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/kubernetes/models.py) +- **Purpose:** Define structured data for Kubernetes identity and session info. +- **Key Kubernetes Models:** + - `KubernetesIdentityInfo`: Holds Kubernetes identity metadata, such as context, cluster, and user. + - `KubernetesSession`: Stores the Kubernetes API client and context information. + +### `KubernetesService` (Service Base Class) + +- **Location:** [`prowler/providers/kubernetes/lib/service/service.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/kubernetes/lib/service/service.py) +- **Purpose:** Abstract base class that all Kubernetes service-specific classes inherit from. This implements the generic service pattern (described in [service page](./services.md#service-base-class)) specifically for Kubernetes. +- **Key Kubernetes Responsibilities:** + - Receives a `KubernetesProvider` instance to access session, identity, and configuration. + - Manages the Kubernetes API client and context. + - Provides a `__threading_call__` method to make API calls in parallel by resource. + - Exposes common audit context (`context`, `api_client`, `audit_config`, `fixer_config`) to subclasses. + +### Exception Handling + +- **Location:** [`prowler/providers/kubernetes/exceptions/exceptions.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/kubernetes/exceptions/exceptions.py) +- **Purpose:** Custom exception classes for Kubernetes-specific error handling, such as session, API, and configuration errors. + +### Session and Utility Helpers + +- **Location:** [`prowler/providers/kubernetes/lib/`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/kubernetes/lib/) +- **Purpose:** Helpers for argument parsing, mutelist management, and other cross-cutting concerns. + +## Specific Patterns in Kubernetes Services + +The generic service pattern is described in [service page](./services.md#service-structure-and-initialisation). You can find all the currently implemented services in the following locations: + +- Directly in the code, in location [`prowler/providers/kubernetes/services/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/kubernetes/services) +- In the [Prowler Hub](https://hub.prowler.com/) for a more human-readable view. + +The best reference to understand how to implement a new service is following the [service implementation documentation](./services.md#adding-a-new-service) and taking other already implemented services as reference. + +### Kubernetes Service Common Patterns + +- Services communicate with Kubernetes using the Kubernetes Python SDK. See the [official documentation](https://github.com/kubernetes-client/python/blob/master/kubernetes/README.md/). +- Every Kubernetes service class inherits from `KubernetesService`, ensuring access to session, identity, configuration, and client utilities. +- The constructor (`__init__`) always calls `super().__init__` with the provider object, and initializes resource containers (typically as dictionaries keyed by resource UID or name). +- Resource discovery and attribute collection can be parallelized using `self.__threading_call__`. +- All Kubernetes resources are represented as Pydantic `BaseModel` classes, providing type safety and structured access to resource attributes. +- Kubernetes API calls are wrapped in try/except blocks, always logging errors. +- Additional attributes that cannot be retrieved from the default call should be collected and stored for each resource using dedicated methods and threading. + +## Specific Patterns in Kubernetes Checks + +The Kubernetes checks pattern is described in [checks page](./checks.md). You can find all the currently implemented checks in: + +- Directly in the code, within each service folder, each check has its own folder named after the name of the check. (e.g. [`prowler/providers/kubernetes/services/rbac/rbac_minimize_wildcard_use_roles/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/kubernetes/services/rbac/rbac_minimize_wildcard_use_roles)) +- In the [Prowler Hub](https://hub.prowler.com/) for a more human-readable view. + +The best reference to understand how to implement a new check is following the [Kubernetes check implementation documentation](./checks.md#creating-a-check) and taking other checks as reference. + +### Check Report Class + +The `Check_Report_Kubernetes` class models a single finding for a Kubernetes resource in a check report. It is defined in [`prowler/lib/check/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/lib/check/models.py) and inherits from the generic `Check_Report` base class. + +#### Purpose + +`Check_Report_Kubernetes` extends the base report structure with Kubernetes-specific fields, enabling detailed tracking of the resource, name, and namespace associated with each finding. + +#### Constructor and Attribute Population + +When you instantiate `Check_Report_Kubernetes`, you must provide the check metadata and a resource object. The class will attempt to automatically populate its Kubernetes-specific attributes from the resource, using the following logic (in order of precedence): + +- **`resource_id`**: + - Uses `resource.uid` if present. + - Otherwise, uses `resource.name` if present. + - Defaults to an empty string if none are available. + +- **`resource_name`**: + - Uses `resource.name` if present. + - Defaults to an empty string if not available. + +- **`namespace`**: + - Uses `resource.namespace` if present. + - Defaults to "cluster-wide" for cluster-scoped resources. + +If the resource object does not contain the required attributes, you must set them manually in the check logic. + +Other attributes are inherited from the `Check_Report` class, from which you **always** have to set the `status` and `status_extended` attributes in the check logic. + +#### Example Usage + +```python +report = Check_Report_Kubernetes( + metadata=check_metadata, + resource=resource_object +) +report.status = "PASS" +report.status_extended = "Resource is compliant." +``` diff --git a/docs/developer-guide/m365-details.md b/docs/developer-guide/m365-details.md new file mode 100644 index 0000000000..0840b9856c --- /dev/null +++ b/docs/developer-guide/m365-details.md @@ -0,0 +1,131 @@ +# Microsoft 365 (M365) Provider + +This page details the [Microsoft 365 (M365)](https://www.microsoft.com/en-us/microsoft-365) provider implementation in Prowler. + +By default, Prowler will audit the Microsoft Entra ID tenant and its supported services. To configure it, follow the [getting started](../index.md#microsoft-365) page. + +--- + +## PowerShell Requirements for M365 Checks + +> **Most Microsoft 365 checks in Prowler require PowerShell, not just the Microsoft Graph API.** + +- **PowerShell is essential** for retrieving data from Exchange Online, Teams, Defender, Purview, and other M365 services. Many checks cannot be performed using only the Graph API. +- **PowerShell 7.4 or higher is required** (7.5 recommended). PowerShell 5.1 and earlier versions are not supported for M365 checks. +- **Required modules:** + - [ExchangeOnlineManagement](https://www.powershellgallery.com/packages/ExchangeOnlineManagement/3.6.0) (≥ 3.6.0) + - [MicrosoftTeams](https://www.powershellgallery.com/packages/MicrosoftTeams/6.6.0) (≥ 6.6.0) +- If you use Prowler Cloud or the official containers, PowerShell is pre-installed. For local or pip installations, you must install PowerShell and the modules yourself. See [Requirements: Supported PowerShell Versions](../getting-started/requirements.md#supported-powershell-versions) and [Needed PowerShell Modules](../getting-started/requirements.md#needed-powershell-modules). +- For more details and troubleshooting, see [Use of PowerShell in M365](../tutorials/microsoft365/use-of-powershell.md). + +--- + +## M365 Provider Classes Architecture + +The M365 provider implementation follows the general [Provider structure](./provider.md). This section focuses on the M365-specific implementation, highlighting how the generic provider concepts are realized for M365 in Prowler. For a full overview of the provider pattern, base classes, and extension guidelines, see [Provider documentation](./provider.md). + +### `M365Provider` (Main Class) + +- **Location:** [`prowler/providers/m365/m365_provider.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/m365/m365_provider.py) +- **Base Class:** Inherits from `Provider` (see [base class details](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/common/provider.py)). +- **Purpose:** Central orchestrator for M365-specific logic, session management, credential validation, region/authority configuration, and identity context. +- **Key M365 Responsibilities:** + - Initializes and manages M365 sessions (supports Service Principal, environment variables, Azure CLI, browser, and user/password authentication). + - Validates credentials and sets up the M365 identity context. + - Manages the Microsoft Graph API client and the PowerShell client. + - Loads and manages configuration, mutelist, and fixer settings. + - Provides properties and methods for downstream M365 service classes to access session, identity, and configuration data. + +### Data Models + +- **Location:** [`prowler/providers/m365/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/m365/models.py) +- **Purpose:** Define structured data for M365 identity, session, region configuration, and credentials. +- **Key M365 Models:** + - `M365IdentityInfo`: Holds M365 identity metadata, including tenant ID, domain(s), user, and location. + - `M365RegionConfig`: Stores the specific region/authority and API base URL for the tenant. + - `M365Credentials`: Represents credentials for authentication (user, password, client ID, client secret, tenant ID, etc.). + +### `M365Service` (Service Base Class) + +- **Location:** [`prowler/providers/m365/lib/service/service.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/m365/lib/service/service.py) +- **Purpose:** Abstract base class for all M365 service-specific classes. +- **Key M365 Responsibilities:** + - Receives an `M365Provider` instance to access session, identity, and configuration. + - Manages the Microsoft Graph API client for the service. + - Initializes a PowerShell client for most services if credentials and identity are available. + - Exposes common audit context (`audit_config`, `fixer_config`) to subclasses. + +### Exception Handling + +- **Location:** [`prowler/providers/m365/exceptions/exceptions.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/m365/exceptions/exceptions.py) +- **Purpose:** Custom exception classes for M365-specific error handling, such as credential, session, region, and argument errors. + +### Session and Utility Helpers + +- **Location:** [`prowler/providers/m365/lib/`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/m365/lib/) +- **Purpose:** Helpers for argument parsing, region/authority setup, mutelist management, PowerShell integration, and other cross-cutting concerns. + + > **Key File: [`m365_powershell.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/m365/lib/powershell/m365_powershell.py)** + > + > This is the core module for Microsoft 365 PowerShell integration. It manages authentication, session handling, and provides a comprehensive set of methods for interacting with Microsoft Teams, Exchange Online, and Defender policies via PowerShell. + > + > This module provides secure credential management and authentication using MSAL and PowerShell. It handles automated installation and initialization of required PowerShell modules. The module offers a rich set of methods for retrieving and managing Teams, Exchange, and Defender configurations. It serves as the central component for all M365 provider operations that require PowerShell automation. + +## Specific Patterns in M365 Services + +The generic service pattern is described in [service page](./services.md#service-structure-and-initialisation). You can find all the currently implemented services in the following locations: + +- Directly in the code, in location [`prowler/providers/m365/services/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/m365/services) +- In the [Prowler Hub](https://hub.prowler.com/) for a more human-readable view. + +The best reference to understand how to implement a new service is by following the [service implementation documentation](./services.md#adding-a-new-service) and by taking other already implemented services as reference. + +### M365 Service Common Patterns + +- Services communicate with Microsoft 365 using the Microsoft Graph API **and/or PowerShell**. See the [official documentation](https://learn.microsoft.com/en-us/graph/api/overview) and [PowerShell reference](https://learn.microsoft.com/en-us/powershell/). +- Every M365 service class inherits from `M365Service`, ensuring access to session, identity, configuration, and client utilities. +- The constructor (`__init__`) always calls `super().__init__` with the provider object, and initializes the Graph client and the PowerShell client. +- Resource containers **must** be initialized in the constructor, typically as objects that represent the different settings of the service. +- All M365 resources are represented as Pydantic `BaseModel` classes, providing type safety and structured access to resource attributes. +- Microsoft Graph API and PowerShell calls are wrapped in try/except blocks, always logging errors. +- To retrieve some data in the services, it is so common that you have to create a new method also in the `m365_powershell.py` file to later be called in the service. + +## Specific Patterns in M365 Checks + +The M365 checks pattern is described in [checks page](./checks.md). You can find all the currently implemented checks in: + +- Directly in the code, within each service folder, each check has its own folder named after the name of the check. (e.g. [`prowler/providers/m365/services/entra/entra_users_mfa_enabled/`](https://github.com/prowler-cloud/prowler/tree/master/prowler/providers/m365/services/entra/entra_users_mfa_enabled)) +- In the [Prowler Hub](https://hub.prowler.com/) for a more human-readable view. + +The best reference to understand how to implement a new check is following the [M365 check implementation documentation](./checks.md#creating-a-check) and by taking other checks as reference. + +### Check Report Class + +The `CheckReportM365` class models a single finding for a Microsoft 365 resource in a check report. It is defined in [`prowler/lib/check/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/lib/check/models.py) and inherits from the generic `Check_Report` base class. + +#### Purpose + +`CheckReportM365` extends the base report structure with M365-specific fields, enabling detailed tracking of the resource, name, and location associated with each finding. + +#### Constructor and Attribute Population + +When you instantiate `CheckReportM365`, you must provide the check metadata and a resource object. The class will attempt to automatically populate its M365-specific attributes from the resource, using the following logic (in order of precedence): + +- **`resource_id`**: A required field that **must** be explicitly set in the constructor to identify the resource being checked. +- **`resource_name`**: A required field that **must** be explicitly set in the constructor to provide a human-readable name for the resource. +- **`location`**: A required field that can be explicitly set in the constructor to indicate where the resource is located. If not specified, defaults to "global". + +If the resource object does not contain the required attributes, you must set them manually in the check logic. + +Other attributes are inherited from the `Check_Report` class, from which you **always** have to set the `status` and `status_extended` attributes in the check logic. + +#### Example Usage + +```python +report = CheckReportM365( + metadata=check_metadata, + resource=resource_object +) +report.status = "PASS" +report.status_extended = "Resource is compliant." +``` diff --git a/mkdocs.yml b/mkdocs.yml index 57f89287a8..bf597ce494 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -107,18 +107,27 @@ nav: - Authentication: tutorials/microsoft365/authentication.md - Use of PowerShell: tutorials/microsoft365/use-of-powershell.md - Developer Guide: - - Introduction: developer-guide/introduction.md - - Providers: developer-guide/provider.md - - Services: developer-guide/services.md - - Checks: developer-guide/checks.md - - Documentation: developer-guide/documentation.md - - Compliance: developer-guide/security-compliance-framework.md - - Outputs: developer-guide/outputs.md - - Integrations: developer-guide/integrations.md - - Testing: + - General Concepts: + - Introduction: developer-guide/introduction.md + - Providers: developer-guide/provider.md + - Services: developer-guide/services.md + - Checks: developer-guide/checks.md + - Outputs: developer-guide/outputs.md + - Integrations: developer-guide/integrations.md + - Compliance: developer-guide/security-compliance-framework.md + - Provider Specific Details: + - AWS: developer-guide/aws-details.md + - Azure: developer-guide/azure-details.md + - Google Cloud: developer-guide/gcp-details.md + - Kubernetes: developer-guide/kubernetes-details.md + - Microsoft 365: developer-guide/m365-details.md + - GitHub: developer-guide/github-details.md + - Miscellaneous: + - Documentation: developer-guide/documentation.md + - Testing: - Unit Tests: developer-guide/unit-testing.md - Integration Tests: developer-guide/integration-testing.md - - Debugging: developer-guide/debugging.md + - Debugging: developer-guide/debugging.md - Security: security.md - Contact Us: contact.md - Troubleshooting: troubleshooting.md