ben.carrasco/prowler
1
0
Fork 0
mirror of https://github.com/prowler-cloud/prowler.git synced 2026-07-24 13:01:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(db-connector): Include env variables (#1236)

Browse Source 
* feat(db-connector): Include env variables

* fix(typo)

* fix(psql-test): Remove PGPASSWORD
This commit is contained in:
Pepe Fagoaga
2022-06-30 08:43:41 +02:00
committed by GitHub
parent 172484cf08
commit d88a136ac3
4 changed files with 98 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+50 -38
View File
@@ -332,50 +332,62 @@ Prowler has two parameters related to regions: `-r` that is used query AWS servi
```
## Database providers connector
You can send the Prowler's output to different databases (right now only PostgreSQL is supported).
You can send the Prowler's output to different databases (right now only PostgreSQL is supported).
Jump into the section for the database provider you want to use and follow the required steps to configure it.
### PostgreSQL
Jump into the section for the database provider you want to use and follow the required steps to configure it.
### PostgreSQL
- Install psql
- Mac -> `brew install libpq`
- Ubuntu -> `sudo apt-get install postgresql-client `
- RHEL/Centos -> `sudo yum install postgresql10`
- Configure a `~/.pgpass` file into the root folder of the user that is going to launch Prowler ([pgpass file doc](https://www.postgresql.org/docs/current/libpq-pgpass.html)), including an extra field at the end of the line, separated by `:`, to name the table, for instance:
`hostname:port:database:username:password:prowler_findings`
#### Credentials
There are two options to pass the PostgreSQL credentials to Prowler:
##### Using a .pgpass file
Configure a `~/.pgpass` file into the root folder of the user that is going to launch Prowler ([pgpass file doc](https://www.postgresql.org/docs/current/libpq-pgpass.html)), including an extra field at the end of the line, separated by `:`, to name the table, for instance:
`hostname:port:database:username:password:prowler_findings`
##### Using environment variables
Configure the following environment variables:
- `POSTGRES_HOST`
- `POSTGRES_PORT`
- `POSTGRES_USER`
- `POSTGRES_PASSWORD`
- `POSTGRES_DB`
- Create a table in your PostgreSQL database to store the Prowler's data. You can use the following SQL statement to create the table:
```
CREATE TABLE IF NOT EXISTS prowler_findings (
profile TEXT,
account_number TEXT,
region TEXT,
check_id TEXT,
result TEXT,
item_scored TEXT,
item_level TEXT,
account_number TEXT,
region TEXT,
check_id TEXT,
result TEXT,
item_scored TEXT,
item_level TEXT,
check_title TEXT,
result_extended TEXT,
check_asff_compliance_type TEXT,
severity TEXT,
service_name TEXT,
check_asff_resource_type TEXT,
check_asff_type TEXT,
risk TEXT,
remediation TEXT,
documentation TEXT,
check_caf_epic TEXT,
resource_id TEXT,
prowler_start_time TEXT,
account_details_email TEXT,
account_details_name TEXT,
account_details_arn TEXT,
account_details_org TEXT,
account_details_tags TEXT
result_extended TEXT,
check_asff_compliance_type TEXT,
severity TEXT,
service_name TEXT,
check_asff_resource_type TEXT,
check_asff_type TEXT,
risk TEXT,
remediation TEXT,
documentation TEXT,
check_caf_epic TEXT,
resource_id TEXT,
prowler_start_time TEXT,
account_details_email TEXT,
account_details_name TEXT,
account_details_arn TEXT,
account_details_org TEXT,
account_details_tags TEXT
);
```
- Execute Prowler with `-d` flag, for example:
`./prowler -M csv -d postgres`
- Execute Prowler with `-d` flag, for example:
`./prowler -M csv -d postgresql`
> *Note*: This command creates a `csv` output file and stores the Prowler output in the configured PostgreSQL DB. It's an example, `-d` flag **does not** require `-M` to run.
## Advanced Usage
@@ -461,7 +473,7 @@ S3 URIs are also supported as custom folders for custom checks, e.g. `s3://bucke
### Show or log only FAILs
In order to remove noise and get only FAIL findings there is a `-q` flag that makes Prowler to show and log only FAILs.
In order to remove noise and get only FAIL findings there is a `-q` flag that makes Prowler to show and log only FAILs.
It can be combined with any other option.
Will show WARNINGS when a resource is excluded, just to take into consideration.
@@ -485,12 +497,12 @@ An easy way to run Prowler to scan your account is using AWS CloudShell. Read mo
## Security Hub integration
Since October 30th 2020 (version v2.3RC5), Prowler supports natively and as **official integration** sending findings to [AWS Security Hub](https://aws.amazon.com/security-hub). This integration allows Prowler to import its findings to AWS Security Hub. With Security Hub, you now have a single place that aggregates, organizes, and prioritizes your security alerts, or findings, from multiple AWS services, such as Amazon GuardDuty, Amazon Inspector, Amazon Macie, AWS Identity and Access Management (IAM) Access Analyzer, and AWS Firewall Manager, as well as from AWS Partner solutions and from Prowler for free.
Since October 30th 2020 (version v2.3RC5), Prowler supports natively and as **official integration** sending findings to [AWS Security Hub](https://aws.amazon.com/security-hub). This integration allows Prowler to import its findings to AWS Security Hub. With Security Hub, you now have a single place that aggregates, organizes, and prioritizes your security alerts, or findings, from multiple AWS services, such as Amazon GuardDuty, Amazon Inspector, Amazon Macie, AWS Identity and Access Management (IAM) Access Analyzer, and AWS Firewall Manager, as well as from AWS Partner solutions and from Prowler for free.
Before sending findings to Prowler, you need to perform next steps:
1. Since Security Hub is a region based service, enable it in the region or regions you require. Use the AWS Management Console or using the AWS CLI with this command if you have enough permissions:
1. Since Security Hub is a region based service, enable it in the region or regions you require. Use the AWS Management Console or using the AWS CLI with this command if you have enough permissions:
- `aws securityhub enable-security-hub --region <region>`.
2. Enable Prowler as partner integration integration. Use the AWS Management Console or using the AWS CLI with this command if you have enough permissions:
2. Enable Prowler as partner integration integration. Use the AWS Management Console or using the AWS CLI with this command if you have enough permissions:
- `aws securityhub enable-import-findings-for-product --region <region> --product-arn arn:aws:securityhub:<region>::product/prowler/prowler` (change region also inside the ARN).
- Using the AWS Management Console:
![Screenshot 2020-10-29 at 10 26 02 PM](https://user-images.githubusercontent.com/3985464/97634660-5ade3400-1a36-11eb-9a92-4a45cc98c158.png)
@@ -506,7 +518,7 @@ or for only one filtered region like eu-west-1:
```sh
./prowler -M json-asff -q -S -f eu-west-1
```
> Note 1: It is recommended to send only fails to Security Hub and that is possible adding `-q` to the command.
> Note 1: It is recommended to send only fails to Security Hub and that is possible adding `-q` to the command.
> Note 2: Since Prowler perform checks to all regions by defaults you may need to filter by region when runing Security Hub integration, as shown in the example above. Remember to enable Security Hub in the region or regions you need by calling `aws securityhub enable-security-hub --region <region>` and run Prowler with the option `-f <region>` (if no region is used it will try to push findings in all regions hubs).
@@ -534,7 +546,7 @@ To use Prowler and Security Hub integration in China regions there is an additio
Either to run Prowler once or based on a schedule this template makes it pretty straight forward. This template will create a CodeBuild environment and run Prowler directly leaving all reports in a bucket and creating a report also inside CodeBuild basedon the JUnit output from Prowler. Scheduling can be cron based like `cron(0 22 * * ? *)` or rate based like `rate(5 hours)` since CloudWatch Event rules (or Eventbridge) is used here.
The Cloud Formation template that helps you doing that is [here](https://github.com/prowler-cloud/prowler/blob/master/util/codebuild/codebuild-prowler-audit-account-cfn.yaml).
The Cloud Formation template that helps you doing that is [here](https://github.com/prowler-cloud/prowler/blob/master/util/codebuild/codebuild-prowler-audit-account-cfn.yaml).
> This is a simple solution to monitor one account. For multiples accounts see [Multi Account and Continuous Monitoring](util/org-multi-account/README.md).
## Allowlist or remove a fail from resources
@@ -783,9 +795,9 @@ Multi Account environments assumes a minimum of two trusted or known accounts. F
![multi-account-environment](/docs/images/prowler-multi-account-environment.png)
## Custom Checks
Using `./prowler -c extra9999 -a` you can build your own on-the-fly custom check by specifying the AWS CLI command to execute.
Using `./prowler -c extra9999 -a` you can build your own on-the-fly custom check by specifying the AWS CLI command to execute.
> Omit the "aws" command and only use its parameters within quotes and do not nest quotes in the aws parameter, --output text is already included in the check.
>
>
Here is an example of a check to find SGs with inbound port 80:
```sh
include/db_connector
+2 -2
View File
@@ -25,9 +25,9 @@ export SUPPORTED_DB_PROVIDERS
#POSTGRESQL connector
postgresql_connector () {
CSV_REGISTRY="${1}"
psql -q -U "${PSQL_USER}" -h "${PSQL_HOSTNAME}" -c "copy ${PSQL_TABLE} from stdin with null as E'\'\'' delimiter ','" <<< "${CSV_REGISTRY}"
psql -q -U "${POSTGRES_USER}" -h "${POSTGRES_HOST}" -c "copy ${POSTGRES_TABLE} from stdin with null as E'\'\'' delimiter ','" <<< "${CSV_REGISTRY}"
}
db_exit_abnormally() {
include/validate_options
+45 -27
View File
@@ -193,46 +193,64 @@ validate_modes() {
# Validate database with -d
validate_database() {
# Check database providers
if [[ ${DATABASE_PROVIDER} ]]
then
# Check if input provider is supported
if ! grep -w -q -E "${DATABASE_PROVIDER}" <<< "${SUPPORTED_DB_PROVIDERS}"
# Check database providers
if [[ ${DATABASE_PROVIDER} ]]
then
db_exit_abnormally "${DATABASE_PROVIDER}" "DB provider not supported, providers supported: ${SUPPORTED_DB_PROVIDERS} - EXITING!"
elif ! command -v "psql" > /dev/null 2>&1
then
db_exit_abnormally "postgresql" "psql tool not installed or not found- EXITING!"
elif [[ $(find "${HOME}/.pgpass" -perm 600 2>/dev/null) != "$HOME/.pgpass" ]]
then
db_exit_abnormally "postgresql" ".pgpass file not found at $HOME/.pgpass or .pgpass file permissions not matching 600 - EXITING!"
else
IFS=':' read -r PSQL_HOSTNAME PSQL_PORT PSQL_DATABASE PSQL_USER PSQL_PASSWORD PSQL_TABLE < "$HOME/.pgpass"
if [[ ! ${PSQL_HOSTNAME} ]] || [[ ! ${PSQL_PORT} ]] || [[ ! ${PSQL_DATABASE} ]] || [[ ! ${PSQL_USER} ]] || [[ ! ${PSQL_PASSWORD} ]] || [[ ! ${PSQL_TABLE} ]]
# Check if input provider is supported
if ! grep -w -q -E "${DATABASE_PROVIDER}" <<< "${SUPPORTED_DB_PROVIDERS}"
then
db_exit_abnormally "postgresql" "Empty field into ${HOME}/.pgpass file, all fields must be filled. Please check Prowler README about .pgpass file format - EXITING!"
db_exit_abnormally "${DATABASE_PROVIDER}" "DB provider not supported, providers supported: ${SUPPORTED_DB_PROVIDERS} - EXITING!"
fi
export PSQL_USER
export PSQL_HOSTNAME
export PSQL_TABLE
# Check psql command
if ! command -v "psql" > /dev/null 2>&1
then
db_exit_abnormally "postgresql" "psql tool not installed or not found- EXITING!"
fi
# Check credentials
if [[ -z "${POSTGRES_HOST}" || -z "${POSTGRES_PORT}" || -z "${POSTGRES_USER}" || -z "${POSTGRES_PASSWORD}" || -z "${POSTGRES_DB}" ]]
then
# echo "${OPTRED}ERROR!$OPTNORMAL Database connector for '${DATABASE_PROVIDER}' failed. Required environment variables for PostgreSQL were not found. Checking .pgpass file"
# If no environment variable present check the "$HOME/.pgpass" file
if [[ $(find "${HOME}/.pgpass" -perm 600 2>/dev/null) != "$HOME/.pgpass" ]]
then
db_exit_abnormally "postgresql" ".pgpass file not found at $HOME/.pgpass or .pgpass file permissions not matching 600 - EXITING!"
else
# Store credentials if file is present
IFS=':' read -r POSTGRES_HOST POSTGRES_PORT POSTGRES_DB POSTGRES_USER POSTGRES_PASSWORD POSTGRES_TABLE < "$HOME/.pgpass"
# Check for emtpy values
if [[ ! ${POSTGRES_HOST} ]] || [[ ! ${POSTGRES_PORT} ]] || [[ ! ${POSTGRES_DB} ]] || [[ ! ${POSTGRES_USER} ]] || [[ ! ${POSTGRES_PASSWORD} ]] || [[ ! ${POSTGRES_TABLE} ]]
then
db_exit_abnormally "postgresql" "Empty field into ${HOME}/.pgpass file, all fields must be filled. Please check Prowler README about .pgpass file format - EXITING!"
fi
fi
else
# Save the environment variables in the "$HOME/.pgpass" file
echo "${POSTGRES_HOST}:${POSTGRES_PORT}:${POSTGRES_DB}:${POSTGRES_USER}:${POSTGRES_PASSWORD}:${POSTGRES_TABLE}" > "$HOME/.pgpass"
chmod 600 "$HOME/.pgpass"
fi
# Save environment variables in the .pgpass file
export POSTGRES_USER
export POSTGRES_HOST
export POSTGRES_TABLE
# Once all the variables are defined and exported test if database instance is reachable
if ! pg_isready -q -U "${PSQL_USER}" -h "${PSQL_HOSTNAME}" -p "${PSQL_PORT}"
if ! pg_isready -q -U "${POSTGRES_USER}" -h "${POSTGRES_HOST}" -p "${POSTGRES_PORT}"
then
db_exit_abnormally "postgresql" "Database listening on host ${PSQL_HOSTNAME} on port ${PSQL_PORT} connected as user ${PSQL_USER} is unreachable - EXITING!"
db_exit_abnormally "postgresql" "Database listening on host ${POSTGRES_HOST} on port ${POSTGRES_PORT} connected as user ${POSTGRES_USER} is unreachable - EXITING!"
# If database instance is ready time to check credentials
elif ! PGPASSWORD="${PSQL_PASSWORD}" psql -U "${PSQL_USER}" -h "${PSQL_HOSTNAME}" -c "\q" > /dev/null 2>&1
elif ! psql -U "${POSTGRES_USER}" -h "${POSTGRES_HOST}" -c "\q" > /dev/null 2>&1
then
db_exit_abnormally "postgresql" "User ${PSQL_USER} invalid or invalid credentials, please check ${HOME}/.pgpass file - EXITING!"
db_exit_abnormally "postgresql" "User ${POSTGRES_USER} invalid or invalid credentials, please check ${HOME}/.pgpass file - EXITING!"
# If credentials are ok -> database exists?
elif ! PGPASSWORD="${PSQL_PASSWORD}" psql -U "${PSQL_USER}" -h "${PSQL_HOSTNAME}" "${PSQL_DATABASE}" -c "\q" > /dev/null 2>&1
elif ! psql -U "${POSTGRES_USER}" -h "${POSTGRES_HOST}" "${POSTGRES_DB}" -c "\q" > /dev/null 2>&1
then
db_exit_abnormally "postgresql" "Database not exists, please check ${HOME}/.pgpass file - EXITING!"
# and finally, if database exists -> table exists ?
elif ! PGPASSWORD="${PSQL_PASSWORD}" psql -U "${PSQL_USER}" -h "${PSQL_HOSTNAME}" "${PSQL_DATABASE}" -c "SELECT * FROM ${PSQL_TABLE};" > /dev/null 2>&1
elif ! psql -U "${POSTGRES_USER}" -h "${POSTGRES_HOST}" "${POSTGRES_DB}" -c "SELECT * FROM ${POSTGRES_TABLE};" > /dev/null 2>&1
then
db_exit_abnormally "postgresql" "Table ${PSQL_TABLE} not exists, please check ${HOME}/.pgpass file - EXITING!"
db_exit_abnormally "postgresql" "Table ${POSTGRES_TABLE} not exists, please check ${HOME}/.pgpass file - EXITING!"
fi
fi
fi
}
prowler
+1 -1
View File
@@ -127,7 +127,7 @@ USAGE:
i,e. -a 'ec2 describe-security-groups --filters Name=ip-permission.to-port,Values=80 --query SecurityGroups[*].GroupId[]]'
-V Show version number & exit.
-h This help.
-d Send output to database through database connectors supported, currently only PostgreSQL. Prowler will get the credentials and table name from your ~/.pgpass file.
-d <provider> Send output to database through database connectors supported, currently only PostgreSQL. Prowler will get the credentials and table name from your ~/.pgpass file.
"
exit
}