ben.carrasco/prowler
1
0 Fork 0
mirror of https://github.com/prowler-cloud/prowler.git synced 2026-10-23 20:04:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
41 Commits 585 Branches 225 Tags
ec67fc12e040b5c20fdc886a76f4d8fc9313faa4
Commit Graph
41 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Víctor Fernández Poyatos ec67fc12e0 feat/PRWLR-4177 Add /tasks endpoints and data model (#35) 
* feat(Backend): PRWLR-4177 add Task model and migrations

* feat(Tasks): PRWLR-4177 add RLSTask class

* feat(API): PRWLR-4177 add Task serializers

* feat(Backend, DB): PRWLR-4177 refactor db variables and add policy on task runner tasks

* feat(API): PRWLR-4177 add Tasks filters and sort fields

* feat(API, Tasks): PRWLR-4177 add deletion tasks and revoke logic to /tasks

* test(Task): PRWLR-4177 add deletion tasks unit tests

* test(Views): PRWLR-4177 add Tasks views unit tests and update outdated ones

* chore(API): PRWLR-4177 improve drf-spectacular annotations

* chore(API): PRWLR-4177 add PROGRESS task state

* chore(API): PRWLR-4177 update spec

* chore(API): PRWLR-4177 remove force query parameter from DELETE /tasks

* feat(Backend): PRWLR-4177 add APITimeoutError and raise when TaskResult is not created

* feat(Backend): PRWLR-4177 add specific error class for task timeouts
 2024-09-06 20:47:51 -04:00
Jon Young f5462c9b27 fix(Scans): rename 'type' to 'trigger'. (#36) 
'type' is a reserved word in JSON:API schemas,
and python.

'trigger' more accurately describes the enum value.
 2024-09-06 15:13:27 +02:00
Víctor Fernández Poyatos 24857eaa7f feat/PRWLR-4002 add /scans endpoints (without SDK integration) (#33) 
* feat(Backend): PRWLR-4992 add Scan db model

* feat(API): PRWLR-4002 add viewset and required tools for /scans endpoint

* feat(API): PRWLR-4002 improve /scans filters

* feat(API): PRWLR-4002 add relationships links

* feat(API): PRWLR-4002 implement POST /scan custom logic

* fix(API): PRWLR-4002 fix Scan.type keyword usage

* feat(API): PRWLR-4002 implement PATCH /scans

* feat(API): PRWLR-4002 refactor serializers for write operations

* fix(API): PRWLR-4002 fix providers on ScanSerializer

* test(API): PRWLR-4002 add unit tests for /scans

* feat(Backend, Test): PRWLR-4002 refactor routing and unit tests db connections

* build(CI): PRWLR-4002 update CI env vars for postgres

* fix(Tests): PRWLR-4002 fix close_db_connections fixture

* feat(Backend, API): PRWLR-4002 apply requested changes to Scan model and filters

* chore(Tests): PRWLR-4002 rename pytest fixtures

* feat(Backend): PRWLR-4002 remove unique constraint from Scan model

* fix(Backend, Tests): PRWLR-4002 fix db routing and migration mechanism for tests

* chore(deps): PRWLR-4002 add uuid6 to poetry deps

* chore(Backend): PRWLR-4002 refactor filter methods

* fix(Tests): PRWLR-4002 fix wrong postgres credentials for testing

* feat(API): PRWLR-4002 implement merging mechanism for Scans.scanner_args

* feat(API): PRWLR-4002 implement merging mechanism for Scans.scanner_args

* feat(Backend): PRWLR-4002 add indexes to Scan model

* feat(Backend): PRWLR-4002 remove id index from Scan model (redundant)

* feat(API): PRWLR-4002 add datetime fields to Scan serializer
 2024-08-30 14:32:13 -04:00
Víctor Fernández Poyatos 8183207802 feat/PRWLR-4014 Implement SDK integration for POST /providers/{provider_id}/connection (#30) 
* chore(deps): PRWLR-4014 include prowler library in python deps

* feat(Backend,API): PRWLR-4014 add AWS provider test_connection through celery tasks

* fix(Backend,API): PRWLR-4014 fix model handling in celery tasks

* test(Tasks): PRWLR-4014 add unit tests for celery tasks

* docs(API): PRWLR-4014 update endpoint tag

* feat(Backend): PRWLR-4014 add decorator for tenant dependant Celery tasks

* chore(Backend): PRWLR-4014 remove TODOs and improve docstrings

* feat(Backend): PRWLR-4014 replace timezone.now for datetime.now(timezone.utc)

* feat(Backend): PRWLR-4014 use SET LOCAL for api.tenant_id setting

* feat(Backend, Tasks): PRWLR-4014 refactor tasks module to abstract business logic

* fix(Backend): PRWLR-4014 use set_config for RLS config and set transaction at request dispatch level

* fix(Tasks): PRWLR-4014 fix tasks tenant decorator
 2024-08-29 09:53:07 -04:00
Víctor Fernández Poyatos 8f2bd45872 feat/PRWLR-4413 Add Postgres Enums for Django and update Provider.provider field (#28) 
* feat(db): PRWLR-4413 add Provider Postgres Enum type for Django

* fix(Backend): PRWLR-4413 Fix initial migration for Providers

* feat(Backend): PRWLR-4413 add provider enum to Provider model

* fix(Backend, API): PRWLR-4413 fix ProviderEnum representation

* chore(Backend): PRWLR-4413 remove max_length constraint from provider enum

* chore(Backend): PRWLR-4413 refactor postgres enum creation to avoid boilerplate

* chore(Backend): PRWLR-4413 improve comments
 2024-08-21 18:02:46 +02:00
Víctor Fernández Poyatos 8a2cfea677 feat/PRWLR-4368 Remove TenantMiddleware and set tenant_id at view level (#31) 
* feat(API): PRWLR-4368 remove TenantMiddleware in favour of transaction based setup

* feat(API): PRWLR-4368 override initial request method to perform atomic transactions on RLS viewsets
 2024-08-21 10:10:01 +02:00
Víctor Fernández Poyatos 308f52c6f9 build(deps): bump gunicorn version to 23.0.0 (#29) 2024-08-13 16:28:39 +02:00
Víctor Fernández Poyatos 4f99518d52 fix(Dockerfile): add curl-dev to base image (#27) 2024-08-12 12:34:55 +02:00
Jon YoungandVíctor Fernández Poyatos a2ab216531 chore(Celery): add basic Celery worker with broker only (#20) 
* chore(Celery): add basic Celery worker with broker only

* chore(Celery): saving progress. Not able to schedule tasks

* fix(Celery): add celery app for use by django

* fix(Celery): register tasks

* fix(Docker): add celery workers to docker-compose

* chore(Celery): add django-celery-results backend

to store results using Django ORM

* fix(Celery): get app config the correct way

* fix(Docker): start connecting docker Celery workers to Valkey

not yet operational

* fix(Celery): get celery & django to work in docker-compose

* docs(Celery): document how to run Celery in development environment

includes changes to support the configuration and deployment
of Celery worker and its dependencies, Valkey and Postgres.

* fix(GHA): add valkey to CI services

* fix(GHA): add valkey to CI services

* fix(GHA): add valkey-cli ping to CI services

* fix(GHA): use right port for valkey

* fix(Views): remove debug task code

* test(Celery): start adding celery task tests

not yet working!

* fix(pyproject): rollback django upgrade

* fix(docker): updated docker runtime and env vars

based on feedback from #20

* fix(Dockerfile): include dependencies for psutil

psutil was introduced by pytest-celery

* fix(Backend): PRWLR-4013 fix celery settings structure

* fix(Celery): update celery app to work with new settings structure

* fix(Views): remove debug task code

* fix(Config): remove debug code

* fix(Celery): update celery app name when running worker

---------

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
 2024-08-09 16:47:35 +02:00
dependabot[bot] 43c9e70a65 chore(deps): bump django from 5.0.7 to 5.0.8 (#26) 
Bumps [django](https://github.com/django/django) from 5.0.7 to 5.0.8.
- [Commits](https://github.com/django/django/compare/5.0.7...5.0.8)

---
updated-dependencies:
- dependency-name: django
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-08-08 10:57:10 -04:00
Víctor Fernández Poyatos e2fc83c81b docs(README): add migration and fixtures commands (#23) 
* docs(README): add migration and fixtures commands

* docs(README): add env variables info
 2024-08-08 10:56:03 -04:00
Víctor Fernández Poyatos 857edd9aa6 chore(Backend,API): PRWLR-4341 restructure project based on versions (#25) 2024-08-07 16:40:18 -04:00
Jon Young 35128b89b1 chore(API): rename project to be Prowler API (#22) 2024-08-07 10:05:05 +02:00
Víctor Fernández Poyatos 4371ad1535 feat/PRWLR-3999 Implement providers endpoints (#21) 
* feat(Backend): PRWLR-3989 add RLS to postgresql db and base models

* feat(API): PRWLR-3989 add TenantMiddleware

* chore(API, Backend): PRWLR-3989 create new db user without RLS bypass on migrations

* chore(Backend): PRWLR-3989 fix RLS bypasser for POST requests

* fix(Backend): PRWLR-3989 fix user permissions when migrating new models

* chore(Backend): PRWLR-3989 add testing view for RLS manual tests

* feat(API): PRWLR-3989 add tenant_id to API logging

* chore(API, Backend): PRWLR-3989 add TODOs

* test(API): PRWLR-3989 add new middleware unit tests

* chore(API): PRWLR-3989 refactor RLS code

* fix(tests): PRWLR-3989 fix testing db connector

* chore: PRWLR-3989 add references to JIRA tickets

* fix: PRWLR-3989 remove bypass logic and fix serializers

* fix: PRWLR-3989 improve drop SQL query for RLS models

* feat(Backend): PRWLR-3989 add specific permissions on each model

* fix(Backend): PRWLR-3989 fix database routing and grant select perms

* fix(test): PRWLR-3989 fix routing issues with unit tests

* chore: PRWLR-3989 remove references to JIRA tickets

* feat(Backend): PRWLR-3999 add Provider model

* feat: PRWLR-3999 add providers view logic

* fix: PRWLR-3999 fix unique index fields

* feat(API): PRWLR-3999 add custom exception handlers

* feat(API): PRWLR-3999 add /providers/{provider_id}/connection endpoint

* test(API): PRWLR-3999 add base unit tests for providers

* fix(API, Backend): PRWLR-3999 fix bugs after rebase

* chore(API, docker): PRWLR-3999 add dev feats for demo

* feat(API): PRWLR-3999 add drf-spectacular-jsonapi and improve docs

* test(API): PRWLR-3999 add providers unit tests

* chore(Backend): PRWLR-3989 adjust privileges for user on table tenant

* chore: PRWLR-3999 rename model custom validation error

* chore: PRWLR-3999 remove Test references

* chore: PRWLR-3999 update API v1 spec

* fix: PRWLR-3999 apply requested changes to filter and models

* feat: PRWLR-3999 add validation to PATCH /providers payload

* fix: PRWLR-3999 fix providers enum description

* chore: add more providers fixtures

* fix: PRWLR-3999 make providers.alias optional
 2024-08-06 12:39:08 -04:00
Víctor Fernández Poyatos e1a12bcb14 feat/PRWLR-3989 Add RLS full implementation (#18) 
* feat(Backend): PRWLR-3989 add RLS to postgresql db and base models

* feat(API): PRWLR-3989 add TenantMiddleware

* chore(API, Backend): PRWLR-3989 create new db user without RLS bypass on migrations

* chore(Backend): PRWLR-3989 fix RLS bypasser for POST requests

* fix(Backend): PRWLR-3989 fix user permissions when migrating new models

* chore(Backend): PRWLR-3989 add testing view for RLS manual tests

* feat(API): PRWLR-3989 add tenant_id to API logging

* chore(API, Backend): PRWLR-3989 add TODOs

* test(API): PRWLR-3989 add new middleware unit tests

* chore(API): PRWLR-3989 refactor RLS code

* fix(tests): PRWLR-3989 fix testing db connector

* chore: PRWLR-3989 add references to JIRA tickets

* fix: PRWLR-3989 remove bypass logic and fix serializers

* fix: PRWLR-3989 improve drop SQL query for RLS models

* feat(Backend): PRWLR-3989 add specific permissions on each model

* fix(Backend): PRWLR-3989 fix database routing and grant select perms

* fix(test): PRWLR-3989 fix routing issues with unit tests

* chore: PRWLR-3989 remove references to JIRA tickets

* chore(Backend): PRWLR-3989 adjust privileges for user on table tenant

* chore: PRWLR-3989 add comments on migrations
 2024-08-02 11:29:05 -04:00
Jon Young 6a135cb47c PRWLR-4236: fix(CI): use postgres for unit tests (#19) 
* fix(Config): use local postgres for unit tests

* fix(CI): use postgres for unit tests in GHA
 2024-07-25 10:00:58 +02:00
Víctor Fernández Poyatos de97b9f298 feat/PRWLR-3999 Add providers endpoint with mock data (#17) 
* chore: PRWLR-3996 add drf-jsonapi library to project

* feat: PRWLR-3996 implement /tenants endpoint and basic JSON:API spec

* feat: PRWLR-3996 implement new middleware to ingest metadata on API responses

* chore: PRWLR-3996 annotate views with drf-spectacular

* feat: PRWLR-3996 add cache-control to tenants endpoints

* feat: PRWLR-3996 update views filters

* chore: PRWLR-3996 remove meta middleware in favour of meta fields in serializers

* test(API): PRWLR-3996 add unit tests for /tenants

* chore: PRWLR-3996 turn DEBUG on on devel environment

* test: PRWLR-3996 rework tenants unit tests

* chore: PRWLR-3996 refactor filter classes

* fix(Backend,API): PRWLR-3996 use correct filter backend

* feat(API): PRWLR-3999 implement mocked /providers endpoints

* feat(API): PRWLR-3999 add /providers/check_connection mocked endpoint and loc headers
 2024-07-23 12:34:48 -04:00
Víctor Fernández Poyatos 7cb3a4e16e feat/PRWLR-3996 implement /tenants endpoint and basic filters logic (#14) 
* chore: PRWLR-3996 add drf-jsonapi library to project

* feat: PRWLR-3996 implement /tenants endpoint and basic JSON:API spec

* feat: PRWLR-3996 implement new middleware to ingest metadata on API responses

* chore: PRWLR-3996 annotate views with drf-spectacular

* feat: PRWLR-3996 add cache-control to tenants endpoints

* feat: PRWLR-3996 update views filters

* chore: PRWLR-3996 remove meta middleware in favour of meta fields in serializers

* test(API): PRWLR-3996 add unit tests for /tenants

* chore: PRWLR-3996 turn DEBUG on on devel environment

* test: PRWLR-3996 rework tenants unit tests

* chore: PRWLR-3996 refactor filter classes

* fix(Backend,API): PRWLR-3996 use correct filter backend
 2024-07-22 16:06:57 -04:00
Jon Young 43f79663d9 chore(CODEOWNERS): Update for team name change (#16) 
* chore(CODEOWNERS): update CODEOWNERS

* chore(CODEOWNERS): update CODEOWNERS

* chore(CODEOWNERS): update CODEOWNERS
 2024-07-22 08:43:50 +02:00
Jon YoungandVíctor Fernández Poyatos cd7d7c303a PRWLR-3997: fix(Models): create BaseModel & TenantModel (#13) 
* fix(Models): create BaseModel & TenantModel

* update description to match class

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>

* fix(Tenants): rename TenantModel to Tenant

---------

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
 2024-07-16 15:44:59 +02:00
Víctor Fernández Poyatos 8f41b38bbf chore: PRWLR-4055 Update project structure and add env vars (#12) 
* chore: PRWLR-4055 update project structure and add env vars

* chore: PRWLR-4055 make production settings default

* chore: PRWLR-4055 change default values to env variables

* chore: PRWLR-4055 apply requested changes on default values

* chore: PRWLR-4055 adapt testing environment
 2024-07-16 09:33:56 -04:00
Víctor Fernández Poyatos 9df759da60 feat: PRWLR-3989 add PostgreSQL to project (#10) 
* feat: PRWLR-3989 add postgresql to project

* chore: PRWLR-3989 add requested changes to docker structure

* chore: PRWLR-3989 update poetry.lock
 2024-07-11 10:39:09 -04:00
dependabot[bot] 157a54f30c chore(deps): bump django from 5.0.6 to 5.0.7 (#11) 
Bumps [django](https://github.com/django/django) from 5.0.6 to 5.0.7.
- [Commits](https://github.com/django/django/compare/5.0.6...5.0.7)

---
updated-dependencies:
- dependency-name: django
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-07-11 10:34:41 +02:00
Víctor Fernández Poyatos f0a276773e feat(backend): PRWLR-4030 Implement Gunicorn as Django WSGI (#9) 
* feat: PRWLR-4030 implement Gunicorn server and logging

* build: PRWLR-4030 update docker files

* docs: PRWLR-4030 update README

* chore: PRWLR-4030 change binding address to 0.0.0.0 by default
 2024-07-10 16:32:56 +02:00
Víctor Fernández Poyatos 457801f752 chore: update CODEOWNERS to include restful-api team (#8) 2024-07-08 13:09:04 +02:00
Víctor Fernández Poyatos 3a502c5b3d feat(Backend): PRWLR-3990 Add custom logging (#6) 
* chore: PRWLR-4022 add django-cors-headers

* chore: PRWLR-4022 configure basic CORS

* feat: PRWLR-4022 add drf-spectacular and spec/docs endpoints

* build: PRWLR-4022 add basic docker development deployment

* chore: PRWLR-4022 update schema generation

* chore: PRWLR-4022 bump required Python version to 3.12

* feat: PRWLR-3990 add basic custom Django logging

* feat: PRWLR-3990 add tasks logger

* chore: PRWLR-3990 update API logging middleware to follow updated syntax

* docs: PRWLR-3990 add docstring for APILoggingMiddleware class

* chore: PRWLR-3990 add testing dependencies to poetry

* test(API): PRWLR-3990 add new middleware unit tests and base setup

* feat(API, Backend): PRWLR-3990 add NDJSONFormatter and transaction_id

* feat(API): PRWLR-3990 add dev/test model, views and urls

* chore(backend): PRWLR-3990 refactor logging settings and modules

* chore(backend): PRWLR-3990 add BackendLogger enum and refactor logging modules

* chore(backend): PRWLR-3990 refactor module importing

* style(backend): PRWLR-3990 remove extra endlines

* chore: PRWLR-3990 update poetry.lock

* chore: PRWLR-3990 run pre-commit
 2024-07-05 10:13:04 -04:00
Jon Young 30775373dc chore(CI): add labeler definition (#7) 2024-07-05 09:30:24 -04:00
Jon Young 7e194407f6 chore(CI): PRWLR-3897: add Continuous Integration workflows (#3) 
* chore(CI): import pre-commit-config.yaml from prowler-cloud/prowler

* chore(CI): import pre-commit-config.yaml from prowler-cloud/prowler

* chore(CI): add safety package to pre-commit

* chore(CI): use poetry run to access venv for local checks

* docs(repo): add community focused docs

* chore(GitHub): add ISSUE_TEMPLATEs

same as prowler-cloud/prowler repo

* chore(CI): add pull-request.yml workflow

runs on PRs and pushes to main

* chore(CI): add additional workflows

same as those on prowler-cloud/prowler

* fix(CI): use correct path

* fix(API): fix flake8 F401 failures

* fix(API, Backend): format with black

* fix( Backend): format with black

* fix(CI): use correct path

* fix(CI): move pyproject.toml

* fix(CI): simplify tools with ruff

* fix(CI): use ruff instead of flake8, black

* fix(CI): change ruff commands for GHA

* fix(CI): use correct ruff command name

* fix(Backend): ruff format

* fix(CI): ignore safety jinja2 vuln id 70612

and add mypy for future usage

* fix(CI): set file path for hadolint

* fix(CI): add dockle

* fix(CI): update dockle version

* fix(CI): fix hadolint errors

* fix(CI): add simple tests for CI

* fix(Docker): multi-stage docker build

* docs(README): update docs for poetry location
 2024-07-05 09:01:02 -04:00
Víctor Fernández Poyatos a69a155679 docs(backend): PRWLR-3988 Include Django development guide (#5) 
* docs: PRWLR-3988 add development guide to README

* docs: PRWLR-3988 add description to README
 2024-07-03 11:45:54 -04:00
Víctor Fernández Poyatos 7ff6d860ce feat(backend): PRWLR-4022 Basic Django and DRF setup and dev tools (#4) 
* chore: PRWLR-4022 add django-cors-headers

* chore: PRWLR-4022 configure basic CORS

* feat: PRWLR-4022 add drf-spectacular and spec/docs endpoints

* build: PRWLR-4022 add basic docker development deployment

* chore: PRWLR-4022 update schema generation

* chore: PRWLR-4022 bump required Python version to 3.12
 2024-07-03 09:34:29 -04:00
dependabot[bot] b440be717c chore(deps): bump djangorestframework from 3.15.1 to 3.15.2 in /src (#2) 
Bumps [djangorestframework](https://github.com/encode/django-rest-framework) from 3.15.1 to 3.15.2.
- [Release notes](https://github.com/encode/django-rest-framework/releases)
- [Commits](https://github.com/encode/django-rest-framework/compare/3.15.1...3.15.2)

---
updated-dependencies:
- dependency-name: djangorestframework
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-07-02 10:00:59 +02:00
Jon Young d8752719c1 Merge pull request #1 from prowler-cloud/PRWLR-3980-project-setup 
feat: PRWLR-3980 Setup base project structure
 2024-07-01 12:20:20 -04:00
Víctor Fernández Poyatos 737a0ff9cb chore: update .gitignore 2024-07-01 18:04:13 +02:00
Víctor Fernández Poyatos 1c8e676822 chore: update CODEOWNERS 2024-07-01 18:03:27 +02:00
Víctor Fernández Poyatos 7b98f0fc92 chore: update pull request template 2024-07-01 18:00:47 +02:00
Víctor Fernández Poyatos 45865f2e71 chore: add required files for project base structure 2024-07-01 17:43:05 +02:00
Víctor Fernández Poyatos eded2df687 chore: add .github folder and related files 2024-07-01 17:42:29 +02:00
Víctor Fernández Poyatos 766d3f6670 chore: PRWLR-3980 add Django project and API 2024-07-01 17:42:03 +02:00
Víctor Fernández Poyatos 3f2d0a13af chore: PRWLR-3980 add poetry files 2024-07-01 17:41:18 +02:00
Víctor Fernández Poyatos 690957e1c3 chore: PRWLR-3980 update .gitignore 2024-07-01 17:19:19 +02:00
Jon Young 3092b56fd6 Initial commit 2024-07-01 10:42:55 -04:00