Commit Graph
11 Commits
Author SHA1 Message Date
Víctor Fernández Poyatos 3de2c47c56 fix(Scan): PRWLR-5145 Update resource storing when performing a scan (#67)
* chore(Resource): PRWLR-5145 remove region from unique constraint

* feat(Resource): PRWLR-5145 update how resources are stored during a scan

* test(Task): PRWLR-5145 add unit tests for scan task resources
2024-11-04 11:05:02 +01:00
Víctor Fernández Poyatos ed427c1352 chore(Scan, Finding): PRWLR-5056 Adjust finding information storing when performing a scan (#57)
* chore: PRWLR-5056 update Prowler dependency

* feat(Scan): PRWLR-5056 adapt scan task code to sdk breaking changes

* test(Scan): PRWLR-5056 fix unit tests

* chore: PRWLR-5056 update fixtures

* chore: PRWLR-5056 update Prowler dependency
2024-10-18 13:56:07 +02:00
Víctor Fernández Poyatos 351132fb5b feat(ProviderSecret): PRWLR-4102 Implement /providers/secrets CRUD (#53)
* feat(Credential): PRWLR-4102 add model definition

* feat(Credential): PRWLR-4102 add serializers, views, urls and decorator

* feat(ProviderSecret): PRWLR-4102 rework credentials into provider secrets

* feat(ProviderSecret): PRWLR-4102 rework urls, views, serializers

* feat(Provider, Scan): PRWLR-4102 refactor modules to adapt connection test and scans to secrets

* feat(ProviderSecret): PRWLR-4102 add GCPProviderSecret serializer

* feat(Provider): PRWLR-4102 validate provider_id when testing connection

* chore(Utils): PRWLR-4987 refactor prowler provider functions

* test(Utils): PRWLR-4102 add unit tests

* test(Scan, Provider): PRWLR-4102 fix unit tests

* chore: PRWLR-4102 add docstrings and update prowler version and API spec

* chore(Provider, Scan): PRWLR-4102 update fixtures

* test(ProviderSecret): PRWLR-4102 add unit tests

* chore: PRWLR-4102 optimize imports
2024-10-11 15:49:45 +02:00
Víctor Fernández Poyatos 8791b7e3f1 feat(Scan, Finding): PRWLR-4987 Adjust findings delta based on previous findings (#52)
* feat(Finding): PRWLR-4987 add uid field to Findings

* feat(Scan): PRWLR-4987 implement logic to calculate Finding delta value

* test(Scan): PRWLR-4987 add unit tests for _create_finding_delta

* chore(Spec): PRWLR-4987 update API spec
2024-10-08 14:43:18 +02:00
Víctor Fernández Poyatos 6bd8a17a5f fix(Task): PRWLR-4970 Fix Celery task issues when status is pending and race conditions (#49)
* fix(Task): PRWLR-4970 add TaskResult entry to database when task reaches broker

* fix(Task, Scan): PRWLR-4970 remove race conditions using atomic transactions

* chore(Django): PRWLR-4970 bump Django version to 5.1.1
2024-10-04 11:54:15 +02:00
Víctor Fernández Poyatos ded28baa2f feat(Scan, Resource, Finding): PRWLR-4015 Implement on demand scans and findings/resources in database (#48)
* feat(Users): PRWLR-4718 make user email the default login username

* feat(Token): PRWLR-4718 add serializers, views and urls for access and refresh tokens

* feat(Token): PRWLR-4718 add first membership tenant in token if not present in json body

* feat(Users): PRWLR-4718 add company_name to model

* feat(Users): PRWLR-4718 create tenant and membership when creating new user

* fix(BaseView): PRWLR-4718 add tenant_id to serializer context

* fix(Tests): PRWLR-4718 use authorization with unit tests

* fix(Views): PRWLR-4718 fix tenant retrieval from request

* fix(Tests): PRWLR-4718 fix tests

* fix(Fixtures): PRWLR-4718 fix tenant memberships ordering

* chore(Tokens): PRWLR-4718 update token url

* chore(Spec): PRWLR-4718 update API spec

* feat(Tokens): PRWLR-4718 enable token refresh blacklisting

* feat(Tokens): PRWLR-4718 implement RS256 algorithm and dev valid keys

* chore(env): PRWLR-4718 update .env.example

* chore(Deps): PRWLR-4015 update prowler dep

* fix(Resources, Findings): PRWLR-4015 fix permission issues on models and migrations

* feat(Scans, Tasks, Resources, Findings): PRWLR-4015 perform whole flow when executing scans

* fix(Settings): PRWLR-4015 fix devel JWT settings

* chore(Scans, Tasks): PRWLR-4015 improve docs and responses format

* test(Scan, Provider): PRWLR-4015 fix unit tests

* chore(Environment): PRWLR-4015 fix .env.example values

* test(Scan): PRWLR-4015 add unit tests for scan task

* chore(Task): PRWLR-4015 give proper format to task result on scan perform

* feat(Provider, Scan): PRWLR-4015 add all provider types to connection check and scan

* fix(Logging): PRWLR-4015 fix API logger middleware to include tenant and user IDs
2024-10-03 18:36:42 +02:00
Jon YoungandVíctor Fernández Poyatos 6a341b88f0 fix(Providers, Resources, Scans): rename provider_id and filter on more provider fields (#42)
* fix(Providers, Resources, Scans): filter on more provider fields

* Apply suggestions from code review

more python-y

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>

---------

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
2024-09-13 16:09:09 +02:00
Víctor Fernández Poyatos ec67fc12e0 feat/PRWLR-4177 Add /tasks endpoints and data model (#35)
* feat(Backend): PRWLR-4177 add Task model and migrations

* feat(Tasks): PRWLR-4177 add RLSTask class

* feat(API): PRWLR-4177 add Task serializers

* feat(Backend, DB): PRWLR-4177 refactor db variables and add policy on task runner tasks

* feat(API): PRWLR-4177 add Tasks filters and sort fields

* feat(API, Tasks): PRWLR-4177 add deletion tasks and revoke logic to /tasks

* test(Task): PRWLR-4177 add deletion tasks unit tests

* test(Views): PRWLR-4177 add Tasks views unit tests and update outdated ones

* chore(API): PRWLR-4177 improve drf-spectacular annotations

* chore(API): PRWLR-4177 add PROGRESS task state

* chore(API): PRWLR-4177 update spec

* chore(API): PRWLR-4177 remove force query parameter from DELETE /tasks

* feat(Backend): PRWLR-4177 add APITimeoutError and raise when TaskResult is not created

* feat(Backend): PRWLR-4177 add specific error class for task timeouts
2024-09-06 20:47:51 -04:00
Víctor Fernández Poyatos 24857eaa7f feat/PRWLR-4002 add /scans endpoints (without SDK integration) (#33)
* feat(Backend): PRWLR-4992 add Scan db model

* feat(API): PRWLR-4002 add viewset and required tools for /scans endpoint

* feat(API): PRWLR-4002 improve /scans filters

* feat(API): PRWLR-4002 add relationships links

* feat(API): PRWLR-4002 implement POST /scan custom logic

* fix(API): PRWLR-4002 fix Scan.type keyword usage

* feat(API): PRWLR-4002 implement PATCH /scans

* feat(API): PRWLR-4002 refactor serializers for write operations

* fix(API): PRWLR-4002 fix providers on ScanSerializer

* test(API): PRWLR-4002 add unit tests for /scans

* feat(Backend, Test): PRWLR-4002 refactor routing and unit tests db connections

* build(CI): PRWLR-4002 update CI env vars for postgres

* fix(Tests): PRWLR-4002 fix close_db_connections fixture

* feat(Backend, API): PRWLR-4002 apply requested changes to Scan model and filters

* chore(Tests): PRWLR-4002 rename pytest fixtures

* feat(Backend): PRWLR-4002 remove unique constraint from Scan model

* fix(Backend, Tests): PRWLR-4002 fix db routing and migration mechanism for tests

* chore(deps): PRWLR-4002 add uuid6 to poetry deps

* chore(Backend): PRWLR-4002 refactor filter methods

* fix(Tests): PRWLR-4002 fix wrong postgres credentials for testing

* feat(API): PRWLR-4002 implement merging mechanism for Scans.scanner_args

* feat(API): PRWLR-4002 implement merging mechanism for Scans.scanner_args

* feat(Backend): PRWLR-4002 add indexes to Scan model

* feat(Backend): PRWLR-4002 remove id index from Scan model (redundant)

* feat(API): PRWLR-4002 add datetime fields to Scan serializer
2024-08-30 14:32:13 -04:00
Víctor Fernández Poyatos 8183207802 feat/PRWLR-4014 Implement SDK integration for POST /providers/{provider_id}/connection (#30)
* chore(deps): PRWLR-4014 include prowler library in python deps

* feat(Backend,API): PRWLR-4014 add AWS provider test_connection through celery tasks

* fix(Backend,API): PRWLR-4014 fix model handling in celery tasks

* test(Tasks): PRWLR-4014 add unit tests for celery tasks

* docs(API): PRWLR-4014 update endpoint tag

* feat(Backend): PRWLR-4014 add decorator for tenant dependant Celery tasks

* chore(Backend): PRWLR-4014 remove TODOs and improve docstrings

* feat(Backend): PRWLR-4014 replace timezone.now for datetime.now(timezone.utc)

* feat(Backend): PRWLR-4014 use SET LOCAL for api.tenant_id setting

* feat(Backend, Tasks): PRWLR-4014 refactor tasks module to abstract business logic

* fix(Backend): PRWLR-4014 use set_config for RLS config and set transaction at request dispatch level

* fix(Tasks): PRWLR-4014 fix tasks tenant decorator
2024-08-29 09:53:07 -04:00
Jon YoungandVíctor Fernández Poyatos a2ab216531 chore(Celery): add basic Celery worker with broker only (#20)
* chore(Celery): add basic Celery worker with broker only

* chore(Celery): saving progress. Not able to schedule tasks

* fix(Celery): add celery app for use by django

* fix(Celery): register tasks

* fix(Docker): add celery workers to docker-compose

* chore(Celery): add django-celery-results backend

to store results using Django ORM

* fix(Celery): get app config the correct way

* fix(Docker): start connecting docker Celery workers to Valkey

not yet operational

* fix(Celery): get celery & django to work in docker-compose

* docs(Celery): document how to run Celery in development environment

includes changes to support the configuration and deployment
of Celery worker and its dependencies, Valkey and Postgres.

* fix(GHA): add valkey to CI services

* fix(GHA): add valkey to CI services

* fix(GHA): add valkey-cli ping to CI services

* fix(GHA): use right port for valkey

* fix(Views): remove debug task code

* test(Celery): start adding celery task tests

not yet working!

* fix(pyproject): rollback django upgrade

* fix(docker): updated docker runtime and env vars

based on feedback from #20

* fix(Dockerfile): include dependencies for psutil

psutil was introduced by pytest-celery

* fix(Backend): PRWLR-4013 fix celery settings structure

* fix(Celery): update celery app to work with new settings structure

* fix(Views): remove debug task code

* fix(Config): remove debug code

* fix(Celery): update celery app name when running worker

---------

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
2024-08-09 16:47:35 +02:00