Commit Graph
17 Commits
Author SHA1 Message Date
Víctor Fernández Poyatos 54bb034cac feat(Tokens): PRWLR-4718 implement user authorization with JWT (#47)
* feat(Users): PRWLR-4718 make user email the default login username

* feat(Token): PRWLR-4718 add serializers, views and urls for access and refresh tokens

* feat(Token): PRWLR-4718 add first membership tenant in token if not present in json body

* feat(Users): PRWLR-4718 add company_name to model

* feat(Users): PRWLR-4718 create tenant and membership when creating new user

* fix(BaseView): PRWLR-4718 add tenant_id to serializer context

* fix(Tests): PRWLR-4718 use authorization with unit tests

* fix(Views): PRWLR-4718 fix tenant retrieval from request

* fix(Tests): PRWLR-4718 fix tests

* fix(Fixtures): PRWLR-4718 fix tenant memberships ordering

* chore(Tokens): PRWLR-4718 update token url

* chore(Spec): PRWLR-4718 update API spec

* feat(Tokens): PRWLR-4718 enable token refresh blacklisting

* feat(Tokens): PRWLR-4718 implement RS256 algorithm and dev valid keys

* fix(Environment): PRWLR-4718 fix jwt keys env vars

* fix(Environment): PRWLR-4718 fix jwt keys env vars (testing)

* chore(Settings): PRWLR-4718 remove drf-spectacular unused settings

* fix(Environment): PRWLR-4718 remove jwt signature keys from dev and testing modules
2024-09-30 17:45:09 +02:00
Víctor Fernández Poyatos 4c83351b26 feat(Tenants, Memberships): PRWLR-4719 Add memberships endpoints and security changes (#44)
* feat(SimpleJWT): PRWLR-4717 add djangorestframework-simplejwt dep

* feat(SimpleJWT): PRWLR-4717 add basic configuration and env variables

* feat(Users): PRWLR-4717 add model and security constraints

* feat(Users): PRWLR-4717 add serializers

* feat(Users): PRWLR-4717 add views

* test(Conftest): PRWLR-4717 add user and authenticated client fixtures

* fix(Unit tests): PRWLR-4717 add automated authentication to all unit tests

* fix(Authentication): PRWLR-4717 add authentication class and update tests

* test(Users): PRWLR-4717 add unit tests

* test(Users): PRWLR-4717 add integration tests

* chore(Schema): PRWLR-4717 update API schema

* fix(User): PRWLR-4717 fix password validation

* feat(Validators): PRWLR-4717 add MaxLength password validator

* fix(User): PRWLR-4717 update User model to delete admin fields

* chore(Serializers): PRWLR-4717 add docstrings and update serializers

* chore(Fixtures): PRWLR-4717 add dev user

* feat(Memberships): PRWLR-4719 add membership.Roles postgres enums

* feat(Memberships): PRWLR-4719 add model

* feat(Tenants): PRWLR-4719 add owner membership when creating a new one

* chore(Users): PRWLR-4717 raise DRF NotFound instead of returning response

* chore(Deps): PRWLR-4719 add drf-nested-routers

* feat(Memberships): PRWLR-4719 add serializers, urls and views

* feat(Tenants): PRWLR-4719 add RLS based on memberships

* chore(Views): PRWLR-4719 unify tenant related views dispatch methods

* test(Tenants): PRWLR-4719 add membership to test user when creating tenants

* test(Users): PRWLR-4719 add name field to model

* chore(Deps): PRWLR-4719 add drf-nested-routers

* feat(Memberships): PRWLR-4719 separate membership endpoints

* feat(Memberships): PRWLR-4719 add filters

* test(Memberships): PRWLR-4719 add unit tests

* test(Tenants): PRWLR-4719 add membership unit tests

* chore(API): PRWLR-4719 update spec

* chore(API): PRWLR-4719 update Django dev fixtures

* test(Memberships): PRWLR-4719 add more unit tests

* chore(Memberships): PRWLR-4719 move /memberships to /users/me/

* feat(Users, Tenants, Memberships): PRWLR-4719 update responses and routes

* feat(Memberships): PRWLR-4719 adjust tenant membership view results under membership role

* chore(Scans): PRWLR-4719 rename django fixture
2024-09-25 13:12:15 +02:00
Jon YoungandVíctor Fernández Poyatos 4ca95b08e2 feat(Findings): Partitioned database tables (#45)
* feat(Findings): initial findings model

* fix(Findings): add view, serializers, migration for enums

* fix(Findings): incomplete jsonb_to_tsvector wrapper

will not run as written

* fix(Findings): use Severity and Status enums from prowler SDK

* tests(Findings): add failing view tests

* fix(Finding): add resource relationship

not returning correct data from serializer, missing links

* fix(FindingSerializer): get Scan & Resource relationships to show up

* fix(FindingFilter): add more filter fields

* fix(FindingFilter): filter on provider id

* fix(FindingSerializer): return Resource in relationship

not ResourceFindingMapping

* fix(FindingModel): update migration

* fix(FindingFilter): full text search on findings

* fix(Resources): include Findings in ResourceSerializer

* fix(FindingFilter): expand text search columns

* fix(DbUtils): docstring, not comment

* fix(BaseViews): remove TODO

comment not applicable right now

* fix(Fixtures): add more findings to fixture file

and change on_delete behavior for resource_finding_mapping

* fix(Resources): rename index to match others

* fix(Findings): update Findigns RLS to allow for full CRUD

eventually we'll let users enter a manual finding
which implies INSERT, UPDATE, DELETE

* fix(Findings): use TextChoices directly for Status enum

* fix(FindingSerializer): build a set instead of a list

* consistency in fixtures

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>

* fix(API): update v1 spec for findings

* feat(Findings): store findings in a partitioned table

* fix(Settings): change unit of partition max age

to match unit when creating

* docs(Partitions): document how to manage partitions via manage.py

* fix(Findings): add tag for spec/docs

* fix(Findings): clean up migrations

* fix(Findings): convert scan_id and inserted_at filters into finding.id filters

* fix(Findings): add default filter for today

and set default sort order

* fix(Findings): add default filter for today

and set default sort order

* fix(Findings): update fixtures so datetime matches id

* fix(Findings): partition the ResourceFindingMapping table to match Findings

* docs(Partitions): document postgres config values more

* docs(UUIDUtils): do not use raw query strigns (typo)

* docs(Partitions): change unit in comment description

* fix(Findings): change resource_name & tags to be Finding

* docs(Partitions): change unit in partitions settings docstring

* fix(Findings): remove conflicting logic & filters

* chore: apply suggested changes

* chore: optimize imports

---------

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
2024-09-23 11:39:03 +02:00
Jon YoungandVíctor Fernández Poyatos 489ac20141 feat(Findings): add the /findings endpoint (#38)
* feat(Findings): initial findings model

* fix(Findings): add view, serializers, migration for enums

* fix(Findings): incomplete jsonb_to_tsvector wrapper

will not run as written

* fix(Findings): use Severity and Status enums from prowler SDK

* tests(Findings): add failing view tests

* fix(Finding): add resource relationship

not returning correct data from serializer, missing links

* fix(FindingSerializer): get Scan & Resource relationships to show up

* fix(FindingFilter): add more filter fields

* fix(FindingFilter): filter on provider id

* fix(FindingSerializer): return Resource in relationship

not ResourceFindingMapping

* fix(FindingModel): update migration

* fix(FindingFilter): full text search on findings

* fix(Resources): include Findings in ResourceSerializer

* fix(FindingFilter): expand text search columns

* fix(DbUtils): docstring, not comment

* fix(BaseViews): remove TODO

comment not applicable right now

* fix(Fixtures): add more findings to fixture file

and change on_delete behavior for resource_finding_mapping

* fix(Resources): rename index to match others

* fix(Findings): update Findigns RLS to allow for full CRUD

eventually we'll let users enter a manual finding
which implies INSERT, UPDATE, DELETE

* fix(Findings): use TextChoices directly for Status enum

* fix(FindingSerializer): build a set instead of a list

* consistency in fixtures

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>

* fix(API): update v1 spec for findings

---------

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
2024-09-18 19:28:03 +02:00
Víctor Fernández Poyatos 9ffde34198 feat(Users): #PRWLR-4717 add /users endpoints and basic auth (#43)
* feat(SimpleJWT): PRWLR-4717 add djangorestframework-simplejwt dep

* feat(SimpleJWT): PRWLR-4717 add basic configuration and env variables

* feat(Users): PRWLR-4717 add model and security constraints

* feat(Users): PRWLR-4717 add serializers

* feat(Users): PRWLR-4717 add views

* test(Conftest): PRWLR-4717 add user and authenticated client fixtures

* fix(Unit tests): PRWLR-4717 add automated authentication to all unit tests

* fix(Authentication): PRWLR-4717 add authentication class and update tests

* test(Users): PRWLR-4717 add unit tests

* test(Users): PRWLR-4717 add integration tests

* chore(Schema): PRWLR-4717 update API schema

* fix(User): PRWLR-4717 fix password validation

* feat(Validators): PRWLR-4717 add MaxLength password validator

* fix(User): PRWLR-4717 update User model to delete admin fields

* chore(Serializers): PRWLR-4717 add docstrings and update serializers

* chore(Fixtures): PRWLR-4717 add dev user

* chore(Users): PRWLR-4717 raise DRF NotFound instead of returning response
2024-09-18 10:19:44 +02:00
Jon YoungandVíctor Fernández Poyatos 6a341b88f0 fix(Providers, Resources, Scans): rename provider_id and filter on more provider fields (#42)
* fix(Providers, Resources, Scans): filter on more provider fields

* Apply suggestions from code review

more python-y

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>

---------

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
2024-09-13 16:09:09 +02:00
Jon YoungandVíctor Fernández Poyatos f7645e8f25 feat(Resources): add the /resources endpoint (#34)
* fix(Resources): add basic Resource MVC

* fix(Resources): implement many-to-many tags via through table

* fix(Resources): tsvector & migration

migration needs work, need to create a custom
django migration field to build the GENERATED ALWAYS
column.

Without it, django will complain about the ts column
not exisiting when we try to create an index.

* test(Resources): add tests for /resources views

* fix(Resources): get basic test cases to pass

need to work on tsvector column
and filtering by provider columns like alias, name

* fix(Resources): provide way to get tags as dict

not a list of ResourceTag models

* fix(Resources): annotate with return type

* fix(Resources): rename search field to not conflict with filters

* fix(Resources): filter by provider.id

* fix(Resources): remove filter and sort by provider

must use provider_id first

* fix(Resources): test adding, clearing tags

* fix(Scans): filter by provider_id

* fix(Resources): remove FIXME comments

* fix(Resources): filter and search on tags

* fix(Resources): full text search

* fix(Resources): full text search to include both resources and tags

and get rid of the distinct() query

* fix(Resources): document text search a bit more

* fix(Resources): remove TODO comments

* fix(Resources): move Resource migrations into 0001_initital

* fix(Config): revert to match main

* fix(Resources): use Django's default query builder for full text search

* Apply suggestions from code review

Mostly formatting changes

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>

* fix(Resources): just use text search

instead of guessing at allowed split characters for each provider

* fix(Resources): remove unused tests

these cases are covered in parameterized lists

* fix(Resources): set a read-only RLS policy

* fix(Resources): rename add_tags to be upsert_or_delete_tags

* fix(Resources): include primary key on through table

django query builder doesn't like tables without primary keys

* fix(Resources): default xx_at filters to date

instead of date time

* docs(Resources): describe what a Resource is

in API docs

* chore(Resources): rename file to match others

---------

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
2024-09-11 18:04:34 +02:00
Víctor Fernández Poyatos 73330ecb1a chore/PRWLR-4775 update datetime default filter for date on inserted_at and updated_at API attributes (#39)
* chore(API): PRWLR-4775 change inserted and updated at filters to filter by date by default

* test(API): PRWLR-4775 replace datetime strings for date

* feat(API): PRWLR-4775 update started_at default behavior and unit tests
2024-09-09 16:52:32 -04:00
Víctor Fernández Poyatos ec67fc12e0 feat/PRWLR-4177 Add /tasks endpoints and data model (#35)
* feat(Backend): PRWLR-4177 add Task model and migrations

* feat(Tasks): PRWLR-4177 add RLSTask class

* feat(API): PRWLR-4177 add Task serializers

* feat(Backend, DB): PRWLR-4177 refactor db variables and add policy on task runner tasks

* feat(API): PRWLR-4177 add Tasks filters and sort fields

* feat(API, Tasks): PRWLR-4177 add deletion tasks and revoke logic to /tasks

* test(Task): PRWLR-4177 add deletion tasks unit tests

* test(Views): PRWLR-4177 add Tasks views unit tests and update outdated ones

* chore(API): PRWLR-4177 improve drf-spectacular annotations

* chore(API): PRWLR-4177 add PROGRESS task state

* chore(API): PRWLR-4177 update spec

* chore(API): PRWLR-4177 remove force query parameter from DELETE /tasks

* feat(Backend): PRWLR-4177 add APITimeoutError and raise when TaskResult is not created

* feat(Backend): PRWLR-4177 add specific error class for task timeouts
2024-09-06 20:47:51 -04:00
Jon Young f5462c9b27 fix(Scans): rename 'type' to 'trigger'. (#36)
'type' is a reserved word in JSON:API schemas,
and python.

'trigger' more accurately describes the enum value.
2024-09-06 15:13:27 +02:00
Víctor Fernández Poyatos 24857eaa7f feat/PRWLR-4002 add /scans endpoints (without SDK integration) (#33)
* feat(Backend): PRWLR-4992 add Scan db model

* feat(API): PRWLR-4002 add viewset and required tools for /scans endpoint

* feat(API): PRWLR-4002 improve /scans filters

* feat(API): PRWLR-4002 add relationships links

* feat(API): PRWLR-4002 implement POST /scan custom logic

* fix(API): PRWLR-4002 fix Scan.type keyword usage

* feat(API): PRWLR-4002 implement PATCH /scans

* feat(API): PRWLR-4002 refactor serializers for write operations

* fix(API): PRWLR-4002 fix providers on ScanSerializer

* test(API): PRWLR-4002 add unit tests for /scans

* feat(Backend, Test): PRWLR-4002 refactor routing and unit tests db connections

* build(CI): PRWLR-4002 update CI env vars for postgres

* fix(Tests): PRWLR-4002 fix close_db_connections fixture

* feat(Backend, API): PRWLR-4002 apply requested changes to Scan model and filters

* chore(Tests): PRWLR-4002 rename pytest fixtures

* feat(Backend): PRWLR-4002 remove unique constraint from Scan model

* fix(Backend, Tests): PRWLR-4002 fix db routing and migration mechanism for tests

* chore(deps): PRWLR-4002 add uuid6 to poetry deps

* chore(Backend): PRWLR-4002 refactor filter methods

* fix(Tests): PRWLR-4002 fix wrong postgres credentials for testing

* feat(API): PRWLR-4002 implement merging mechanism for Scans.scanner_args

* feat(API): PRWLR-4002 implement merging mechanism for Scans.scanner_args

* feat(Backend): PRWLR-4002 add indexes to Scan model

* feat(Backend): PRWLR-4002 remove id index from Scan model (redundant)

* feat(API): PRWLR-4002 add datetime fields to Scan serializer
2024-08-30 14:32:13 -04:00
Víctor Fernández Poyatos 8183207802 feat/PRWLR-4014 Implement SDK integration for POST /providers/{provider_id}/connection (#30)
* chore(deps): PRWLR-4014 include prowler library in python deps

* feat(Backend,API): PRWLR-4014 add AWS provider test_connection through celery tasks

* fix(Backend,API): PRWLR-4014 fix model handling in celery tasks

* test(Tasks): PRWLR-4014 add unit tests for celery tasks

* docs(API): PRWLR-4014 update endpoint tag

* feat(Backend): PRWLR-4014 add decorator for tenant dependant Celery tasks

* chore(Backend): PRWLR-4014 remove TODOs and improve docstrings

* feat(Backend): PRWLR-4014 replace timezone.now for datetime.now(timezone.utc)

* feat(Backend): PRWLR-4014 use SET LOCAL for api.tenant_id setting

* feat(Backend, Tasks): PRWLR-4014 refactor tasks module to abstract business logic

* fix(Backend): PRWLR-4014 use set_config for RLS config and set transaction at request dispatch level

* fix(Tasks): PRWLR-4014 fix tasks tenant decorator
2024-08-29 09:53:07 -04:00
Víctor Fernández Poyatos 8a2cfea677 feat/PRWLR-4368 Remove TenantMiddleware and set tenant_id at view level (#31)
* feat(API): PRWLR-4368 remove TenantMiddleware in favour of transaction based setup

* feat(API): PRWLR-4368 override initial request method to perform atomic transactions on RLS viewsets
2024-08-21 10:10:01 +02:00
Víctor Fernández Poyatos 4371ad1535 feat/PRWLR-3999 Implement providers endpoints (#21)
* feat(Backend): PRWLR-3989 add RLS to postgresql db and base models

* feat(API): PRWLR-3989 add TenantMiddleware

* chore(API, Backend): PRWLR-3989 create new db user without RLS bypass on migrations

* chore(Backend): PRWLR-3989 fix RLS bypasser for POST requests

* fix(Backend): PRWLR-3989 fix user permissions when migrating new models

* chore(Backend): PRWLR-3989 add testing view for RLS manual tests

* feat(API): PRWLR-3989 add tenant_id to API logging

* chore(API, Backend): PRWLR-3989 add TODOs

* test(API): PRWLR-3989 add new middleware unit tests

* chore(API): PRWLR-3989 refactor RLS code

* fix(tests): PRWLR-3989 fix testing db connector

* chore: PRWLR-3989 add references to JIRA tickets

* fix: PRWLR-3989 remove bypass logic and fix serializers

* fix: PRWLR-3989 improve drop SQL query for RLS models

* feat(Backend): PRWLR-3989 add specific permissions on each model

* fix(Backend): PRWLR-3989 fix database routing and grant select perms

* fix(test): PRWLR-3989 fix routing issues with unit tests

* chore: PRWLR-3989 remove references to JIRA tickets

* feat(Backend): PRWLR-3999 add Provider model

* feat: PRWLR-3999 add providers view logic

* fix: PRWLR-3999 fix unique index fields

* feat(API): PRWLR-3999 add custom exception handlers

* feat(API): PRWLR-3999 add /providers/{provider_id}/connection endpoint

* test(API): PRWLR-3999 add base unit tests for providers

* fix(API, Backend): PRWLR-3999 fix bugs after rebase

* chore(API, docker): PRWLR-3999 add dev feats for demo

* feat(API): PRWLR-3999 add drf-spectacular-jsonapi and improve docs

* test(API): PRWLR-3999 add providers unit tests

* chore(Backend): PRWLR-3989 adjust privileges for user on table tenant

* chore: PRWLR-3999 rename model custom validation error

* chore: PRWLR-3999 remove Test references

* chore: PRWLR-3999 update API v1 spec

* fix: PRWLR-3999 apply requested changes to filter and models

* feat: PRWLR-3999 add validation to PATCH /providers payload

* fix: PRWLR-3999 fix providers enum description

* chore: add more providers fixtures

* fix: PRWLR-3999 make providers.alias optional
2024-08-06 12:39:08 -04:00
Víctor Fernández Poyatos e1a12bcb14 feat/PRWLR-3989 Add RLS full implementation (#18)
* feat(Backend): PRWLR-3989 add RLS to postgresql db and base models

* feat(API): PRWLR-3989 add TenantMiddleware

* chore(API, Backend): PRWLR-3989 create new db user without RLS bypass on migrations

* chore(Backend): PRWLR-3989 fix RLS bypasser for POST requests

* fix(Backend): PRWLR-3989 fix user permissions when migrating new models

* chore(Backend): PRWLR-3989 add testing view for RLS manual tests

* feat(API): PRWLR-3989 add tenant_id to API logging

* chore(API, Backend): PRWLR-3989 add TODOs

* test(API): PRWLR-3989 add new middleware unit tests

* chore(API): PRWLR-3989 refactor RLS code

* fix(tests): PRWLR-3989 fix testing db connector

* chore: PRWLR-3989 add references to JIRA tickets

* fix: PRWLR-3989 remove bypass logic and fix serializers

* fix: PRWLR-3989 improve drop SQL query for RLS models

* feat(Backend): PRWLR-3989 add specific permissions on each model

* fix(Backend): PRWLR-3989 fix database routing and grant select perms

* fix(test): PRWLR-3989 fix routing issues with unit tests

* chore: PRWLR-3989 remove references to JIRA tickets

* chore(Backend): PRWLR-3989 adjust privileges for user on table tenant

* chore: PRWLR-3989 add comments on migrations
2024-08-02 11:29:05 -04:00
Víctor Fernández Poyatos 7cb3a4e16e feat/PRWLR-3996 implement /tenants endpoint and basic filters logic (#14)
* chore: PRWLR-3996 add drf-jsonapi library to project

* feat: PRWLR-3996 implement /tenants endpoint and basic JSON:API spec

* feat: PRWLR-3996 implement new middleware to ingest metadata on API responses

* chore: PRWLR-3996 annotate views with drf-spectacular

* feat: PRWLR-3996 add cache-control to tenants endpoints

* feat: PRWLR-3996 update views filters

* chore: PRWLR-3996 remove meta middleware in favour of meta fields in serializers

* test(API): PRWLR-3996 add unit tests for /tenants

* chore: PRWLR-3996 turn DEBUG on on devel environment

* test: PRWLR-3996 rework tenants unit tests

* chore: PRWLR-3996 refactor filter classes

* fix(Backend,API): PRWLR-3996 use correct filter backend
2024-07-22 16:06:57 -04:00
Víctor Fernández Poyatos 3a502c5b3d feat(Backend): PRWLR-3990 Add custom logging (#6)
* chore: PRWLR-4022 add django-cors-headers

* chore: PRWLR-4022 configure basic CORS

* feat: PRWLR-4022 add drf-spectacular and spec/docs endpoints

* build: PRWLR-4022 add basic docker development deployment

* chore: PRWLR-4022 update schema generation

* chore: PRWLR-4022 bump required Python version to 3.12

* feat: PRWLR-3990 add basic custom Django logging

* feat: PRWLR-3990 add tasks logger

* chore: PRWLR-3990 update API logging middleware to follow updated syntax

* docs: PRWLR-3990 add docstring for APILoggingMiddleware class

* chore: PRWLR-3990 add testing dependencies to poetry

* test(API): PRWLR-3990 add new middleware unit tests and base setup

* feat(API, Backend): PRWLR-3990 add NDJSONFormatter and transaction_id

* feat(API): PRWLR-3990 add dev/test model, views and urls

* chore(backend): PRWLR-3990 refactor logging settings and modules

* chore(backend): PRWLR-3990 add BackendLogger enum and refactor logging modules

* chore(backend): PRWLR-3990 refactor module importing

* style(backend): PRWLR-3990 remove extra endlines

* chore: PRWLR-3990 update poetry.lock

* chore: PRWLR-3990 run pre-commit
2024-07-05 10:13:04 -04:00