* chore(partitions): add env to create partitions
* chore(partitions): rename env to DJANGO_MANAGE_DB_PARTITIONS
* chore(partitions): use True|False as value
* fix: begin comment with uppercase
* feat(User): PRWLR-4988 make User.email case insensitive
* test(User): PRWLR-4988 update unit tests
* feat(User): PRWLR-4988 include email validation in serializer
* feat(Finding): PRWLR-4987 add uid field to Findings
* feat(Scan): PRWLR-4987 implement logic to calculate Finding delta value
* test(Scan): PRWLR-4987 add unit tests for _create_finding_delta
* chore(Spec): PRWLR-4987 update API spec
* chore(Django-filter): PRWLR-4948 bump django-filter version to 24.3
* fix(Middleware): PRWLR-4948 fix logging middleware when auth is not present in request
* fix(Task): PRWLR-4948 fix serializer state values
* chore(Filters): PRWLR-4948 use ChoiceFilter instead of implementing method for basic filters
* chore(Filters): PRWLR-4948 refactor complex filters to reduce complexity
* feat(Users): PRWLR-4718 make user email the default login username
* feat(Token): PRWLR-4718 add serializers, views and urls for access and refresh tokens
* feat(Token): PRWLR-4718 add first membership tenant in token if not present in json body
* feat(Users): PRWLR-4718 add company_name to model
* feat(Users): PRWLR-4718 create tenant and membership when creating new user
* fix(BaseView): PRWLR-4718 add tenant_id to serializer context
* fix(Tests): PRWLR-4718 use authorization with unit tests
* fix(Views): PRWLR-4718 fix tenant retrieval from request
* fix(Tests): PRWLR-4718 fix tests
* fix(Fixtures): PRWLR-4718 fix tenant memberships ordering
* chore(Tokens): PRWLR-4718 update token url
* chore(Spec): PRWLR-4718 update API spec
* feat(Tokens): PRWLR-4718 enable token refresh blacklisting
* feat(Tokens): PRWLR-4718 implement RS256 algorithm and dev valid keys
* chore(env): PRWLR-4718 update .env.example
* chore(Deps): PRWLR-4015 update prowler dep
* fix(Resources, Findings): PRWLR-4015 fix permission issues on models and migrations
* feat(Scans, Tasks, Resources, Findings): PRWLR-4015 perform whole flow when executing scans
* fix(Settings): PRWLR-4015 fix devel JWT settings
* chore(Scans, Tasks): PRWLR-4015 improve docs and responses format
* test(Scan, Provider): PRWLR-4015 fix unit tests
* chore(Environment): PRWLR-4015 fix .env.example values
* test(Scan): PRWLR-4015 add unit tests for scan task
* chore(Task): PRWLR-4015 give proper format to task result on scan perform
* feat(Provider, Scan): PRWLR-4015 add all provider types to connection check and scan
* fix(Logging): PRWLR-4015 fix API logger middleware to include tenant and user IDs
* feat(Findings): initial findings model
* fix(Findings): add view, serializers, migration for enums
* fix(Findings): incomplete jsonb_to_tsvector wrapper
will not run as written
* fix(Findings): use Severity and Status enums from prowler SDK
* tests(Findings): add failing view tests
* fix(Finding): add resource relationship
not returning correct data from serializer, missing links
* fix(FindingSerializer): get Scan & Resource relationships to show up
* fix(FindingFilter): add more filter fields
* fix(FindingFilter): filter on provider id
* fix(FindingSerializer): return Resource in relationship
not ResourceFindingMapping
* fix(FindingModel): update migration
* fix(FindingFilter): full text search on findings
* fix(Resources): include Findings in ResourceSerializer
* fix(FindingFilter): expand text search columns
* fix(DbUtils): docstring, not comment
* fix(BaseViews): remove TODO
comment not applicable right now
* fix(Fixtures): add more findings to fixture file
and change on_delete behavior for resource_finding_mapping
* fix(Resources): rename index to match others
* fix(Findings): update Findigns RLS to allow for full CRUD
eventually we'll let users enter a manual finding
which implies INSERT, UPDATE, DELETE
* fix(Findings): use TextChoices directly for Status enum
* fix(FindingSerializer): build a set instead of a list
* consistency in fixtures
Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
* fix(API): update v1 spec for findings
* feat(Findings): store findings in a partitioned table
* fix(Settings): change unit of partition max age
to match unit when creating
* docs(Partitions): document how to manage partitions via manage.py
* fix(Findings): add tag for spec/docs
* fix(Findings): clean up migrations
* fix(Findings): convert scan_id and inserted_at filters into finding.id filters
* fix(Findings): add default filter for today
and set default sort order
* fix(Findings): add default filter for today
and set default sort order
* fix(Findings): update fixtures so datetime matches id
* fix(Findings): partition the ResourceFindingMapping table to match Findings
* docs(Partitions): document postgres config values more
* docs(UUIDUtils): do not use raw query strigns (typo)
* docs(Partitions): change unit in comment description
* fix(Findings): change resource_name & tags to be Finding
* docs(Partitions): change unit in partitions settings docstring
* fix(Findings): remove conflicting logic & filters
* chore: apply suggested changes
* chore: optimize imports
---------
Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
* feat(Findings): initial findings model
* fix(Findings): add view, serializers, migration for enums
* fix(Findings): incomplete jsonb_to_tsvector wrapper
will not run as written
* fix(Findings): use Severity and Status enums from prowler SDK
* tests(Findings): add failing view tests
* fix(Finding): add resource relationship
not returning correct data from serializer, missing links
* fix(FindingSerializer): get Scan & Resource relationships to show up
* fix(FindingFilter): add more filter fields
* fix(FindingFilter): filter on provider id
* fix(FindingSerializer): return Resource in relationship
not ResourceFindingMapping
* fix(FindingModel): update migration
* fix(FindingFilter): full text search on findings
* fix(Resources): include Findings in ResourceSerializer
* fix(FindingFilter): expand text search columns
* fix(DbUtils): docstring, not comment
* fix(BaseViews): remove TODO
comment not applicable right now
* fix(Fixtures): add more findings to fixture file
and change on_delete behavior for resource_finding_mapping
* fix(Resources): rename index to match others
* fix(Findings): update Findigns RLS to allow for full CRUD
eventually we'll let users enter a manual finding
which implies INSERT, UPDATE, DELETE
* fix(Findings): use TextChoices directly for Status enum
* fix(FindingSerializer): build a set instead of a list
* consistency in fixtures
Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
* fix(API): update v1 spec for findings
---------
Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
* fix(Resources): add basic Resource MVC
* fix(Resources): implement many-to-many tags via through table
* fix(Resources): tsvector & migration
migration needs work, need to create a custom
django migration field to build the GENERATED ALWAYS
column.
Without it, django will complain about the ts column
not exisiting when we try to create an index.
* test(Resources): add tests for /resources views
* fix(Resources): get basic test cases to pass
need to work on tsvector column
and filtering by provider columns like alias, name
* fix(Resources): provide way to get tags as dict
not a list of ResourceTag models
* fix(Resources): annotate with return type
* fix(Resources): rename search field to not conflict with filters
* fix(Resources): filter by provider.id
* fix(Resources): remove filter and sort by provider
must use provider_id first
* fix(Resources): test adding, clearing tags
* fix(Scans): filter by provider_id
* fix(Resources): remove FIXME comments
* fix(Resources): filter and search on tags
* fix(Resources): full text search
* fix(Resources): full text search to include both resources and tags
and get rid of the distinct() query
* fix(Resources): document text search a bit more
* fix(Resources): remove TODO comments
* fix(Resources): move Resource migrations into 0001_initital
* fix(Config): revert to match main
* fix(Resources): use Django's default query builder for full text search
* Apply suggestions from code review
Mostly formatting changes
Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
* fix(Resources): just use text search
instead of guessing at allowed split characters for each provider
* fix(Resources): remove unused tests
these cases are covered in parameterized lists
* fix(Resources): set a read-only RLS policy
* fix(Resources): rename add_tags to be upsert_or_delete_tags
* fix(Resources): include primary key on through table
django query builder doesn't like tables without primary keys
* fix(Resources): default xx_at filters to date
instead of date time
* docs(Resources): describe what a Resource is
in API docs
* chore(Resources): rename file to match others
---------
Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
* chore(API): PRWLR-4775 change inserted and updated at filters to filter by date by default
* test(API): PRWLR-4775 replace datetime strings for date
* feat(API): PRWLR-4775 update started_at default behavior and unit tests
* build(Dockerfile): PRWLR-4781 remove versions from dockerfile deps
* fix(Docker): PRWLR-4781 fix celery worker container and apply dev fixtures by default
* build(docker-compose): PRWLR-4781 set default django settings to production in worker service
* chore(deps): PRWLR-4014 include prowler library in python deps
* feat(Backend,API): PRWLR-4014 add AWS provider test_connection through celery tasks
* fix(Backend,API): PRWLR-4014 fix model handling in celery tasks
* test(Tasks): PRWLR-4014 add unit tests for celery tasks
* docs(API): PRWLR-4014 update endpoint tag
* feat(Backend): PRWLR-4014 add decorator for tenant dependant Celery tasks
* chore(Backend): PRWLR-4014 remove TODOs and improve docstrings
* feat(Backend): PRWLR-4014 replace timezone.now for datetime.now(timezone.utc)
* feat(Backend): PRWLR-4014 use SET LOCAL for api.tenant_id setting
* feat(Backend, Tasks): PRWLR-4014 refactor tasks module to abstract business logic
* fix(Backend): PRWLR-4014 use set_config for RLS config and set transaction at request dispatch level
* fix(Tasks): PRWLR-4014 fix tasks tenant decorator
* feat(API): PRWLR-4368 remove TenantMiddleware in favour of transaction based setup
* feat(API): PRWLR-4368 override initial request method to perform atomic transactions on RLS viewsets
* chore(Celery): add basic Celery worker with broker only
* chore(Celery): saving progress. Not able to schedule tasks
* fix(Celery): add celery app for use by django
* fix(Celery): register tasks
* fix(Docker): add celery workers to docker-compose
* chore(Celery): add django-celery-results backend
to store results using Django ORM
* fix(Celery): get app config the correct way
* fix(Docker): start connecting docker Celery workers to Valkey
not yet operational
* fix(Celery): get celery & django to work in docker-compose
* docs(Celery): document how to run Celery in development environment
includes changes to support the configuration and deployment
of Celery worker and its dependencies, Valkey and Postgres.
* fix(GHA): add valkey to CI services
* fix(GHA): add valkey to CI services
* fix(GHA): add valkey-cli ping to CI services
* fix(GHA): use right port for valkey
* fix(Views): remove debug task code
* test(Celery): start adding celery task tests
not yet working!
* fix(pyproject): rollback django upgrade
* fix(docker): updated docker runtime and env vars
based on feedback from #20
* fix(Dockerfile): include dependencies for psutil
psutil was introduced by pytest-celery
* fix(Backend): PRWLR-4013 fix celery settings structure
* fix(Celery): update celery app to work with new settings structure
* fix(Views): remove debug task code
* fix(Config): remove debug code
* fix(Celery): update celery app name when running worker
---------
Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>
* feat(Backend): PRWLR-3989 add RLS to postgresql db and base models
* feat(API): PRWLR-3989 add TenantMiddleware
* chore(API, Backend): PRWLR-3989 create new db user without RLS bypass on migrations
* chore(Backend): PRWLR-3989 fix RLS bypasser for POST requests
* fix(Backend): PRWLR-3989 fix user permissions when migrating new models
* chore(Backend): PRWLR-3989 add testing view for RLS manual tests
* feat(API): PRWLR-3989 add tenant_id to API logging
* chore(API, Backend): PRWLR-3989 add TODOs
* test(API): PRWLR-3989 add new middleware unit tests
* chore(API): PRWLR-3989 refactor RLS code
* fix(tests): PRWLR-3989 fix testing db connector
* chore: PRWLR-3989 add references to JIRA tickets
* fix: PRWLR-3989 remove bypass logic and fix serializers
* fix: PRWLR-3989 improve drop SQL query for RLS models
* feat(Backend): PRWLR-3989 add specific permissions on each model
* fix(Backend): PRWLR-3989 fix database routing and grant select perms
* fix(test): PRWLR-3989 fix routing issues with unit tests
* chore: PRWLR-3989 remove references to JIRA tickets
* chore(Backend): PRWLR-3989 adjust privileges for user on table tenant
* chore: PRWLR-3989 add comments on migrations
Pepe Fagoaga