chore(Celery): add basic Celery worker with broker only (#20)

* chore(Celery): add basic Celery worker with broker only

* chore(Celery): saving progress. Not able to schedule tasks

* fix(Celery): add celery app for use by django

* fix(Celery): register tasks

* fix(Docker): add celery workers to docker-compose

* chore(Celery): add django-celery-results backend

to store results using Django ORM

* fix(Celery): get app config the correct way

* fix(Docker): start connecting docker Celery workers to Valkey

not yet operational

* fix(Celery): get celery & django to work in docker-compose

* docs(Celery): document how to run Celery in development environment

includes changes to support the configuration and deployment
of Celery worker and its dependencies, Valkey and Postgres.

* fix(GHA): add valkey to CI services

* fix(GHA): add valkey to CI services

* fix(GHA): add valkey-cli ping to CI services

* fix(GHA): use right port for valkey

* fix(Views): remove debug task code

* test(Celery): start adding celery task tests

not yet working!

* fix(pyproject): rollback django upgrade

* fix(docker): updated docker runtime and env vars

based on feedback from #20

* fix(Dockerfile): include dependencies for psutil

psutil was introduced by pytest-celery

* fix(Backend): PRWLR-4013 fix celery settings structure

* fix(Celery): update celery app to work with new settings structure

* fix(Views): remove debug task code

* fix(Config): remove debug code

* fix(Celery): update celery app name when running worker

---------

Co-authored-by: Víctor Fernández Poyatos <victor@prowler.com>

.github/workflows/pull-request.yml

@@ -15,6 +15,9 @@ env:
   POSTGRES_USER: postgres
   POSTGRES_PASSWORD: postgres
   POSTGRES_DB: prowler_db_test
+  VALKEY_HOST: localhost
+  VALKEY_PORT: 6379
+  VALKEY_DB: 0
 
 
 jobs:
@@ -43,6 +46,20 @@ jobs:
           --health-interval 10s
           --health-timeout 5s
           --health-retries 5
+      valkey:
+        image: valkey/valkey:7-alpine3.19
+        env:
+          VALKEY_HOST: ${{ env.VALKEY_HOST }}
+          VALKEY_PORT: ${{ env.VALKEY_PORT }}
+          VALKEY_DB: ${{ env.VALKEY_DB }}
+        # Set health checks to wait until postgres has started
+        ports:
+          - 6379:6379
+        options: >-
+          --health-cmd "valkey-cli ping"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
 
     steps:
       - uses: actions/checkout@v4