v3.0.0

Release v3.0.0

.gitignore

@@ -116,3 +116,8 @@ ___*
 *.rdb
 *.json
 .DS_Store
+
+
+# local docker settings and config
+.docker/*
+!.docker/.gitkeep

Pipfile

@@ -9,7 +9,7 @@ pip = "*"
 celery = "*"
 requests = "*"
 redis = "*"
-django = "==5.0.4"
+django = "==5.0.7"
 pymysql = "*"
 django-registration = "*"
 gunicorn = "*"
@@ -18,4 +18,4 @@ gunicorn = "*"
 "flake8" = "*"
 
 [requires]
-python_version = "3.12.2"
+python_version = "3.12.4"

README.md

@@ -1,19 +1,58 @@
 # impresso-user-admin
 
 A basic django application to manage user-related information contained in [Impresso's Master DB](https://github.com/impresso/impresso-master-db).
-We use `pipenv`for development and `docker` for production. Please look at the relevant sections in the documentation.
+We use `pipenv` for development together with `docker`. Please look at the relevant sections in the documentation.
 
-To start _django admin_ in development with pipenv:
+## Development
 
-    ENV=dev pipenv run ./manage.py runserver
+Take the time to explore the `.example.env` file and the related `./impresso/settings.py` to understand the settings that can be configured via environment variables for your specific environment. We have configured `dotenv` in `./impresso/base.py` to allow the loading of different `.env` files. For example, you can use `.env` or `.dev.env` for development, and `.prod.env` to test production settings.
 
-or to test tags:
+```sh
+# our .dev.env file, that connects to the local redis instance
+REDIS_HOST=localhost:6379
+IMPRESSO_DB_HOST=localhost
+IMPRESSO_DB_PORT=3306
+# Then don't forget to fill all SOLR related settings accordiung to your impresso configuration
+IMPRESSO_SOLR_URL=http://localhost:8983/solr/impresso
+IMPRESSO_SOLR_USER=your-user-reader-only
+IMPRESSO_SOLR_PASSWORD=our-user-reader-only-password
+IMPRESSO_SOLR_USER_WRITE=your-user-write-allowed
+IMPRESSO_SOLR_PASSWORD_WRITE=your-user-write-allowed-password
+IMPRESSO_SOLR_PASSAGES_URL=http://localhost:8983/solr/impresso-tr-passages
+```
+
+To start the Django admin, you need to have Redis and MySQL running. You can start them by running the command `docker compose up`. Please note that in our YAML file, the ports for Redis and MySQL are exposed to facilitate local development and testing.
+
+```sh
+docker compose up -d --env-file=.dev.env
+```
+
+Then you can start the development server, e.g. with pipenv and the `dev.env` file:
+
+```sh
+ENV=dev pipenv run ./manage.py runserver
+```
+
+or with Makefile:
+
+```sh
+ENV=dev make run-dev
+```
+
+To start _celery_ task manager in development with pipenv, in a new terminal:
 
-    ENV=dev make run-dev
+```sh
+ENV=dev pipenv run celery -A impresso worker -l info
+```
 
-To start _celery_ task manager in development with pipenv:
+Of course, you can also use a generic `.env file` on development, in this case you don't need to specify the `ENV` variable:
 
-    ENV=dev pipenv run celery -A impresso worker -l info
+```sh
+docker compose up -d
+pipenv run ./manage.py runserver
+# and in another terminal, to start the celery worker
+pipenv run celery -A impresso worker -l info
+```
 
 ### setup with pyenv + pipenv
 
@@ -30,7 +69,7 @@ The last command gives you the version of the local python. If it doesn't meet t
 use pyenv install command:
 
 ```
-pyenv install 3.6.9
+pyenv install 3.12.4
 ```
 
 Use pip to install Pipenv:
@@ -91,12 +130,30 @@ Export query as csv using (first argument being `user_id` then the solr query):
 ENV=dev ./manage.py exportqueryascsv 1 "content_txt_fr:\"premier ministre portugais\""
 ```
 
+Create (or get) a collection:
+
+```sh
+ENV=dev pipenv run ./manage.py createcollection "name of the collection" my-username
+```
+
+Then once you get the collection id, usually a concatenation of the creator profile uid and of the slugified version of the desired name, you can add query results to the collection:
+
+```sh
+ENV=dev pipenv run python ./manage.py addtocollectionfromquery local-user_name-of-the-collection "content_txt_fr:\"premier ministre portugais\""
+```
+
 Index a collection from a list of tr-passages ids resulting from a solr query:
 
 ```sh
 ENV=dev pipenv run python ./manage.py addtocollectionfromtrpassagesquery local-dg-abcde "cluster_id_s:tr-nobp-all-v01-c8590083914"
 ```
 
+Stop a specific job from command line:
+
+```sh
+ENV=dev pipenv run python ./manage.py stopjob 1234
+```
+
 ## Use in production
 
 Please check the included Dockerfile to generate your own docker image or use the docker image available on impresso dockerhub.

docker-compose.yml

@@ -0,0 +1,29 @@
+services:
+  redis:
+    image: redis:alpine
+    restart: always
+    volumes:
+      - ./.docker/redis:/data
+    entrypoint: redis-server --appendonly yes
+    ports:
+      - 6379:6379
+  # mysql:
+  #   image: mariadb:lts
+  #   restart: always
+  #   volumes:
+  #     - ./.docker/mysql:/var/lib/mysql
+  #   ports:
+  #     - ${DB_PORT:-3308}:3306
+  #   environment:
+  #     MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD:-root}
+  #     MARIADB_DATABASE: ${IMPRESSO_DB_NAME}
+  #     MARIADB_USER: ${IMPRESSO_DB_USER}
+  #     MARIADB_PASSWORD: ${IMPRESSO_DB_PASSWORD}
+  mysql-tunnel:
+    image: kroniak/ssh-client
+    restart: always
+    volumes:
+      - ./.docker/config/ssh:/root/.ssh
+    ports:
+      - ${IMPRESSO_DB_PORT:-3306}:3306
+    command: ssh -N impresso-mysql-tunnel