0L-log-indexer

Toolset to index log entries from n number of 0L validator log files based upon a series of time ranges.

Prerequisites

1. Have docker installed properly.
2. Have the requirements installed:

python3 -m venv ./venv
source ./venv/bin/activate
pip install -r requirements.txt

Spin up a postgres db

First pull the latest postgres image:

docker pull postgres

Run the image:

docker run\
    --name researchDB\
    -p 5432:5432\
    -e POSTGRES_USER=research\
    -e POSTGRES_PASSWORD=research\
    -e POSTGRES_DB=research\
    -d postgres

Start indexing log files

1. Check if the db is receiving connections:

docker logs researchDB

2. Define the desired configurations in config.py
3. Run app.py

Some docker commands that might come in handy...

Get into the the container shell:

docker exec -it researchDB /bin/bash

Enter the db container to make queries directly on db:

docker exec -it ol-intel-db /bin/bash
/ # su postgres
/ $ psql postgresql://research:research@localhost:5432/research
...
research=# select _json from validatorlog where _json->>'name' = 'progress_check';
...
research=# exit
/ $ exit
/ # exit

Take down the container (and lose all data in it):

docker rm -f researchDB

Some other commands that might come in handy:

Create a db dump:

docker exec -t researchDB pg_dumpall -c -U research | gzip > ./assets/db_dumps/dump_all.gz

Restore a db dump:

gunzip < ./assets/db_dumps/dump_all.gz | docker exec -i researchDB psql -U research -d research

In postgres, one can access json elements inside a JSONB field by using the -> and/or --> operator:

TODOs

1. Optimize recurring code
2. Make test async, if possible
3. Remove code blocks never visited, if any
4. Add date regex generator