Merge pull request #1 from Paulito123/main

Version 1.0.0

README.md

@@ -1,2 +1,70 @@
-# log-indexer
-Code to index log entries from n number of 0L validator log files based upon a series of time ranges.
+# 0L-log-indexer
+Toolset to index log entries from n number of 0L validator log files based upon a series of time ranges.
+
+## Prerequisites
+1. Have docker installed properly.
+2. Have the requirements installed:
+```bash
+python3 -m venv ./venv
+source ./venv/bin/activate
+pip install -r requirements.txt
+```
+## Spin up a postgres db
+First pull the latest postgres image:
+```bash
+docker pull postgres
+```
+Run the image:
+```bash
+docker run\
+    --name researchDB\
+    -p 5432:5432\
+    -e POSTGRES_USER=research\
+    -e POSTGRES_PASSWORD=research\
+    -e POSTGRES_DB=research\
+    -d postgres
+```
+## Start indexing log files
+1. Check if the db is receiving connections:
+```bash
+docker logs researchDB
+```
+2. Define the desired configurations in config.py
+3. Run app.py
+## Some docker commands that might come in handy...
+Get into the the container shell:
+```bash
+docker exec -it researchDB /bin/bash
+```
+Enter the db container to make queries directly on db:
+```bash
+docker exec -it ol-intel-db /bin/bash
+/ # su postgres
+/ $ psql postgresql://research:research@localhost:5432/research
+...
+research=# select _json from validatorlog where _json->>'name' = 'progress_check';
+...
+research=# exit
+/ $ exit
+/ # exit
+```
+Take down the container (and lose all data in it):
+```bash
+docker rm -f researchDB
+```
+## Some other commands that might come in handy:
+Create a db dump:
+```bash
+docker exec -t researchDB pg_dumpall -c -U research | gzip > ./assets/db_dumps/dump_all.gz
+```
+Restore a db dump:
+```bash
+gunzip < ./assets/db_dumps/dump_all.gz | docker exec -i researchDB psql -U research -d research
+```
+In postgres, one can access json elements inside a JSONB field by using the -> and/or --> operator:
+![JSONB example](assets/images/_json.png)
+## TODOs
+1. Optimize recurring code
+2. Make test async, if possible
+3. Remove code blocks never visited, if any
+4. Add date regex generator

app.py

@@ -0,0 +1,33 @@
+#!/usr/bin/python3
+
+import os
+
+from src.regexgen import generate_regex_for_ts, get_matches_from_file
+from src.config import ProdConfig
+from src.models import ValidatorLog
+
+
+def crawl_log_files() -> None:
+    directory = ProdConfig.PATH_VALIDATOR_LOG_FILES
+    print(f"Crawling directory: {directory}")
+
+    for r in ProdConfig.TIME_GAP_LIST:
+        regex = generate_regex_for_ts(r[0], r[1])
+        print(f"REGEX for range {r[0]} > {r[1]}:")
+        print(f"{regex}")
+        print(f"-"*50)
+
+        for filename in os.listdir(directory):
+            f = os.path.join(directory, filename)
+            print(f"Crawling file {filename}")
+
+            if os.path.isfile(f):
+                matches = get_matches_from_file(regex, f)
+                ValidatorLog.load_list_matches(filename, matches)
+                print(f"{len(matches)} matches found!")
+
+    print("Done!")
+
+
+if __name__ == "__main__":
+    crawl_log_files()

app_test.py

@@ -0,0 +1,7 @@
+#!/usr/bin/python3
+
+from src.test_full_range import test_ts_time_range
+
+
+if __name__ == "__main__":
+    test_ts_time_range("2022-01-01T00:00:00", "2022-01-01T21:00:01")