Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cria cronjob com horario randomico para atualizar indices do solr #3637

Open
wants to merge 2 commits into
base: 3.1.x
Choose a base branch
from
Open

cria cronjob com horario randomico para atualizar indices do solr #3637

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
595613f
cria cronjob com horario randomico para atualizar indices do solr
Mar 24, 2023
fdb06f4
fix: mudanças no cronjob após revisao de codigo
Mar 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ ENV DEBIAN_FRONTEND noninteractive

ENV BUILD_PACKAGES apt-utils apt-file libpq-dev graphviz-dev build-essential git pkg-config \
python3-dev libxml2-dev libjpeg-dev libssl-dev libffi-dev libxslt1-dev \
libcairo2-dev software-properties-common python3-setuptools python3-pip
libcairo2-dev software-properties-common python3-setuptools python3-pip \
cron

## NAO EH PRA TIRAR O vim DA LISTA DE COMANDOS INSTALADOS!!!
ENV RUN_PACKAGES graphviz python3-lxml python3-magic postgresql-client python3-psycopg2 \
Expand Down Expand Up @@ -71,3 +72,9 @@ EXPOSE 80/tcp 443/tcp
VOLUME ["/var/interlegis/sapl/data", "/var/interlegis/sapl/media"]

CMD ["/var/interlegis/sapl/start.sh"]

COPY cronjob /etc/cron.d/update_solr_index
RUN chmod 0644 /etc/cron.d/update_solr_index
RUN crontab /etc/cron.d/update_solr_index
RUN touch /var/log/cron.log
CMD cron && tail -f /var/log/cron.log
18 changes: 18 additions & 0 deletions docker/random_cron_job.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/usr/bin/env bash

SOLR_URL=$1
# Define the interval of time to run the cronjob
RANDOM_MINUTE_MIN=0
RANDOM_MINUTE_MAX=60
RANDOM_HOUR_MIN=0
RANDOM_HOUR_MAX=3

# Generate a random minute within the interval
RANDOM_MINUTE=$((RANDOM % ($RANDOM_MINUTE_MAX-$RANDOM_MINUTE_MIN+1) + $RANDOM_MINUTE_MIN))
RANDOM_HOUR=$((RANDOM % ($RANDOM_HOUR_MAX-$RANDOM_HOUR_MIN+1) + $RANDOM_HOUR_MIN))

# Add the cronjob to the crontab
echo "$RANDOM_MINUTE $RANDOM_HOUR * * * /path/to/command" >> /etc/crontab

# Start the cron daemon
crond -f -L /dev/stdout
13 changes: 12 additions & 1 deletion docker/solr_cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import argparse
import datetime
import logging
import re
import secrets
Expand Down Expand Up @@ -277,9 +278,10 @@ def setup_embedded_zk(solr_url):
help='Replication factor (default=1)', default=1)
parser.add_argument('-ms', type=int, dest='max_shards_per_node', nargs='?',
help='Max shards per node (default=1)', default=1)

parser.add_argument("--embedded_zk", default=False, action="store_true",
help="Embedded ZooKeeper")
parser.add_argument("--rebuild_index", default=False, action="store_true",)
parser.add_argument("--update_index", default=False, action="store_true",)

try:
args = parser.parse_args()
Expand Down Expand Up @@ -315,3 +317,12 @@ def setup_embedded_zk(solr_url):
if num_docs == 0:
print("Performing a full reindex of '%s' collection..." % collection)
p = subprocess.call(["python3", "manage.py", "rebuild_index", "--noinput"])

if args.rebuild_index:
print("Rebuilding index of '%s' collection..." % collection)
p = subprocess.call(["python3", "manage.py", "rebuild_index", "--noinput"])

if args.update_index:
print("Updating index of '%s' collection..." % collection)
p = subprocess.call(["python3", "manage.py", "update_index", "--noinput"])

15 changes: 15 additions & 0 deletions docker/start.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,21 @@ if [ "${USE_SOLR-False}" == "True" ] || [ "${USE_SOLR-False}" == "true" ]; then
fi

python3 solr_cli.py -u $SOLR_URL -c $SOLR_COLLECTION -s $NUM_SHARDS -rf $RF -ms $MAX_SHARDS_PER_NODE $ZK_EMBEDDED &

RANDOM_MINUTE_MIN=0
RANDOM_MINUTE_MAX=60
RANDOM_HOUR_MIN=0
RANDOM_HOUR_MAX=3

# Generate a random minute within the interval
RANDOM_MINUTE=$((RANDOM % ($RANDOM_MINUTE_MAX-$RANDOM_MINUTE_MIN+1) + $RANDOM_MINUTE_MIN))
RANDOM_HOUR=$((RANDOM % ($RANDOM_HOUR_MAX-$RANDOM_HOUR_MIN+1) + $RANDOM_HOUR_MIN))
Comment on lines +88 to +96
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aqui eu estava pensando no seguinte esquema: ter uma hora base a noite e variar a hora e minuto a partir do qual a atualização do índice vai ocorrer todo dia. Isso pode ser resolvido conforme o script em bash abaixo:


BASE_HOUR=20

random_time() {
  RANDOM_HOUR=$((RANDOM % 4))
  RANDOM_MIN=$(printf "%02d" $((RANDOM % 60)))
  RANDOM_TIME=$((BASE_HOUR + RANDOM_HOUR))":"$RANDOM_MIN
  echo $RANDOM_TIME
}

# gera sequencia de 10 horários aleatórios para teste
for i in {1..10}; do
  echo `random_time`
done

Neste caso, a partir das 20h ele vai gerar um horário aleatório pro cronjob rodar. Pode ser criada duas funções uma pra retornar a hora e houra pra retornar o minuto.


# Add the cronjob to the crontab
echo "$RANDOM_MINUTE $RANDOM_HOUR * * * python3 solr_cli.py -u $SOLR_URL -c $SOLR_COLLECTION --update-index" >> /etc/cron.daily/update_solr_index

# Start the cron daemon
crond -f -L /dev/stdout
else
echo "Solr is offline, not possible to connect."
fi
Expand Down