Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds inactivity detection #27

Merged
merged 53 commits into from
Apr 8, 2024
Merged
Show file tree
Hide file tree
Changes from 50 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
40bc314
service exposes inactivity
Oct 23, 2023
5c52d5e
inverting logic
Oct 23, 2023
874d99f
renaming
Oct 23, 2023
d311da0
refactor
Oct 23, 2023
be8989e
replacing docker compose strings
Oct 23, 2023
5bf7f71
using new API interface
Oct 24, 2023
aaf69b9
using different random port
Oct 24, 2023
712afcd
refactor
Mar 7, 2024
0c43510
rename
Mar 7, 2024
39f7156
update kernel checker script
Mar 18, 2024
40e1503
rename
Mar 18, 2024
d4050c0
Merge branch 'main' into adds-inactivity-detection
Mar 18, 2024
6a8d74e
refactor lowe usage percentage
Mar 19, 2024
e92fe5b
bump library version
Mar 19, 2024
3ea2298
updating
Mar 19, 2024
1ef6b75
wip
Mar 19, 2024
a6d8c9b
cpu usage test
Mar 21, 2024
c16219c
refactor tests
Mar 21, 2024
bdef8b3
refactor and add disk usage
Mar 21, 2024
7869229
adding tests for disk usage
Mar 21, 2024
0ebfc44
refactor tests
Mar 21, 2024
7ae3faa
added new tests
Mar 21, 2024
b85c80c
refactor tests
Mar 21, 2024
6cd4ccf
final version of tests
Mar 21, 2024
6a5ea59
fixed tornado shutdown
Mar 21, 2024
7235d5c
fixed tests
Mar 21, 2024
162c972
added ci scripts
Mar 21, 2024
5e578de
fix tests
Mar 21, 2024
cbafd57
refactor
Mar 21, 2024
a648f43
fix test
Mar 21, 2024
c616915
refactor
Mar 21, 2024
9c34107
add coverage and fail rate
Mar 22, 2024
b4dfd74
fix test script
Mar 22, 2024
b8ee98b
removed failing test
Mar 22, 2024
4fea5b1
update gitignore
Mar 22, 2024
0158da6
refactor
Mar 22, 2024
ee9e253
refactor CPUUsage to be none blocking
Mar 22, 2024
9045dfa
refactor tests
Mar 25, 2024
53544b3
refactor with non blocking pattern
Mar 25, 2024
e7b8284
added network_manager and tests
Mar 25, 2024
8994bf8
fix issues
Mar 25, 2024
699f427
updated thresholds
Mar 25, 2024
69d1998
refactor limits
Mar 25, 2024
f113c79
refactor to use threading
Mar 25, 2024
04c8de7
replaced tornado with builtin server
Mar 25, 2024
bec3433
removing the need to specify the interpreter
Mar 25, 2024
9ba6c70
update to use service-activity-monitor
Mar 26, 2024
9e55138
remove unused
Apr 2, 2024
c188c2a
added todo
Apr 2, 2024
f67ba54
upgrade build tooling
Apr 5, 2024
72cb231
math reacts to inactivity
Apr 8, 2024
350fa7d
remove unused
Apr 8, 2024
c021506
bump service version as major
Apr 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .coverage
Binary file not shown.
8 changes: 4 additions & 4 deletions .github/workflows/check-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ jobs:
- name: Checkout repo content
uses: actions/checkout@v2
- name: ooil version
uses: docker://itisfoundation/ci-service-integration-library:v1.0.3-dev-4
uses: docker://itisfoundation/ci-service-integration-library:v1.0.4
with:
args: ooil --version
- name: Assemble docker-compose spec
uses: docker://itisfoundation/ci-service-integration-library:v1.0.3-dev-4
- name: Assemble docker compose spec
uses: docker://itisfoundation/ci-service-integration-library:v1.0.4
with:
args: ooil compose
- name: Build all images if multiple
uses: docker://itisfoundation/ci-service-integration-library:v1.0.3-dev-4
uses: docker://itisfoundation/ci-service-integration-library:v1.0.4
with:
args: docker compose build
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -53,4 +53,4 @@ docker-compose.yml
# ignore all files in validation
validation/

*.ignore.*
*.ignore.*
5 changes: 5 additions & 0 deletions .osparc/jupyter-math/runtime.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,8 @@ paths-mapping:
outputs_path: /home/jovyan/work/outputs
state_paths:
- /home/jovyan/work/workspace
callbacks-mapping:
inactivity:
service: container
command: ["python", "/usr/local/bin/service-monitor/activity.py"]
timeout: 1
27 changes: 25 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ RUN jupyter serverextension enable voila && \
# Import matplotlib the first time to build the font cache.
ENV XDG_CACHE_HOME /home/$NB_USER/.cache/
RUN MPLBACKEND=Agg .venv/bin/python -c "import matplotlib.pyplot" && \
fix-permissions /home/$NB_USER
# run fix permissions only once. This can be probably optimized, so it is faster to build
fix-permissions /home/$NB_USER
# run fix permissions only once. This can be probably optimized, so it is faster to build

# copy README and CHANGELOG
COPY --chown=$NB_UID:$NB_GID CHANGELOG.md ${NOTEBOOK_BASE_DIR}/CHANGELOG.md
Expand All @@ -92,6 +92,29 @@ ENV JP_LSP_VIRTUAL_DIR="/home/${NB_USER}/.virtual_documents"
# Copying boot scripts
COPY --chown=$NB_UID:$NB_GID docker /docker

# install service activity monitor
ARG ACTIVITY_MONITOR_VERSION=v0.0.1

# Detection thresholds for application
ENV ACTIVITY_MONITOR_BUSY_THRESHOLD_CPU_PERCENT=0.5
ENV ACTIVITY_MONITOR_BUSY_THRESHOLD_DISK_READ_BPS=0
ENV ACTIVITY_MONITOR_BUSY_THRESHOLD_DISK_WRITE_BPS=0
ENV ACTIVITY_MONITOR_BUSY_THRESHOLD_NETWORK_RECEIVE_BPS=1024
ENV ACTIVITY_MONITOR_BUSY_THRESHOLD_NETWORK_SENT_BPS=1024

# TODO: above values require some tewaking otherwise this will always be considered in use

# install service activity monitor
RUN apt-get update && \
apt-get install -y curl && \
# install using curl
curl -sSL https://raw.githubusercontent.com/ITISFoundation/service-activity-monitor/main/scripts/install.sh | \
bash -s -- ${ACTIVITY_MONITOR_VERSION} && \
# cleanup and remove curl
apt-get purge -y --auto-remove curl && \
rm -rf /var/lib/apt/lists/*


RUN echo 'export PATH="/home/${NB_USER}/.venv/bin:$PATH"' >> "/home/${NB_USER}/.bashrc"

EXPOSE 8888
Expand Down
5 changes: 3 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,14 @@ devenv: .venv ## create a python virtual environment with tools to dev, run and
requirements: devenv ## runs pip-tools to build requirements.txt that will be installed in the JupyterLab
# freezes requirements
pip-compile kernels/python-maths/requirements.in --resolver=backtracking --output-file kernels/python-maths/requirements.txt
pip-compile requirements/test.in --resolver=backtracking --output-file requirements/test.txt
GitHK marked this conversation as resolved.
Show resolved Hide resolved

# Builds new service version ----------------------------------------------------------------------------
define _bumpversion
# upgrades as $(subst $(1),,$@) version, commits and tags
@docker run -it --rm -v $(PWD):/${DOCKER_IMAGE_NAME} \
-u $(shell id -u):$(shell id -g) \
itisfoundation/ci-service-integration-library:v1.0.3-dev-4 \
itisfoundation/ci-service-integration-library:v1.0.4 \
sh -c "cd /${DOCKER_IMAGE_NAME} && bump2version --verbose --list --config-file $(1) $(subst $(2),,$@)"
endef

Expand All @@ -49,7 +50,7 @@ version-patch version-minor version-major: .bumpversion.cfg ## increases service
compose-spec: ## runs ooil to assemble the docker-compose.yml file
@docker run -it --rm -v $(PWD):/${DOCKER_IMAGE_NAME} \
-u $(shell id -u):$(shell id -g) \
itisfoundation/ci-service-integration-library:v1.0.3-dev-4 \
itisfoundation/ci-service-integration-library:v1.0.4 \
sh -c "cd /${DOCKER_IMAGE_NAME} && ooil compose"

build: | compose-spec ## build docker image
Expand Down
1 change: 1 addition & 0 deletions docker/entrypoint.bash
Original file line number Diff line number Diff line change
Expand Up @@ -74,4 +74,5 @@ chmod gu-w "/home/${NB_USER}/work"

echo
echo "$INFO" "Starting notebook ..."
exec gosu "$NB_USER" python /usr/local/bin/service-monitor/activity_monitor.py &
exec gosu "$NB_USER" /docker/boot_notebook.bash
2 changes: 1 addition & 1 deletion kernels/python-maths/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# pip-compile --output-file=kernels/python-maths/requirements.txt --resolver=backtracking kernels/python-maths/requirements.in
# pip-compile --output-file=kernels/python-maths/requirements.txt kernels/python-maths/requirements.in
#
aiofiles==22.1.0
# via ypy-websocket
Expand Down
Loading