pickle -> onnx, IsolationForest -> AADForest

[Knispel2]

Changes:

• Model format changed from .pickle to .onnx
• Used model changed from sklearn.IsolationForest to AADForest

Currently having trouble running tests on my computer. I created PR to see how the code behaves on workflow tests (my python-worker crashes when trying to test it).
To be continued...

• Identify all problems
• ID -> lc_features
• Slack channel access?

[JulienPeloton]

Thanks -- I enable tests so that you can test here :-)

[JulienPeloton]

You need to install the onnx package though:

Traceback (most recent call last):
  File "fink_science/anomaly_detection/processor.py", line 25, in <module>
    from onnx import load
ModuleNotFoundError: No module named 'onnx'

Just edit the .github/workflows/run_test.yml, and add:

 name: Run test suites
      run: |
        pip uninstall -y actsnfink
        pip install git+https://github.com/emilleishida/fink_sn_activelearning.git@bf8d4e263e02d42781642f872f7bc030c24792bc#egg=actsnfink
+        pip install <whatever you need>
        ./run_tests.sh
        curl -s https://codecov.io/bash | bash

[Knispel2]

pip install

Thanks!

[Knispel2]

Didn't notice that the new archive didn't load...

[Knispel2]

I don't know what I broke :(

2023-10-31 14:18:43.407634: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory

Do I have to upload this library to the repository?

[JulienPeloton]

Hum, usually it is not required (other modules use tensorflow without the need). I am inspecting.

[JulienPeloton]

Actually, this is just a warning, meaning there is no GPU support here (make sense). Look at the line after:

2023-10-31 14:18:43.407670: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.

The job failed because another thing:

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

Try to downgrade protobuf:

pip install protobuf==3.20.3

to see if that solves the problem.

[JulienPeloton]

Ah wait -- the problem is not from your code... So this means the protobuf version in the Docker image is too recent (images change recently -- but apparently I did not catch that). I will correct this later.

[JulienPeloton]

OK, we have a problem. onnx 1.15.0 requires protobuf>=3.20.2, but we are using tensorflow==2.9.2 which requires protobuf<3.20... I am checking if one can upgrade tensorflow and release the constraint on protobuf.

[JulienPeloton]

I will check if I can upgrade tensorflow, but in the meantime, could you check if installing onnx==1.12.0 would work for you?

[Knispel2]

I will check if I can upgrade tensorflow, but in the meantime, could you check if installing onnx==1.12.0 would work for you?

I don't think there should be a problem (seems like the model should work with 1.12.0). I specified the version in the run_test.yml file

[JulienPeloton]

Thanks @Knispel2 -- the CI is all green :-)

One question: are you still using anomaly_detection_forest.zip? Or this can be removed from the repository?

[Knispel2]

Thanks for your help!

One question: are you still using anomaly_detection_forest.zip? Or this can be removed from the repository?

We want to keep the option to use the old model. It may be useful in the future. For this I added the model_type parameter.

AADForest allows us to use active learning. We want to use reactions from experts under notification messages in Telegram and Slack channels as a signal. If an expert puts a thumbs up, it is indeed an anomaly, and if it puts a thumbs down, it is not an anomaly. We plan to periodically update the model separately using a module located here: https://github.com/Knispel2/fink_anomaly_detection_model, and then just update the .zip archive with the model in the fink-science repository.

This raises two questions for me:

• I need to restore the lc_features by objectID. Is it possible to add lc_features column to https://fink-portal.org/api/v1/anomaly?
• To collect reactions from a channel with anomalies in Slack my Slack App will need to access this channel: would it be possible to add this?

[JulienPeloton]

I need to restore the lc_features by objectID. Is it possible to add lc_features column to https://fink-portal.org/api/v1/anomaly?

Yes definitely possible! I will handle this.

To collect reactions from a channel with anomalies in Slack my Slack App will need to access this channel: would it be possible to add this?

Sure -- just let me know what is your app.

[Knispel2]

Sure -- just let me know what is your app.

The app is called reactions_reader. I added it to the workspace and gave you access to this app

[codecov-commenter]

Codecov Report

Attention: 21 lines in your changes are missing coverage. Please review.

Comparison is base (f69329f) 53.40% compared to head (07e9b91) 53.58%.

❗ Current head 07e9b91 differs from pull request most recent head a1e6f57. Consider uploading reports for the commit a1e6f57 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #356      +/-   ##
==========================================
+ Coverage   53.40%   53.58%   +0.17%     
==========================================
  Files          33       33              
  Lines        1511     1536      +25     
==========================================
+ Hits          807      823      +16     
- Misses        704      713       +9     

Files	Coverage Δ
fink_science/anomaly_detection/processor.py	73.07% <58.82%> (-4.29%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[JulienPeloton]

Thanks -- I'm merging and this will appear in the next release