Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

877 automated e2e tests #886

Closed
wants to merge 5 commits into from
Closed

877 automated e2e tests #886

wants to merge 5 commits into from

Conversation

fjammes
Copy link
Contributor

@fjammes fjammes commented Sep 13, 2024

IMPORTANT: Please create an issue first before opening a Pull Request.
Linked to issue(s): Closes #877

What changes were proposed in this pull request?

Tools to run self-hosted CI on a linux server and GHA integration.

How was this patch tested?

On fink server @lpc

- Add github action dispatch job
- Add line_profiler as fink-science dep
- Import science module only if needed
- Move fink-science dep to science requirements
- Set py4j log level to INFO
- Add logging messages for raw2science and distribute
- Remove useless fink wait script
@fjammes fjammes self-assigned this Sep 13, 2024
@fjammes fjammes force-pushed the 877-automated-e2e-tests branch 4 times, most recently from 519c2c5 to d7bb151 Compare September 13, 2024 13:57
Copy link
Member

@JulienPeloton JulienPeloton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @fjammes , it looks good to me! Thanks for the refactoring of the scientific part of the code 💪

Concerning the fact that there are no topics in the output, try to increase the number of alerts launched by the simulator:

https://github.com/astrolabsoftware/fink-alert-simulator/blob/dcc8ff1d1412cc67f78f5594d04b0234506d8c18/chart/templates/configmap.yaml#L40

if args.noscience:
fsvsn = "no-science"
else:
# Do not import fink_science if --noscience is set
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would put this comment above in the logger:

if args.noscience:
  logger.debug("Do not import fink_science as --noscience is set")
  fsvsn = "no-science"
else:
  ...

/s5cmd --log debug --no-verify-ssl ls -H "s3://fink-broker-online/raw/2020/01/01/"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be:

s3://fink-broker-online/raw/20200101

@fjammes fjammes force-pushed the 877-automated-e2e-tests branch 2 times, most recently from 486a9e9 to 0a7a209 Compare September 16, 2024 08:52
Copy link

codecov bot commented Sep 16, 2024

Codecov Report

Attention: Patch coverage is 74.64789% with 36 lines in your changes missing coverage. Please review.

Project coverage is 84%. Comparing base (70a2540) to head (0a7a209).
Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
fink_broker/mm_utils.py 72% 16 Missing ⚠️
bin/raw2science.py 59% 15 Missing ⚠️
bin/distribute.py 80% 4 Missing ⚠️
fink_broker/spark_utils.py 66% 1 Missing ⚠️
Additional details and impacted files
@@          Coverage Diff          @@
##           master   #886   +/-   ##
=====================================
- Coverage      84%    84%   -1%     
=====================================
  Files          21     21           
  Lines        1292   1330   +38     
=====================================
+ Hits         1092   1121   +29     
- Misses        200    209    +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@fjammes fjammes force-pushed the 877-automated-e2e-tests branch from 0a7a209 to 433b557 Compare September 16, 2024 09:22
@fjammes fjammes force-pushed the 877-automated-e2e-tests branch from 433b557 to 0bca0c3 Compare September 16, 2024 09:37
Copy link

@fjammes
Copy link
Contributor Author

fjammes commented Sep 16, 2024

@JulienPeloton , FYI I have added a "noscience" pipeline, because the Github one is down.

@fjammes fjammes closed this Sep 16, 2024
@fjammes fjammes deleted the 877-automated-e2e-tests branch September 16, 2024 09:41
@fjammes
Copy link
Contributor Author

fjammes commented Sep 16, 2024

LGTM

@JulienPeloton
Copy link
Member

Why did you close this PR?

@fjammes
Copy link
Contributor Author

fjammes commented Sep 16, 2024

My mistake, I wanted to merge it, I have merged it manually...

@JulienPeloton
Copy link
Member

:D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automate Execution of Fink-Broker E2E Tests with Science Algorithm on Linux Server and Report to GitHub Action
2 participants