Update to Spark 3.4.1

[JulienPeloton]

IMPORTANT: Please create an issue first before opening a Pull Request.
Linked to issue(s): Closes #758

Contains also:

• Automatic generation of the SSOFT #760
• add fast transient module to the science pipeline #757

[JulienPeloton]

Requires spark>=3.2

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (47e4b0f) 90% compared to head (4e20e5b) 90%.

Additional details and impacted files

@@          Coverage Diff          @@
##           master   #759   +/-   ##
=====================================
  Coverage      90%    90%           
=====================================
  Files          17     17           
  Lines         939    948    +9     
=====================================
+ Hits          847    857   +10     
+ Misses         92     91    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[JulienPeloton]

All tests OK, including the self-hosted ones, but the GHA:

##[debug]Evaluating condition for step: 'Check results'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Check results
##[debug]Loading inputs
##[debug]Loading env
Run ./itest/check-results.sh
##[debug]/usr/bin/bash -e /home/runner/work/_temp/ec3e39b3-7317-421c-877d-a6ee9d20e29f.sh
error: timed out waiting for the condition on topics
Error: Process completed with exit code 1.
##[debug]Finishing: Check results

This error seems a constant unfortunately, and unrelated to the code :-(