Add Spark monitoring

[fjammes]

Describe the issue
There is currently no real-time monitoring system integrated with Spark within the fink-broker infrastructure. This limits the ability to track job performance, resource usage, and detect issues during processing in real-time.

Expected behaviour

The system should provide real-time metrics and monitoring for Spark jobs, including but not limited to:

• Job execution time
• Resource (CPU, memory) utilization
• Task success/failure rates
• Data processing rates (throughput)

This will allow administrators to easily visualize performance metrics and troubleshoot any issues more efficiently.

Actual behaviour

Currently, no monitoring tools are integrated into the fink-broker Spark processes, so resource usage and job performance need to be tracked manually, or users must rely on log files without real-time insights.

Screenshots
N/A

Additional context
The integration could leverage existing Spark monitoring tools like Spark UI, or Prometheus, to enable comprehensive insights into the processing jobs.