Scaling and profiling experiments

[tms-epcc]

From cycle-6 wiki page this comprises:

• get it working first with acceptance tests for Lasair 4.0
  then enhance as follows:
• See Running pipeline at speed, scaling experiments below
• Numbers of nodes, processes, threads
• Batch and buffer sizes
• Database tuning
• LSST rates
• process 10 million alerts in 12 hours
• Trans atlantic link (blocked)
• Maximising the pipeline throughput and the web/api response

[tms-epcc]

05/OCT/22

• @RoyWilliams reported development system works with Lasair v4.0 acceptance tests
• @tms-epcc and @RoyWilliams agreed to change ticket name from "Build complete development system" to "Scaling and profiling experiments"

[tms-epcc]

06/OCT/22

• Cycle 7 plan added the following tests to the task and hence the check box list above
  • Trans atlantic link (blocked)
  • Maximising the pipeline throughput and the web/api response

[tms-epcc]

14/DEC/22

• @gpfrancis will provide initial thoughts to promote discussion on the test plan for this

[tms-epcc]

25/JAN/23

• @RoyWilliams reported that the Performance Testing Plan was now ready and the next step was to agree responsibilities for executing the various tests

[tms-epcc]

16/AUG/23

• as discussed with @RoyWilliams this will be deferred to cycle 9 (starts 01/OCT/23) due to the impact of RSP synergy, Oxford user meeting and LSST@Europe

[tms-epcc]

14/AUG/24

• 10 million alerts in 48 hours was achieved as part of the OR4 : see July'24 newsletter

[tms-epcc]

06/NOV/24

• currently analysing performance of nodes and processes - see meeting mins