Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-evaluate breaking-IID experiments from checkpoints #23

Closed
mcw92 opened this issue Jan 7, 2025 · 0 comments · Fixed by #20
Closed

Re-evaluate breaking-IID experiments from checkpoints #23

mcw92 opened this issue Jan 7, 2025 · 0 comments · Fixed by #20
Assignees
Labels
experiments Tasks related to experiments to run

Comments

@mcw92
Copy link
Member

mcw92 commented Jan 7, 2025

Additional metrics for breaking-IID series

As we decided to analyze a variety of performance metrics in addition to the plain accuracy, we need to re-evaluate all 16-node breaking-IID experiments from their respective model checkpoints to obtain the local and global confusion matrices.

Relevant scripts are:

  • Actual Python script to run: ./scripts/examples/evaluate_from_checkpoint_breaking_iid.py
  • SLURM utility function to obtain correct checkpoint paths for each parameter combination, starting from given base path: find_checkpoint_dir_and_uuid in ./specialcouscous/utils/slurm.py
  • Job script generation script: ./scripts/experiments/generate_parallel_evaluation_from_breaking_iid_ckpt_job_scripts.py
  • Path to actual results on HoreKa: ${BASEDIR}/results/ (BASEDIR corresponds to our workspace)
@mcw92 mcw92 added the experiments Tasks related to experiments to run label Jan 7, 2025
@mcw92 mcw92 self-assigned this Jan 7, 2025
@mcw92 mcw92 linked a pull request Jan 7, 2025 that will close this issue
@mcw92 mcw92 closed this as completed in #20 Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
experiments Tasks related to experiments to run
Projects
None yet
1 participant