Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encouraging results density #270

Open
psyhtest opened this issue Mar 28, 2023 · 2 comments
Open

Encouraging results density #270

psyhtest opened this issue Mar 28, 2023 · 2 comments

Comments

@psyhtest
Copy link
Contributor

According to the current rules, a submitter may intentionally or unintentionally introduce sparsity in the results table.

For example, if they choose a different system name for each workload they submit on essentially the same system:

awesome_system_for_resnet50
awesome_system_for_retinanet
...

the results table will contain one row per workload. One issue is that such a submitter may get an unfair advantage when their system is picked for audit - with only one workload being subject to audit.

We should mould the rules to encourage results density by carefully defining what constitutes "essentially the same system" and what does not.

@psyhtest
Copy link
Contributor Author

Notable examples in v3.0 and v3.1 include CPU-only submissions with names 1-node-2S-SPR-PyTorch-INT8 and 1-node-2S-SPR-PyTorch-MIX, which could be folded into the same row.

On the other hand, 1-node-2S-SPR-PyTorch-INT4+INT8 and 1-node-2S-SPRHBM-PyTorch-BF16 could be viewed as sufficiently different.

@mrmhodak
Copy link
Contributor

Recommendation can be added to the Rules. Intel will make sure this will not happen again.

@psyhtest psyhtest reopened this Feb 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants