Skip to content

Commit

Permalink
updating audit rules to ensure fair voting
Browse files Browse the repository at this point in the history
  • Loading branch information
itayhubara committed Oct 24, 2023
1 parent d850817 commit 6a3f1d3
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions inference_rules.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,10 @@ In each round, up to two submissions will be audited: one at random from all sub

The process of random selection is in two stages: first a submitter is randomly chosen from all submitters with auditable submissions, then one of those submissions is randomly chosen. A submission is not a candidate for the randomly chosen audit if the system is equivalent to a system audited in the previous round. For the purposes of this rule, equivalent systems have the same CPU, NIC, accelerator, and accelerator count, with the same configuration of those components as per the system configuration JSON. For LoadGen Over Network submission the Networking must be the same. The review committee may determine that additional systems are equivalent to those audited in a previous round and exempt them from random audit. As a guidance for this exemption, if an accelerator is audited in one of the previous rounds, then the systems using the same accelerator can be excluded from random audit, if the aggregate system performance and the performance per accelerator are not more than 10% from those submitted during last audit time. For systems with power metrics, in addition to the performance, power efficiency must also be within 10% from the last audit time to be eligible for an exclusion from random audit. If any new result like a new model, an additional non-inferred scenario measurement or a new power measurement is submitted from the last audit time, then the exclusion is not applicable unless the review committee decides otherwise.

To ensure equity, if a submitter undergoes consecutive audits spanning two or more rounds, the committee must compose a brief explanation outlining the discrepancies between the current submission and the prior submission outcomes. If the auditee perceives this as unjust, they retain the option to submit an appeal to MLCommons board.

In addtion, if a submitter receive their code base from another submitter and run on a similar hardware, they can nominate systems for an audit but are ineligible to participate in ranked choice voting.

During the review process, a github issue shall be opened where submitters can nominate systems for audit. Each nomination shall contain a reason, such as new HW or SW, unusual or interesting features, performance outside of expectations, etc. Review committee chairs evaluate the nominations and compile a list of systems at the end of the review period. Any systems with new accelerators are added to the list by the chairs if not nominated. The review committee will select a submission for audit by ranked choice voting using a simple majority. An option "No Selected Audit This Round" may be added if requested by a majority of the review committee.

An auditor shall be chosen by the review committee who has no conflict of interest with the submitter. The process of auditor selection will take no more than 28 days from selection of the submitter.
Expand Down

0 comments on commit 6a3f1d3

Please sign in to comment.