Skip to content

Commit

Permalink
Update inference_rules.adoc
Browse files Browse the repository at this point in the history
  • Loading branch information
mrmhodak authored Feb 6, 2024
1 parent c954aeb commit 456686e
Showing 1 changed file with 19 additions and 0 deletions.
19 changes: 19 additions & 0 deletions inference_rules.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,25 @@ MLCommons shall retain a library of past audit reports and send copies to MLComm

An audit is expected to be completed within a 90 day period. Audits failing to meet this timeline can be requested to be invalidated by the auditee. The final decision to accept such a request will be taken by the Working Group.

=== Assigned Peer Review Process

In addition to general MLCommons review rules outlined https://github.com/mlperf/policies/blob/master/submission_rules.adoc[here], Inference workgroup uses Assigned Peer Review to improve the scrutiny of the results. The goal is to ensure that each submission is reviewed. The process, executed by Results Chair, is as follows:

* All submitters are ordered (alphabetically, or any other way) and are assigned a number according to their position starting with 1

* A random generator that reorders (shuffled) the list is used live during a review meeting. List Randomizer from random.org can be used.

* Peer Reviews are assigned based on the new order with the last submitter in the new order assigned to review the first one. For example, with companies Company1, Company2, and Company3 participating, their assigned numbers will be 1, 2, and 3, respectively. The list gets reordered 2, 1, 3. The Assigned Reviews will be: Company2 will review Company1, Company1 will review Company3, Company3 will review Company2.

* Chair will open an github issue against each company that has review assignment. Issues will be closed once reviewers indicate tha they finished their tasks.

* During the review, reviwers are asked to pay special attention to: (1) results validity, (2) methodology, (3) instructions for reproducibility, and (4) content of json files in systems directory.

* Any issues discovered should be filed as github issues and resolved as usual.

* Submitters are encouraged to review other submissions beyond their assigned review.


== Scenarios

In order to enable representative testing of a wide variety of inference
Expand Down

0 comments on commit 456686e

Please sign in to comment.