From 456686eb6a69df41e49eb5ea4a76bce802b270ef Mon Sep 17 00:00:00 2001 From: Miro Date: Tue, 6 Feb 2024 02:23:28 -0500 Subject: [PATCH] Update inference_rules.adoc --- inference_rules.adoc | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/inference_rules.adoc b/inference_rules.adoc index a403543..5ab5869 100644 --- a/inference_rules.adoc +++ b/inference_rules.adoc @@ -127,6 +127,25 @@ MLCommons shall retain a library of past audit reports and send copies to MLComm An audit is expected to be completed within a 90 day period. Audits failing to meet this timeline can be requested to be invalidated by the auditee. The final decision to accept such a request will be taken by the Working Group. +=== Assigned Peer Review Process + +In addition to general MLCommons review rules outlined https://github.com/mlperf/policies/blob/master/submission_rules.adoc[here], Inference workgroup uses Assigned Peer Review to improve the scrutiny of the results. The goal is to ensure that each submission is reviewed. The process, executed by Results Chair, is as follows: + +* All submitters are ordered (alphabetically, or any other way) and are assigned a number according to their position starting with 1 + +* A random generator that reorders (shuffled) the list is used live during a review meeting. List Randomizer from random.org can be used. + +* Peer Reviews are assigned based on the new order with the last submitter in the new order assigned to review the first one. For example, with companies Company1, Company2, and Company3 participating, their assigned numbers will be 1, 2, and 3, respectively. The list gets reordered 2, 1, 3. The Assigned Reviews will be: Company2 will review Company1, Company1 will review Company3, Company3 will review Company2. + +* Chair will open an github issue against each company that has review assignment. Issues will be closed once reviewers indicate tha they finished their tasks. + +* During the review, reviwers are asked to pay special attention to: (1) results validity, (2) methodology, (3) instructions for reproducibility, and (4) content of json files in systems directory. + +* Any issues discovered should be filed as github issues and resolved as usual. + +* Submitters are encouraged to review other submissions beyond their assigned review. + + == Scenarios In order to enable representative testing of a wide variety of inference