From 456686eb6a69df41e49eb5ea4a76bce802b270ef Mon Sep 17 00:00:00 2001 From: Miro Date: Tue, 6 Feb 2024 02:23:28 -0500 Subject: [PATCH 1/5] Update inference_rules.adoc --- inference_rules.adoc | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/inference_rules.adoc b/inference_rules.adoc index a403543..5ab5869 100644 --- a/inference_rules.adoc +++ b/inference_rules.adoc @@ -127,6 +127,25 @@ MLCommons shall retain a library of past audit reports and send copies to MLComm An audit is expected to be completed within a 90 day period. Audits failing to meet this timeline can be requested to be invalidated by the auditee. The final decision to accept such a request will be taken by the Working Group. +=== Assigned Peer Review Process + +In addition to general MLCommons review rules outlined https://github.com/mlperf/policies/blob/master/submission_rules.adoc[here], Inference workgroup uses Assigned Peer Review to improve the scrutiny of the results. The goal is to ensure that each submission is reviewed. The process, executed by Results Chair, is as follows: + +* All submitters are ordered (alphabetically, or any other way) and are assigned a number according to their position starting with 1 + +* A random generator that reorders (shuffled) the list is used live during a review meeting. List Randomizer from random.org can be used. + +* Peer Reviews are assigned based on the new order with the last submitter in the new order assigned to review the first one. For example, with companies Company1, Company2, and Company3 participating, their assigned numbers will be 1, 2, and 3, respectively. The list gets reordered 2, 1, 3. The Assigned Reviews will be: Company2 will review Company1, Company1 will review Company3, Company3 will review Company2. + +* Chair will open an github issue against each company that has review assignment. Issues will be closed once reviewers indicate tha they finished their tasks. + +* During the review, reviwers are asked to pay special attention to: (1) results validity, (2) methodology, (3) instructions for reproducibility, and (4) content of json files in systems directory. + +* Any issues discovered should be filed as github issues and resolved as usual. + +* Submitters are encouraged to review other submissions beyond their assigned review. + + == Scenarios In order to enable representative testing of a wide variety of inference From b1cda853cf88e626da5f71bd42923efaae8cb74b Mon Sep 17 00:00:00 2001 From: Miro Date: Tue, 13 Feb 2024 01:12:59 -0500 Subject: [PATCH 2/5] Update inference_rules.adoc --- inference_rules.adoc | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/inference_rules.adoc b/inference_rules.adoc index 5ab5869..197956d 100644 --- a/inference_rules.adoc +++ b/inference_rules.adoc @@ -129,19 +129,19 @@ An audit is expected to be completed within a 90 day period. Audits failing to m === Assigned Peer Review Process -In addition to general MLCommons review rules outlined https://github.com/mlperf/policies/blob/master/submission_rules.adoc[here], Inference workgroup uses Assigned Peer Review to improve the scrutiny of the results. The goal is to ensure that each submission is reviewed. The process, executed by Results Chair, is as follows: +In addition to general MLCommons review rules outlined https://github.com/mlperf/policies/blob/master/submission_rules.adoc[here], Inference workgroup uses Assigned Peer Review to improve scrutiny of the results. The goal is to ensure that each submission is reviewed. The process, executed by Results Chair, is as follows: -* All submitters are ordered (alphabetically, or any other way) and are assigned a number according to their position starting with 1 +* All submitters are compiled into an ordered list. Alphabetical, or any other way order is fine -* A random generator that reorders (shuffled) the list is used live during a review meeting. List Randomizer from random.org can be used. +* A list randomizer is used to generate a re-ordered list live during a review meeting. List Randomizer from random.org can be used. -* Peer Reviews are assigned based on the new order with the last submitter in the new order assigned to review the first one. For example, with companies Company1, Company2, and Company3 participating, their assigned numbers will be 1, 2, and 3, respectively. The list gets reordered 2, 1, 3. The Assigned Reviews will be: Company2 will review Company1, Company1 will review Company3, Company3 will review Company2. +* Peer Reviews are assigned based on the new order with the last submitter in the new order assigned to review the first one. For example, with companies Company1, Company2, and Company3 participating, the list can be reordered to Company2, Company1, Company3. The Assigned Reviews will be: Company2 will review Company1, Company1 will review Company3, Company3 will review Company2. -* Chair will open an github issue against each company that has review assignment. Issues will be closed once reviewers indicate tha they finished their tasks. +* Chair will open a github issue against each company that has review assignment. Issues will be closed once reviewers indicate that they had finished their tasks. -* During the review, reviwers are asked to pay special attention to: (1) results validity, (2) methodology, (3) instructions for reproducibility, and (4) content of json files in systems directory. +* During the review, reviewers are asked to pay special attention to: (1) results validity, (2) methodology, (3) instructions for reproducibility, and (4) content of json files in systems directory. -* Any issues discovered should be filed as github issues and resolved as usual. +* Any issues discovered should be filed as github issues and resolved as usual. The issues should be filed promptly before the issue filing deadline. * Submitters are encouraged to review other submissions beyond their assigned review. From f795002be5df57c3a19004c522fd93009ccdfe31 Mon Sep 17 00:00:00 2001 From: Miro Date: Tue, 20 Feb 2024 01:32:21 -0500 Subject: [PATCH 3/5] Update inference_rules.adoc --- inference_rules.adoc | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/inference_rules.adoc b/inference_rules.adoc index 197956d..00b9ad6 100644 --- a/inference_rules.adoc +++ b/inference_rules.adoc @@ -131,17 +131,13 @@ An audit is expected to be completed within a 90 day period. Audits failing to m In addition to general MLCommons review rules outlined https://github.com/mlperf/policies/blob/master/submission_rules.adoc[here], Inference workgroup uses Assigned Peer Review to improve scrutiny of the results. The goal is to ensure that each submission is reviewed. The process, executed by Results Chair, is as follows: -* All submitters are compiled into an ordered list. Alphabetical, or any other way order is fine - -* A list randomizer is used to generate a re-ordered list live during a review meeting. List Randomizer from random.org can be used. - -* Peer Reviews are assigned based on the new order with the last submitter in the new order assigned to review the first one. For example, with companies Company1, Company2, and Company3 participating, the list can be reordered to Company2, Company1, Company3. The Assigned Reviews will be: Company2 will review Company1, Company1 will review Company3, Company3 will review Company2. +* Each submitter will be assigned another submitter to review. * Chair will open a github issue against each company that has review assignment. Issues will be closed once reviewers indicate that they had finished their tasks. * During the review, reviewers are asked to pay special attention to: (1) results validity, (2) methodology, (3) instructions for reproducibility, and (4) content of json files in systems directory. -* Any issues discovered should be filed as github issues and resolved as usual. The issues should be filed promptly before the issue filing deadline. +* Any issues discovered are to be filed as github issues and resolved as usual. The issues should be filed before the deadline to raise objections. * Submitters are encouraged to review other submissions beyond their assigned review. From a13c9638d64df3fcfdc98798e244940ce06cfdd2 Mon Sep 17 00:00:00 2001 From: Miro Date: Tue, 20 Feb 2024 01:51:17 -0500 Subject: [PATCH 4/5] Update inference_rules.adoc --- inference_rules.adoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/inference_rules.adoc b/inference_rules.adoc index 00b9ad6..df842e1 100644 --- a/inference_rules.adoc +++ b/inference_rules.adoc @@ -141,6 +141,8 @@ In addition to general MLCommons review rules outlined https://github.com/mlperf * Submitters are encouraged to review other submissions beyond their assigned review. +During the Peer Review Process, a reviewer can ask for access to a submission system to reproduce the submission. If the request is granted and the reviewer is satisfied with the results, this can be used to waive the reviewed system from the audit, subject to the approval of the review group. + == Scenarios From 663da6b91828ac5530b4a098711abcc6d74824e8 Mon Sep 17 00:00:00 2001 From: Miro Date: Thu, 22 Feb 2024 14:45:49 -0500 Subject: [PATCH 5/5] Update inference_rules.adoc Removing waiving audit requirements --- inference_rules.adoc | 2 -- 1 file changed, 2 deletions(-) diff --git a/inference_rules.adoc b/inference_rules.adoc index df842e1..00b9ad6 100644 --- a/inference_rules.adoc +++ b/inference_rules.adoc @@ -141,8 +141,6 @@ In addition to general MLCommons review rules outlined https://github.com/mlperf * Submitters are encouraged to review other submissions beyond their assigned review. -During the Peer Review Process, a reviewer can ask for access to a submission system to reproduce the submission. If the request is granted and the reviewer is satisfied with the results, this can be used to waive the reviewed system from the audit, subject to the approval of the review group. - == Scenarios