Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Peer Review Rules #290

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Peer Review Rules #290

Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
456686e
Update inference_rules.adoc
mrmhodak Feb 6, 2024
b1cda85
Update inference_rules.adoc
mrmhodak Feb 13, 2024
f795002
Update inference_rules.adoc
mrmhodak Feb 20, 2024
a13c963
Update inference_rules.adoc
mrmhodak Feb 20, 2024
663da6b
Update inference_rules.adoc
mrmhodak Feb 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions inference_rules.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,25 @@ MLCommons shall retain a library of past audit reports and send copies to MLComm

An audit is expected to be completed within a 90 day period. Audits failing to meet this timeline can be requested to be invalidated by the auditee. The final decision to accept such a request will be taken by the Working Group.

=== Assigned Peer Review Process
mrmhodak marked this conversation as resolved.
Show resolved Hide resolved

In addition to general MLCommons review rules outlined https://github.com/mlperf/policies/blob/master/submission_rules.adoc[here], Inference workgroup uses Assigned Peer Review to improve scrutiny of the results. The goal is to ensure that each submission is reviewed. The process, executed by Results Chair, is as follows:

* All submitters are compiled into an ordered list. Alphabetical, or any other way order is fine

* A list randomizer is used to generate a re-ordered list live during a review meeting. List Randomizer from random.org can be used.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we simplify the wording here? I suspect it is easier if we just said "submitters will be randomly assigned another to review"... I think as long as it's random, we don't need to get into implementation details in the rules.


* Peer Reviews are assigned based on the new order with the last submitter in the new order assigned to review the first one. For example, with companies Company1, Company2, and Company3 participating, the list can be reordered to Company2, Company1, Company3. The Assigned Reviews will be: Company2 will review Company1, Company1 will review Company3, Company3 will review Company2.

* Chair will open a github issue against each company that has review assignment. Issues will be closed once reviewers indicate that they had finished their tasks.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the github issue process, but again don't want to prescribe the process into the rules. Perhaps this could go into a 1 pager document off the rules for future reference.

I think it is okay to leave these exact details to the discretion of the chair and working group to adjust as necessary and is practical.


* During the review, reviewers are asked to pay special attention to: (1) results validity, (2) methodology, (3) instructions for reproducibility, and (4) content of json files in systems directory.

* Any issues discovered should be filed as github issues and resolved as usual. The issues should be filed promptly before the issue filing deadline.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I 100% agree with the importantance of "The issues should be filed promptly before the issue filing deadline." I'd like to limit the number of suggestions we write as rules. What to you think of combining all the suggestions / encouragements into a single statement like,

"Reviewers will be given these directions:

  • You are encouraged to pay special attention to: (1) results validity, (2) methodology, (3) instructions for reproducibility, and (4) content of json files in systems directory.
  • You are encouraged to file your review early
  • You are encouraged to review other submitters beyond your assignments."


* Submitters are encouraged to review other submissions beyond their assigned review.


== Scenarios

In order to enable representative testing of a wide variety of inference
Expand Down
Loading