Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Peer Review Rules #290

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Peer Review Rules #290

Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
456686e
Update inference_rules.adoc
mrmhodak Feb 6, 2024
b1cda85
Update inference_rules.adoc
mrmhodak Feb 13, 2024
f795002
Update inference_rules.adoc
mrmhodak Feb 20, 2024
a13c963
Update inference_rules.adoc
mrmhodak Feb 20, 2024
663da6b
Update inference_rules.adoc
mrmhodak Feb 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions inference_rules.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,23 @@ MLCommons shall retain a library of past audit reports and send copies to MLComm

An audit is expected to be completed within a 90 day period. Audits failing to meet this timeline can be requested to be invalidated by the auditee. The final decision to accept such a request will be taken by the Working Group.

=== Assigned Peer Review Process
mrmhodak marked this conversation as resolved.
Show resolved Hide resolved

In addition to general MLCommons review rules outlined https://github.com/mlperf/policies/blob/master/submission_rules.adoc[here], Inference workgroup uses Assigned Peer Review to improve scrutiny of the results. The goal is to ensure that each submission is reviewed. The process, executed by Results Chair, is as follows:

* Each submitter will be assigned another submitter to review.

* Chair will open a github issue against each company that has review assignment. Issues will be closed once reviewers indicate that they had finished their tasks.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the github issue process, but again don't want to prescribe the process into the rules. Perhaps this could go into a 1 pager document off the rules for future reference.

I think it is okay to leave these exact details to the discretion of the chair and working group to adjust as necessary and is practical.


* During the review, reviewers are asked to pay special attention to: (1) results validity, (2) methodology, (3) instructions for reproducibility, and (4) content of json files in systems directory.

* Any issues discovered are to be filed as github issues and resolved as usual. The issues should be filed before the deadline to raise objections.

* Submitters are encouraged to review other submissions beyond their assigned review.

During the Peer Review Process, a reviewer can ask for access to a submission system to reproduce the submission. If the request is granted and the reviewer is satisfied with the results, this can be used to waive the reviewed system from the audit, subject to the approval of the review group.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this language is good, "If the request is granted and the reviewer is satisfied with the results, this can be used to waive the reviewed system from the audit"

Because a supplier could use an OEM to self review and then exempt themselves from the audit. I'd support language where the random audit is random.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of saving effort, but it seems too open ended and not specific. I'd favor simpler and easier rules.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if its waived from the selection audit but not the random audit for this round?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bitfort "subject to the approval of the review group" condition is still there. If the random audit is random, most of the time it is a waste of everyone's time like happening now.

@peladodigital The waiver is meant for random audit. Because the selection audit is done by the review committee and automatically the committee will be eliminating uninteresting and already reproduced submissions.



== Scenarios

In order to enable representative testing of a wide variety of inference
Expand Down
Loading