Merge pull request #285 from krai/krai-open-datasets

Update the open dataset requirement

inference_rules.adoc

@@ -322,14 +322,13 @@ For 3DUNet, the logical destination for the benchmark output is considered to be
 ==== Relaxed constraints for the Open division
 
 1. An Open benchmark must perform a task matching an existing Closed benchmark, and be substitutable in LoadGen for that benchmark.
-1. The accuracy dataset must be the same as used in an existing Closed benchmark, or must be pre-approved and added to the following list: ImageNet 2012 validation dataset for Image Classification; COCO 2017 validation dataset for Object Detection. From v3.0, if a submitter provides any results with any models trained on a pre-approved dataset,
-the submitter must also provide at least one result with the corresponding Closed model trained
-(or finetuned) on the same pre-approved dataset, and instructions to reproduce the training (or finetuning) process.
+1. The validation dataset must be the same as used in an existing Closed benchmark, or must be pre-approved and added to the following list: ImageNet 2012 validation dataset for Image Classification; COCO 2017 validation dataset for Object Detection.
+When seeking such pre-approval, it is recommended that a potential submitter convincingly demonstrates the accuracy of the corresponding Closed model on the same validation dataset, which may involve retraining or finetuning the Closed model if required.
 1. Accuracy constraints are not applicable: instead the submission must report the accuracy obtained.
 1. Latency constraints are not applicable: instead the submission must report the latency constraints under which the reported performance was obtained.
 1. Scenario constraints are not applicable: any combination of scenarios is permitted.
 1. A open submission must be classified as "Available", "Preview", or "Research, Development, or Internal".
-1. The model can be of any origin (trained on any dataset, quantized in any way, and sparsified in anyway).
+1. The model can be of any origin (trained on any dataset, except the validation dataset; quantized in any way; sparsified in any way).
 
 ==== Additional inference parameters