[Add] files via upload

Author: testing-cs

README.md

@@ -1,2 +1,81 @@
-# LaF-comparison-testing
-LaF focuses on the comparion testing of multiple deep learning models without manual labeling.
+# LaF: labeling-free comparison testing of deep learning models
+
+## Problem definition
+
+Given N pre-trained deep learning models, the task is to estimate the rank of models regrading their performance on an unlabeled test set.
+
+## Dependency
+
+- python 3.6.10
+- keras 2.6.0
+- tensorflow 2.5.1
+- scipy 1.5.4
+- numpy 1.19.5
+
+## Download the dataset
+
+###ID data###
+MNIST, CIFAR-10, and Fashion-MNIST are available in Keras.
+
+Amazon and iwildcam are taken from [WILDS](https://github.com/p-lambda/wilds).
+ 
+Java250 and C++1000 are taken from [Project CodeNet](https://github.com/IBM/Project_CodeNet).
+
+###OOD data###
+
+Download the OOD data of MNIST from [Google drive]() or generate it by <pre><code>python gene_mnist.py</code></pre>
+
+Download the OOD data of CIFAR-10 from [Google drive]() or generate it by <pre><code>python gene_cifar10.py</code></pre>
+
+Download the OOD data of Amazon and iwildCam from [WILDS](https://github.com/p-lambda/wilds).
+
+Download the OOD data of Java250 from [Google drive]().
+
+## Download Pre-trained deep learning models
+
+Download all the models from [Google drive]().
+
+You can also train the models for MNIST and CIFAR-10 by running the scripts in **trainModel/mnist** and **trainModel/cifar10**. 
+
+## How to use
+
+To speed the execution and avoid calling the model repeatedly, we first get the model prediction. E.g.:
+
+```
+python main_ground.py --dataName mnist
+```
+
+To get the results by baseline methods (SDS, Random, CES), run the following code:
+
+```
+python main_selection.py --dataName mnist --metric random
+```
+
+Besides, to get the final results of CES, you need to run:
+
+```
+python main_ces_best.py --dataName mnist
+```
+
+To get the results by LaF, run the following code:
+```
+python main_laf.py --dataName mnist --dataType id
+```
+
+To get the evaluation on kendall's tau, spearman's coefficients, jaccard similarity, run the following code:
+
+```
+python main_eva.py --dataName mnist 
+```
+
+**[Notice] Be careful with the saving directories.**
+
+## Reference
+<pre><code>@misc{guo2022labelingfree,
+      title={Labeling-Free Comparison Testing of Deep Learning Models}, 
+      author={Yuejun Guo and Qiang Hu and Maxime Cordy and Xiaofei Xie and Mike Papadakis and Yves Le Traon},
+      year={2022},
+      eprint={2204.03994},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG}
+}</code></pre>

config.py

@@ -0,0 +1,60 @@
+import os
+
+
+class hyperparameters():
+    def __init__(self, dataName, dataType="original", severity=0):
+        self.dataName = dataName
+        self.dataType = dataType
+        self.severity = severity
+        if dataName == "fashion":
+            self.class_num = 10
+            self.batch_size = 100
+            self.model_num = 25
+        elif dataName == "mnist":
+            self.class_num = 10
+            self.batch_size = 100
+            self.model_num = 30
+        elif "mnist-" in dataName:
+            self.class_num = 10
+            self.batch_size = 100
+            self.model_num = 10
+        elif dataName == "cifar10":
+            self.class_num = 10
+            self.batch_size = 100
+            self.model_num = 30
+        elif dataName == "java250":
+            self.class_num = 250
+            self.batch_size = 200
+            self.model_num = 20
+        elif dataName == "cpp1000":
+            self.class_num = 1000
+            self.batch_size = 512
+            self.model_num = 20
+        elif dataName == "iwildcamO":
+            self.class_num = 182
+            self.batch_size = 100
+            self.model_num = 20
+        elif dataName == "amazon":
+            self.class_num = 5
+            self.batch_size = 100
+            self.model_num = 20
+
+        root_path = "/Volumes/1T/dnnCompare/"
+        self.save_model_root = root_path + "{0}/savedM/".format(dataName)
+        self.save_model_pre_root = root_path + "{0}/savedP/{1}/".format(dataName, dataType)
+        self.save_log_root_test = root_path + "{0}/savedL/{1}/".format(dataName, dataType)
+        self.save_result_root = root_path + "{0}/savedR/{1}/".format(dataName, dataType)
+        self.save_ground_root = root_path + "{0}/savedG/{1}/".format(dataName, dataType)
+        self.save_data_root_adv = root_path + "{0}/savedD/".format(dataName)
+        if not os.path.isdir(self.save_model_root):
+            os.makedirs(self.save_model_root)
+        if not os.path.isdir(self.save_log_root_test):
+            os.makedirs(self.save_log_root_test)
+        if not os.path.isdir(self.save_data_root_adv):
+            os.makedirs(self.save_data_root_adv)
+        if not os.path.isdir(self.save_ground_root):
+            os.makedirs(self.save_ground_root)
+        if not os.path.isdir(self.save_result_root):
+            os.makedirs(self.save_result_root)
+        if not os.path.isdir(self.save_model_pre_root):
+            os.makedirs(self.save_model_pre_root)