This dataset contains results of the third development round of [NIPS 2017 Adversarial learning competition](https://www.kaggle.com/nips-2017-adversarial-learning-competition). ## Content Matrices with intermediate results Following matrices with intermediate results are provided: * **accuracy_matrix.csv** - matrix with number of correctly classified images for each pair of attack (targeted and non-targeted) and defense * **error_matrix.csv** - matrix with number of misclassified images for each pair of attack (targeted and non-targeted) and defense * **hit_target_class_matrix.csv** - matrix with number of times image was classified as specific target class for each pair of attack (targeted and non-targeted) and defense In each of these matrices, rows correspond to defenses, columns correspond to attack. Also first row and column are headers with Kaggle Team IDs (or baseline ID). Scores and run time statistics of submissions Following files contain scores and run time stats of the submissions: * **non_targeted_attack_results.csv** - scores and run time statistics of all non-targeted attacks * **targeted_attack_results.csv** - scores and run time statistics of all targeted attacks * **defense_results.csv** - scores and run time statistics of all defenses Each row of these files correspond to one submission. Columns have following meaning: * KaggleTeamId - either Kaggle Team ID or ID of the baseline. * TeamName - human readable team name * Score - raw score of the submission * NormalizedScore - normalized (to be between 0 and 1) score of the submission * MinEvalTime - minimum evaluation time of 100 images * MaxEvalTime - maximum evaluation time of 100 images * MedianEvalTime - median evaluation time of 100 images * MeanEvalTime - average evaluation time of 100 images ## Notes about the data * Due to team mergers, team name in these files might be different from the leaderboard. * Not all attacks were used to compute scores of defenses and not all defenses were used to compute scores of attacks. Thus if you simply sum-up values in rows/columns of the corresponding matrix you won't obtain exact score of the submission (however number you obtain will be very close to actual score). * Few targeted and non-targeted attacks exceeded 500 seconds time limit on all batches of images. These submissions received score 0 in the official leaderboard. We still were able to compute "real" score for these submissions and include it into non_targeted_attack_results.csv and targeted_attack_results.csv files. However these scores are negated in the provided files to emphasize that these submissions violate the time limit.