BlackboxBench

ImageNet Leaderboard

    We set the maxium queries to be 10000 on all tests and the attack budget will be set uniformly by

    ImageNet: l_inf: 0.05 = 12.75/255, l_2: 5 = 1275/255.



　Model  → ResNet-50 Inception-v3 AT model FastAT(56.90) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
NES 1031.9 720 1 1571.2 840 0.95 2113.4 1500 0.714 2578.8 1560 0.727 3084.5 1745 0.763
ZS 2013 1220 0.765 1316.7 671 0.896 1688.7 1037 0.875 1956.3 1152 0.897 2266.6 1243 0.645
Bandit 392.2 58 0.98 903.1 142 0.95 1091.5 402 0.584 1700.5 612 0.643 789.4 567 0.453
Parsimonious 347.7 241 1 853.2 258 0.973 1440.6 646 0.8 1374.2 901 0.842 2423.5 1765 0.465
Sign Hunter 264.1 85 0.974 557.2 108 0.944 1522.7 258 0.811 1520.9 199 0.722 665.7 323 0.745
Square 76.5 13 1 247.1 23 0.997 1109 143 0.841 693.8 108 0.842 95.6 23 0.894
　Model  → ResNet-50 Inception-v3 AT model FastAT(56.90) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
NES 1335.2 1020 1 2048.6 1200 0.744 2365.1 1620 0.218 2194.3 1320 0.261 2023.5 1200 0.5
ZS 843.3 549 0.464 1642.6 1240 0.735 639.3 610 0.082 765.3 876 0.101 2134.7 145.6 0.355
Bandit 856.5 512 1 1327.2 636 0.912 2769.2 1756 0.456 2434.8 1673 0.502 889.6 565 0.654
SimBA 1234.5 1120 0.686 2077.4 1398 0.805 2638.2 2081 0.349 2183.2 1934.5 0.401 1235.6 1120 0.644
Square 612.1 174 0.99 1107.1 310 0.89 1404.3 330 0.472 1593.5 392 0.483 345.5 133 0.447
　Model  → ResNet-50 Inception-v3 AT model FastAT(56.90) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
NES 28776.5 17640.5 0.885 33456.1 17665 0.795 32345.6 23460 0.556 36549.4 24598 0.602 33567.1 21454 0.68
ZS 17640.5 8546 0.502 27894.6 13456 0.634 35673.2 24565 0.512 32122.4 26754 0.564 38746.5 20989 0.54
Bandit 22454.7 16721 0.935 26789.3 18067 0.926 32134.5 30235 0.475 35233.4 29104 0.575 57894 35567 0.545
Parsimonious 6946.7 4536 1 7076 5053 1 30456.4 14566 0.854 31245.3 15334 0.861 32176 25053 0.532
Sign Hunter 8345.6 4788 1 8814.7 5481 0.987 24578.3 12434 0.833 29876.5 13443 0.865 23454.6 18081 0.79
Square 4184.5 2784 1 4584.8 2859 1 21345.5 9865 0.852 22135.2 10334 0.875 23447.7 21345 0.786
　Model  → ResNet-50 Inception-v3 AT model FastAT(56.90) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
NES 21094.4 12114 0.633 18544.2 13268 0.574 0 0 54234.2 32368 0.342
ZS 19993.4 12234.5 0.504 19875.4 13453 0.435 0 0 54878.3 34453 0.323
Bandit 22367.8 15443 0.782 21345.6 16589 0.745 22670.4 15443 0.113 24553.4 14877 0.141 40984.3 37645 0.622
SimBA 19887.4 11234.5 0.652 23425.3 12347 0.604 21093.6 13446 0.076 21886.7 12345 0.121 39085.5 27644 0.442
Square 18944.3 14556.3 0.721 19884.3 15946.4 0.665 20448.4 13442 0.124 22010.2 11990 0.162 45875.6 38755 0.503




　Model  → ResNet-50 Inception-v3 AT model(robust library) FastAT RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
OPT 2547.5 2023 0.21 2235.6 1978 0.17 0 0 0
Sign-OPT 2399.4 1980 0.36 2458.4 1756 0.31 0 0 0
GeoDA 1239.4 240 0.8 1143.5 229 0.75 3123.5 1863 0.42 3087.5 1853 0.47 2345.6 1443 0.34
HSJA 846.3 182 0.21 953.5 225 24 0 0 0
Sign Flip Attack 1575.4 701 0.97 1627.6 845 0.97 2478.5 1755 0.69 2254.2 1355 0.71 2675.6 1265 0.79
Rays 1109.4 653 0.98 1135.6 753 0.98 2245.6 1465 0.66 2134.6 1547 0.72 2987.6 1675 0.76
　Model  → ResNet-50 Inception-v3 AT model(robust library) FastAT RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
Boundary Attack 2347.6 435 0.1 2267.3 339 0.04 0 0 0
Evolutionary Attack 2208.6 610 0.24 2156.5 598 0.21 0 0 0
OPT 1656.4 1237 0.18 1786.5 1323 0.21 0 0 0
Sign-OPT 2234.5 1498 0.41 2019.2 1598 0.37 3476.5 2245 0.11 3214.6 2323 0.18 0
GeoDA 1345.7 724 0.6 1473.5 612 0.6 1876.4 1087 0.32 1985.4 1125 0.35 18756.6 7894 0.21
HSJA 3125.3 2135 0.81 3339.6 2550 0.79 3246.5 2534 0.46 3546.7 2344 0.48 23456.8 10875 0.35
　Model  → ResNet-50 Inception-v3 AT model(robust library) FastAT RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
OPT 50893.5 48734 0.04 0 0 0 0
Sign-OPT 43567.8 42459 0.15 46334.5 43987 0.02 0 0 0
GeoDA 23098.6 15639 0.93 25635.6 14336 0.9 28634.3 15604 0.42 27984.5 14335 0.47 48756.7 34234 0.75
HSJA 41256.6 36875 0.86 42387.6 32456 0.82 37584.9 29874 0.28 36446.5 27653 0.33 60987.5 45564 0.32
Sign Flip Attack 21873.5 16072 1 22367.8 14554 1 24504 18765 0.71 23456.5 19065 0.75 43456.5 26219 0.82
Rays 19876.4 13497 1 21010.4 12908 1 22456.4 14587 0.76 22087.5 15349 0.79 48764.5 26219 0.82
　Model  → ResNet-50 Inception-v3 AT model(robust library) FastAT RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
Boundary Attack 65445 57344 0.12 74753.9 66748 0.03 0 0 0
Evolutionary Attack 23243 19844 0.09 25456.8 15854 0.04 0 0 0
OPT 59854 53225 0.11 0 0 0 0
Sign-OPT 60143.4 55096 0.22 59807.6 54347 0.13 0 0 0
GeoDA 35657.5 30346 0.54 42135.6 29856 0.48 52457.6 42456 0.29 48974.6 38596 0.3 48974.6 45316 0.21
HSJA 58946.4 56467 0.74 59684.5 53567 0.63 60984.5 53556 0.35 58946.4 58764 0.4 74946.4 67632 0.22

If you would like to report your results here, please follow instructions at Google Form to make a submission. The results will be posted after we check the model validity (expect to take about a week).