BlackboxBench

CIFAR-10 Leaderboard

    We set the maxium queries to be 10000 on all tests and the attack budget will be set uniformly by

    CIFAR: l_inf：0.05 = 12.75/255, l_2: 1 = 255/255



　Model  → VGG-16 ResNet-50 WideResNet-28 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
NES 267.3 180 1 395.2 190 0.998 465.5 210 0.988 711.5 280 0.218 821.6 330 0.193 843.5 445 0.94
ZS 257.4 155 1 421.5 186 1 481.2 195 1 634.5 195 0.162 614.9 217 0.138 935.2 545 0.84
Bandit 111.3 54 1 156.3 63 0.988 210.2 72 0.97 1498.2 418 0.384 1451.5 444 0.367 213.6 134 0.523
Parsimonious 207.5 146 1 380.5 183 1 457.8 201 0.997 1119.6 320 0.543 1180.4 359 0.516 875.7 346 0.312
Sign Hunter 106.8 57 1 152.3 68 1 167.6 74 0.997 716.6 229 0.557 766.3 241 0.524 875.7 346 0.312
Square 64.7 26 1 78.6 26 0.986 90.2 28 0.965 1085.2 276 0.534 1135.4 293 0.501 734.6 232.6 0.812
　Model  → VGG-16 ResNet-50 WideResNet-28 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
NES 374.8 270 1 685.3 345 0.988 729.1 360 0.975 1093.1 612 0.256 1155.4 630 0.235 1345.6 570 0.91
ZS 340.1 275 1 596.3 255 0.995 675.3 240 0.989 400.3 374 0.172 397.5 372 0.144 1853.5 894 0.85
Bandit 391 186 1 539.8 248 0.986 619 262 0.967 2078.4 1286 0.334 2163.2 1318 0.312 1656.4 645 0.385
SimBA 308 136 0.994 426.8 185 0.989 457.2 190 0.96 1487.4 987 0.378 1523.2 1013 0.365 1355.6 678 0.456
Square 407.1 160 0.997 586.3 185 0.982 640 200 0.975 1903.6 759 0.341 1935.4 766 0.323 1453.5 345 0.39
　Model  → VGG-16 ResNet-50 WideResNet-28 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
NES 2045.2 1023 1 2435.6 1365 0.978 2134.5 1509 0.897 0 0 10884.5 7567 0.75
ZS 1983.5 972 1 2234.5 1056 0.982 22134.6 1298 0.887 0 0 12357.3 7834 0.76
Bandit 846.6 454 1 1098.4 632 1 1456.5 985 0.911 0 0 7865.4 2345 0.351
Parsimonious 964.5 589 1 1174.6 678 1 1345.2 1098 0.915 5556.8 2989 0.086 5876.5 2768 0.053 12567.4 7665 0.122
Sign Hunter 738.7 432 1 604.5 445 1 1098.4 765 0.917 8745.7 3298 0.092 7953.7 3064 0.064 10056.7 3245 0.662
Square 598.4 356 1 604.5 389 1 990.4 754 0.911 9253.4 3985 0.112 1135.4 3867 0.076 12574.4 4233 0.687
　Model  → VGG-16 ResNet-50 WideResNet-28 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
NES 3198.4 1365 0.917 2988.7 1236 0.899 3124.5 1345 0.845 0 0 31224.6 13645 0.723
ZS 3245.6 1456 0.932 3123.6 1546 0.905 3213.7 1455 0.835 0 0 32913.7 18455 0.715
Bandit 2987.4 1204 0.921 3098.4 1356 0.914 3234.5 1634 0.897 0 0 34234.5 16934 0.421
SimBA 3234.5 938 0.873 2987.6 865 0.834 2987.7 1232 0.806 8765.4 3009 0.05 8865.6 2897 0.04 32954.7 21232 0.523
Square 2768.9 875 0.923 3123.4 992 0.918 2675.5 967 0.899 9866.5 4566 0.07 9657.7 4355 0.05 20675.5 5967 0.36

Note: Two pretrained AT models, including AT models(ddpm) and AT model(optimization trick), are downloaded from Adversarial Robustness, and the download links are AT model(optimization trick)[1] and AT model(ddpm)[2]

[1]Gowal, S., Qin, C., Uesato, J., Mann, T., & Kohli, P. (2020). Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593.

[2]Rebuffi, S. A., Gowal, S., Calian, D. A., Stimberg, F., Wiles, O., & Mann, T. (2021). Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.01946.



　Model  → VGG-16 ResNet-50 WideResNet-28 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
OPT 2103 1399 0.81 2100 1353 0.77 2287 1542 0.76 2203 1132 0.25 2241 1356 0.23 0
Sign-OPT 1803 1561 0.96 1803 1560 0.96 1803 1561 0.96 1878 1603 0.31 1833 1647 0.23 0
GeoDA 760 347 0.96 786 222 0.91 832 376 0.95 811 332 0.49 833 401 0.51 534.7 345 0.454
HSJA 780 557 1 1906 539 0.68 878 557 0.99 2501 1489 0.65 2576 1502 0.69 3370.8 1874 0.347
Sign Flip Attack 200 120 1 203 120 1 200 120 1 1102 675 0.77 1217 691 0.76 564.7 345 0.46
Rays 510 338 1 512 339 1 510 338 1 2216 1301 0.79 2171 1245 0.79 876.4 435 0.75
K)
　Model  → VGG-16 ResNet-50 WideResNet-28 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
Boundary Attack 2147 1766 0.06 2103 1566 0.05 2038 1496 0.05 2002 1277 0.02 2103 1327 0.02 0
Evolutionary Attack 1814 1033 0.24 1877 1115 0.24 1933 1211 0.26 1906 1168 0.21 1947 1265 0.20 0
OPT 1701 1176 0.79 1688 1145 0.76 1719 1200 0.77 1711 1187 0.2 1787 1108 0.19 0
SignOPT 1317 1061 0.98 1367 1225 1 1426 1282 1 1421 1089 0.32 1429 1131 0.31 0
GeoDA 1355 575 0.66 1381 580 0.66 1444 580 0.69 1419 601 0.36 1461 603 0.36 2341.5 1074 0.21
HSJA 1220 974 1 2692 1557 0.9 1208 961 1 3172 2088 0.52 3087 2101 0.53 3189.5 1873 0.43
　Model  → VGG-16 ResNet-50 WideResNet-28 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
OPT 6456.7 5302 0.48 6536.6 5123 0.42 6453.3 4987 0.4 0 0 0
Sign-OPT 5553.5 4976 0.59 5257.4 4872 0.56 5442.3 4762 0.51 0 0 0
GeoDA 2167.5 1356 1 2346.4 1223 1 2254.6 1313 1 18784.3 5234 0.287 17636.4 5230 0.247 4568.7 3466 0.785
HSJA 3731.4 3197 0.9 3890.2 3209 0.88 3878.4 3280 0.86 18745.2 4645 0.198 18637.2 4829 0.182 14567.4 7896 0.666
Sign Flip Attack 1907.4 1343 1 2083.4 1298 1 2124.5 1283 1 16830.3 5534 0.298 16894.4 5345 0.262 9876.7 3475 0.754
Rays 1897.6 1274 1 1982.4 1254 1 2014.6 1278 1 15235.5 5323 0.308 16534.5 5125 0.287 8965.5 3423 0.786
　Model  → VGG-16 ResNet-50 WideResNet-28 AT mode(ddpm) AT model(optimization trick) RND
Blackbox  Attack↓ average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR average number medium number ASR
Boundary Attack 9084.5 6734 0.09 0 0 0 0 0
Evolutionary Attack 10984.5 8270 0.26 12381.3 8654 0.22 12676.5 9884.3 0.18 35354.5 23455 0.1 32356.5 24311 0.11 0
OPT 19846.5 10873 0.73 18743.4 11823 0.71 19846.5 10873 0.69 36554.3 25466 0.24 37233.3 25565 0.25 0
SignOPT 18945.6 13245 0.77 15874.6 12356 0.75 18739.5 12434 0.77 18739.5 12434 0.31 19834.6 13676 0.33 0
GeoDA 13364.4 8934 0.64 16535.6 7896 0.63 13456.5 8732 0.6 17586.4 12343 0.41 18585.6 13454 0.43 42345.7 34553 0.32
HSJA 20945.6 17345 0.78 21345.6 17636 0.74 21234.5 14556 0.71 39534.5 23145 0.59 38737.5 24355 0.6 54345.5 42356 0.43