Authors:
Dávid Szeghy
1
;
2
;
Mahmoud Aslan
3
;
Áron Fóthi
3
;
Balázs Mészáros
3
;
Zoltán Ádám Milacski
4
and
András Lőrincz
3
Affiliations:
1
AImotive Inc., 18-22 Szépvölgyi út, Budapest, 1025, Hungary
;
2
Department of Geometry, Faculty of Natural Sciences, ELTE Eötvös Loránd University, 1/C. Pázmány Péter sétány, Budapest, 1117, Hungary
;
3
Department of Artificial Intelligence, Faculty of Informatics, ELTE Eötvös Loránd University, 1/A. Pázmány Péter sétány, Budapest, 1117, Hungary
;
4
Former Member of Department of Geometry, Faculty of Natural Sciences, ELTE Eötvös Loránd University, 1/C. Pázmány Péter sétány, Budapest, 1117, Hungary
Keyword(s):
Sparse Coding, Group Sparse Coding, Stability Theory, Adversarial Attack.
Abstract:
While deep neural networks are sensitive to adversarial noise, sparse coding using the Basis Pursuit (BP) method is robust against such attacks, including its multi-layer extensions. We prove that the stability theorem of BP holds upon the following generalizations: (i) the regularization procedure can be separated into disjoint groups with different weights, (ii) neurons or full layers may form groups, and (iii) the regularizer takes various generalized forms of the `1 norm. This result provides the proof for the architectural generalizations of (Cazenavette et al., 2021) including (iv) an approximation of the complete architecture as a shallow sparse coding network. Due to this approximation, we settled to experimenting with shallow networks and studied their robustness against the Iterative Fast Gradient Sign Method on a synthetic dataset and MNIST. We introduce classification based on the `2 norms of the groups and show numerically that it can be accurate and offers considerable
speedups. In this family, linear transformer shows the best performance. Based on the theoretical results and the numerical simulations, we highlight numerical matters that may improve performance further. The proofs of our theorems can be found in the supplementary material a .
(More)