Non-linear Random Matrix Theory (RMT) has recently emerged as a powerful paradigm for the theoretical understanding of deep learning theory. Throughout recent works, a universality principle, the \textit{Gaussian Equivalence Theorem} (GET), has become an indispensable tool allowing for the behavior of complex nonlinear neural networks to be understood through tractable linear kernel models. This thesis contributes to this emerging field, first by using the GET universality principle to derive a novel scaling law in Neural Tangent Kernel (NTK) regression, and second by studying the implications of this idealized linear equivalence on a high-dimensional nonlinearly separable dataset.

We propose a novel framework for solving nonlinear partial differential equations (PDEs) using sparse radial basis function (RBF) networks. Sparsity-promoting regularization is employed to prevent over-parameterization and reduce redundant features. This work is motivated by longstanding challenges in traditional RBF collocation methods, together with the limitations of physics-informed neural networks (PINNs) and Gaussian process (GP) approaches, and aims to combine their respective strengths in a unified framework. The theoretical foundation of our approach lies in the function space associated with reproducing kernel Banach spaces (RKBSs) induced by one-hidden-layer neural networks of possibly infinite width. We prove a representer theorem showing that the sparse optimization problem in the RKBS admits a finite-dimensional solution, and establish error bounds that provide a foundation for extending classical numerical analysis to this setting. We also discuss the function-space characterization of sparse RBF networks, including connections with Besov spaces that are largely independent of the particular radial kernel. On the computational side, the method is implemented through an adaptive three-phase algorithm combining feature selection, second-order optimization, and pruning of inactive neurons. The explicit kernel-based structure further enables efficient quasi-analytical evaluation of differential and nonlocal operators, including fractional Laplacians. Numerical experiments on PDE benchmarks demonstrate the effectiveness of the proposed method, its ability to produce accurate sparse representations, and its potential advantages over related GP- and PINN-based approaches. This work opens new directions for adaptive PDE solvers grounded in rigorous analysis with efficient, learning-inspired implementation.