First of all, the question asked is what basic knowledge is needed to understand the mathematical derivation of the paper on deep learning theory.

The answer to this question is actually very broad. Here I assume that the subject has mastered the theoretical foundations of fractions, advanced algebra, probability theory, necessary statistical knowledge and Bayesian aspects. In addition, in recent years, some papers on ICLR and ICML have used more and more profound mathematical knowledge, including but not limited to real variables, functionals, point set topology, differential geometry, and abstract algebra. If you have an engineering background, you will often get confused when you look at people’s definitions of a bunch of Greek letters and a bunch of fancy letters. If you want to catch up on some mathematical foundations, you don’t know where to start.

My personal experience is that it is unrealistic and a waste of time for engineering students majoring in deep learning to learn all these courses. However, even engineering students are recommended to learn the following:

Real variable function (this is the part that appears the most in papers. At least you must know what is a measurable set and what is an unmeasurable set, What is integrability, further Riemann integrability and Lebesgue integrability; understand the concept of measure)

Functional, calculus of variations (this course is really difficult, I only learned it Part of it, but to do machine learning, you must know the variational Euler-Lagrange equation)

Basic topological concepts (everyone likes to use the word manifest in current papers, as long as it talks about high-dimensional data Just the manifest, the source is here. Another example is the proof of the perfect classifier in WGAN, which is actually a very basic proof of metric space and Hausdorff space in the textbook)

Some basic metric knowledge (appears in the paper) There are also a lot of them, but I feel that knowing metric tensors, exponential mappings, and geodesic equations is enough. Some deeper geometric concepts rarely appear)

As for deeper mathematics, I also Never learned. For example, I was confused about this year's ICLR article on spherical CNN, but the above is basically enough for you to look at most in-depth theoretical articles from a high-level perspective.