This code repository comprises a series of fundamental Deep Neural Network (DNN) experiments designed to assist you in comprehending and mastering the basic concepts of deep learning. Additionally, we have also verified two crucial phenomena: the Frequency Principle and the Condensation Phenomenon.
DNNs often fit target functions from low to high frequencies.
The first figure shows the evolution of the function in spatial domain, the red line is the target function, and the blue line is the DNN output. Ordinate vs. Abscissa : y vs. x.
The second figure shows the evolution of the function in Fourier domain, the red line is the FFT of the target function, and the blue line is the FFT of DNN output. Ordinate vs. Abscissa: amplitude vs. frequency.
-
DNN 1D
-
DNN nD
[1] Zhi-Qin John Xu* , Yaoyu Zhang, and Yanyang Xiao, Training behavior of deep neural network in frequency domain, arXiv preprint: 1807.01251, (2018), 26th International Conference on Neural Information Processing (ICONIP 2019). pdf and web
[2] Zhi-Qin John Xu* , Yaoyu Zhang, Tao Luo, Yanyang Xiao, Zheng Ma, Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks, arXiv preprint: 1901.06523, Communications in Computational Physics (CiCP). pdf and in web, some code is in github
[3] Zhi-Qin John Xu*, Yaoyu Zhang, Tao Luo, Overview frequency principle/spectral bias in deep learning. arxiv 2201.07395 (2022) . pdf, and in arxiv.
For more details, refer to Zhi-Qin John Xu's homepage
The above picture is the illstration of the condensation.
[1] Tao Luo#, Zhi-Qin John Xu #, Zheng Ma, Yaoyu Zhang*, Phase diagram for two-layer ReLU neural networks at infinite-width limit, arxiv 2007.07497 (2020), Journal of Machine Learning Research (2021) pdf, and in arxiv.
[2] Hanxu Zhou, Qixuan Zhou, Tao Luo, Yaoyu Zhang*, Zhi-Qin John Xu*, Towards Understanding the Condensation of Neural Networks at Initial Training. arxiv 2105.11686 (2021) pdf, and in arxiv, see slides and video talk in Chinese, NeurIPS2022.
For more details, see xuzhiqin condense
If you have any questions or suggestions, you can reach us via email at:
Zhi-Qin John Xu : [email protected]
Zhongwang Zhang : [email protected]
Zhangchen Zhou : [email protected]