High-dimensional mediation analysis has gained increasing interest in various fields, particularly in genetic and medical research. Compared with existing works that focus mainly on high-dimensional mediators, this paper advocates a new statistical inference framework, named PRIME, when both exposures and mediators are high-dimensional. Estimated direct and indirect effects are established using a group-wise partially penalized least squares method, incorporating a double-layer latent factor structure. F-type and Wald tests for the high-dimensional direct and indirect effects, respectively, are advocated based on the proposed estimators. Both theoretical and numerical performance of PRIME have been carefully studied. PRIME is also applied to investigating direct effects of genetic variants on Alzheimer's disease and indirect effects of them mediated by changes in brain activity intensity.