In this paper, we are interested in recovering the latent group structures and sparsity patterns in high dimensional panel data model. The slope coefficients of the model are subject dependent and there exists group structures where the slope coefficients are homogeneous within groups and heterogenous between groups. We emphasize that the sparsity structures may also be different across groups, so that the important covariates are different across groups. We develop a penalized approach for recovering the group structures and sparsity patterns simultaneously, in which pairwise difference of the slope parameters are shrunk to recover the group structures. Due to the high computational complexity in the pairwise penalty, we propose a strategy to prune the number of penalty terms in the objective function, which could also improve the ability of recovering group structures and achieve higher estimation accuracy. The proposed estimator is able to recover the latent group structures and the sparsity patterns consistently in large samples. The finite sample performance of the proposed estimator is evaluated through Monte Carlo studies and illustrated with a real data set.