Single-photon imaging (SPI), with its singlephoton sensitivity, enables detection and imaging at long distances and extremely low-flux, offering significant potential for applications. This work presents a novel multi-branch architecture for SPI that leverages the aggregate statistics of photon detection time over N illumination periods. Our analysis reveals a reflectivity-dependent intensity distribution for normalized background noise photon histogram, motivating a dual-branch decoder that jointly reconstructs depth and reflectivity. We introduce a data-driven approach to learn the censoring mechanism inherent in the mixture probability density distribution of photon detection times. This allows us to estimate noise photon level implicitly encoded within the normalized histogram by leveraging depth information. This, in turn, facilitates effective censoring of the processed histogram for improved reflectance estimation. To further exploit the significant spatiotemporal correlations present in 3D single-photon data, we extend the architecture to a three-branch encoder. A novel block-based 3D transformer branch captures global spatiotemporal dependencies within the photon cube, while a parallel branch employs dilated convolutions for local feature extraction. A third branch captures residual information, and the outputs of all three branches are fused for the final reconstruction. Extensive experiments on simulated, public, and newly acquired multiexposure datasets demonstrate state-of-the-art performance across a range of signal-to-background ratios. Our model significantly improves the accuracy and robustness of reflectivity and depth map estimation in complex environments.