Gamma-ray telescopes measure the direction and energy of incoming photons, resulting in photon-count maps that contain both spatial and spectral information. A major goal when analyzing such data is to determine source-count distributions (SCDs), which characterize the brightness of point-sources too faint to be detected individually. Existing statistical and machine learning methods for this task exist; however, they typically neglect the photon energy. We present a deep learning framework able to jointly reconstruct the spectra of different emission components and the SCDs of point-source populations.In a proof-of-concept example, we show that our method accurately extracts even complex-shaped spectra and SCDs from simulated maps.