Recently, paired single-cell sequencing technology has allowed the measurement of multiple modalities of molecular data simultaneously, at single-cell resolution. Along with the advances in these technologies, many methods have been developed aiming at integrating these paired single-cell multi-omics data have been developed. However, how to incorporate prior biological understanding of the properties of data into the existing model remains an open question in the field.Here, we propose a novel probabilistic learning framework that explicitly incorporates the conditional independent relationships between multi-modal data as a directed acyclic graph using a generalized hierarchical variational autoencoder. We show that our method can identify cell clusters that might be of interest. We anticipate our proposed framework could help construct flexible graphical models that reflect biological hypotheses with ease and unravel the interactions between different biological data types, such as different modalities of paired single-cell multi-omics data.