Causal representation learning has been proposed to encode causal relationships between factors presented in the high dimensional data. Existing methods are limited to being trained and fully supervised by ground-truth generative factors. In this paper, we seek to reduce supervision strength by leveraging intervention on either the cause factor or effect factor for reducing supervision strength. Applying interventions on cause factors and effect factors will lead to different results since intervention on effect factors will change the causal graph. In contrast, intervention on cause factors will not change the relationships. The intervention can also be called \emph{do-operation}. Based on this attribute of \emph{do-operation}, we propose a framework called Do-VAE, which implements \emph{do-operation} by swapping latent cause factors and effect factors encoded from a pair of inputs and utilizing the supervision signal from a pair of inputs by comparing original inputs and reconstructions. Moreover, we also identify the inadequacy of existing causal representation metrics and introduce new metrics for better evaluation.