We develop a deep generative model that generalizes feed-forward, rectified linear neural networks with stochastic activations. We call these models bow tie networks because of the shape of their activation distributions. Then we leverage the Pólya-gamma augmentation scheme to render the model conditionally conjugate, and we derive a block Gibbs sampling algorithm based to approximate the posterior distribution over activations and model parameters. The resulting algorithm is massively parallelizable. We show a proof-of-concept of this model and Bayesian inference algorithm on a variety of standard regression benchmarks.