Bayesian Optimization with Gaussian Process (GP) and Matérn and Radial Basis Function (RBF) covariance functions is commonly used to optimize black-box functions. The Matérn and the RBF kernels do not make any assumptions about the domain of the function, which may limit their applicability in bounded domains. To address the limitation issue, we introduce a non-stationary Beta Unit Hyper-Cube (BUC) kernel, which is induced by a product of Beta distribution density functions, and allows to model functions on bounded domains. To provide theoretical insights, we provide analyses of information gain and cumulative regret bounds when using the GP Upper Confidence Bound (GP-UCB) algorithm with the BUC kernel. Our experiments show that the BUC kernel consistently outperforms the well-known Matérn and RBF kernels in different problems, including synthetic function optimization and the compression of vision and language models.