Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method to adapt large foundation models on downstream tasks. However, the weight updates are constrained to be low-rank structures, limiting their expressiveness. Alternatively, low-displacement rank (LDR)-based structured matrices are rank unrestricted, while requiring fewer parameters and supporting fast matrix-vector multiplication. We propose a new PEFT strategy to construct the weight updates with block-wise LDR matrices by sampling parameters from a hyper network framework. Our method, hyper low-displacement rank adaptation (HyDRA), offers high flexibility for choosing the size of a pool of trainable parameters, while not being restricted by the displacement rank. Our experiments demonstrate that the HyDRA can boost the classification accuracy by up to 3.8% and achieve two-fold improvement in parameter efficiency on an image classification benchmark compared with other PEFT variants.