AI-assisted drug discovery has revolutionized healthcare by accelerating virtual screening methods as compared to traditional processes. Many advanced AI models have been developed to predict and generate drug candidates, with potential applications across various diseases. However, challenges still remain in applying AI models in clinical settings. These include the lack of heterogeneity and insufficient consideration of patient-specific treatment plans. To mitigate these challenges, we propose PanRX, a cell-line-specific pancreatic cancer drug effect model using multi-modal knowledge graphs. It aims at achieving a personalized drug discovery framework by incorporating rich genetic and chemical information. We first construct a multi-modal knowledge graph dataset PanCan-DrugsGenes. It extracts textual genetic information from NCBI, mutation status from the Genomics of Drug Sensitivity in Cancer (GDSC) dataset, textual descriptions of drugs from PubChem, and chemical geometry from the PM6 dataset. Then, PanRX utilizes a geometric model to learn chemical conformation, a language model to learn textual description, and a graph neural network to fuse all information and predict the target drug effects. We verify the effectiveness of PanRX by achieving the generalization performance with very low MSE (< 0.0000) and MAE (0.0009). This work emphasizes the potential of merging knowledge graphs and deep learning in the fields of genomics and medicine, enriching the intersection of human biological expertise and AI in drug discovery and design tasks.