Agent-based models (ABMs) are discrete simulators comprising agents that can act and interact in a computational world. ABMs are relevant across several disciplines as these agents can be cells in bio-electric networks, humans in physical networks, or even AI avatars in digital networks. Despite wide applicability, research in ABMs has been extremely fragmented and has not benefited from modern computational advances, especially automatic differentiation. This paper presents AgentTorch: a framework to design, simulate, and optimize agent-based models. AgentTorch definition can be used to build stochastic, non-linear ABMs across digital, biological, and physical realms; while ensuring gradient flow through all simulation steps. AgentTorch simulations are fully tensorized, execute on GPUsand can range from a few hundred agents in synthetic grids to millions of agents in real-world contact graphs. The end-to-end differentiability of AgentTorch enables automatic differentiation of simulation parameters and integration with deep neural networks (DNNs) in several ways, for both supervised and reinforcement learning. We validate AgentTorch through multiple case studies that study cell morphogenesis over bio-electric networks, infection disease epidemiology over physical networks and opinion dynamics over social networks. AgentTorch is designed to be a viable toolkit for scientific exploration and real-world policy decision-making. We hope AgentTorch can help bridge research in AI and agent-based modeling.