Both time-domain and gravitational-wave (GW) astronomy have gone through a revolution in the last decade. These two previously disjoint fields converged when the electromagnetic (EM) counterpart of a binary neutron star merger, GW170817, was discovered in 2017. However, despite the discovery rate of GWs steadily increasing, by several folds in each observing run of the LIGO/Virgo GW instruments, GW170817 remains the only success story of EMGW astronomy. While future GW detectors will detect even larger number of events, this does not guarantee corresponding increase in the number of EM counterparts discovered. In fact, the growing number is overwhelming since wide-field telescope surveys will have to contend with distinguishing the optical EM counterpart, called a kilonova, from the ever increasing number of ``vanilla'' transients objects they encounter during a GW follow-up operation. To this end, we present a novel tool based on a temporal convolutional network (TCN) architecture for Electromagnetic Counterpart Identification (El-CID). The overarching goal of El-CID is to slice through list of objects that are consistent with the GW sky localization, and determine which sources are consistent with kilonovae, allowing limited and judicious use of telescope and spectroscopic resources. Our classifier is trained on sparse early-time photometry and contextual information available during discovery. Apart from verifying our model on an extensive testing sample, we also show succesful results on real events during the previous LIGO/Virgo observing runs.