Automatically generated emotion arcs that capture how an individual or a population feels towards a product or entity overtime are widely used in industry and across research disciplines. However, there is little work on evaluating the generated arcs. This is in part due to the difficulty of establishing the true (gold) emotion arc. Our work, for the first time, systematically and quantitatively evaluates automatically generated emotion arcs. We also compare two common ways of generating emotion arcs: lexicon-based methods and machine learning models for sentiment analysis. Along the way, we systematically study the relationship between the quality of an emotion lexicon and the quality of the emotion arc that can be generated with it. We show that despite being markedly poor at instance-level, the lexicon-only method has extremely high predictive power when it comes to aggregating information from hundreds of instances and generating emotion arcs. This work has wide-spread implications for commercial development, as well as research in psychology, public health, digital humanities, etc. that values simple interpretable methods without the need for domain-specific training data, programming expertise, and high-carbon-footprint neural models.