Generative capabilities of large language models have grown beyond the wildest imagination of the broader AI research community, leading many to speculate whether these successes may be attributed to the training data or different factors concerning the model. At the same time however, LLMs continue to exhibit many shortcomings, which might contain important clues to understanding their behavior as well as attribution. I will present some work from my group which has revealed unique successes and shortcomings in the generative capabilities of LLMs, on knowledge-oriented tasks, tasks with human and social utility and tasks that reveal more than surface-level understanding of language. I will end with a brief discussion of the implications for attribution in the peculiar domain that natural language occupies.