We introduce a way to get a chatbot to improve using a unique type of reinforcement learning. We get the chatbot itself to evaluate its responses and indicate alternate responses that would be better in quality.Here both the actor and the critic are the same system. We then teacher force the better response against the utterance that was parsed to the chatbot. Our experiments show that this may be a good way to optimize a chatbots ”policy”.