This study applied the sequence to sequence (seq2seq) learning method for recurrent neural networks (RNN) to learning for interactive robots, which respond to a human's linguistic instructions by generating appropriate behavior. This study extended the method by constructing target data not as unimodal language sequences, but as multimodal sequences of words, vision, and the robot's joint angles. By using them to train the RNN, the robot can acquire the ability to deal with interactive tasks online. In this scheme, not only the relationships between instructions and corresponding behaviors but also the task progression pattern, that is, the repetition of instruction, behavior, and waiting for subsequent instructions, can be autonomously learned from the data, so the execution of the task is achieved by continuous forward propagation alone. This proposal has the following novelty: (1) We implemented a long short-term memory (LSTM)-RNN model trained by the seq2seq method, which is mainly used in the field of natural language processing, for interactive robots in the aforementioned extended way. (2) We dealt with the logical operators "true," "false," "and," and "or," which have not been dealt with in previous studies on integrative learning of language and robot behavior in the research field called symbol emergence in robotics.