Spotlight
in
Workshop: All Things Attention: Bridging Different Perspectives on Attention
Hierarchical Abstraction for Combinatorial Generalization in Object Rearrangement
Michael Chang · Alyssa L Dayan · Franziska Meier · Tom Griffiths · Sergey Levine · Amy Zhang
Keywords: [ objects ] [ slot attention ] [ compositionality ] [ Symmetry ] [ abstraction ] [ binding ] [ rearrangement ] [ Combinatorial Generalization ] [ graph search ] [ hierarchy ] [ independence ]
Object rearrangement is a challenge for embodied agents because solving these tasks requires generalizing across a combinatorially large set of underlying entities that take the value of object states. Worse, these entities are often unknown and must be inferred from sensory percepts. We present a hierarchical abstraction approach to uncover these underlying entities and achieve combinatorial generalization from unstructured inputs. By constructing a factorized transition graph over clusters of object representations inferred from pixels, we show how to learn a correspondence between intervening on states of entities in the agent's model and acting on objects in the environment. We use this correspondence to develop a method for control that generalizes to different numbers and configurations of objects, which outperforms current offline deep RL methods when evaluated on a set of simulated rearrangement and stacking tasks.