We present a precise form of structural realism, called group structural realism, which identifies ‘structure’ in quantum theory with symmetry groups. However, working out the details of this view actually illuminates a major problem for structural realism; namely, a structure can itself have structure. This article argues that, once a precise characterization of structure is given, the ‘metaphysical hierarchy’ on which group structural realism rests is overly extravagant and ultimately unmotivated.