Wormegay Castle was originally a Late Saxon manorial complex, which was fortified in the 11th century, by Hermer de Ferrers. In the 12th century, Reginald de Warenne modified the fortification and founded the earthwork motte and bailey castle. The large sub circular motte, is encased by a wide ditch and still supports the slight remains of an encompassing rampart. To the east, encased by a rampart and ditch, is an extensive bailey, which supports on its northern edge, a sub rectangular building platform. 4 miles north is Middleton Castle and 11 miles south-east is Oxburgh Hall.