Wilton Castle was possibly an early 12th century earth and timber fortress, founded by Hugo de Longchamp. In the 14th century, the de Grey family founded an irregular stone castle, encased by a roughly rectangular moat. The curtain walls of the quadrilateral court are flanked the angles and the east wall, by round towers and a later octagonal tower. In the 16th century, Charles Brydges demolished the large south-west gatehouse tower, the south-east angle tower and the south curtain wall when building an Elizabethan mansion. Attacked and burnt during the Civil War, the west end of the ruined mansion, was restored back into a dwelling house in the 19th century. 5 miles south is Goodrich Castle.