Toddington Castle is a Norman earthwork motte and bailey fortress, founded by the Tracy family. The motte, encased by a wide ditch, supports an irregular depression, which may indicate the site of buildings. To the west and south, the bailey is partially built over, with only the slightest of ramparts now remaining. The site was once the 13th century stronghold of Sir Paulinus Pegure and the name Conger Hill originates, from its use as a 16th century rabbit warren. 8 miles north-east is Higham Gobion Castle and 15 miles south is Berkhamsted Castle.