Middleton Towers was originally a 14th century fortified manor house, founded by the Scales family. In the 15th century, Thomas, Lord Scales erected a brick courtyard fortress, on the large moated platform. Sadly all that remains, is a fragment of the west range and the large three storey gatehouse, which is flanked on the angles by four storey octagonal turrets. In the 19th century, Sir Lewis Whincop Jarvis integrated the derelict gatehouse, in the construction of a Tudor style mansion. A mile south-west is Middleton Castle and 7 miles north is Castle Rising Castle.