Brougham Castle was originally a stone three storey keep and courtyard fortress, founded by Robert de Vieuxpont. The 13th century keep with its range of service buildings, stands on the angle of the Roman fort of Brocavum and was encased by a timber palisade. In the 14th century, Robert Clifford added a third floor to the keep, an inner and outer gatehouse and a stone curtain wall, flanked by the Tower of League. In the 17th century, Lady Anne Clifford restored the castle and converted it into a country mansion. A mile south-west is Brougham Hall and 2 miles north-west is Hutton Hall.