Scargill Castle was originally a late 12th or early 13th century stone fortified manor house, founded by Warren de Scargill. A defensible tower stood in the south-east angle of a small walled courtyard, with several other ranges and a hall opposite the western gateway. An adjacent southern rectangular platform, supported a walled barmkin, which gave protection to the village during border raids. By the early 14th century the de Scargill family had abandoned the castle and moved south to the manor of Saddleworth in the West Riding. In 1531 the castle and manor of Scargill passed of Marmaduke Tunstall, on his marriage to Mary Scargill and he gave the castle a fortified façade. When adding a new Tudor hall and flanking the high curtain wall of the courtyard, with a three storey gabled undefended gatehouse. Sadly by the late 17th century the castle had once again been abandoned, however there is still a story of a subterranean passage, which leads to Egglestone Abbey. The gatehouse and its projecting semi-circular stair tower, are now adjoined by fragmentary lengths of curtain wall and on the site of the old hall, are the remains of a magnificent Tudor fireplace, which once graced the great hall. 4 miles north-west is Bowes Castle and 11 miles south-east is Richmond Castle.