Hermitage Castle is a stone multi-towered castle, the bleak surroundings give it an extraordinary atmosphere. Fragments of the early castle lie at the centre of the building, round arched doorways, large ashlar blocks in the spiral stairs give its position away. Built up to become the main central keep, with towers added at each corner, it became the largest and strongest castle on the Border. The east and west sides of the towers are joined at the top, by an arched curtain, so that a wooden hoarding could encompass the castle. 16 miles to the north is Hawick Castle and 6 miles to the south is Liddle Castle.