Ely Tower is the northern portion of the late 11th century earthwork motte and bailey fortress of Brecon, founded by Bernard de Neufmarché. In the 12th century, the Braose family founded the stone castle when crowning the motte with a round tower and a polygonal shell keep. The site is the residence of the Bishop of Swansea and Brecon and the mound occupy's the north-east angle of its post medieval gardens. The high motte is crowned by the remains of a large shell keep, which was part of the tower complex, known as Ely's Tower. Flanking the wall is a small turret, with a deeply spurred base and a barrel-vaulted room, pierced by a small light to the east. In 1483, Henry Stafford, the second duke of Buckingham was an unconditional supporter of King Richard III but he retired here with his prisoner, John Morton, Bishop of Ely, to plot against the King. His rebellion failed and the duke was executed and Ely Tower is named after his accomplice, who fled overseas to join Henry Tudor.