West Port House was originally a late 16th century stone L-plan tower house, founded by James Hamilton of Silvertonhill. The three storey laird's house, with a rectangular stair turrets corbelled-out in the re-entrant angle, was defended by gunloops and shot-holes. The house was lowered when absorbed into a 18th century rubble-built town house, crowned by crow-stepped gables and swept dormers on a steeply pitched roof. The lowering of the High Street and the addition of a retaining wall two centuries ago, added extra elevation to the house. Nearby is Linlithgow Palace.