The earliest record from 1137 gives the owner of the estate as Lambert de Scoteni. Roger Ashburnham is credited with building the castle c.1378-80.
Construction of the castle began as a roughly rectangular fortified house with towers in each corner. The original plan may never have been finished, and by 1558 it is likely only the southern tower remained. In 1580 the south wing was rebuilt in Elizabethan architecture style, and around 1630 the eastern range was rebuilt in three story Inigo Jones style. The Elizabethan wing remained a bailiff's residence until 1905, but the eastern range was partly dismantled on the completion of the new house in 1843, leaving the ruin as a garden feature