In the early eighteenth century the house was the seat of Richard Bulkeley. During the English Civil War, Richard Bulkeley’s successor, Colonel Thomas Bulkeley apparently invited King Charles I to take possession of the house and set up his court there.

The house was reconstructed in 1776 by architect Samuel Wyatt, with a curved façade, terraces and balconies. An ice house and lodge were also constructed in the gardens. In the nineteenth century the occupants of Baron Hill remained the dominant Anglesey landowners, possessing estate also at Llanfairfechan and other parts of Caernarfonshire.

During World War I, death duties exhausted the family fortune and made it impossible for the family (by then known as the Williams-Bulkeley’s) to maintain the house. In World War II, it was requisitioned by The Royal Engineers and used as temporary housing for Polish soldiers. They found the home too cold and decided to start a fire within the mansion so they would be moved to new accommodation. The fire destroyed a large part of the interior and the soldiers achieved their aim and were removed from the house but only to tents on the estate. 

The mansion was then abandoned and just the shell of the house now survives.