This castle was destroyed in the mid-17th century during Cromwell's conquest. Later the Orpen family buit a mansion house here, within the remains of the old castle.

In 1847, Sir Richard Orpen Townsend demolished the earlier house and the remains of the castle, replacing it with a fine 5-bay 2-storey Scottish-Baronial style house of which we see the remains today. It had 27 rooms, a circular 3-storey battlemented corner tower on the southeast corner and a 3-storey corbelled circular turret on the east corner.

In 1921, during the Irish War of Independence, Ardtully House was burned down by the IRA. It was never rebuilt.