Thomas Holy Suckley & his wife Catherine bought the land on the Hudson River in 1852 and called it Wilderstein, meaning "wild man’s stone". They had John Warren Ritch design an Italianate villa.