Rough Point was built over a five years period starting in 1887, in the style of an English Manor. The owner, Frederick William Vanderbilt, sold it in 1906 to William Bateman Leeds who left it to his wife when he died in 1910. In 1922, she sold it to James Buchanan Duke, of Duke University fame. On his death in 1925, he left the house and his entire fortune to his 12 year old daughter, Doris.