Completed in 1852 in granite, Chateau-sur-Mer was built for William Shepard Wetmore in the style of a French villa. His son, George Peabody Wetmore, inherited the house in 1862, aged 16, and by the 1870s, he had remodeled it in the Second Empire style.