The Mansion began as a more humble home built in 1884, by Eli Sims Shorter II (1858 - 1908) and his wife Wileyna Lamar (1861 - 1927) of Macon, Georgia. An extensive renovation, began in 1901 and lasting through 1906, transformed it into the outstanding Greek Revival Mansion it is today, enabling it to be listed on the National Register. Eli and Wileyna had three children and were well respected in the community.
The Mansion was a social center for the small town. After Eli's death, just two years after the renovation, Wileyna continued living there until 1927, when the home was passed to daughter Fannie S. Upshaw. The home was passed to her daughter, Wileyna S. Kennedy, in 1959. Because of health reasons, Wileyna's family moved away from Eufaula and the Mansion suffered neglect.
In 1965, Wileyna put the home up for public auction. A group of civic-minded citizens organized the Eufaula Heritage Association and were able to buy the Mansion at this auction for $33,000. A state grant of $75,000 was awarded the Heritage Association to honor six governors of the State of Alabama from Barbour County. This grant was used for the purchasing of antique furnishings for the Mansion and making it a house museum and cultural center open to the public. The home was refurnished by a decorator and donations. Plans were soon in place for an annual Pilgrimage tour of other historic homes in Eufaula to offset the cost of maintaining the house and to have the resources necessary to pursue an active preservation effort.
The first Eufaula Pilgrimage was held April 11 - 17, 1966.
David Kennedy, great grandson of the builder, continues to be involved with the Mansion, having shared memorabilia and memories that add to the overall history of this great place, home to many weddings, receptions, teas, and other special occasions.