The Semmes family were early settlers of Madison County, having taken a migration path from Charles County, Maryland that included Wilkes County and Savannah, in Georgia. Included among its family members in Mississippi and elsewhere, were physicians, merchants, and at least one Catholic priest.
A biography of the Semmes family can be read here.