The Swan House, with its soaring columns and manicured gardens, stands as an iconic symbol of Buckhead’s elegance and Southern grandeur. For decades, it has captivated visitors as the quintessential monument to one of Buckhead’s founding families and their vast fortune. But to focus solely on its architectural splendor is to miss the far more compelling and complex story of the family who built it—and one remarkable woman who worked within its walls.
The Inman family narrative, particularly when viewed through the lives of wife Emily, and their maid, Lizzie McDuffie, reveals a deeper, more nuanced history of Buckhead. This is a story that goes beyond opulence, exploring the intricate social fabric, the seeds of philanthropy, and the unexpected connections to national movements for equality that were woven into the very foundations of this community.
The story begins with Edward Inman, the heir to a vast cotton brokerage fortune. In the early 20th century, he was a pivotal figure in Atlanta’s commercial landscape, with business interests spanning real estate, transportation, and banking. He was a partner in the Kimball House Company, a founder of the Atlanta Woolen Mills, and a director of the First National Bank of Atlanta. His civic contributions were equally significant, serving on the Atlanta City Council and the Fulton County Board of Commissioners. Inman was known for his “disdain of government extravagance and his concern for the poor,” a reputation that cemented his role as a key player in the city’s development. The Inman family’s philanthropy was a cornerstone of their identity, contributing to institutions that are still cornerstones of Atlanta today, including Georgia Tech, Agnes Scott College, and Grady Memorial Hospital.
But if Edward was the engine of the family’s economic and civic influence, his wife, Emily, was the driving force behind its social and cultural legacy. A woman of extraordinary independence and vision, Emily Inman came from a family of strong-willed women. Her mother was a founder and president of the Equal Suffrage Party of Georgia, a movement in which Emily actively participated. Emily was a shrewd and capable investor, tending to her own estate with a remarkable flair. She famously once stated that her financial independence allowed her to “live the kind of life she wanted to live,” a profound declaration for the time when women had not yet won the right to vote.
This independence was never more evident than in the creation of the Swan House. While Edward’s name is often associated with the mansion, it was Emily who was the true “decision maker.” She collaborated closely with renowned architect Philip Trammell Shutze and interior designer Ruby Ross Wood, guiding every detail to create a home that was both a testament to Southern classical architecture and a sanctuary for her family. Following Edward’s untimely death in 1931, Emily became the sole steward of the family’s wealth and the Swan House itself. She lived in the mansion until her death in 1965, at which point she orchestrated its sale to the Atlanta Historical Society for a half-million dollars—an act of preservation that ensured her home, and a piece of Buckhead’s history, would be accessible to the public forever.
Beneath the grandeur of the Swan House, a different, equally compelling narrative was unfolding on the third floor and in the back servants’ quarters. This was the world of the domestic staff, a world meticulously organized by the unspoken rules of the Southern social hierarchy. The Swan House employed a staff of four to seven servants, including cooks, maids, a butler, a chauffeur, and a governess. Their lives, while integrated into the daily rhythms of the Inman family, were spatially and socially distinct. They lived in quarters on the third floor or above the six-car garage, accessed by a back staircase, and their days were governed by a strict schedule that allowed for little time off.
This was the world Elizabeth “Lizzie” McDuffie knew intimately. Born in Covington, Georgia to formerly enslaved parents, Lizzie began her service with the Inman family as a nursemaid and maid around 1901. For 32 years, she was a part of their household, experiencing firsthand the racial and social dynamics of early 20th-century Atlanta. She lived through a time when the Atlanta Washerwomen’s Strike of 1881 had shown the power of organized Black labor, and the devastating Atlanta Race Riot of 1906 had underscored the city’s deep racial tensions. These experiences, combined with the daily realities of domestic work in a wealthy white household, would profoundly shape her worldview.
While specific anecdotes about her time with the Inmans are scarce, the physical layout of the Swan House tells its own story. The back staircases, the annunciator call-board in the maid’s room, and the separate living quarters all serve as silent testaments to the rigid social order. It was a world of subservience, but as historian Catherine Stewart notes, domestic workers found ways to assert their agency, whether through refusing tasks or eschewing honorifics like “ma’am.” It was within this environment that Lizzie McDuffie would cultivate the quiet strength and determination that would fuel her future.
In 1933, as the Great Depression deepened, Lizzie McDuffie left the Inman family’s service and embarked on an extraordinary new chapter. Through her husband’s position as the president’s personal valet, she was offered a position as a maid on the third floor of the White House under President Franklin D. Roosevelt and First Lady Eleanor Roosevelt.
It was here that Lizzie McDuffie’s true calling as a political activist blossomed. She quickly became an invaluable and trusted figure to the Roosevelts, especially to Eleanor, who saw in her a direct line to the concerns of ordinary African Americans. Lizzie used this unique access to become a powerful voice for racial equality. She was a tireless advocate for her community, regularly bringing the issues of racial injustice, employment discrimination, and segregation directly to the First Lady. So relentless was she in her advocacy that she took to calling herself FDR’s “SASOCPA”—his “self-appointed secretary on colored people’s affairs.” The president, in turn, affectionately called her “Doll.”
Lizzie’s influence was not limited to behind-the-scenes conversations. She became a vocal champion for the rights of her fellow domestic workers, leading a successful campaign to form the United Government Employees, a union for lower-paid workers in the White House. With Eleanor Roosevelt’s apparent blessing, she organized her colleagues and fought for fair wages and better working conditions. In 1936 and 1940, she even took to the campaign trail, traveling with the Roosevelts and speaking to African American audiences, urging them to vote for FDR. Lizzie was at Warm Springs Georgia with the president when he died on April 12, 1945 and rode the funeral train to Washington.
Her activism was rooted in her own lived experience. In 1934, she testified before a government panel on behalf of laundry workers, decrying the low wages and unfair treatment that mirrored the struggles she had witnessed and endured. She spoke with a clear and powerful voice, stating that she was “a true friend of the Negro race without paternalism” and that she was “not an automaton.”
The stories of Edward, Emily, and Lizzie offer a far more compelling historical portrait of Buckhead than the simple narrative of a wealthy family and their magnificent home. The Inman family’s influence went beyond the grand estates and society pages, leaving a legacy of civic engagement and philanthropy that continues to define Atlanta. At the same time, the life of Lizzie McDuffie provides a powerful, often-overlooked counter-narrative, revealing the complex human stories and social hierarchies that coexisted alongside the grandeur.
The Swan House is more than just a monument to architectural history; it serves as a historical record of the lives lived within its walls. It stands as a testament to the pioneering spirit of Emily Inman, who defied societal expectations to become a financial leader and cultural preservationist. Furthermore, it offers a crucial starting point for comprehending the extraordinary trajectory of Lizzie McDuffie, whose experiences in Buckhead shaped her into a national advocate for social change. Her story illustrates that a person of any means or background is capable of great impact in the world.