Thomas Bailey Aldrich was a child when his father moved to New Orleans from Portsmouth, New Hampshire. After 10 years, Aldrich was sent back to Portsmouth to prepare for college. This period of his life is partly described in his semi-autobiographical novel The Story of a Bad Boy (1870), in which "Tom Bailey" is the juvenile hero. Critics have said that this novel contains the first realistic depiction of childhood in American fiction and prepared the ground for Adventures of Huckleberry Finn. Aldrich went on to associate with many of the literati of his time in New York City, and was editor of the Atlantic Monthly in the 1880's.

His son died of Tuberculosis in Saranac lake, NY, and the family house is now a Bed and Breakfast there. The Aldrich house in Portsmouth is part of the Strawberry Banke Museum in Portsmouth, New Hampshire.