Robert Carmichael is the eldest child of William Howland and Margaret Carmichael Howland. He is one of three siblings and their only son. He has two younger sisters, Nina and Christine. In addition to his sisters, he was raised alongside his father's granddaughter, Abigail. Robert's parents came from very different backgrounds. William was descended from a wealthy family. The Howlands were influential and provided the cornerstone for the local economy. Margaret was from a less fortunate family that hardly missed her when she left. The most important difference was that William was white and Margaret was black. Although her entire family was legally free, people of color were still looked down upon and interracial marriage was prohibited. Margaret went to the Howland estate to work as a housekeeper but quickly became a mistress, then, just before the birth of Robert, William's secret wife. Due to their pairing, Robert spent his younger years on the Howland estate and was known simply as Margaret's child, and was given her surname, Carmichael.