Boyd was the son of William Boyd, 4th Earl of Kilmarnock (1705-1746), and Anne Livingston (d. 1747).
He fought on the Jacobite side with his father at the Battle of Culloden in 1746, while his elder brother James (1726-1778) and twin-brother William (1728-1780) both fought on the Government side. Their father was captured and beheaded at the Tower of London, while Charles escaped and went into hiding on Arran before going into exile in France for about 20 years until pardoned. While waiting for his execution at the Tower, the 4th Earl wrote to his eldest son, James, the following about Charles who had by then gone into hiding:
Use all your interest to get your brother pardoned and brought home as soon as possible, that his circumstances, and the bad influence of those he is among, may not induce him to accept of foreign service, and lose him both to his country and his family. If money can be found to support him, I wish you would advise him to go to Geneva, where his principles of religion and liberty will be confirmed, and where he may stay till you see if a pardon can be procured for him. 1
Following his return to Scotland, Boyd lived for the most part of his time in Aberdeen and at Slains Castle north of Aberdeen with his older brother and their respective wives and children. According to Boswell, during his stay on Arran in 1745-46, Boyd "had found a chest of medical books left by a surgeon there, and had read them till he acquired some skill, in consequence of which he is often consulted."
He married, firstly, Jeanne Antoinette Wyandt, a French lady whom he met during his exile, with whom he had at least two children. He married, secondly, Anne Lockhart, a sister of his brother's first wife and daughter of Alexander Lockhart of Covington.
- 1(Source: Historic Earls and Earldoms of Scotland)
Charles Boyd and Boswell were fourth cousins, sharing a common ancestor in William Cochrane, 1st Earl of Dundonald (1605-1685). They do not appear to have met each other until August 24, 1773, when Boswell and Johnson were invited to Slains Castle, the seat of Boyd's older brother James, 15th Earl of Erroll. In his journal, Boswell described Boyd's informal medical practice, which he found "but a foolish amusement of vanity, and no doubt of benevolence too."
In the afternoon, Boyd accompanied Boswell and Johnson in a coach ride first to Dunbuy, and then on to Bullers of Buchan, two remarkable natural features a few miles north of Slains.