Hay was one of Boswell's closest friends for some years until the mid-1770s, when they seem to have fallen out with each other.

Boswell dined with Hay, William Forbes, William Scott and Robert Arbuthnot on August 15, 1773, during Dr Johnson's stay in Edinburgh, before Johnson and Boswell set out on their tour of Scotland and the Hebrides. Hay has, however, the dubious honour of being one of a few people in Boswell's Journal of a Tour to the Hebrides, who is not mentioned by name but merely referred to as "another gentleman". His identity is established by Boswell's notes and journals.

Boswell refers to Hay a couple of times in his journal in 1777, writing that "I was disgusted with the vulgarity of Braxfield and with Hay’s fulsome flattery of him" (January 25, 1777), and "Charles Hay there. Disgusting meanness. Vexed to recollect intimacy with such." (July 12, 1777)