You are quite right in querying whether it was Robert Graham of Gartmore (later Cunninghame Graham of Gartmore & Finlaystone), though not for the reason you give for your unease.
I can assure you that it most certainly was not he who strolled with Boswell.
Robert Graham was the 2nd son of Nicol Graham of Gartmore and left for Jamaica in 1752. He was Receiver General for Taxes the following year, married a local girl in 1764, was elected to the Assembley for the District of St Davids in 1765 and did not return to the UK until 1770 when he inherited the Ardoch estate, biding in London until 1772 to settle his affairs.
The most likely "young Graham" would have been William Graham of Gartmore Ygr (1734-1774), Robert's older brother and heir to the Gartmore estates, who along with Robert had been educated at Glasgow University.
Less likely to my mind, though possible, would be one of the younger sons, John, Walter or Thomas all born in the 1740s, though why they'd be "young Graham" when there were three of them no more than 3-4 years apart in age eludes me.
Hope this information is helpful.
W R B Cunninghame Graham of Gartmore