Son of James McAdam (1662?-1744?) and Margaret Reid. Married (1745) to Susanna (or Susannah) Cochrane, with whom he had several children, including the renowned engineer and inventor John Loudon McAdam.
In 1760 he moved with his family to the newly-built Lagwine Castle, which was on the ancestral family estate of Waterhead, but more accessible than the old castle. Lagwine burned down in December 1762, just a few months after Boswell's visit there. The McAdam family moved to Blairquhan, a castle leased by Sir John Whiteford. Lagwine Castle was never rebuilt and is today in ruins. Sometime before his death, James McAdam had to sell the Whitehead estate, because of a poor economy.
Boswell was related to the family, James McAdam's wife Susanna being a Cochrane, and thus sharing a common ancestor with James Boswell. Sir John Cochrane (d. 1707) was Boswell's maternal great-great-grandfather and Susanna Cochrane's grandfather.
Boswell arrived at Lagwine Castle on September 14, 1762, on the first leg of his harvest jaunt of that year. McAdam himself wasn't present (he "was at the moor selling lambs, not to return till next day"), but at least two of his daughters were there (who cannot be properly identified, as McAdam had eight daughters), and Boswell very much enjoyed the company, performing scenes from The Beggar's Opera together with one of the McAdam daughters. He stayed the next day as well, and in the evening McAdam arrived back home, together with the Laird of Daltoune. Boswell described McAdam as "a sensible well-bred gentleman".