Son of James Drummond of Blair Drummond (1673-1739) and Jean Carre (1681-1766). Married firstly (1738) to Jane Grey (d. 1762), daughter of the Earl of Stamford. Married secondly to Frances Moray, daughter of James Moray, 13th of Abercairny.
George Drummond died on March 17, 1765, just a few weeks after the birth of his only child, James. The infant outlived his father by just over a year, dying in May 1766, causing George's sister Agatha Drummond, and subsequently, her son, George Home, to inherit the lands and title of Blair Drummond.
Boswell met Drummond at Kames on October 16, 1762. He described him as having "a little haughtiness which renders him not very agreeable at first sight. So much for the shade of his character. He is at the same time a man of sense, very honest and very friendly when he has an attachment. He has done several generous things. He loves company and has humour, though it is sometimes a little dry".