Son of George Henry Lee (1690-1742), the 2nd Earl of Lichfield, and Frances Hales (d. 1769). Married (1744/5) Dinah Frankland (1719-1779), daughter of Thomas Frankland, 3rd Bart of Thirkelby. MP for the county of Oxford (1740-1743). Lord of the Bed-Chamber (1760). Chancellor of the University of Oxford (1762-1772).
Lee died without issue, and following his death in 1772, the Earldom of Lichfield passed to his uncle, Robert Lee (1706-1776). The 4th Earl also died childless, and the title became extinct.
Coincidentally, George Henry Lee and his wife Dinah were both descendants in the fourth degree of King Charles II and Oliver Cromwell respectively. The 1st Earl of Lichfield married Charlotte Fitzroy, daughter of King Charles II, while Dinah's grandfather Sir Thomas Frankland married Elizabeth Russell, whose mother was Frances, youngest daughter of Cromwell.
Boswell and Lee met at Lord Eglinton's in London on June 24, 1763.