Downing Street, adjacent to Whitehall, was built in the 1680s by Sir George Downing (1623-1684), 1st Baronet of East Hatley. It soon became a center of power and Downing Street no. 10 has been the official residence of the British Prime Minister since 1735.
In the 1760s, when Boswell lived there, it was home to the Office of Plantations and to some private housing, accomodating among others civil servants from the various government offices in the vicinity.
Access to Downing Street by the general public was restricted in the 1970s and further tightened in 1982 and 1989.
The street itself bears little resemblance to how it looked in the 1760s. The houses between Number 10 and Whitehall were taken over by the government and demolished in 1824 to allow the construction of the Privy Council Office, Board of Trade and Treasury offices.
In 1861 the houses on the west side of Downing Street gave way to new purpose-built government offices for the Foreign Office, India Office, Colonial Office and the Home Office.
Boswell rented a room in the house of Thomas Terrie in Downing Street from November 26, 1762 until July 7, 1763.