Like all significant collections of pictures and sculpture in country houses, those at Stourhead have two main elements: the family portraits accumulated over a number of generations, and the paintings systematically commissioned or acquired by deliberate collectors. In the case of the Hoares of Stourhead there were really just two of these: Henry Hoare II (1705–1785), known as 'The Magnificent', and his grandson, Sir Richard Colt Hoare (1758–1738), 2nd Bt.

The collection was, however, affected by three other things: most crucially, by the Heirlooms sales of 1883, in which what were then thought to be the most valuable pictures were put up for auction. These included Rembrandt's nocturnal "Flight into" "Egypt "(now in the National Gallery of Ireland, Dublin), Fra Bartolomeo's "The Virgin and Saint" "Joseph Adoring the Christ Child" (now held by the National Gallery, London), Poussin's "Rape of the Sabines" (now at the Metropolitan Museum of Art, New York), and Carlo Dolci's "Salome" (now in Glasgow Museums).Then there was the fire that gutted the house in 1902. The paintings were saved, but when the Saloon was rebuilt it was reduced in size, so the large paintings that had occupied its stucco frames were either banished to the upper landing of the Staircase or given to the newly created Victoria Art Gallery in Bath. Though all but one were copies, mostly of great Seicento paintings, they gave the collection part of its character (two of them can now be seen in the Guildhall in Bath). Finally, to fill some of the gaps left by the Heirlooms sales, Sir Henry Hoare (1865–1947), 6th Bt and his wife Alda, who gave Stourhead and its contents to the National Trust in 1946, brought with them a number of pictures from the seat of their line of the family, Wavendon, in Buckinghamshire.

The character of Henry Hoare II's collection emerges best from the account of it given by Horace Walpole, after a visit in July 1762. It was chiefly composed of pictures by Italian artists, or ones painted in Italy, leavened by a number of landscapes, no longer in the collection, by John Wootton, (who had also painted the horse in the great equestrian portrait of Henry Hoare II as a young man, then as now, dominating the Hall), and by pastels by Hoare of Bath (who was no relation, although his daughter later married into the family).

Amongst the original pictures listed by Walpole that are still in the collection are: the two Sebastiano Riccis (one of which has since turned out to be a copy), Carlo Maratti's "Marchese Niccolo Maria Pallavicini Guided to the Temple of Virtue by Apollo" and "The Flight into Egypt", Anton Raphael Mengs's "Octavius Caesar (Later the Emperor Augustus), and Cleopatra," the pair of Francesco Fernandis, and "The Choice of Hercules" by Nicolas Poussin. Henry Hoare II's most significant acquisitions subsequent to Walpole's visit are rather singular for a British collection: the Lagrenées (which had originally been commissioned by the disgraced duc de Choiseul).

The paintings were distributed throughout the ground-floor rooms of the house, unsystematically – apart from the seven large copies and one original, Sébastien Bourdon's "Judgement of Midas" (which was with the two copies later given to the Victoria Art Gallery, but whose present whereabouts is unknown), in the Saloon. In what was advanced taste for the time, Henry Hoare II had a sky-lit picture gallery, but even this does not seem to have been reserved either for the best pictures, or for a particular type of picture. The large paintings by, and attributed to, Gaspar Dughet, and the great work by Maratti were acquired directly from Italy via an agent, but – apart from the paintings that he commissioned – he seems to have made most of his acquisitions through London dealers and salerooms.

Sir Richard Colt Hoare – generally known as Colt Hoare – was a collector in a very different mould. He acquired pictures more eclectically, but he arranged them more systematically. He set out the principles governing the allocation of certain types of picture to particular rooms, in his description of Stourhead in "The History of Modern" "Wiltshire: The Hundred of Mere "(1822). Colt Hoare also differed from his grandfather in his evident desire to encourage a native school of painting, by giving commissions to British artists for 'fancy' paintings, rather than just for portraits or landscapes (an approach shared by the 3rd Earl of Egremont at Petworth House, National Trust). He also encouraged artists such as Ducros – and, above all, J.M.W. Turner – to paint watercolours on a scale, and with an ambition, to rival oils.

Unlike Henry Hoare II, Colt Hoare himself made the Grand Tour to Italy, on two separate journeys, in 1785–1787 and 1788–1791, visiting places off the beaten track, such as Elba and the further stretches of the Appian Way. He kept, and partially published, journals of his travels, and both made sketches of views and sites himself, and took other artists along with him to do so; above all, Carlo Labruzzi, but also Don 'Titta' Lusieri, and the Hackert brothers. Colt Hoare was criticised by artists in Rome for only commissioning landscape paintings, and no sculpture, but he did acquire major history pictures in oils from earlier periods – notably Cigoli's great altarpiece of "The Adoration of the Magi" from a destroyed church in Florence.

After he got back home, he decided to add a library to Stourhead in 1792, semi-detached (prophetically, in case of the main block of the house catching fire) and then a semi-detached Picture Gallery (completed 1802) – on a much greater scale than Henry Hoare II's, but regrettably not top-lit. Having done so, he proceeded to rearrange the pictures systematically. The key changes were: to devote the three main walls of the Entrance Hall to portraits; to house most of the important Old Master paintings in the collection (and later two large pictures by Henry Thomson, '"Distress by Sea'" and '"Distress by Land') "in the Picture Gallery; and to turn the Picture Cabinet from a mixed room of history pictures into a room exclusively devoted to landscape subjects.

The substitute for the pictures forfeited in the Heirlooms sales made by Sir Henry and Alda, Lady Hoare, which they had brought from Wavendon, were mostly not of great significance. However, one of the two triptychs that they called by the generic name of 'Memling' has turned out to be by that singular artist, Jan Provost. Their taste seems particularly to have been for architectural capricci, complementing the Francis Harding pastiches of Panini – "The Interior of St Peter's at Rome" and "Architectural Capriccio" – introduced as overmantels by Henry Hoare II. In addition – possibly fostered by some particular rapport between artist and patroness – the artist and novelist St George Hare not only painted portraits of Sir Henry, Lady Hoare, and their only son, whose death in the First World War was to lead to their donation of Stourhead to the National Trust, but also a series of subject pictures, some of whose themes have now come to seem curiously equivocal – notably '"Miserere"", domini'' (Christians in Prison)" and "The Gilded Cage, a Female Captive. "In the early years of its ownership of Stourhead, the National Trust thinned the hangs of pictures – especially in the Picture Gallery – and lent a number of them to other properties – such as Mompesson House, Salisbury – without picture collections of their own. More recently, it has clawed back these pictures, and brought others down from attic stores, to recreate – so far as is possible, given the losses in 1883, and the subsequent introductions by Sir Henry and Alda, Lady Hoare – the hangs of the house in its heyday: at the time of Colt Hoare's death in 1838. The Saloon, being the one room that was not rebuilt as it had been before the fire in 1902, is largely given over to the introductions and commissions of Sir Henry and Lady Hoare. Some hangs still remain to be recreated: above all, that of the large sepia copies of classic paintings, and of details in them, by Seydelmann and Rouby in the Italian Room.