The first Cole to come to Ireland was William Cole of Devonshire, who became one of the prime movers in James VI & I's Plantation of Ulster. Assigned 1,320 acres of escheated lands in 1611/1612, 80 were designated for the town of Enniskillen, of which he was appointed Governor and first Provost. Knighted in 1617, he controlled it from the Castle until his death in 1653. In about 1719 his great-grandson, Sir John Cole (1680–1726) began the first house on the present site, named after his wife, Florence, only daughter of Sir Bourchier Wrey, Bt, also of a Devon family. The main block and colonnades of the existing house were built by his son, John (1709–1767), created 1st Baron Mount Florence in 1760. His son, William Willoughby (1736–1803), created Viscount Enniskillen in 1776, and 1st Earl of Enniskillen in 1789, completed it with the pavilions. The 5th Earl (1876–1963), together with his son, Viscount Cole (1921–1956), gave it to the National Trust in 1954. A year later, the central block was gutted by fire caused by an electrical fault, but all the contents were rescued. The Trust immediately set about restoring the house, under the supervision of Sir Albert Richardson. Fortunately, all but the two important stucco ceilings of the attic floor – the one over the school room and the dome over the bedroom landing – had been photographed by the Irish Georgian Society in 1911, so re-instatement was largely possible.

Most of the contents were, however, removed in 1973 by Viscount Cole's cousin, David Lowry-Cole (1918–1989) (who had succeeded to the title as 7th Earl of Enniskillen, having up until then been a rancher in Kenya, where he had been born) and his second wife, the American journalist, Nancy MacLennan, when they left Florence Court for Scotland, having fallen out with the National Trust over the management of the property. Nancy (1917–1998), Countess of Enniskillen had already hinted at these difficulties, in her book, "Florence Court – My Irish Home" (Enniskillen, 1972). She made a splendid restitution of the major contents in 1997, however, when, knowing that she did not have long to live, she invited the National Trust's Historic Buildings representative, Peter Marlow, to Kinloch House, Perthshire,to see what she was proposing to return, provided that he had organised its transport and re-installation in the house within a fewmonths. This he did, and it revitalised the interiors of the house. Other things, which Lady Enniskillen had not regarded as of intrinsic significance for the house, were acquired at auction in Scotland and at Christie's South Kensington after her death.

Amongst the most significant purchases were watercolours and sepia wash drawings of classical sites in Italy by Ducros, Keiserman and Klengel,that had been collected by John Willoughby Cole (1768–1840), Viscount Cole and later 2nd Earl of Enniskillen, when on the Grand Tour in 1792, just before which he was painted by Thomas Robinson. There are also a number of portraits of his distinguished younger brother, General the Honourable Sir Galbraith Lowry Cole (1772–1842), who fought in the Peninsular War and later became Governor successively of Mauritius and the Cape of Good Hope, and who is commemorated by a huge columnar monument at Enniskillen. But the most intriguing pictures are the three large views of: "Lough Erne at Enniskillen with Castle Barracks in the Middle Distance", "Devenish Island" and "Belleek Falls (The Falls of Ballyshannon, County Donegal) "(a fourth picture from the set was presented long ago to the Earl of Erne), which are still in search of an author – possibly an immigrant scenery-painter employed by Dublin's vigorous Smock Alley Theatre.

The most important member of the family was William Willoughby (1807–1887), 3rd Earl of Enniskillen, the palaeontologist, whose unrivalled collection of almost 10,000 fossil fish, once housed in the South Pavilion, each one of which he could recognise after going totally blind, was bought by special grant for the newly built Natural History Museum of the British Museum in 1883 (the North Pavilion was lived in by his alienated bachelor younger brother, Colonel the Honourable Henry Arthur Cole (1809–1890) and his manservant). But the most eccentric member of the family was John Henry Michael (1876–1963), 5th Earl of Enniskillen, who was given to throwing toast across the dining room when it was too dry for his taste, and who, when woken in his club in distant Belfast on the morning of the fire to be told of it, exploded with irritation, saying 'What the devil do you expect me to do about it?', before turning over and going back to sleep.