The collection now cared for at Annan Museum began with a small number of local history objects and paintings gathered together by Annan Rotary Club. In 1955 Annan Town Council took over responsibility for this collection, and located it in a room in Annan Library in Bank Street. A few years later the museum collection was moved to Moat House in Bruce Street. It continued to be administered by an Honorary Curator until 1979 when it was moved to Dumfries Museum for safe-keeping.
In 1991 the collection was returned to Annan Council Chambers for storage, but a couple of years later the District Council agreed to establish a post of Curator. The Council also agreed to amend plans for the renovation of the former library, built in 1906 in Bank Street, to allow for the creation of a museum with permanent and temporary exhibition spaces located on the ground and first floors. Between 1994 and the present day staff have developed a small but varied collection of work by local artists, or of local scenes, often with financial support from The Friends of Annandale and Eskdale Museums. Most works date from the latter half of the 19th century or the early part of the 20th century. Artists represented include George Wright (1851–1916) and William Ewart Lockhart (1846–1900).