Only four of the 56 species studied in the UK Butterfly Monitoring Scheme saw an increase in population size. The relentless rain and cold of 2012 meant that in particular summer species struggled to find food, shelter and mating opportunities.
The scheme looked at how a range of species did at more than 1,000 sites across the UK. The black hairstreak, one of the UK's rarest butterflies, did badly. It saw its population fall by 98% compared to 2011. The green hairstreak was down by 68% and the white-letter hairstreak down 71%.
The critically-endangered high brown fritillary saw numbers fall by 46% compared with 2011, and the endangered heath fritillary saw numbers halve. More widespread butterflies also suffered - common blue numbers fell by 60% and cabbage whites fell by half, said the charity.
Dr Tom Brereton, head of monitoring at Butterfly Conservation which carries out the monitoring with the Centre for Ecology and Hydrology, said: "2012 was a catastrophic year for almost all of our butterflies, halting progress made through our conservation efforts in recent years.
"Butterflies have proved before that given favourable conditions and the availability of suitable habitat they can recover, but with numbers in almost three-quarters of UK species at a historically low ebb any tangible recovery will be more difficult than ever."