Analysis of strains from 2012 shows the parts of the pertussis bacterium that the vaccine primes the immune system to recognise are changing.
It may have “serious consequences” in future outbreaks, UK researchers state in the Journal of Infectious Diseases.
But experts stressed the vaccine remains highly effective in protecting the most vulnerable young babies.
There has been a global resurgence of whooping cough in recent years.
In 2012, there were almost 10,000 confirmed cases in England and Wales – a dramatic increase from the last “peak” of 900 cases in 2008.
The outbreak led to 14 deaths in babies under three months of age – the group who are most vulnerable to infection.
Rising figures prompted health officials to recommend vaccination of pregnant women so immunity could be passed to their newborns – a strategy that a recent study showed was working well.
But there has been much debate among experts about whether the introduction of a new vaccine in 2004 has been a factor in rising rates of whooping cough.
One issue is that immunity from the newer acellular vaccine – which contains specific proteins from the surface of the bacteria – does not seem to last as long as the previous whole cell version, leaving teenagers and adults lacking protection.
In the latest study, researchers analysed the genes coding for the proteins on the surface of the pertussis bacterium responsible for the UK outbreak.
They found proteins being targeted by the vaccine were mutating at a faster rate than other surface proteins not included in the vaccine.
Potentially it means the bacteria is changing quickly to get around immune system’s defences put in place with immunisation.
But the researchers are still trying to work out what the changes mean in reality – for example do the mutations boost the ability of the bacteria to cause infection.
“We wanted to look at strains from the UK to see if there was anything sudden that had occurred that had led to these really large outbreaks,” said study leader Dr Andrew Preston from the University of Bath.
The “million dollar question” he said was what, if anything, could be done to improve the vaccine – which is still the best defence we have – and prevent future outbreaks.
Options to consider include adding more or different proteins to the vaccine, adding novel adjuvants – chemicals which boost the immune response, or even revisiting the old-style whole cell vaccine, he said.
“Pertussis has a cyclical nature and other big question is are we going to see another increase in late 2015,” he added
Prof Adam Finn, a paediatric immunology expert at the University of Bristol said the importance – or not – of the subtle changes found in the study was as yet unclear.
“But the control of pertussis is a significant worry,” he added.
Only 60% of pregnant women have had the pertussis vaccine and we should be doing more to raise awareness of its benefits, he said.
“There is very good new evidence that vaccinating pregnant women protects their babies. And the group we really want to protect is newborn babies,” he said.