But he's now back in his element, training for Tokyo 2020, and says there's life after the Olympics for his swimming career - whether he wins or not.
"I will always be a swimmer" - Florent Manaudou to le Parisen
Falling in love with swimming again
After two years in handball, working his way up to France's second division - with a professional contract on the table - scoring 15 goals in 10 appearances in his final season, Manaudou decided that swimming was calling him back.
In November 2018, Rio 2016 double silver medallist was invited to swim at a meet with his old Marseilles club Cercle des Nageurs in Istres and swam a 50m in 20.62s.
But it wasn't just the time that called him back, it was the feeling of being back in the water.
Before the lockdown, he was back training, but with a different mindset and a renewed approach.
France's '20 Minutes' asked him recently how he's enjoyed being back in the pool.
"It's not the same kind of fun as handball: there are no balls, no friends, no goals," he replies, "but there is more fun than in my previous [swimming] career."
"Of the 300 days of the year spent in the water, there are only four or five that are interesting in competition. So, if you don't enjoy training, it makes life difficult."
But aside from training, the swim star has realised that what he does between sessions may be even more important.
Oreos for Breakfast?
"When you start a week, if you haven't decompressed on the weekend, you won't perform well in the water. For me, anyway. I need to get away, have a drink with my friends."
The 29-year-old reportedly has previously indulged in 'oreos for breakfast, and a beer at noon to wash down the chips covered in mayonnaise'.
Even if the diet is a little eccentric, he seems to be enjoying his swimming a lot more after his return to the pool. It is helping to prevent the burnout which saw him walk away from the sport entirely after Rio 2016.
"With experience, cutting things when I want to cut, talking with my coaches without being closed to my own reading of things, it's easier to avoid that."
Manaudou swam for the Energy Standard team, playing an integral part in their 2019 title win.
"It's different, more like a show. With competitions like ISL, we travel, we see things, we meet people and there is less weariness. So, inevitably, burnout is less present."
Silver at the 2019 European Championships
His form towards the end of 2019 was proof of Manaudou's progress.
A world-leading time of 20.77s in the 50m freestyle at the Indianapolis ISL meet in October, for example.
In December, the European Short Course Championships in Glasgow pitted him against the best on the continent.
It was a quietly confident comeback. No big gold medal headlines splashing across France's sports dailies, but plenty of encouraging signs.
Friday brought silver in the 50m freestyle, and Saturday a fifth place in the final of the 50m butterfly with a bronze medal in the 4x50m mixed relay.
A bronze and a silver isn't a bad return after three years out, but it isn't enough for the ultra-competitive Manaudou.
"I almost forgot that I had stopped for three years and it annoys me to be four tenths off my best time," he told L'Equipe after winning the silver.
His coach, James Gibson, who was with the swimmer for that defining moment at London 2012, said that silver was a step forward.
"We are not going to cry over this medal even if, obviously, we wanted gold.
"But this competition allows him to get back into the process, competing in the heats, semi-finals and finals as at the Games."
"I am no longer the hunted but the hunter" - Manaudou
For Manaudou himself, he felt fast.
“I had the sensation that I was flying above the water, not in terms of speed, but it was a thrill, I don't feel tense, it feels good."
One thing was evident: his killer instinct is back.
"In the morning, I wake up thinking of my opponents who are going faster than me and I want to beat them. It's more motivating than trying to fight against yourself.
"My approach has changed a bit, in the sense that I want to win more than in 2016, I have this animal in me."
U.S. veteran Anthony Ervin took his chance to surprise the world at Rio 2016 in the 50m freestyle.
Aged 35, the American won a memorable Olympic final ahead of Manaudou in second place.
The Frenchman will have turned 30 by the time the Olympics in Tokyo take place in 2021.
So will we see an experienced Manaudou return to become Olympic champion in Japan?
Manaudou welcomed the news of the postponement of the Games following the global coronavirus:
"It was the best decision for all the athletes," he told Le Figaro.
"As for me, it will allow me a few more months to train".
Before the new Olympic dates were set, it was his intention to carry on swimming as long as he could.
"I want to continue, so if I am in the same physical and mental state, I will be a swimmer."