The 23-year-old Quick Step rider timed his sprint to perfection to overtake German Andre Greipel and hold off a fast-finishing Peter Sagan, who just got second, in a photo finish.
"But we'll be working hard as a team again to protect the yellow jersey", said Van Avermaet, who many pundits feel may have a couple of stage wins in him before the Tour turns to the Alps.
Gaviria's compatriot Rigoberto Uran nearly lost time after being caught in a late crash around 5km from the finish, but he was paced back to the main group by his EF Education First-Drapac team to stay 10th, 35 seconds behind Van Avermaet.
Maximiliano Richeze delivered a flawless ride-along performance for Quick-Step comrade Gaviria, powering him past the Dimension Data team and into the forefront for a challenge on the stage victory.
"It's the first big stage of this kind and we are going to see time difference in the overall standings, it just depends who wants to go for it", he said.
Fellow Britons Froome and Thomas didn't fare any better but will have ended Stage 4 close to content with their results, nestled among a pack in which Van Avermaet was also happy to reside.
But there were no changes among the title hopefuls.
Froome was left 55 seconds behind in the overall standings with another week of nervy rolling stages before hitting the Alps. Movistar teammates Alejandro Valverde and Mikel Landa were 15th and 16th (:53), narrowly ahead of Froome.
Man United boss Jose Mourinho opens up about Ivan Perisic
United open the 2018-19 Premier League season as they take on Leicester City at Old Trafford on August 10. Angel Gomes is a young player who may want to show he can be a part of the first team.
Today, though, it was back to the more typical fare of the Tour's opening week.
A crash with just 5 kilometres remaining of the 195 kilometre stage from La Baule split the peloton, with last year's runner-up, Rigoberto Uran of EF Education First-Drapac, having to battle back to rejoin the bunch ahead of the final 3 kilometres to avoid losing time.
BMC Racing kept the Franco-Belgian breakaway under control, however, and with the pace down there was little to report through much of the stage.
"It's not an easy stage by any means but, I think stage six will be more of a test for me". "(Depending) on how it goes there will be differences between the riders". During Tuesday's stage, several syringes were seen alongside the course route in apparent protest by anti-Froome spectators.
Froome is aiming to join Jacques Anquetil, Eddy Merckx, Bernard Hinault and Miguel Indurain as the only riders to win the Tour five times.
The three-week Tour ends July 29 in Paris.