In fact, if players are looking to be challenged by Calamity Ganon (a notoriously easy final boss), they may want to avoid some or all of these dungeons until after defeating him.
However, much like finding all the memories, completing the Divine Beasts in BOTW is not required to beat the story Calamity Ganon will simply have more hit points depending on how few Divine Beasts were freed from the Blight Ganons. If the player completed any Divine Beasts, their magical energy will beam towards Hyrule Castle and deplete Ganon's overall health pool.
Regardless of how many memories have been recovered, Link delves into the heart of Hyrule Castle to face Calamity Ganon.