Checkpointing - saving the state of the computation at intermediate points - can help a lot with this.
BUT, beware that checkpointing can also be unreliable if the system being simulated exhibits chaotic dynamics; truncating off "unimportant" decimal places during checkpointing can produce results that diverge rapidly between the "original" and "restarted from checkpoint" versions.
Yes, checkpoints help a ton. Even if this means dealing with a slower algorithm. Then make sure to never overwrite the intermediate steps and back them up as-is if needed.
Comments
Checkpointing - saving the state of the computation at intermediate points - can help a lot with this.
BUT, beware that checkpointing can also be unreliable if the system being simulated exhibits chaotic dynamics; truncating off "unimportant" decimal places during checkpointing can produce results that diverge rapidly between the "original" and "restarted from checkpoint" versions.
Yes, checkpoints help a ton. Even if this means dealing with a slower algorithm. Then make sure to never overwrite the intermediate steps and back them up as-is if needed.