Do you ever write checkpoint code? Like, dump current state into a json file, and use it to resume?
Also, that's pretty clever with the rsync. I presume you have a script that just rsyncs and ssh's an execution. I usually just use git to sync my code across machines.
Do you ever write checkpoint code? Like, dump current state into a json file, and use it to resume?
In my experience, this is often not feasible because it is usually not possible to serialize every object that the program uses. For example, many projects are primarily scripted in a higher level language such as Python and make use of some library written in C++ and the only way to access the C++ objects is through an interface that doesn't expose all of the internal data needed to reconstruct those objects.
I'm not saying that it wouldn't work for you, but this is the type of problem that you might encounter.
Well then how about you run your workload in a virtual machine and add a pause method in your code base? Snapshot the virtual machine while you're paused and restore the snapshot when you need to return to that exact state.
Comments
Do you ever write checkpoint code? Like, dump current state into a json file, and use it to resume?
Also, that's pretty clever with the rsync. I presume you have a script that just rsyncs and ssh's an execution. I usually just use git to sync my code across machines.
In my experience, this is often not feasible because it is usually not possible to serialize every object that the program uses. For example, many projects are primarily scripted in a higher level language such as Python and make use of some library written in C++ and the only way to access the C++ objects is through an interface that doesn't expose all of the internal data needed to reconstruct those objects.
I'm not saying that it wouldn't work for you, but this is the type of problem that you might encounter.
Well then how about you run your workload in a virtual machine and add a pause method in your code base? Snapshot the virtual machine while you're paused and restore the snapshot when you need to return to that exact state.