Why don't we see more general purpose computing environments where the whole interpreter state is persistent and serializable by default? The only ones I'm aware of is R's .RData mechanism, some Smalltalk environments, and solutions at the container/VM level.
What if an interpreter wrote every event down to a variable change to a database-like WAL, making it resumable after a crash? Downside: it would be slow, and there would need to be provision for transaction boundaries and for access to network sockets and other non-serializable resources. Upside: develop incrementally in the Smalltalk REPL style, never worry about the data layer at all, backup or give developers a copy of the whole application state - for utilities or LOB apps of limited scope or complexity, why not?
There's this wonderful paper from Usenix OSDI 2014 [1] in which the authors present an eidetic system: "a computer system [should] provide the ability to recall any past state that existed on the computer, and further, [should] be able to provide the lineage of any byte in a current or past state."
They implemented and evaluated it, too: "Preliminary data from several weeks of continuous use [show that] storage requirements for 4 or more years of usage can be satisfied by adding a 4 TB hard drive to the system. Further, the performance overhead on almost all workloads we measured was under 8%."
[1]: Devecsery, David, MIchael Chow, Xianzheng Dou, Jason Flinn, and Peter M. Chen. “Eidetic Systems.” In Proc. 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), 2014. https://www.usenix.org/conference/osdi14/technical-sessions/...
You generally only want to serialize a part of the program state.
I built a workflow engine in the past on the JVM that uses serializable continuations, a bit like what Goblins is doing. The important part is not the VM-level engineering, the important part is defining the boundaries of what gets serialized. If you actually checkpoint everything then you can't upgrade your software anymore, because developers think about program upgrades as reloading from a blank slate, not hot-patching a running program in memory. OK maybe some Lisp devs think in the latter way, but most people don't, and the moment you upgrade a library from one version to another of course there is no instructions for how to incrementally patch your way in memory from one to another.
It's also not clear why you'd want to. A lot of program state is plumbing. Logging frameworks, HTTP stacks, thread pools, system configuration, JIT compiler state, etc. There's no point in saving that to disk. It belongs in transient RAM naturally.
You'd still need a mechanism for persisting separately, or you'd end up with a broken image/"data layer" due to post-crash persists.
It's common to do something similar by writing most of an application as functions over a database and static files, and using scaffolding tools to reinitialise the database when things break.
Comments
Why don't we see more general purpose computing environments where the whole interpreter state is persistent and serializable by default? The only ones I'm aware of is R's .RData mechanism, some Smalltalk environments, and solutions at the container/VM level.
What if an interpreter wrote every event down to a variable change to a database-like WAL, making it resumable after a crash? Downside: it would be slow, and there would need to be provision for transaction boundaries and for access to network sockets and other non-serializable resources. Upside: develop incrementally in the Smalltalk REPL style, never worry about the data layer at all, backup or give developers a copy of the whole application state - for utilities or LOB apps of limited scope or complexity, why not?
It wouldn't necessarily be slow!
There's this wonderful paper from Usenix OSDI 2014 [1] in which the authors present an eidetic system: "a computer system [should] provide the ability to recall any past state that existed on the computer, and further, [should] be able to provide the lineage of any byte in a current or past state."
They implemented and evaluated it, too: "Preliminary data from several weeks of continuous use [show that] storage requirements for 4 or more years of usage can be satisfied by adding a 4 TB hard drive to the system. Further, the performance overhead on almost all workloads we measured was under 8%."
[1]: Devecsery, David, MIchael Chow, Xianzheng Dou, Jason Flinn, and Peter M. Chen. “Eidetic Systems.” In Proc. 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), 2014. https://www.usenix.org/conference/osdi14/technical-sessions/...
You generally only want to serialize a part of the program state.
I built a workflow engine in the past on the JVM that uses serializable continuations, a bit like what Goblins is doing. The important part is not the VM-level engineering, the important part is defining the boundaries of what gets serialized. If you actually checkpoint everything then you can't upgrade your software anymore, because developers think about program upgrades as reloading from a blank slate, not hot-patching a running program in memory. OK maybe some Lisp devs think in the latter way, but most people don't, and the moment you upgrade a library from one version to another of course there is no instructions for how to incrementally patch your way in memory from one to another.
It's also not clear why you'd want to. A lot of program state is plumbing. Logging frameworks, HTTP stacks, thread pools, system configuration, JIT compiler state, etc. There's no point in saving that to disk. It belongs in transient RAM naturally.
You'd still need a mechanism for persisting separately, or you'd end up with a broken image/"data layer" due to post-crash persists.
It's common to do something similar by writing most of an application as functions over a database and static files, and using scaffolding tools to reinitialise the database when things break.