Richard Gabriel's "The Rise of Worse is Better," written around the same time period the Unix-HATERS Handbook was written, gives some clues. Unix was contrasted with environments such as Scheme (which is rather small, but because it's designed as a "crown jewel"), Common Lisp (the exemplar of a "big complex system" that is complete and correct, but large and complex), and the ITS operating system (https://en.wikipedia.org/wiki/Incompatible_Timesharing_Syste...). Thankfully there are many open-source Scheme and Common Lisp implementations, and ITS is also available as open source (https://github.com/PDP-10/its).
Of course, modern Unix-like systems these days are large and complex, though Plan 9 and Inferno are quite architecturally refined, reducing some of the complexities that you'll see in contemporary Unix-like systems.
At the time The Rise of Worse Is Better was written, Scheme had:
- no error handling. The specification described that certain situations result in an error, without defining what that means, or how you can catch and recover from one programmatically.
- no module system. The Scheme report didn't describe any feature for decomposing a Scheme program into multiple files. No rules governing how literal data goes into compiled files and what the rules are for recovering symbols. No evaluation controls (like don't evaluate this form when compiling, but only deposit it into a compiled file and such).
Unix was a useful production system used for running businesses, with error handling in its API's and compilation of programs decomposed into multiple files.
The "PC losering" problem was nicely resolved. Unixes developed sigaction, where you can specify it both ways: interrupts can bail the system call (which is sometimes what you want) or restart it (which is what you want at other times). Neither is inherently better. Sometimes you really don't want a long system call to go back to sleep as if nothing happened. A third solution is possible though: your signal can resume the system call when it returns normally, or else abandon it with a siglongjmp, which restores the signal mask to what it was at the sigsetjmp point. Then there is is various other signal paraphernalia. Basically the Unix people had the right intuitions in this area, and took an incremental approach whereby they gradually rolled in more correctness.
That's kinda the whole point of the essay: getting something up and running quick that is 'complete' (if not particularly elegant) is better than slowly and carefully working on a Good System. Unix ate Lisp's lunch because you could build things on it right now.
That essay definitely was a bit of a complaint about the crudeness of Unix. But I read it primarily as a wake up call (not to say epitaph) for the Ivory Tower Lisp developers who were dreaming about the perfect system while Unix ate their lunch.
What ate Lisp's lunch wasn't necessarily the Lispers dreaming of a perfect system. Rather, Unix was far more accessible in terms of availability than the contemporary Lisp systems coming out of Symbolics and Xerox at the time Unix really started to take off. Before the breakup of AT&T in 1984, Unix was available to universities under comparatively generous licensing terms. While source licenses were very expensive for companies and (after 1984) for universities, binary licenses were relatively inexpensive, and beginning in the 1990s we saw open source Unix-like operating systems such as Linux and the BSDs. Unix was not restricted to a particular architecture; it ran on a wide variety of hardware.
Contrast this with Symbolics Genera and Interlisp-D, the premier Lisp operating systems of the 1980s. A Symbolics workstation can easily cost five figures in mid-1980s dollars, and cheaper alternatives such as the MacIvory boards and OpenGenera on DEC Alpha machines weren't released until later in Symbolics' history. Interlisp-D originally ran on Xerox workstations that also cost five figures, though it was later ported to the Sun SPARC architecture (though I have no idea what Interlisp-D licenses cost). Both Symbolics Genera and Interlisp-D missed out on the open source revolution of the 1980s and 1990s. To this day Symbolics Genera remains proprietary, though thankfully Interlisp-D was made open-source recently (https://interlisp.org/). There are open-source Common Lisp compilers such as SBCL, but they are not full-fledged operating systems.
Part of the reason why Unix took off had less to do with Unix's design and had more to do with Unix's lower costs. DOS and Windows had even lower costs than Unix before open source Unix clones appeared, and they were (and, in the case of Windows, still is) widely used. While Unix's design characteristics certainly play a role, we shouldn't ignore the impact of cost and licensing.
In an alternative universe, imagine had RMS had set out on building a FOSS Lisp operating system (GNU Emacs doesn't count) instead of building GNU. RMS was a Lisp hacker at MIT who started the GNU project due to his frustrations with the proprietary Lisp machine companies like Symbolics that were spun off from MIT's AI lab, so it's not unlikely to think about an alternative situation where RMS decided to clone a Lisp OS instead of cloning Unix. I wonder if a community could have rallied behind an open source Lisp OS to be a challenging contender to Unix back in the 1980s?
Lisp machines were mostly just a single type of computer: a graphical workstation for Lisp developers. Those were mostly in R&D, typically in AI - largely financed by DARPA and similar, hoping for the AI software providing leadership in tech and military. The market for their software&hardware was on the leading edge, which later was taken from offerings growing on cheaper platforms.
They had one/two operating system to fork others from: single user, no terminal story, no security, needing large amounts of memory, needing specific expensive hardware (graphics, disks, custom boards, ...), not easy to port, needed complex (and hard to debug) memory management, scarce number of Lisp system programmers accumulated in a small number of companies, no public source story, ...
Many open source operating systems are copies or forks. There was not much original research. Writing a new operating system (say, a portable Lisp OS with device drivers, multi-user capabilities, security, terminal and GUI usage) would have needed a lot of work and there are not enough combined Lisp AND systems programmers to do that - educating them takes a lot of money.
Even today, the state of the 70s/80s Lisp OS hasn't been reached again: there is no comparable operating system written in Lisp and the old ones exist only as emulators of the past.
The tech base of the early Lisp system was too focused, there was no room to mutate and grow to different platforms in different incarnations. There were attempts (like embedded Lisp hard- & software) but that had no effect in the market.
UNIX was developed for a completely different market: simpler base software, very portable, multi-user, client&server, modular programs, terminal, ... a bunch of companies used UNIX as a base for their workstation offerings (SUN, SGI, HP, IBM, NeXT, ...) - the rest is history.
But Lisp OS like software on the UNIX systems was working well in the 80s and early 90s. They still needed large amount of RAM, large amount of virtual memory, responsive GUIs, advanced GC support, ... some high-end machines could offer that and they fixed enough OS bugs to run large Lisp systems. Every larger UNIX vendor had some Lisp story as part of their general offerings.
The Lisp people absolutely got something up and running quickly, which is why Lisp is one of the earliest languages. Big, institutional Lisp systems were in a different state of maturity when Unix was coming up.
Early Lisp didn't have defmacro with destructuring, backquote, lexical scope, condition handling, structures, hash tables, ... you wouldn't want to use it today.
Sometimes I wonder, if you wanted to create an operating system like ITS for more modern platforms, what it would look like?
ITS is written in PDP-10 assembly, what if someone wrote a compiler which read in PDP-10 assembly language and spit out C code? Could that be a first step to porting it?
It is surely a lot more complicated than that. It contains self-modifying code, which would obviously break that translation strategy. A lot of hardware-specific code would have to be rewritten. 6 character filenames without nested directories might have been acceptable in the 1970s, but few could endure it today. A multi-user system with a near-total absence of security was acceptable back then, obviously not in today's very different world.
ITS does have some interesting features contemporary systems don't:
A process can submit commands to be run by the shell that spawned it – actually MS-DOS COMMAND.COM also had that feature (INT 2E), but I haven't seen anything else with it. A Unix shell could implement this by creating a Unix domain socket, and passing its path to subprocesses via an environment variable–but I've never seen that done.
Another was that a program being debugged could call an API to run commands in its own debugger – I've never seen that in any other debugger, although I suppose you could write a GDB plugin to implement it (have a magic do-nothing function, then a GDB Python script sets a breakpoint on that function, and interprets its argument as commands for GDB.) Actually, in ITS these two features were the exact same feature, since the debugger was used as the command shell.
Another was that a program had named subprocesses (by default the subprocess name was the same as the executable name, but it didn't have to be.) Compare that to most Unix shells, where it is easy to forget what you are running as background jobs 1, 2 or 3.
A process can submit commands to be run by the shell that spawned it – actually MS-DOS COMMAND.COM also had that feature (INT 2E), but I haven't seen anything else with it. A Unix shell could implement this by creating a Unix domain socket, and passing its path to subprocesses via an environment variable–but I've never seen that done.
What would you use it for? How is that superior to simply spawning a new shell process?
Norton Utilities for DOS came with a program called NCD (Norton Change Directory). It was a full-screen replacement for the CD command, you could browse through the filesystem to get to the directory you wanted, then it would exit and return you to COMMAND.COM's prompt, with you already in that directory. That worked because under DOS, the current directory was system-wide, [0] not per-process, so a subprocess could change the parent process' current directory. On Unix systems, and the Windows NT family, the current directory is per-process, so that doesn't work any more.
Now, you can implement the same idea on Unix–but it is more complex. You need to define a shell function in your .profile/.bash_profile/.zprofile/whatever. That shell function then executes the external change-directory program, and passes the destination directory back to the shell function somehow. The shell function then actually changes the directory.
What if shells had exposed some kind of standard API to their subprocesses? Maybe something as simple as an environment variable containing the path to a Unix domain socket which accepts a simple text-based protocol. You could then implement an NCD-style command, by having it send the shell which called it a request to change its directory. That way, you would not have to install such a command by modifying your profile and restarting your shell, the command would just work the first time you ran it.
Similarly, there are a lot of tools out there for supporting multiple versions of development tools concurrently (Environment Modules, venv for Python, nvm for Node, conda, etc.) Most of these tools work by manipulating your shell environment. Since, on Unix, environment variables are per-process, and a process cannot modify the environment of its parent, you have to install some shell function in your profile and restart your shell before using one of them. Once again, if shells exported some kind of "get/set shell environment" API to their subprocesses, these kind of tools would work the first time you ran them, without any need to modify your shell profile.
[0] Actually, in MS-DOS and Windows 3.x/9x/Me, there is both a system-wide current drive, and a separate system-wide current directory for each drive – Windows NT family's cmd.exe simulates this by using a hidden environment variable per a drive to store its current directory, but that is a convention which only cmd.exe supports, the rest of Windows knows nothing about it.
That worked because under DOS, the current directory was system-wide, [0] not per-process, so a subprocess could change the parent process' current directory.
For people who don't know DOS: DOS didn't really have multiple processes. COMMAND.COM made part of it resident (TSR), which is how it "survived" invoking other applications. At any given point, there was exactly one process running, and all the state you'd typically associate with a process was "system-wide".
For people who don't know DOS: DOS didn't really have multiple processes.
Well, compared to CP/M, it did. CP/M only supported a single program being loaded into memory at a time [0]–because every program was loaded into memory at the same absolute memory address (0x100). By contrast, DOS could have multiple processes in memory simultaneously, even though only one of them at a time could be the active foreground process. DOS processes can launch child processes; the parent will be suspended (while remaining in memory) while the child process executes, but will then resume execution once the child process finishes. Under CP/M, that was impossible, there was no API to do that, the architecture didn't support the existence of one.
Indeed, while the main execution of the parent process would be suspended while the child process ran, any interrupt handlers installed by the parent would still be invoked, so some of the parent's code could still run while the child was executing. A parent process can even expose an API to its children – which is exactly what COMMAND.COM does (such as INT 2E which I mentioned)
That's what makes TSRs possible. A TSR is essentially just an ordinary program, the only difference is that when it exits, it tells the OS to leave the process in memory rather than unloading it. By contrast, the CP/M equivalent to TSRs, RSXs (Resident System Extensions), are completely different from ordinary CP/M programs. They are loaded at a variable address near the top of memory, instead of a fixed address at the bottom.
COMMAND.COM made part of it resident (TSR), which is how it "survived" invoking other applications.
COMMAND.COM isn't a TSR. COMMAND.COM's "residence" is essentially the same as any other program which spawns a child program under DOS, the parent always remains in memory while the child runs. Unlike a TSR, when a COMMAND.COM instance terminates (the initial COMMAND.COM instance won't, but inferior instances will), it doesn't stay in memory, it is unloaded.
What makes COMMAND.COM somewhat unusual, is that it splits itself into two portions, a "resident" portion and a "transient" portion. The resident porition is loaded at the bottom of memory, the transient portion is loaded at the top of free memory, without being officially allocated. Since DOS normally allocates memory in a bottom-up manner, if the child program doesn't use much memory, the transient portion will not be overwritten. When the chlid program returns, COMMAND.COM tests the integrity of the transient portion – if it finds it has been overwritten (because the child program needed that much memory), it reloads it from disk before continuing. That's quite unusual behaviour, but still rather different from how TSRs behave.
At any given point, there was exactly one process running, and all the state you'd typically associate with a process was "system-wide".
That's not true. When you spawn a child process, you can either let it inherit your environment variables, or you can provide it with a new environment segment – thus each process in the system can potentially have different environment variables. Similarly, file handles are per-process. Similar to Unix – indeed, the DOS 2.x file handle design was copied from Unix – every handle has an "inherit" flag, which determines whether child processses inherit that handle from their parent. DOS keeps track of which processes have which files open, so when the last process with that file open terminates, the file is closed. Likewise, allocated memory blocks are per-process, so when a process is unloaded, DOS frees its memory blocks, but not those of ancestor processes or TSRs.
If DOS has per-process environment variables, file handles and memory allocations – why not current directories as well? I don't know, but I can speculate: DOS 2.0 copied the idea of a current directory from Unix, so like Unix they probably would have initially planned to make it per-process. However, requirements for backward compatiblity with DOS 1.x programs pushed them towards having a separate current directory per each drive. Keeping a separate current directory per-process might have been feasible if it was just a single current directory, but having to do so separately for each drive would have made it a lot more complex and wasteful of memory. I think that is why they made it system-wide instead of per-process.
Microsoft always planned to make MS-DOS a multi-tasking operating system, and much of its internal architecture was designed to support gradual evolution towards multi-tasking [1] – which makes its architecture somewhat closer to a multi-tasking operating system than that of a true single-tasking OS such as CP/M. In fact, they even developed a multi-tasking version of MS-DOS (the so-called "European MS-DOS 4.0" [2]) but it was largely unsuccessful. Eventually Microsoft gave up on the idea of making MS-DOS multitasking, but much of the ideas and experience they developed in trying to do so ended up going into OS/2 and Windows instead.
[0] CP/M later evolved into MP/M, which was a true multi-tasking operating system, but I'm not talking about that here
That's what makes TSRs possible. A TSR is essentially just an ordinary program, the only difference is that when it exits, it tells the OS to leave the process in memory rather than unloading it.
Yeah, I remember programming a bunch of those in TurboPascal. There were compiler directives to limit the heap size and some inline assembly was needed to save/restore the stack inside the interrupt handler, but once it worked, the rest was like a normal program.
It was very useful to have a calculator, taking notes, etc. Now we take multitasking for granted, of course.
On Unix systems, and the Windows NT family, the current directory is per-process, so that doesn't work any more.
But still it bites me everyonce in a while with cd's per-drive curent dir. pushd doesn't have this issue but I need to remember to use it. Usually after a failed cd on the other drive.
Correct and complete systems are large and complex, sure, but they are almost always comprised of small and simple parts. Unix in the 90s was around 300k LOC, hardly a small and simple system, but if we're to believe the author, it must have been made of small and simple parts.
Multics. The Unix philosophy (and name) is a direct response to the failure of Multics. And they were right. It's true that by the 80's, we were starting to see real competition. But really what distinguished Unix in the 70's was that it worked at all on the kind of interactive hardware its users wanted to buy. Most stuff was either missing huge features, aimed at IBM style batch environments, or priced out of reach. Unix worked.
It worked because it was necessary. And occasionally still is.
But we have workable alternatives now. We have bigger teams with version control servers, static analysis, and safer languages.
We can make it general enough to be attract 1000 developers, then those 1000 developers can all decide to go for correctness in all cases with high feature richness.
"Software" barely existed at the time of UNIX. UNIX philosophy basically says "Don't write software, assemble it onsite as needed just for one task".
UNIX was amazing, and provided a great base to eventually evolve Linux and co, but it no longer seems to be the only way to do things.
We have bigger teams with version control servers, static analysis, and safer languages.
That’s why enterprise software is such a crap - what you are describing is part of a problem, not a solution.
Also, at that time big teams definitely existed: see OS/360 and Fred Brooks’ book. High level languages were there too, eg Lisp. What you want to replace Unix with is exactly what Unix itself had replaced.
I'm not sure I've everyone been on the user side of true "enterprise software" since I don't do business side work, but basically all the commercial-inspirex stuff is made with the same idea.
Feature rich, high level languages, high integration, opinionated workflow. Krita, FreeCAD, LibreOffice, Ardour, browsers, and VSCode mostly all seem to fully ignore anything UNIXy, and are amazing.
I don't really see software doing significant amounts of sucking these days. When I do, it's usually because it's subscription or cloud dependent, or because they left out something important for simplicity's sake.
Software basically runs the entire modern world. The expectations of computers are no longer to just "Compute". Everything is interactive, embedded systems are everywhere, and the stuff that really is "Computing" is often GPU accelerated or distributed.
Computers were their own separate thing in the UNIX era. They weren't fully replacements for any other device yet, outside of research. People used real filing cabinets and records or cassettes.
Large complex software enables huge classes of applications that would otherwise have enough friction that nobody would want them, you'd juat get a pen and paper rather than reading a man page to do what should be a 4 second task.
And when you have dozens or hundreds of them, not many people are going to want to learn them all, and actively go out of their way to stay current with them.
People complained CONSTANTLY about computers into the early 2000s. Somehow it was always a challenge, you always had to spend time figuring out how to get the computer to do what you wanted, and you might often wonder if it was worth using them at all.
UNIX users have a very different perspective from average users. They spend a lot of time dealing with text and processing data, and mostly like computers for academic reasons or for tasks that are completely impractical by hand. Not many of them seem to actually want an ubiquitous IoT type future with an app for anything.
With an appreciation of simplicity seems to come a love of the analog, because that's the ultimate UNIX philosophy, not using a computer at all.
1. It's been the case for many, many decades; nothing new there, and
2. All your examples are end-user applications, not operating systems.
Operating systems - and perhaps frameworks in general - are different, in that complexity is much more of a problem. Operating system is not just another application, it's the foundation of the entire thing. All this text processing stuff you've mentioned: this is just a part of Unix, the "icing". What matters lies below.
Computers were their own separate thing in the UNIX era.
Back when computers were their own separate thing, Unix was just one of a bunch of vastly different systems. Very large market share - perhaps even bigger than Unix - was VMS, and MVS was a thing too; then you had a bunch of less common ones, like Pr1me.
Now, however, literally every phone runs Unix.
With an appreciation of simplicity seems to come a love of the analog, because that's the ultimate UNIX philosophy, not using a computer at all.
Or mechanical :-) Again, this is very true, but it is like this for a reason: it's because we understand the complexity involved and its consequences.
Linux really blurs the lines with what is part of the OS, and what is part of the application, since there's now a common set of middleware daemons that are used in most mainstream distros.
It used to be much more loose and modular, and also much more unpredictable with no real stable platform.
Now it's more like GNU/Linux/systemd/DBus/xdg, and most of what I love about Linux comes from the fact we've eventually evolved a real "platform" that is vaguely standard between popular distros, with just enough modularity to make devuan and void possible so we don't get rioting.
Android is even more like that. In practice it's inseparable from all the other stuff on top, and you never interact with UNIXy concepts. They don't really even want you to directly deal with files.
And.... it all works fine. Android is a bit of a developer nightmare, but it's wonderful for users, and Ubuntu-likes are great for both users and developers(As long as they don't want to customize stuff under the hood too much).
I rarely meet a techie who doesn't have quite the fondness for the mechanical though, and they all seem to be very smart, and very capable, so there must be some sort of reason.
I've rarely had trouble with the more complex things, either at home or at work, so I suspect some of it comes down to how much you value a sense of control and understanding.
You're probably gonna have a bad time with a mega-ultra-framework if you want to design an architecture to fit the task, and your own vision of Good Code, and then implement that.
But you'll probably have a great time if you enjoy finding ways to fit the application into The One True Way the framework is built for.
It seems like a lot of people who enjoy analog and especially paper notetaking specifically really like the lack of any predetermined structure.
Simple things always claim to be very logical and consistent and built from a small set of concepts, but it seems like the real result, and probably why people like them, is that they invite you to build more yourself, so you have in practice a near infinite set of features with a different subset in every project.
I'm reading over NASA's 10 rules for safety critical code and it's littered with the word simple and keeping things small. I don't think NASA is an example for large and complex at all.
Not to mention NASA is known for many of the the most famous software bugs in history, but I'm guessing that's just related to space exploration more than anything particularly novel/incorrect about their approach to software.
Comments
Where is this "complete and correct" code that came at the expense of being large and complex? I've certainly never seen it.
Richard Gabriel's "The Rise of Worse is Better," written around the same time period the Unix-HATERS Handbook was written, gives some clues. Unix was contrasted with environments such as Scheme (which is rather small, but because it's designed as a "crown jewel"), Common Lisp (the exemplar of a "big complex system" that is complete and correct, but large and complex), and the ITS operating system (https://en.wikipedia.org/wiki/Incompatible_Timesharing_Syste...). Thankfully there are many open-source Scheme and Common Lisp implementations, and ITS is also available as open source (https://github.com/PDP-10/its).
Of course, modern Unix-like systems these days are large and complex, though Plan 9 and Inferno are quite architecturally refined, reducing some of the complexities that you'll see in contemporary Unix-like systems.
At the time The Rise of Worse Is Better was written, Scheme had:
- no error handling. The specification described that certain situations result in an error, without defining what that means, or how you can catch and recover from one programmatically.
- no module system. The Scheme report didn't describe any feature for decomposing a Scheme program into multiple files. No rules governing how literal data goes into compiled files and what the rules are for recovering symbols. No evaluation controls (like don't evaluate this form when compiling, but only deposit it into a compiled file and such).
Unix was a useful production system used for running businesses, with error handling in its API's and compilation of programs decomposed into multiple files.
The "PC losering" problem was nicely resolved. Unixes developed sigaction, where you can specify it both ways: interrupts can bail the system call (which is sometimes what you want) or restart it (which is what you want at other times). Neither is inherently better. Sometimes you really don't want a long system call to go back to sleep as if nothing happened. A third solution is possible though: your signal can resume the system call when it returns normally, or else abandon it with a siglongjmp, which restores the signal mask to what it was at the sigsetjmp point. Then there is is various other signal paraphernalia. Basically the Unix people had the right intuitions in this area, and took an incremental approach whereby they gradually rolled in more correctness.
That's kinda the whole point of the essay: getting something up and running quick that is 'complete' (if not particularly elegant) is better than slowly and carefully working on a Good System. Unix ate Lisp's lunch because you could build things on it right now.
That essay definitely was a bit of a complaint about the crudeness of Unix. But I read it primarily as a wake up call (not to say epitaph) for the Ivory Tower Lisp developers who were dreaming about the perfect system while Unix ate their lunch.
What ate Lisp's lunch wasn't necessarily the Lispers dreaming of a perfect system. Rather, Unix was far more accessible in terms of availability than the contemporary Lisp systems coming out of Symbolics and Xerox at the time Unix really started to take off. Before the breakup of AT&T in 1984, Unix was available to universities under comparatively generous licensing terms. While source licenses were very expensive for companies and (after 1984) for universities, binary licenses were relatively inexpensive, and beginning in the 1990s we saw open source Unix-like operating systems such as Linux and the BSDs. Unix was not restricted to a particular architecture; it ran on a wide variety of hardware.
Contrast this with Symbolics Genera and Interlisp-D, the premier Lisp operating systems of the 1980s. A Symbolics workstation can easily cost five figures in mid-1980s dollars, and cheaper alternatives such as the MacIvory boards and OpenGenera on DEC Alpha machines weren't released until later in Symbolics' history. Interlisp-D originally ran on Xerox workstations that also cost five figures, though it was later ported to the Sun SPARC architecture (though I have no idea what Interlisp-D licenses cost). Both Symbolics Genera and Interlisp-D missed out on the open source revolution of the 1980s and 1990s. To this day Symbolics Genera remains proprietary, though thankfully Interlisp-D was made open-source recently (https://interlisp.org/). There are open-source Common Lisp compilers such as SBCL, but they are not full-fledged operating systems.
Part of the reason why Unix took off had less to do with Unix's design and had more to do with Unix's lower costs. DOS and Windows had even lower costs than Unix before open source Unix clones appeared, and they were (and, in the case of Windows, still is) widely used. While Unix's design characteristics certainly play a role, we shouldn't ignore the impact of cost and licensing.
In an alternative universe, imagine had RMS had set out on building a FOSS Lisp operating system (GNU Emacs doesn't count) instead of building GNU. RMS was a Lisp hacker at MIT who started the GNU project due to his frustrations with the proprietary Lisp machine companies like Symbolics that were spun off from MIT's AI lab, so it's not unlikely to think about an alternative situation where RMS decided to clone a Lisp OS instead of cloning Unix. I wonder if a community could have rallied behind an open source Lisp OS to be a challenging contender to Unix back in the 1980s?
Lisp machines were mostly just a single type of computer: a graphical workstation for Lisp developers. Those were mostly in R&D, typically in AI - largely financed by DARPA and similar, hoping for the AI software providing leadership in tech and military. The market for their software&hardware was on the leading edge, which later was taken from offerings growing on cheaper platforms.
They had one/two operating system to fork others from: single user, no terminal story, no security, needing large amounts of memory, needing specific expensive hardware (graphics, disks, custom boards, ...), not easy to port, needed complex (and hard to debug) memory management, scarce number of Lisp system programmers accumulated in a small number of companies, no public source story, ...
Many open source operating systems are copies or forks. There was not much original research. Writing a new operating system (say, a portable Lisp OS with device drivers, multi-user capabilities, security, terminal and GUI usage) would have needed a lot of work and there are not enough combined Lisp AND systems programmers to do that - educating them takes a lot of money.
Even today, the state of the 70s/80s Lisp OS hasn't been reached again: there is no comparable operating system written in Lisp and the old ones exist only as emulators of the past.
The tech base of the early Lisp system was too focused, there was no room to mutate and grow to different platforms in different incarnations. There were attempts (like embedded Lisp hard- & software) but that had no effect in the market.
UNIX was developed for a completely different market: simpler base software, very portable, multi-user, client&server, modular programs, terminal, ... a bunch of companies used UNIX as a base for their workstation offerings (SUN, SGI, HP, IBM, NeXT, ...) - the rest is history.
But Lisp OS like software on the UNIX systems was working well in the 80s and early 90s. They still needed large amount of RAM, large amount of virtual memory, responsive GUIs, advanced GC support, ... some high-end machines could offer that and they fixed enough OS bugs to run large Lisp systems. Every larger UNIX vendor had some Lisp story as part of their general offerings.
The Lisp people absolutely got something up and running quickly, which is why Lisp is one of the earliest languages. Big, institutional Lisp systems were in a different state of maturity when Unix was coming up.
Early Lisp didn't have defmacro with destructuring, backquote, lexical scope, condition handling, structures, hash tables, ... you wouldn't want to use it today.
Sometimes I wonder, if you wanted to create an operating system like ITS for more modern platforms, what it would look like?
ITS is written in PDP-10 assembly, what if someone wrote a compiler which read in PDP-10 assembly language and spit out C code? Could that be a first step to porting it?
It is surely a lot more complicated than that. It contains self-modifying code, which would obviously break that translation strategy. A lot of hardware-specific code would have to be rewritten. 6 character filenames without nested directories might have been acceptable in the 1970s, but few could endure it today. A multi-user system with a near-total absence of security was acceptable back then, obviously not in today's very different world.
ITS does have some interesting features contemporary systems don't:
A process can submit commands to be run by the shell that spawned it – actually MS-DOS COMMAND.COM also had that feature (INT 2E), but I haven't seen anything else with it. A Unix shell could implement this by creating a Unix domain socket, and passing its path to subprocesses via an environment variable–but I've never seen that done.
Another was that a program being debugged could call an API to run commands in its own debugger – I've never seen that in any other debugger, although I suppose you could write a GDB plugin to implement it (have a magic do-nothing function, then a GDB Python script sets a breakpoint on that function, and interprets its argument as commands for GDB.) Actually, in ITS these two features were the exact same feature, since the debugger was used as the command shell.
Another was that a program had named subprocesses (by default the subprocess name was the same as the executable name, but it didn't have to be.) Compare that to most Unix shells, where it is easy to forget what you are running as background jobs 1, 2 or 3.
I wish we had more of this kind of content on Hacker News.
Interesting context thanks!
What would you use it for? How is that superior to simply spawning a new shell process?
Norton Utilities for DOS came with a program called NCD (Norton Change Directory). It was a full-screen replacement for the CD command, you could browse through the filesystem to get to the directory you wanted, then it would exit and return you to COMMAND.COM's prompt, with you already in that directory. That worked because under DOS, the current directory was system-wide, [0] not per-process, so a subprocess could change the parent process' current directory. On Unix systems, and the Windows NT family, the current directory is per-process, so that doesn't work any more.
Now, you can implement the same idea on Unix–but it is more complex. You need to define a shell function in your .profile/.bash_profile/.zprofile/whatever. That shell function then executes the external change-directory program, and passes the destination directory back to the shell function somehow. The shell function then actually changes the directory.
What if shells had exposed some kind of standard API to their subprocesses? Maybe something as simple as an environment variable containing the path to a Unix domain socket which accepts a simple text-based protocol. You could then implement an NCD-style command, by having it send the shell which called it a request to change its directory. That way, you would not have to install such a command by modifying your profile and restarting your shell, the command would just work the first time you ran it.
Similarly, there are a lot of tools out there for supporting multiple versions of development tools concurrently (Environment Modules, venv for Python, nvm for Node, conda, etc.) Most of these tools work by manipulating your shell environment. Since, on Unix, environment variables are per-process, and a process cannot modify the environment of its parent, you have to install some shell function in your profile and restart your shell before using one of them. Once again, if shells exported some kind of "get/set shell environment" API to their subprocesses, these kind of tools would work the first time you ran them, without any need to modify your shell profile.
[0] Actually, in MS-DOS and Windows 3.x/9x/Me, there is both a system-wide current drive, and a separate system-wide current directory for each drive – Windows NT family's cmd.exe simulates this by using a hidden environment variable per a drive to store its current directory, but that is a convention which only cmd.exe supports, the rest of Windows knows nothing about it.
For people who don't know DOS: DOS didn't really have multiple processes. COMMAND.COM made part of it resident (TSR), which is how it "survived" invoking other applications. At any given point, there was exactly one process running, and all the state you'd typically associate with a process was "system-wide".
Well, compared to CP/M, it did. CP/M only supported a single program being loaded into memory at a time [0]–because every program was loaded into memory at the same absolute memory address (0x100). By contrast, DOS could have multiple processes in memory simultaneously, even though only one of them at a time could be the active foreground process. DOS processes can launch child processes; the parent will be suspended (while remaining in memory) while the child process executes, but will then resume execution once the child process finishes. Under CP/M, that was impossible, there was no API to do that, the architecture didn't support the existence of one.
Indeed, while the main execution of the parent process would be suspended while the child process ran, any interrupt handlers installed by the parent would still be invoked, so some of the parent's code could still run while the child was executing. A parent process can even expose an API to its children – which is exactly what COMMAND.COM does (such as INT 2E which I mentioned)
That's what makes TSRs possible. A TSR is essentially just an ordinary program, the only difference is that when it exits, it tells the OS to leave the process in memory rather than unloading it. By contrast, the CP/M equivalent to TSRs, RSXs (Resident System Extensions), are completely different from ordinary CP/M programs. They are loaded at a variable address near the top of memory, instead of a fixed address at the bottom.
COMMAND.COM isn't a TSR. COMMAND.COM's "residence" is essentially the same as any other program which spawns a child program under DOS, the parent always remains in memory while the child runs. Unlike a TSR, when a COMMAND.COM instance terminates (the initial COMMAND.COM instance won't, but inferior instances will), it doesn't stay in memory, it is unloaded.
What makes COMMAND.COM somewhat unusual, is that it splits itself into two portions, a "resident" portion and a "transient" portion. The resident porition is loaded at the bottom of memory, the transient portion is loaded at the top of free memory, without being officially allocated. Since DOS normally allocates memory in a bottom-up manner, if the child program doesn't use much memory, the transient portion will not be overwritten. When the chlid program returns, COMMAND.COM tests the integrity of the transient portion – if it finds it has been overwritten (because the child program needed that much memory), it reloads it from disk before continuing. That's quite unusual behaviour, but still rather different from how TSRs behave.
That's not true. When you spawn a child process, you can either let it inherit your environment variables, or you can provide it with a new environment segment – thus each process in the system can potentially have different environment variables. Similarly, file handles are per-process. Similar to Unix – indeed, the DOS 2.x file handle design was copied from Unix – every handle has an "inherit" flag, which determines whether child processses inherit that handle from their parent. DOS keeps track of which processes have which files open, so when the last process with that file open terminates, the file is closed. Likewise, allocated memory blocks are per-process, so when a process is unloaded, DOS frees its memory blocks, but not those of ancestor processes or TSRs.
If DOS has per-process environment variables, file handles and memory allocations – why not current directories as well? I don't know, but I can speculate: DOS 2.0 copied the idea of a current directory from Unix, so like Unix they probably would have initially planned to make it per-process. However, requirements for backward compatiblity with DOS 1.x programs pushed them towards having a separate current directory per each drive. Keeping a separate current directory per-process might have been feasible if it was just a single current directory, but having to do so separately for each drive would have made it a lot more complex and wasteful of memory. I think that is why they made it system-wide instead of per-process.
Microsoft always planned to make MS-DOS a multi-tasking operating system, and much of its internal architecture was designed to support gradual evolution towards multi-tasking [1] – which makes its architecture somewhat closer to a multi-tasking operating system than that of a true single-tasking OS such as CP/M. In fact, they even developed a multi-tasking version of MS-DOS (the so-called "European MS-DOS 4.0" [2]) but it was largely unsuccessful. Eventually Microsoft gave up on the idea of making MS-DOS multitasking, but much of the ideas and experience they developed in trying to do so ended up going into OS/2 and Windows instead.
[0] CP/M later evolved into MP/M, which was a true multi-tasking operating system, but I'm not talking about that here
[1] See for example this design document in the MS-DOS 2.x source code: https://github.com/microsoft/MS-DOS/blob/master/v2.0/bin/DEV...
[2] https://www.pcjs.org/software/pcx86/sys/dos/microsoft/4.0M/
That's what makes TSRs possible. A TSR is essentially just an ordinary program, the only difference is that when it exits, it tells the OS to leave the process in memory rather than unloading it.
Yeah, I remember programming a bunch of those in TurboPascal. There were compiler directives to limit the heap size and some inline assembly was needed to save/restore the stack inside the interrupt handler, but once it worked, the rest was like a normal program.
It was very useful to have a calculator, taking notes, etc. Now we take multitasking for granted, of course.
But still it bites me everyonce in a while with cd's per-drive curent dir. pushd doesn't have this issue but I need to remember to use it. Usually after a failed cd on the other drive.
If you have a program 'finddir' that just prints out the directory, you could simply to 'cd `finddir`' or make that a shell alias 'ncd'.
Correct and complete systems are large and complex, sure, but they are almost always comprised of small and simple parts. Unix in the 90s was around 300k LOC, hardly a small and simple system, but if we're to believe the author, it must have been made of small and simple parts.
Multics. The Unix philosophy (and name) is a direct response to the failure of Multics. And they were right. It's true that by the 80's, we were starting to see real competition. But really what distinguished Unix in the 70's was that it worked at all on the kind of interactive hardware its users wanted to buy. Most stuff was either missing huge features, aimed at IBM style batch environments, or priced out of reach. Unix worked.
It worked because it was necessary. And occasionally still is.
But we have workable alternatives now. We have bigger teams with version control servers, static analysis, and safer languages.
We can make it general enough to be attract 1000 developers, then those 1000 developers can all decide to go for correctness in all cases with high feature richness.
"Software" barely existed at the time of UNIX. UNIX philosophy basically says "Don't write software, assemble it onsite as needed just for one task".
UNIX was amazing, and provided a great base to eventually evolve Linux and co, but it no longer seems to be the only way to do things.
That’s why enterprise software is such a crap - what you are describing is part of a problem, not a solution.
Also, at that time big teams definitely existed: see OS/360 and Fred Brooks’ book. High level languages were there too, eg Lisp. What you want to replace Unix with is exactly what Unix itself had replaced.
I'm not sure I've everyone been on the user side of true "enterprise software" since I don't do business side work, but basically all the commercial-inspirex stuff is made with the same idea.
Feature rich, high level languages, high integration, opinionated workflow. Krita, FreeCAD, LibreOffice, Ardour, browsers, and VSCode mostly all seem to fully ignore anything UNIXy, and are amazing.
I don't really see software doing significant amounts of sucking these days. When I do, it's usually because it's subscription or cloud dependent, or because they left out something important for simplicity's sake.
Software basically runs the entire modern world. The expectations of computers are no longer to just "Compute". Everything is interactive, embedded systems are everywhere, and the stuff that really is "Computing" is often GPU accelerated or distributed.
Computers were their own separate thing in the UNIX era. They weren't fully replacements for any other device yet, outside of research. People used real filing cabinets and records or cassettes.
Large complex software enables huge classes of applications that would otherwise have enough friction that nobody would want them, you'd juat get a pen and paper rather than reading a man page to do what should be a 4 second task.
And when you have dozens or hundreds of them, not many people are going to want to learn them all, and actively go out of their way to stay current with them.
People complained CONSTANTLY about computers into the early 2000s. Somehow it was always a challenge, you always had to spend time figuring out how to get the computer to do what you wanted, and you might often wonder if it was worth using them at all.
UNIX users have a very different perspective from average users. They spend a lot of time dealing with text and processing data, and mostly like computers for academic reasons or for tasks that are completely impractical by hand. Not many of them seem to actually want an ubiquitous IoT type future with an app for anything.
With an appreciation of simplicity seems to come a love of the analog, because that's the ultimate UNIX philosophy, not using a computer at all.
What you are saying is true, however:
1. It's been the case for many, many decades; nothing new there, and
2. All your examples are end-user applications, not operating systems.
Operating systems - and perhaps frameworks in general - are different, in that complexity is much more of a problem. Operating system is not just another application, it's the foundation of the entire thing. All this text processing stuff you've mentioned: this is just a part of Unix, the "icing". What matters lies below.
Back when computers were their own separate thing, Unix was just one of a bunch of vastly different systems. Very large market share - perhaps even bigger than Unix - was VMS, and MVS was a thing too; then you had a bunch of less common ones, like Pr1me.
Now, however, literally every phone runs Unix.
Or mechanical :-) Again, this is very true, but it is like this for a reason: it's because we understand the complexity involved and its consequences.
Linux really blurs the lines with what is part of the OS, and what is part of the application, since there's now a common set of middleware daemons that are used in most mainstream distros.
It used to be much more loose and modular, and also much more unpredictable with no real stable platform.
Now it's more like GNU/Linux/systemd/DBus/xdg, and most of what I love about Linux comes from the fact we've eventually evolved a real "platform" that is vaguely standard between popular distros, with just enough modularity to make devuan and void possible so we don't get rioting.
Android is even more like that. In practice it's inseparable from all the other stuff on top, and you never interact with UNIXy concepts. They don't really even want you to directly deal with files.
And.... it all works fine. Android is a bit of a developer nightmare, but it's wonderful for users, and Ubuntu-likes are great for both users and developers(As long as they don't want to customize stuff under the hood too much).
I rarely meet a techie who doesn't have quite the fondness for the mechanical though, and they all seem to be very smart, and very capable, so there must be some sort of reason.
I've rarely had trouble with the more complex things, either at home or at work, so I suspect some of it comes down to how much you value a sense of control and understanding.
You're probably gonna have a bad time with a mega-ultra-framework if you want to design an architecture to fit the task, and your own vision of Good Code, and then implement that.
But you'll probably have a great time if you enjoy finding ways to fit the application into The One True Way the framework is built for.
It seems like a lot of people who enjoy analog and especially paper notetaking specifically really like the lack of any predetermined structure.
Simple things always claim to be very logical and consistent and built from a small set of concepts, but it seems like the real result, and probably why people like them, is that they invite you to build more yourself, so you have in practice a near infinite set of features with a different subset in every project.
Indeed, Microsoft Xenix was implemented on 80286 processors, which allowed a maximum of 64k of program text per process.
The POSIX shell was designed to fit within that limit.
NASA has written such code.
I'm reading over NASA's 10 rules for safety critical code and it's littered with the word simple and keeping things small. I don't think NASA is an example for large and complex at all.
Not to mention NASA is known for many of the the most famous software bugs in history, but I'm guessing that's just related to space exploration more than anything particularly novel/incorrect about their approach to software.