Lots of little programs that can be easily chained together.
I think that's usually described as "the Unix philosophy." It's not limited to GNU (nor does it originate with GNU). See, for example, the Wikipedia article on "Unix philosophy"[1]:
Doug McIlroy, the inventor of Unix pipes and one of the founders of the Unix tradition, summarized the philosophy as follows:[2]
This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
This is usually abridged to "Write programs that do one thing and do it well".
An external program communicating over pipes can let the OS handle buffering, run on another process core, be swapped out for another program that speaks the same protocol (perhaps in a faster language), won't crash the whole system, etc.
A program running as a library subroutine has a bit less overhead, and can use more context (library-native data structures, rather than piped text), but this is also usually more language-specific. Working within a "full environment" language like Smalltalk has a lot of advantages, but it also needs comprehensive libraries for your problem domain, or you're back to using external programs.
There's an insightful aside about this in Joe Armstrong's _Programming Erlang_, in the chapter about ports - Erlang code can load foreign code as linked-in libraries, but a buggy library will make the whole system unstable in a way that code running in a foreign process and communicating via message passing will not. He argues for running code in an external process (a "port") by default.
Of course, having a comprehensive (but low-level) library in C with wrappers in higher-level languages is an option. High-level languages' type systems / object models can be very different, though, and it takes experience to translate a C API to feel native to Python/Lua/Ruby/etc.
It also works to structure a program as a C library, but provide a small standalone program which gives it a command line interface. SQLite and Lua are good examples of the latter approach.
I (still) think that message passing is a vastly undervalued mechanism. Erlang is a really interesting language by the way, I wished I had more time to devote to learning it.
One strategy that I've used before is to write every program as a library, then put a very thin command-line program in front of it. All new functionality goes into the library, then gets exposed.
That gives the best of both worlds. I have my usable program. And later I can easily integrate it into any other library that can benefit.
One particular area where the external program bit is annoying is when you're querying or setting system/package parameters or defaults on a unix machine, there must be 100 incompatible ways of doing that.
Usually the only way to get the job done is to escape to a shell, which really feels kludgy.
I can see some of the charm of 'images' such as used by smalltalk.
Or maybe the appeal of glue languages like Bash or (one style of) Perl. It is very nice to have the ability to treat arbitrary programs like libraries for your program. (Unfortunately, because of the GNU/BSD split - among other things - this style of programming is also completely brittle. One non-standard flag, and boom.)
I suppose the trick, then, would be to take a bare system call and somehow automatically promote it to a process in the operating system?
You could do this for a given function signature, perhaps. But it's not clear how it would work in the general case. Is there some universal function signature that makes sense for any kind of API, that makes sense both for pipes and for the semantics of the specific program?
You're on the way to re-inventing Erlang I think ;)
That would be one way to do it, but system calls are generally assumed to be atomic, promoting them to process status would be quite tricky.
Multi-threaded kernels effectively do some of this by allowing you to execute multiple system calls in parallel.
The typical way in which an 'image' based system works is that all your persistence (including compiled code) continues to reside in the image, over time you get the same kind of build-up that you typically see in file systems, only there there is no difference between 'programs' and 'subroutines'.
But that's not 'natural', that's an external process with a whole pile of start-up and shut-down overhead. Incidentally, some of the worst C code I've ever seen used piped unix shell commands all over the place, as if there is no penalty to doing this.
That's true. I think the tricky part is providing segregation where it's needed. That is, the problem is in supporting systems and programs which don't conform to the standard. Process segregation and scheduling is one way of solving this problem. Another is to simply enforce the design uniformly across the entire system. IIRC Plan9 does this slightly, but a better example may be a system which uses e.g., Haskell as the base language and simply requires all programs (modules) to be interfaced with the system with public facing symbols. A 'grep' IO monad may be an interesting way of linking and segregation.
It does seem that GNU is bigger on it than the BSDs these days, though. The modern BSD approach is to write libraries, and then have shell utilities just be frontends on those libraries (sometimes with multiple front ends sharing the same library).
Stop! Whoever crosseth the bridge of Death, must answer first these questions three, ere the other side he see!
"What... is your name?"
"Sir Brian of Bell."
"What... is your quest?"
"I seek the Holy Grail."
"What... are four lowercase letters that are not legal flag arguments to the Berkeley UNIX version of `ls'?"
"I, er.... AIIIEEEEEE!"
Comments
This is essentially the same as the "RISC patent" - a patent that essentially said "if you make something simpler, it'll go faster"
Quoted from James Gosling, http://nighthacks.org/roller/jag/entry/quite_the_firestorm
Also, kudo's to the 15 years maintainer of GNU grep.
Isn't that a big part of the GNU mentality too? Lots of little programs that can be easily chained together.
Lots of little programs that can be easily chained together.
I think that's usually described as "the Unix philosophy." It's not limited to GNU (nor does it originate with GNU). See, for example, the Wikipedia article on "Unix philosophy"[1]:
Doug McIlroy, the inventor of Unix pipes and one of the founders of the Unix tradition, summarized the philosophy as follows:[2]
This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
This is usually abridged to "Write programs that do one thing and do it well".
[1]http://en.wikipedia.org/wiki/Unix_philosophy#McIlroy:_A_Quar...
[2]http://www.faqs.org/docs/artu/ch01s06.html
There is this strange wall between 'program' and 'subroutines' that at times feels completely artificial.
Why shouldn't 'grep' be automatically available as a routine once programmed?
I can see some of the charm of 'images' such as used by smalltalk.
There are benefits to both approaches.
An external program communicating over pipes can let the OS handle buffering, run on another process core, be swapped out for another program that speaks the same protocol (perhaps in a faster language), won't crash the whole system, etc.
A program running as a library subroutine has a bit less overhead, and can use more context (library-native data structures, rather than piped text), but this is also usually more language-specific. Working within a "full environment" language like Smalltalk has a lot of advantages, but it also needs comprehensive libraries for your problem domain, or you're back to using external programs.
There's an insightful aside about this in Joe Armstrong's _Programming Erlang_, in the chapter about ports - Erlang code can load foreign code as linked-in libraries, but a buggy library will make the whole system unstable in a way that code running in a foreign process and communicating via message passing will not. He argues for running code in an external process (a "port") by default.
Of course, having a comprehensive (but low-level) library in C with wrappers in higher-level languages is an option. High-level languages' type systems / object models can be very different, though, and it takes experience to translate a C API to feel native to Python/Lua/Ruby/etc.
It also works to structure a program as a C library, but provide a small standalone program which gives it a command line interface. SQLite and Lua are good examples of the latter approach.
I (still) think that message passing is a vastly undervalued mechanism. Erlang is a really interesting language by the way, I wished I had more time to devote to learning it.
One strategy that I've used before is to write every program as a library, then put a very thin command-line program in front of it. All new functionality goes into the library, then gets exposed.
That gives the best of both worlds. I have my usable program. And later I can easily integrate it into any other library that can benefit.
That's a really good approach.
One particular area where the external program bit is annoying is when you're querying or setting system/package parameters or defaults on a unix machine, there must be 100 incompatible ways of doing that.
Usually the only way to get the job done is to escape to a shell, which really feels kludgy.
I can see some of the charm of 'images' such as used by smalltalk.
Or maybe the appeal of glue languages like Bash or (one style of) Perl. It is very nice to have the ability to treat arbitrary programs like libraries for your program. (Unfortunately, because of the GNU/BSD split - among other things - this style of programming is also completely brittle. One non-standard flag, and boom.)
I suppose the trick, then, would be to take a bare system call and somehow automatically promote it to a process in the operating system?
You could do this for a given function signature, perhaps. But it's not clear how it would work in the general case. Is there some universal function signature that makes sense for any kind of API, that makes sense both for pipes and for the semantics of the specific program?
Interesting to think about, though.
You're on the way to re-inventing Erlang I think ;)
That would be one way to do it, but system calls are generally assumed to be atomic, promoting them to process status would be quite tricky.
Multi-threaded kernels effectively do some of this by allowing you to execute multiple system calls in parallel.
The typical way in which an 'image' based system works is that all your persistence (including compiled code) continues to reside in the image, over time you get the same kind of build-up that you typically see in file systems, only there there is no difference between 'programs' and 'subroutines'.
'grep' is available once programmed. Just use popen(3). Until you naturally think this way, you will not understand the Unix philosophy.
I (obviously) realize that you can do that.
But that's not 'natural', that's an external process with a whole pile of start-up and shut-down overhead. Incidentally, some of the worst C code I've ever seen used piped unix shell commands all over the place, as if there is no penalty to doing this.
That's true. I think the tricky part is providing segregation where it's needed. That is, the problem is in supporting systems and programs which don't conform to the standard. Process segregation and scheduling is one way of solving this problem. Another is to simply enforce the design uniformly across the entire system. IIRC Plan9 does this slightly, but a better example may be a system which uses e.g., Haskell as the base language and simply requires all programs (modules) to be interfaced with the system with public facing symbols. A 'grep' IO monad may be an interesting way of linking and segregation.
It does seem that GNU is bigger on it than the BSDs these days, though. The modern BSD approach is to write libraries, and then have shell utilities just be frontends on those libraries (sometimes with multiple front ends sharing the same library).
As telemachos says, that's the Unix philosophy. The GNU philosophy is to add options to programs until you run out of letters in the alphabet.
Stop! Whoever crosseth the bridge of Death, must answer first these questions three, ere the other side he see!
"What... is your name?" "Sir Brian of Bell." "What... is your quest?" "I seek the Holy Grail." "What... are four lowercase letters that are not legal flag arguments to the Berkeley UNIX version of `ls'?" "I, er.... AIIIEEEEEE!"