An external program communicating over pipes can let the OS handle buffering, run on another process core, be swapped out for another program that speaks the same protocol (perhaps in a faster language), won't crash the whole system, etc.
A program running as a library subroutine has a bit less overhead, and can use more context (library-native data structures, rather than piped text), but this is also usually more language-specific. Working within a "full environment" language like Smalltalk has a lot of advantages, but it also needs comprehensive libraries for your problem domain, or you're back to using external programs.
There's an insightful aside about this in Joe Armstrong's _Programming Erlang_, in the chapter about ports - Erlang code can load foreign code as linked-in libraries, but a buggy library will make the whole system unstable in a way that code running in a foreign process and communicating via message passing will not. He argues for running code in an external process (a "port") by default.
Of course, having a comprehensive (but low-level) library in C with wrappers in higher-level languages is an option. High-level languages' type systems / object models can be very different, though, and it takes experience to translate a C API to feel native to Python/Lua/Ruby/etc.
It also works to structure a program as a C library, but provide a small standalone program which gives it a command line interface. SQLite and Lua are good examples of the latter approach.
I (still) think that message passing is a vastly undervalued mechanism. Erlang is a really interesting language by the way, I wished I had more time to devote to learning it.
Comments
There are benefits to both approaches.
An external program communicating over pipes can let the OS handle buffering, run on another process core, be swapped out for another program that speaks the same protocol (perhaps in a faster language), won't crash the whole system, etc.
A program running as a library subroutine has a bit less overhead, and can use more context (library-native data structures, rather than piped text), but this is also usually more language-specific. Working within a "full environment" language like Smalltalk has a lot of advantages, but it also needs comprehensive libraries for your problem domain, or you're back to using external programs.
There's an insightful aside about this in Joe Armstrong's _Programming Erlang_, in the chapter about ports - Erlang code can load foreign code as linked-in libraries, but a buggy library will make the whole system unstable in a way that code running in a foreign process and communicating via message passing will not. He argues for running code in an external process (a "port") by default.
Of course, having a comprehensive (but low-level) library in C with wrappers in higher-level languages is an option. High-level languages' type systems / object models can be very different, though, and it takes experience to translate a C API to feel native to Python/Lua/Ruby/etc.
It also works to structure a program as a C library, but provide a small standalone program which gives it a command line interface. SQLite and Lua are good examples of the latter approach.
I (still) think that message passing is a vastly undervalued mechanism. Erlang is a really interesting language by the way, I wished I had more time to devote to learning it.