RIP. I remember when I was a young guy learning programming, reading the source code of various DOS based LZ, LZH or LZW compression utilities and implementing myself a LZH algorithm thinking it will bring me great fame and fortune. Lempel, Ziv and Eugene Roshall were my heroes.
At that time many young programmers regarded compression, compiler writing and anti-virus writing as very important. Now people just learn React. :)
Compression and compiler writing are just as important today as they were back then. And the closer to the real end of cheap hardware improvements we get the more important they will be. They are where real progress lies, the rest was just a temporary freebie.
Take cars: cars have a reasonable upper limit in physics so low that the difference between 'mild' (100 kph on the highway) and insane (300 kph+, no matter where) is a mere factor of three. And it took a good 100 years before that barrier was broken. It won't progress much beyond the standard of 100 or so kph under normal circumstances. There are just too many things that work against you when you go much faster than that including energy consumption, braking distance, the willingness of your vehicle to stay on the ground, attention requirements (ok, AI... someday...) and so on. So even if it is possible in exceptional cases it likely won't be the norm, even on closed over and evacuated freeways.
Aircraft did a little better, but even there the supersonic passenger jet came and went. Though it may be back one day.
In comparison computers have gotten many orders of magnitude faster in a much shorter time. But there is a physical limit, and we are very likely to run into that soon. And once that happens the only way to progress is to make more efficient use of the resources you've already got.
I don’t agree. Aircraft had their growth phase earlier, first flight in 1903 to landing on the moon in 1969. Airliners look like they did in the 60s because there’s nothing left to improve on with regards to efficiency. How many orders of magnitude are there between the wright flier and a 747?
We’re reaching the diminishing returns portion of the logistic curve for computing with regards to both things like hardware and compiling, compression, etc. It’s just hard to compare industries which are done growing with ones that still feel like they might be exponential forever.
It’s just hard to compare industries which are done growing with ones that still feel like they might be exponential forever.
Some industries never evolved too much. I don't see a big functional difference between cars a 100 years ago and cars now. And there were electric cars in 1900s.
A car is constraint by lots of things just like GP comment affirmed. Putting more gadgets in a car won't change what a car is or how it works at a fundamental level.
From the very first microprocessors to today's there is a big leap. If we apply that leap to cars they should be using nuclear fusion and take us to moon and back in seconds. And it's not like trillions of dollars weren't invested in car R&D in more than a century.
Software growth seems exponential only because of hardware and the Moore's Law. Which will come to an end soon.
Thanks to Moore's Law, we programmers, afforded to be careless, not think about performance or use unnecessary abstractions just out of esthetical or ideologic considerations. It allowed businesses to say that time to market trumps everything and customers should simply buy more powerful boxes or more boxes. Now this will come to an end soon and we will have to think about efficiency, speed and performance.
Or maybe chips will move from silicon to some alternative processes where growth is still possible.
You are making my point stronger rather than weaker: Aircraft have only become more efficient in the last 50 years or so and that's precisely the equivalent of better compilers and compression.
And the diminishing returns for computing are for the most part driven by the lack of investment in new ways of computing because we got so much for free. But once the free ride is over you can expect the research into efficiency to be picked up again because that will be the only field where real progress can be made. There have been a few half-assed (apologies to those involved, it was no doubt a ton of work) attempts at making computing fabrics and clockless machinery. And that's because by the time you have something working the scaling advantages have already overtaken your work and absolutely nobody is going to give up a working architecture for something unproven if it doesn't given an immediate return. Some pretty good ideas died like that.
Yes, for digital computing. But computing is on the cusp of becoming more non-determistic, which will allow analog computing to absolutely demolish digital computing in performance.
In comparison computers have gotten many orders of magnitude faster in a much shorter time. But there is a physical limit, and we are very likely to run into that soon. And once that happens the only way to progress is to make more efficient use of the resources you've already got.
Good points. And I will even argue that while we didn't hit a physical limit in hardware yet, it's still important to have a reasonable performance because not having performance means resources are wasted, money are wasted and it might lead to user frustration.
But it seems to not be the majority's opinion or at least not most of the business owners opinion.
Unfortunately until crappy and slow performing software starts losing business to well-implemented alternatives, it will never change. The current PM model is purely based on shipping fast, and any thought to performance is immediately dismissed as "pre-mature optimization." I've been consulting for companies that had to rolling reboot their entire cluster of app servers every couple of hours because a host of memory leaks (that also added O(n) runtime) would slowly exhaust resources. Rather than have someone fix the problem, they chose to invest in devops to automate the rolling restarts. This is not all that unusual in my experience, because customers will tolerate very sluggish response times so it's not worth fixing to the business.
Entirely my experience. I worked for a very large company and since our microservices were running in Kubernetes it was totally OK for the services to crash. We just logged the things in elastic stack, used data dog to monitor, used some SRE to restart services when things were messy.
I've lost days to solve a memory bug in one of the services and no one cared.
The original architect and programmers were gone, the engineering managers were pushing Clean, SOLID and design patterns hopping that will help and the domain was hard because we had to deal with real money while respecting tens of different laws and locales.
We tried at our best to do the impossible, we were understaffed by a factor of 5X.
And the services were crashing like crazy but it looked like a no downtime because we provisioned another POD in Kubernetes.
Provisioning was like: how many pods do we need? 3? Let's make it nine. How much memory do we need? Half a gig? Let's make it 4 gig to be sure.
So it was a mess and it worked and that mess continues to work somehow.
Not sure if mess driving engineering is a sound bussineses idea, though.
Erlang has taken 'let it crash' to entirely new heights, but that's from a top level viewpoint, components are allowed to crash but the service isn't. And you're still supposed to figure out why your service crashed but I'm pretty sure not everybody does that.
So as long as your supervision mechanism is bulletproof (supervision trees) then you can get away with this for very long.
At that time many young programmers regarded compression […] as very important.
They still do regard it as important, just in a different sense. Shannon formally proved that compression is congruent to statistical inference — an efficient compression algorithm seeks to minimize the statistical entropy of the scheme used to encode a message. Compression is thus loosely congruent to a branch of statistical inference you may have heard a thing or two about called "artificial intelligence" [0], which many young programmers are very much engaged with.
Take the Hutter Prize [1], which awards money for the ability to compress the entire English Wikipedia, or the Large Text Compression Benchmark [2], the latter of which's current top performer is a transformer-based neural network [3]. As the LTCP's rationale says [0],
Given a probability distribution P over strings s, the Shannon capacity theorem states that the optimal (shortest average) code for s has length log 1/P(s) bits. If the distribution P is known, then such codes are in fact easy to find and compute. The problem is that for natural language, we do not know how to explicitly compute the model P(s) for a given string s.
However humans must implicitly know P(s) because our speech and writing, by definition, follows this distribution. Now suppose that a compute can compute P(s) for all s. Then for any question q (or dialog ending in a question) and any answer a, the machine can compute P(a|q) = P(q,a)/P(q), and use this distribution to answer any question q posed by the judge. By definition, this distribution would be identical to the distribution of answers given by the human, and the two would be indistinguishable.
While compression is congruent with some cases of statistical inference, no one is trying to do statistical inference using compression. Even the example you gave of Fabrice Bellard (which I highly admire) does the converse and uses neural networks to compress.
Even for realizing what you've said one needs a bit of theory, basic computer science and some math. And most people today have a "do" philosophy and are result oriented. They are not learn oriented, they hate CS and math and they hate low level programming.
I don't think compression is highly relevant today, but at the time I started learning it was a good tool to sharpen one's mind. Learn algorithms, learn about memory management, learn about hardware, learn low level constructs, learn a bit of math.
Whenever a young person wanting to undertake a programmer career is asking me for advice I'll ask what their motivation is. If they just want money I advice them to learn the tools in fashion today, maybe Javascript and React, maybe Python, maybe C#. But if they respond they want to do it for both enjoyment and money, I put them on the hard path. I ask they enroll an BS CS program, or if that is not possible, give them long lists of books, tutorials, courses which cover CS basics, math used in CS basics, hardware, low level programming and the programming hot topics du jour. In that last case I also design their learning paths to accommodate their goals and I am always taking time to answer them questions in the future if I can, if not I will at least indicate where should they look or which is the right person to ask.
no one is trying to do statistical inference using compression
Well, maybe no one sensible? I still think it's quite cool that you can take any general purpose compression algorithm, and abuse it to do e.g. image classification. (Just append the image you want to classify to each of the classes' sets in turn, and see which compresses best!)
And actually I do remember a paper that tried to use ideas from PAQ to do online learning. Gated Linear Networks, out of DeepMind, in 2019:
Well stochastic processes are a much wider category than mere algebraic operations applied on group theory that losseles data compression use.
I'd think that calculus or functional theory or category theory can find more bijections towards statistics or even congruences than mere arithmetics or algebra ever will. Ok, you can explain or derive any mathematics construct using only algebra, and there were efforts to do so, but does it makes sense?
Comments
RIP. I remember when I was a young guy learning programming, reading the source code of various DOS based LZ, LZH or LZW compression utilities and implementing myself a LZH algorithm thinking it will bring me great fame and fortune. Lempel, Ziv and Eugene Roshall were my heroes.
At that time many young programmers regarded compression, compiler writing and anti-virus writing as very important. Now people just learn React. :)
Compression and compiler writing are just as important today as they were back then. And the closer to the real end of cheap hardware improvements we get the more important they will be. They are where real progress lies, the rest was just a temporary freebie.
Take cars: cars have a reasonable upper limit in physics so low that the difference between 'mild' (100 kph on the highway) and insane (300 kph+, no matter where) is a mere factor of three. And it took a good 100 years before that barrier was broken. It won't progress much beyond the standard of 100 or so kph under normal circumstances. There are just too many things that work against you when you go much faster than that including energy consumption, braking distance, the willingness of your vehicle to stay on the ground, attention requirements (ok, AI... someday...) and so on. So even if it is possible in exceptional cases it likely won't be the norm, even on closed over and evacuated freeways.
Aircraft did a little better, but even there the supersonic passenger jet came and went. Though it may be back one day.
In comparison computers have gotten many orders of magnitude faster in a much shorter time. But there is a physical limit, and we are very likely to run into that soon. And once that happens the only way to progress is to make more efficient use of the resources you've already got.
I don’t agree. Aircraft had their growth phase earlier, first flight in 1903 to landing on the moon in 1969. Airliners look like they did in the 60s because there’s nothing left to improve on with regards to efficiency. How many orders of magnitude are there between the wright flier and a 747?
We’re reaching the diminishing returns portion of the logistic curve for computing with regards to both things like hardware and compiling, compression, etc. It’s just hard to compare industries which are done growing with ones that still feel like they might be exponential forever.
Some industries never evolved too much. I don't see a big functional difference between cars a 100 years ago and cars now. And there were electric cars in 1900s.
A car is constraint by lots of things just like GP comment affirmed. Putting more gadgets in a car won't change what a car is or how it works at a fundamental level.
From the very first microprocessors to today's there is a big leap. If we apply that leap to cars they should be using nuclear fusion and take us to moon and back in seconds. And it's not like trillions of dollars weren't invested in car R&D in more than a century.
Software growth seems exponential only because of hardware and the Moore's Law. Which will come to an end soon.
Thanks to Moore's Law, we programmers, afforded to be careless, not think about performance or use unnecessary abstractions just out of esthetical or ideologic considerations. It allowed businesses to say that time to market trumps everything and customers should simply buy more powerful boxes or more boxes. Now this will come to an end soon and we will have to think about efficiency, speed and performance.
Or maybe chips will move from silicon to some alternative processes where growth is still possible.
You are making my point stronger rather than weaker: Aircraft have only become more efficient in the last 50 years or so and that's precisely the equivalent of better compilers and compression.
And the diminishing returns for computing are for the most part driven by the lack of investment in new ways of computing because we got so much for free. But once the free ride is over you can expect the research into efficiency to be picked up again because that will be the only field where real progress can be made. There have been a few half-assed (apologies to those involved, it was no doubt a ton of work) attempts at making computing fabrics and clockless machinery. And that's because by the time you have something working the scaling advantages have already overtaken your work and absolutely nobody is going to give up a working architecture for something unproven if it doesn't given an immediate return. Some pretty good ideas died like that.
But I suspect their day will come.
Yes, for digital computing. But computing is on the cusp of becoming more non-determistic, which will allow analog computing to absolutely demolish digital computing in performance.
Good points. And I will even argue that while we didn't hit a physical limit in hardware yet, it's still important to have a reasonable performance because not having performance means resources are wasted, money are wasted and it might lead to user frustration.
But it seems to not be the majority's opinion or at least not most of the business owners opinion.
Unfortunately until crappy and slow performing software starts losing business to well-implemented alternatives, it will never change. The current PM model is purely based on shipping fast, and any thought to performance is immediately dismissed as "pre-mature optimization." I've been consulting for companies that had to rolling reboot their entire cluster of app servers every couple of hours because a host of memory leaks (that also added O(n) runtime) would slowly exhaust resources. Rather than have someone fix the problem, they chose to invest in devops to automate the rolling restarts. This is not all that unusual in my experience, because customers will tolerate very sluggish response times so it's not worth fixing to the business.
Entirely my experience. I worked for a very large company and since our microservices were running in Kubernetes it was totally OK for the services to crash. We just logged the things in elastic stack, used data dog to monitor, used some SRE to restart services when things were messy.
I've lost days to solve a memory bug in one of the services and no one cared.
The original architect and programmers were gone, the engineering managers were pushing Clean, SOLID and design patterns hopping that will help and the domain was hard because we had to deal with real money while respecting tens of different laws and locales.
We tried at our best to do the impossible, we were understaffed by a factor of 5X.
And the services were crashing like crazy but it looked like a no downtime because we provisioned another POD in Kubernetes.
Provisioning was like: how many pods do we need? 3? Let's make it nine. How much memory do we need? Half a gig? Let's make it 4 gig to be sure.
So it was a mess and it worked and that mess continues to work somehow.
Not sure if mess driving engineering is a sound bussineses idea, though.
Erlang has taken 'let it crash' to entirely new heights, but that's from a top level viewpoint, components are allowed to crash but the service isn't. And you're still supposed to figure out why your service crashed but I'm pretty sure not everybody does that.
So as long as your supervision mechanism is bulletproof (supervision trees) then you can get away with this for very long.
They still do regard it as important, just in a different sense. Shannon formally proved that compression is congruent to statistical inference — an efficient compression algorithm seeks to minimize the statistical entropy of the scheme used to encode a message. Compression is thus loosely congruent to a branch of statistical inference you may have heard a thing or two about called "artificial intelligence" [0], which many young programmers are very much engaged with.
Take the Hutter Prize [1], which awards money for the ability to compress the entire English Wikipedia, or the Large Text Compression Benchmark [2], the latter of which's current top performer is a transformer-based neural network [3]. As the LTCP's rationale says [0],
[0] http://www.mattmahoney.net/dc/rationale.html
[1] http://prize.hutter1.net/
[2] http://www.mattmahoney.net/dc/text.html
[3] https://bellard.org/nncp/
While compression is congruent with some cases of statistical inference, no one is trying to do statistical inference using compression. Even the example you gave of Fabrice Bellard (which I highly admire) does the converse and uses neural networks to compress.
Even for realizing what you've said one needs a bit of theory, basic computer science and some math. And most people today have a "do" philosophy and are result oriented. They are not learn oriented, they hate CS and math and they hate low level programming.
I don't think compression is highly relevant today, but at the time I started learning it was a good tool to sharpen one's mind. Learn algorithms, learn about memory management, learn about hardware, learn low level constructs, learn a bit of math.
Whenever a young person wanting to undertake a programmer career is asking me for advice I'll ask what their motivation is. If they just want money I advice them to learn the tools in fashion today, maybe Javascript and React, maybe Python, maybe C#. But if they respond they want to do it for both enjoyment and money, I put them on the hard path. I ask they enroll an BS CS program, or if that is not possible, give them long lists of books, tutorials, courses which cover CS basics, math used in CS basics, hardware, low level programming and the programming hot topics du jour. In that last case I also design their learning paths to accommodate their goals and I am always taking time to answer them questions in the future if I can, if not I will at least indicate where should they look or which is the right person to ask.
Well, maybe no one sensible? I still think it's quite cool that you can take any general purpose compression algorithm, and abuse it to do e.g. image classification. (Just append the image you want to classify to each of the classes' sets in turn, and see which compresses best!)
And actually I do remember a paper that tried to use ideas from PAQ to do online learning. Gated Linear Networks, out of DeepMind, in 2019:
https://arxiv.org/abs/1910.01526
All compression is related to AI, but especially sample efficient online learning is basically what data compression is all about.
Well stochastic processes are a much wider category than mere algebraic operations applied on group theory that losseles data compression use.
I'd think that calculus or functional theory or category theory can find more bijections towards statistics or even congruences than mere arithmetics or algebra ever will. Ok, you can explain or derive any mathematics construct using only algebra, and there were efforts to do so, but does it makes sense?