Congratulations. Studying one field means you know that field.
This link is not meant for you. It is meant for a scientist, and most scientists do not also have an EE degree or CS degree.
How much graduate level biology, oceanography, physics, geology, chemistry, meteorology, or other scientific field do you know?
All of those have subfields where computational performance is important. My experience is scientists are more likely to pick up the software skills than EEs are willing to pick up the science background. (In part because scientific software development generally pays less well than commercial software development.)
"Math" is such a wide topic that you certainly must qualify your statement.
The standard entomologist curriculum does not require calculus, while a physics curriculum does. Both produce scientists. (For example, https://cals.cornell.edu/education/degrees-programs/entomolo... under "Major Requirements" says "One semester of college statistics or biometry", and the listed physics requirement doesn't require calculus.)
On the other hand, an entomologist interested in population ecology may need to know differential equations.
Your use of "study program" suggests your experience is at the undergrad level, and not at the grad school level, which is how most scientists I know got their training.
At the undergrad level the study programs do reflect what's needed for a solid education. If a student is interested in computational biology, that program will emphasize taking more CS courses than the program for a student interested in marine biology.
But at the grad level, the "study program" is much less formalized. You might take graduate level classes the first couple of years, but then you are expected to pick up the missing bits on your own.
Once you have your PhD and are a working scientist, you rarely have the luxury of following any study program.
And if you've been a scientist for 20 years, any CS training you had likely did not cover SIMD, and emphasized practices which are no longer relevant. (For example, the link points out "That advice [about HDDs] is mostly outdated today [with SSDs]".)
Those latter categories are who the linked-to piece is for, not undergrads in a well-defined study program.
The article basically implies that some non-professional coder will be doing assembly and basically doing the work of an optimizing compiler. I think the point of the parent is that if you are at this point already, and you're in an academic setting, you might as well real a full computer architecture textbook front to back.
I would be curious to know of all the "scientific coders" what percentage of them understood the entire article. I'd be similarly curious how much your typical "bootcamp" developers would understand of it. I know everything presented, so it basically comes off as a "lecture notes" for someone that already knows it. Someone that doesn't understand SIMD, CPU fundamentals, assembly, and compilers, I'd imagine their eyes would glaze right when the assembly code appeared.
And while SSDs are MUCH FASTER than HDDs, the basics of interacting with storage is the same, just that rather than waiting a million years for data to arrive from the CPU's perspective, it comes in 10,000s of years.
Latency numbers all programmers should be aware of:
I am a professional coder and my eyes glaze over with assembly. Still, I don't think there's that much assembly. What I saw was to show how the code is implemented, with very little about "doing assembly" outside of a simple example.
I can't judge background - I don't have a sense of who uses Julia, and I've been programming for too long, without exposure to the target audience.
Since you mentioned "academic setting", I'll point out there are also scientists-who-program in industrial settings. However, none of the ones I know about use Julia.
My belief is that most scientists-who-program aren't going to read text books from other fields. They are under pressure to produce NOW, and don't think it's worth the time to acquire an entirely new mindset. Instead, I think this sort of knowledge transfer is by jerks and fits, as someone figures out an optimization, and passes it along, with domain-specific context that makes it easier for others in the field to understand.
Which means, like you, I don't think this notebook will be all that useful, though in my case that's because I think it's too generic.
what percentage of them understood the entire article
I don't think that's a telling metric. Only some scientific coders are interested in writing fast code (vs. fast-enough code), and only some of those use Julia.
I agree with this sentiment, like the majority of CS people are telling statisticians that a lot of Julia remains a kind of snake oil or otherwise mystical thinking, it is very unfortunate. Even in the first page of the documentation "No need to vectorize code for performance; devectorized code is fast" is some kind of category error redefinition of how programming languages work in my opinion.
Even in the first page of the documentation "No need to vectorize code for performance; devectorized code is fast" is some kind of category error redefinition of how programming languages work in my opinion.
Can you elaborate a bit? I don't really get what you are trying to say.
If the code can easily be vectorized then it has the potential to vectorize it incorrectly or there is some automation happening that is hidden. If they're just saying their non-vectorized operations are just as quick, then how quick could true vectorization be. Also, this is how Octave, NumPy, Matlab, R, etc work by making vectorized math operations happen with whole matrices using statements that look like simple non-vector operations.
Further, usually when people are having these kinds of issues it's because they started with a non-parallelizable concept of their problem in our trying to redo it... And no amount of magic is going to fix a bad concept of the problem space.
I think the problem here is that there are two very different meanings of "vectorized" at play. The first (and what the Julia docs are talking about here) is the pattern of writing "vector-operations" (i.e. rather than writing a loop, writing an expression that works across an entire array). The second meaning is using SIMD instructions (e.g. AVX2). What the julia docs are trying to say is unlike in languages like Python/R/Matlab where loops have a high overhead do to an interpreter, in Julia, loops are fast because the language is compiled. There are lots of algorithms that are easy to express in an iterative fashion that are pretty much impossible to vectorize efficiently (dataflow analyais/differential equations etc).
The docs here aren't trying to talk about SIMD instructions at all here. (although Julia/LLVM are pretty good at producing SIMD instructions from loops where possible).
This existed quite a while before Julia, how is it that Julia claims this differentiation then? "We pride ourselves by not using the slow thing you're probably not actually using."
Because Julia claims to get usability and performance in one language, rather than the two Python needs (Python for the user-facing API, C/C++/Fortran/etc. for performance).
This should also help with optimization, and a larger amount of code can be optimized together, while (C)Python can only optimize up to the Python/C boundary.
Again redefining these things here... if the language has tools for all of inline raw chip specific instructions, compiler optimized versions of those instructions, virtualized and then optimized instructions, a jit compiler, and then also high level interpreter operations, then how can it be in anyway an encompassing system and how can that be coherent among all levels without requiring someone to know all levels at which point, I'm going to say no thanks and stick to tools at their respective levels of abstraction where I can reason them to be coherent rather than a dynamically allocated string which is sometimes tied to only working on amd64 because someone wanted an assembler way of pattern matching for some reason.
Maybe within the scope of scientific computing this seems logical if you just restrict yourself to matrix multiplication or whatever, but I don't see how that makes a "language" and it can be coherent, composable, etc...
I've seen a ton of examples, however, over the years of "stunt driven" technological advantages that people thought would be interesting but turned out to be the wrong solution for the wrong problem, but none that claim to break the laws of physics and reason with their evangelism than Julia. Even this article starts with "you must understand the quirks" and I hope that isn't a goal for their design because they are also claiming that I shouldn't have to know the quirks so which is it.
Ummm, CPython is also leaky abstraction. Parts of the C implementation, like garbage collection, id(), 'a is b' checks, the ast and dis modules, and more.
It even has the beginnings of JIT support.
The leaky abstraction thesis is that all layers leak.
Julia's argument is that if you have all of these levels anyway, do it in one language instead of two. If you don't like leaky abstractions, you should prefer a system with one less layer of abstraction.
You also reject Rust, yes? It has many of the same abilities.
And JITed Lisp implementations with user access to the JITted code?
then how can it be in anyway an encompassing system and how can that be coherent among all levels without requiring someone to know all levels at which point
That sounds like an argument from incredulity.
Just because you don't see how something can be true, that doesn't mean it isn't true.
which is sometimes tied to only working on amd64 because someone wanted an assembler way of pattern matching for some reason
I believe all of the big C compiler vendors support ways to embed assembly. I use it in my code, for better support for x86-64, and a fallback for other platforms.
I also used Turbo Pascal's inline assembly in the early 1990s.
but none that claim to break the laws of physics and reason with their evangelism than Julia
I guess you're too young to remember Lisp evangelists.
You seem to be reacting to something beyond what is in the linked-to essay. What breaks the laws of physics? Again, appealing to gut instinct isn't that good of an argument.
Just because you don't see how something can be true, that doesn't mean it isn't true.
This is also a non-answer and I don't mean to be flippant but if you have any further justification I'd love to read it. It is the core of the argument you're handwaving away.
While I could certainly put most of this together from my undergrad CS education, I would not say this "subject [was] taught" to me during undergrad. Instead, as with much of undergrad, you get pieces of it along the way - but collecting it together and writing for a somewhat-lay audience has a lot of value. This is also more up to date than my under grad education from ~14 years ago! It has clear explanations for things like Hyperthreading, which existed at the time I was in undergrad, but hadn't really made its way into the curriculum yet.
Well, apparently I offended a lot of people by merely providing a piece of information about computer architecture curriculum. I apologize for commenting.
With no context about your emotional state, it came off as a bit brusque, and I think most people read comments like that in a snarky, condescending tone.
If you had prepended the comment with something like "I love this topic!" to show enthusiasm or approval, you probably would have gotten a much different response.
Comments
This subject is taught in undergrad computer architecture courses along with machine coding. As an EE, I learned it in grad school.
Congratulations. Studying one field means you know that field.
This link is not meant for you. It is meant for a scientist, and most scientists do not also have an EE degree or CS degree.
How much graduate level biology, oceanography, physics, geology, chemistry, meteorology, or other scientific field do you know?
All of those have subfields where computational performance is important. My experience is scientists are more likely to pick up the software skills than EEs are willing to pick up the science background. (In part because scientific software development generally pays less well than commercial software development.)
Math was always a must for a scientist, todays computer science is also a must. The study programmes should reflect that.
"Math" is such a wide topic that you certainly must qualify your statement.
The standard entomologist curriculum does not require calculus, while a physics curriculum does. Both produce scientists. (For example, https://cals.cornell.edu/education/degrees-programs/entomolo... under "Major Requirements" says "One semester of college statistics or biometry", and the listed physics requirement doesn't require calculus.)
On the other hand, an entomologist interested in population ecology may need to know differential equations.
Your use of "study program" suggests your experience is at the undergrad level, and not at the grad school level, which is how most scientists I know got their training.
At the undergrad level the study programs do reflect what's needed for a solid education. If a student is interested in computational biology, that program will emphasize taking more CS courses than the program for a student interested in marine biology.
But at the grad level, the "study program" is much less formalized. You might take graduate level classes the first couple of years, but then you are expected to pick up the missing bits on your own.
Once you have your PhD and are a working scientist, you rarely have the luxury of following any study program.
And if you've been a scientist for 20 years, any CS training you had likely did not cover SIMD, and emphasized practices which are no longer relevant. (For example, the link points out "That advice [about HDDs] is mostly outdated today [with SSDs]".)
Those latter categories are who the linked-to piece is for, not undergrads in a well-defined study program.
The article basically implies that some non-professional coder will be doing assembly and basically doing the work of an optimizing compiler. I think the point of the parent is that if you are at this point already, and you're in an academic setting, you might as well real a full computer architecture textbook front to back.
I would be curious to know of all the "scientific coders" what percentage of them understood the entire article. I'd be similarly curious how much your typical "bootcamp" developers would understand of it. I know everything presented, so it basically comes off as a "lecture notes" for someone that already knows it. Someone that doesn't understand SIMD, CPU fundamentals, assembly, and compilers, I'd imagine their eyes would glaze right when the assembly code appeared.
And while SSDs are MUCH FASTER than HDDs, the basics of interacting with storage is the same, just that rather than waiting a million years for data to arrive from the CPU's perspective, it comes in 10,000s of years.
Latency numbers all programmers should be aware of:
https://gist.github.com/jboner/2841832
I am a professional coder and my eyes glaze over with assembly. Still, I don't think there's that much assembly. What I saw was to show how the code is implemented, with very little about "doing assembly" outside of a simple example.
I can't judge background - I don't have a sense of who uses Julia, and I've been programming for too long, without exposure to the target audience.
Since you mentioned "academic setting", I'll point out there are also scientists-who-program in industrial settings. However, none of the ones I know about use Julia.
My belief is that most scientists-who-program aren't going to read text books from other fields. They are under pressure to produce NOW, and don't think it's worth the time to acquire an entirely new mindset. Instead, I think this sort of knowledge transfer is by jerks and fits, as someone figures out an optimization, and passes it along, with domain-specific context that makes it easier for others in the field to understand.
Which means, like you, I don't think this notebook will be all that useful, though in my case that's because I think it's too generic.
I don't think that's a telling metric. Only some scientific coders are interested in writing fast code (vs. fast-enough code), and only some of those use Julia.
I agree with this sentiment, like the majority of CS people are telling statisticians that a lot of Julia remains a kind of snake oil or otherwise mystical thinking, it is very unfortunate. Even in the first page of the documentation "No need to vectorize code for performance; devectorized code is fast" is some kind of category error redefinition of how programming languages work in my opinion.
Can you elaborate a bit? I don't really get what you are trying to say.
If the code can easily be vectorized then it has the potential to vectorize it incorrectly or there is some automation happening that is hidden. If they're just saying their non-vectorized operations are just as quick, then how quick could true vectorization be. Also, this is how Octave, NumPy, Matlab, R, etc work by making vectorized math operations happen with whole matrices using statements that look like simple non-vector operations. Further, usually when people are having these kinds of issues it's because they started with a non-parallelizable concept of their problem in our trying to redo it... And no amount of magic is going to fix a bad concept of the problem space.
I think the problem here is that there are two very different meanings of "vectorized" at play. The first (and what the Julia docs are talking about here) is the pattern of writing "vector-operations" (i.e. rather than writing a loop, writing an expression that works across an entire array). The second meaning is using SIMD instructions (e.g. AVX2). What the julia docs are trying to say is unlike in languages like Python/R/Matlab where loops have a high overhead do to an interpreter, in Julia, loops are fast because the language is compiled. There are lots of algorithms that are easy to express in an iterative fashion that are pretty much impossible to vectorize efficiently (dataflow analyais/differential equations etc).
The docs here aren't trying to talk about SIMD instructions at all here. (although Julia/LLVM are pretty good at producing SIMD instructions from loops where possible).
But no one in science uses Python loops for example, they use NumPy / Jax / Polars etc so that is an unfair and disingenuous comparison.
That's exactly the point. People are changing their coding style to work around the fact that loops in the language are slow.
This existed quite a while before Julia, how is it that Julia claims this differentiation then? "We pride ourselves by not using the slow thing you're probably not actually using."
Because Julia claims to get usability and performance in one language, rather than the two Python needs (Python for the user-facing API, C/C++/Fortran/etc. for performance).
This should also help with optimization, and a larger amount of code can be optimized together, while (C)Python can only optimize up to the Python/C boundary.
I can't imagine anything worse than something that is not appropriate for the abstraction... eg https://en.m.wikipedia.org/wiki/Leaky_abstraction
Again redefining these things here... if the language has tools for all of inline raw chip specific instructions, compiler optimized versions of those instructions, virtualized and then optimized instructions, a jit compiler, and then also high level interpreter operations, then how can it be in anyway an encompassing system and how can that be coherent among all levels without requiring someone to know all levels at which point, I'm going to say no thanks and stick to tools at their respective levels of abstraction where I can reason them to be coherent rather than a dynamically allocated string which is sometimes tied to only working on amd64 because someone wanted an assembler way of pattern matching for some reason.
Maybe within the scope of scientific computing this seems logical if you just restrict yourself to matrix multiplication or whatever, but I don't see how that makes a "language" and it can be coherent, composable, etc...
I've seen a ton of examples, however, over the years of "stunt driven" technological advantages that people thought would be interesting but turned out to be the wrong solution for the wrong problem, but none that claim to break the laws of physics and reason with their evangelism than Julia. Even this article starts with "you must understand the quirks" and I hope that isn't a goal for their design because they are also claiming that I shouldn't have to know the quirks so which is it.
Ummm, CPython is also leaky abstraction. Parts of the C implementation, like garbage collection, id(), 'a is b' checks, the ast and dis modules, and more.
It even has the beginnings of JIT support.
The leaky abstraction thesis is that all layers leak.
Julia's argument is that if you have all of these levels anyway, do it in one language instead of two. If you don't like leaky abstractions, you should prefer a system with one less layer of abstraction.
You also reject Rust, yes? It has many of the same abilities.
And JITed Lisp implementations with user access to the JITted code?
That sounds like an argument from incredulity.
Just because you don't see how something can be true, that doesn't mean it isn't true.
I believe all of the big C compiler vendors support ways to embed assembly. I use it in my code, for better support for x86-64, and a fallback for other platforms.
I also used Turbo Pascal's inline assembly in the early 1990s.
I guess you're too young to remember Lisp evangelists.
You seem to be reacting to something beyond what is in the linked-to essay. What breaks the laws of physics? Again, appealing to gut instinct isn't that good of an argument.
This is also a non-answer and I don't mean to be flippant but if you have any further justification I'd love to read it. It is the core of the argument you're handwaving away.
I know little about Julia. I do know that advocates for Lisp also make similar claims, so I don't see it as that exceptional.
Not surprisingly, Julia draws on Lisp's macro abilities to achieve similar goals. Julia is also influenced by Dylan, another ALGOL-like Lisp variant.
If Julia does do what you say you don't believe it can, how would you learn that you were wrong?
While I could certainly put most of this together from my undergrad CS education, I would not say this "subject [was] taught" to me during undergrad. Instead, as with much of undergrad, you get pieces of it along the way - but collecting it together and writing for a somewhat-lay audience has a lot of value. This is also more up to date than my under grad education from ~14 years ago! It has clear explanations for things like Hyperthreading, which existed at the time I was in undergrad, but hadn't really made its way into the curriculum yet.
Well, apparently I offended a lot of people by merely providing a piece of information about computer architecture curriculum. I apologize for commenting.
With no context about your emotional state, it came off as a bit brusque, and I think most people read comments like that in a snarky, condescending tone.
If you had prepended the comment with something like "I love this topic!" to show enthusiasm or approval, you probably would have gotten a much different response.
Not everyone does.