I haven't looked closely enough to be sure if they literally had complex activations, but this seems like an obvious use. Maybe they would have, if only tensorflow made it easy.
Why would you ever want to represent activations directly as complex numbers in that case? I can’t think of any good reason, compared with putting them in rotation matrix form or some other equivalent form that actually maps to the domain modeling problem.
Even when working in signal processing problems that require complex arithmetic, the underlying representations are just based on tuples of doubles and operator conventions, and you always need to map to real spaces (real part, imaginary part, angle, or magnitude) for any type of analytical representation that can be human readable.
In all these cases, the idea that what we should optimize for is overhead-free easy expression of cutesy math domain verbiage is a bad idea.
Writing libraries that expose an API that matches the user’s domain mental model is a great thing. But enforcing a particular abstraction and extensibility hierarchy so those things can be “autogenerated” just by parameterizing over a new type turns out to be actually much worse than just writing that type separately, with helper functions and converters, and customizing its API to be efficient from a domain mental model perspective.
A better way, for example, might be to use mixin patterns or decorators and other metaprogramming, while writing a custom data type and its associated methods.
It didn’t require zero lines of code. It required a huge amount of backend code to set up the abstraction and make lots of built-in types that adhere to the abstraction. And in cases when the abstraction fails to offer the exact type of extensibility needed (which is most of the time unless you’re authoring yet another highly abstracted library that can tie its use cases to that underlying abstraction, which is never in practice), then it was wasted effort, and “no overhead” is a false description, because you still have to dig into the guts of all the stuff that gets auto-generated if you plugged into the abstraction and change the mechanism of how it gets auto-generated for your special case, or else (usually easier), just write separate data structures outside of the abstraction vortex and have a few small converters or helpers that marshal your custom data type into and out of the abstraction for the really tiny anount of auto-generated features that actually matter to the use case.
The “but it requires zero lines of code” thing is so misleading once you hit real use cases where the choices of how the abstraction auto-generates things end up being unusable for some specific situation.
Right here we have a classic case of someone on the internet anonymously saying something is impossible to do while there are many many examples of exactly this working. I recommend readers of these posts to ignore the FUD and do some Google searches to look through some Julia code repositories to see it in action. There are some great tutorials and fact-based discussions out there that can lead you to some useful examples with tricks you can employ in your own code.
It looks like you are just posting knee-jerk defensive posts about julia I guess. Whatever this is, it’s clearly not related to my earlier comments in the thread.
What are you talking about? Where did I say any of this was not possible? It’s obviously possible.
It just turns out to be bad when you do it. It causes problems that the company line memo about zero overhead never is upfront about.
Premature abstraction is a problem for engineering.
But what about scientists who are not too fussed with engineering considerations but would like to explore such things? Then this extensibility can be valuable.
Comments
This sounds like premature abstraction to me...
There is work on rotationally invariant networks, e.g. for identifying galaxies, or cells under a microscope. For example:
https://arxiv.org/abs/1612.04642
https://arxiv.org/abs/1805.12301
I haven't looked closely enough to be sure if they literally had complex activations, but this seems like an obvious use. Maybe they would have, if only tensorflow made it easy.
Why would you ever want to represent activations directly as complex numbers in that case? I can’t think of any good reason, compared with putting them in rotation matrix form or some other equivalent form that actually maps to the domain modeling problem.
Even when working in signal processing problems that require complex arithmetic, the underlying representations are just based on tuples of doubles and operator conventions, and you always need to map to real spaces (real part, imaginary part, angle, or magnitude) for any type of analytical representation that can be human readable.
In all these cases, the idea that what we should optimize for is overhead-free easy expression of cutesy math domain verbiage is a bad idea.
Writing libraries that expose an API that matches the user’s domain mental model is a great thing. But enforcing a particular abstraction and extensibility hierarchy so those things can be “autogenerated” just by parameterizing over a new type turns out to be actually much worse than just writing that type separately, with helper functions and converters, and customizing its API to be efficient from a domain mental model perspective.
A better way, for example, might be to use mixin patterns or decorators and other metaprogramming, while writing a custom data type and its associated methods.
Something like this? https://github.com/andyferris/Traitor.jl
That is definitely a cool direction to take it!
How is it premature abstraction if it takes zero extra lines of code to support it and have it optimized? That's kind of the beauty of Julia.
It didn’t require zero lines of code. It required a huge amount of backend code to set up the abstraction and make lots of built-in types that adhere to the abstraction. And in cases when the abstraction fails to offer the exact type of extensibility needed (which is most of the time unless you’re authoring yet another highly abstracted library that can tie its use cases to that underlying abstraction, which is never in practice), then it was wasted effort, and “no overhead” is a false description, because you still have to dig into the guts of all the stuff that gets auto-generated if you plugged into the abstraction and change the mechanism of how it gets auto-generated for your special case, or else (usually easier), just write separate data structures outside of the abstraction vortex and have a few small converters or helpers that marshal your custom data type into and out of the abstraction for the really tiny anount of auto-generated features that actually matter to the use case.
The “but it requires zero lines of code” thing is so misleading once you hit real use cases where the choices of how the abstraction auto-generates things end up being unusable for some specific situation.
Right here we have a classic case of someone on the internet anonymously saying something is impossible to do while there are many many examples of exactly this working. I recommend readers of these posts to ignore the FUD and do some Google searches to look through some Julia code repositories to see it in action. There are some great tutorials and fact-based discussions out there that can lead you to some useful examples with tricks you can employ in your own code.
It looks like you are just posting knee-jerk defensive posts about julia I guess. Whatever this is, it’s clearly not related to my earlier comments in the thread.
What are you talking about? Where did I say any of this was not possible? It’s obviously possible.
It just turns out to be bad when you do it. It causes problems that the company line memo about zero overhead never is upfront about.
Premature abstraction is a problem for engineering.
But what about scientists who are not too fussed with engineering considerations but would like to explore such things? Then this extensibility can be valuable.
At some point one of those explorations becomes useful and it's no longer premature.