Anyone have any experience using a Java treeshaker? I have a Clojure GUI/JFX App that's bloated and I've only managed to slim it down with manual package pruning.
I couldn't get Proguard to work with Clojure. Granted proguard is quite baroque so I probably did something wrong. But maybe there is some simpler solution out there for the JVM? Or would this simply be impossible b/c Clojure uses reflection? Though so does Javascript from what I understand..
Shouldn’t the JVM already be doing this, as part of dead code analysis during the compile step? I believe tree shaking in JavaScript also only handles static analysis — eg imports and function calls.
I imagine though usage of something like exec throws DCE right out the window, poisoning the whole program
No, the Java compiler does not do this. You can try it yourself and see. All dependencies are bundled into the JAR. If you have some large library as a dependency (like BoofCV or JavaFX for instance) then all of the classes and dependencies are bundled. Even if you don't use 99% of them. That's why a lot of Java libraries are broken up into many smaller sub-libraries which you can then manually pick and choose from. It's also why you sometimes see people re-implement smaller sections of larger libraries - b/c you don't want to be forced to drag in the whole mess of code
As far as I understand, and I'm murky on the details, but if you're language supports reflection then you can call any dependency during runtime. And so this is not amenable to static analysis. So the Java compiler doesn't have the same guarantees as say a C++ compiler.
Most people are running JVM on the server so they just don't care about executable size (and on Android people use proguard). But reflection in general is a source of headaches..
If you use something like GraalVM then it will try to prune dead code but it will not work well with reflections.
That all being said, on the specifics, I'm actually not entirely sure how you'd use reflection to arbitrarily call library code at run time. If anyone knows, I'd be curious to see an example
In Java you could make an instance of a class doing
c = Class.forName("tld.something.SomeClass").newInstance();
and if that string is created runtime, it's impossible to rule out usage of any dependency. So you basically have to tell the tools what to remove and what not to.
Oh that's cool. I'm not intimately familiar with Java, so it's interesting to see how it's done. I figured it was something like that. Coming from C++, reflection seems like a giant useless codesmell - and there are all these issues further down the line because of it. And I get the impression most Java code doesn't even rely on it
Most Java code actually does rely on it, but behind the scenes. So you could go years between writing any reflection code yourself, but libraries and frameworks use it to dynamically search for known stuff in the classpath and change behavior if it finds other class X, or use it to serialize to and from various formats and java objects, do stuff based on annotations etc.
Java's reflection capabilities are pretty far down the "almost anything is possible" side of things, so yeah - unless you can rule out ALL reflection in the whole application, there's no way to safely shake a tree.
Crazy to think they choose an unshrinkable language as a mobile platform :)
It doesn't seem like that powerful of a language feature (and more of a code smell) and it creates this mess down the line. I wonder what fraction of libraries even require it.. I'm guessing it's b/c of the flag soup i never got ProGuard working. I know the Clojure language/runtime itself relies on some reflection - unfortunately. That probably complicates things
Java's is extremely powerful, especially when you include stuff like bytecode-weaving (e.g. AspectJ: https://www.lenar.io/logging-method-invocations-in-java-with...). Which I mostly include because support for it effectively built into the language and all major IDEs, whether it's a run-time or compile-time construct (or a blend of both). And with a sufficiently-complex custom class loader, you can do darn near anything at runtime, as you have control over how the system loads new code.
Android specifically though: an unbelievable number of libraries use reflection at least a little, if not deeply. E.g. the Android UI framework is absolutely riddled with it - ever used XML to inflate views, i.e. the default way to build a UI? That's all based on reflection. Many (many!) of the most-popular libraries use it as well, though the efficiency-minded ones generally use compile-time codegen (annotation processors) where at all possible.
Yeah, I mean from a IDE/developer point of view reflection is fantastic - really can't argue with that. At dev time it's a life saver. But coming from C++ my gut reaction is that for compilation/distribution time this is mostly a crutch/code-smell with a huge price. I've for instance noticed there is seemingly a lot of library rewriteing and fragmentation. It's common for people to choose a smaller one-trick-pony library instead of a more mature large library - to avoid dragging in a enormous dependency. That ends up as a huge bummer for the ecosystem
Thanks for the insight into Java internals and Android. It's a shame it's so tightly coupled at the roots but it's pure fantasy on my part to have a reflection-less JVM :) At the end of the day, there isn't really anything else like Clojure, so I just try to accept the warts and features.
Out of curiosity I took a cursory look at Qt and they seem to handle QML without using RTTI - so think it's all solvable - at least in most cases (I'm guessing by enumerating your possible types at compile time?)
I recommend checking out the jdeps command line tool, that comes bundled with the JDK. Though you probably have to read up on modules a bit first. It can create very lean “JREs” that can be bundled with the program and only load the required classes.
As I understood it, jdeps is only for shrinking the runtime and bundling it with the program. This part seems fine.. or at least they're doing the best they can :)
I ended up skipping this step and distributing a JAR - which is a bit of no-no .. but i just ask user to install the Java runtime. (it's better than making executable for every platform and then spinning up VMs to test)
However, jdeps or not, the rest of the executable still remains as bloated as before
You are right in that it doesn’t do dead code elimination per se.
But if my understanding is correct, java classes are loaded as needed. So other than JAR size, you don’t really need it as much as in the case of JS for example.
That's an important distinction. I guess my annoyance is mostly cosmetic. The app is pretty small and does a simple data processing task. Had I written the app in C++ it would have been a couple of megabytes (and it would have taken me 10 times longer to write..). But with Clojure/Java/JVM/jdeps it comes out as some monstrous 100MB installer. It just feels a bit gross ...
End user apps are second class citizens on the JVM (except weirdly on Android.. but they're in a weird bubble of their own)
Comments
Anyone have any experience using a Java treeshaker? I have a Clojure GUI/JFX App that's bloated and I've only managed to slim it down with manual package pruning.
I couldn't get Proguard to work with Clojure. Granted proguard is quite baroque so I probably did something wrong. But maybe there is some simpler solution out there for the JVM? Or would this simply be impossible b/c Clojure uses reflection? Though so does Javascript from what I understand..
Shouldn’t the JVM already be doing this, as part of dead code analysis during the compile step? I believe tree shaking in JavaScript also only handles static analysis — eg imports and function calls.
I imagine though usage of something like exec throws DCE right out the window, poisoning the whole program
No, the Java compiler does not do this. You can try it yourself and see. All dependencies are bundled into the JAR. If you have some large library as a dependency (like BoofCV or JavaFX for instance) then all of the classes and dependencies are bundled. Even if you don't use 99% of them. That's why a lot of Java libraries are broken up into many smaller sub-libraries which you can then manually pick and choose from. It's also why you sometimes see people re-implement smaller sections of larger libraries - b/c you don't want to be forced to drag in the whole mess of code
As far as I understand, and I'm murky on the details, but if you're language supports reflection then you can call any dependency during runtime. And so this is not amenable to static analysis. So the Java compiler doesn't have the same guarantees as say a C++ compiler.
Most people are running JVM on the server so they just don't care about executable size (and on Android people use proguard). But reflection in general is a source of headaches..
If you use something like GraalVM then it will try to prune dead code but it will not work well with reflections.
That all being said, on the specifics, I'm actually not entirely sure how you'd use reflection to arbitrarily call library code at run time. If anyone knows, I'd be curious to see an example
In Java you could make an instance of a class doing
and if that string is created runtime, it's impossible to rule out usage of any dependency. So you basically have to tell the tools what to remove and what not to.Oh that's cool. I'm not intimately familiar with Java, so it's interesting to see how it's done. I figured it was something like that. Coming from C++, reflection seems like a giant useless codesmell - and there are all these issues further down the line because of it. And I get the impression most Java code doesn't even rely on it
Most Java code actually does rely on it, but behind the scenes. So you could go years between writing any reflection code yourself, but libraries and frameworks use it to dynamically search for known stuff in the classpath and change behavior if it finds other class X, or use it to serialize to and from various formats and java objects, do stuff based on annotations etc.
Ages ago, I dimly recall debug tracing the classloader to see what is actually used. Something about troubleshooting Spring or some such.
Same strategy could be used for "tree shaking"?
I'll ponder how unused functions might be identified (during runtime).
Well, there are techniques. One of the hackier ones is probably the one described here: https://fgiesen.wordpress.com/2012/04/08/metaprogramming-for...
In theory the same applies for JavaScript, just eval an obfuscated piece of code and the shaker is none the wiser.
Java's reflection capabilities are pretty far down the "almost anything is possible" side of things, so yeah - unless you can rule out ALL reflection in the whole application, there's no way to safely shake a tree.
Which is exactly why ProGuard and R8 have so many flags and options, and basically every use of it requires customization: https://developer.android.com/studio/build/shrink-code
Crazy to think they choose an unshrinkable language as a mobile platform :)
It doesn't seem like that powerful of a language feature (and more of a code smell) and it creates this mess down the line. I wonder what fraction of libraries even require it.. I'm guessing it's b/c of the flag soup i never got ProGuard working. I know the Clojure language/runtime itself relies on some reflection - unfortunately. That probably complicates things
Java's is extremely powerful, especially when you include stuff like bytecode-weaving (e.g. AspectJ: https://www.lenar.io/logging-method-invocations-in-java-with...). Which I mostly include because support for it effectively built into the language and all major IDEs, whether it's a run-time or compile-time construct (or a blend of both). And with a sufficiently-complex custom class loader, you can do darn near anything at runtime, as you have control over how the system loads new code.
Android specifically though: an unbelievable number of libraries use reflection at least a little, if not deeply. E.g. the Android UI framework is absolutely riddled with it - ever used XML to inflate views, i.e. the default way to build a UI? That's all based on reflection. Many (many!) of the most-popular libraries use it as well, though the efficiency-minded ones generally use compile-time codegen (annotation processors) where at all possible.
Yeah, I mean from a IDE/developer point of view reflection is fantastic - really can't argue with that. At dev time it's a life saver. But coming from C++ my gut reaction is that for compilation/distribution time this is mostly a crutch/code-smell with a huge price. I've for instance noticed there is seemingly a lot of library rewriteing and fragmentation. It's common for people to choose a smaller one-trick-pony library instead of a more mature large library - to avoid dragging in a enormous dependency. That ends up as a huge bummer for the ecosystem
Thanks for the insight into Java internals and Android. It's a shame it's so tightly coupled at the roots but it's pure fantasy on my part to have a reflection-less JVM :) At the end of the day, there isn't really anything else like Clojure, so I just try to accept the warts and features.
Out of curiosity I took a cursory look at Qt and they seem to handle QML without using RTTI - so think it's all solvable - at least in most cases (I'm guessing by enumerating your possible types at compile time?)
I recommend checking out the jdeps command line tool, that comes bundled with the JDK. Though you probably have to read up on modules a bit first. It can create very lean “JREs” that can be bundled with the program and only load the required classes.
As I understood it, jdeps is only for shrinking the runtime and bundling it with the program. This part seems fine.. or at least they're doing the best they can :)
I ended up skipping this step and distributing a JAR - which is a bit of no-no .. but i just ask user to install the Java runtime. (it's better than making executable for every platform and then spinning up VMs to test)
However, jdeps or not, the rest of the executable still remains as bloated as before
You are right in that it doesn’t do dead code elimination per se.
But if my understanding is correct, java classes are loaded as needed. So other than JAR size, you don’t really need it as much as in the case of JS for example.
That's an important distinction. I guess my annoyance is mostly cosmetic. The app is pretty small and does a simple data processing task. Had I written the app in C++ it would have been a couple of megabytes (and it would have taken me 10 times longer to write..). But with Clojure/Java/JVM/jdeps it comes out as some monstrous 100MB installer. It just feels a bit gross ...
End user apps are second class citizens on the JVM (except weirdly on Android.. but they're in a weird bubble of their own)
Perhaps give native compilation a try? Not sure how well does it work with Clojure, but that might shrink the app size a bit as well?