Scala Spark - a bit absurd if you don't need the parallelism, though. Most of the development can be done simply in quick compilation iterations or copied from the sbt REPL. Python/pandas feels Stone Age in comparison - you absolutely waste a lot of time iterating with run-time testing.
Comments
Scala Spark - a bit absurd if you don't need the parallelism, though. Most of the development can be done simply in quick compilation iterations or copied from the sbt REPL. Python/pandas feels Stone Age in comparison - you absolutely waste a lot of time iterating with run-time testing.
Why scala spark over pyspark?
Scala (and Java) has a typed Dataset api.[0] pyspark only provides the Dataframe API, which is not typed.
[0] https://spark.apache.org/docs/latest/sql-programming-guide.h...
thanks!