This is great work, but certain part of the comparison is not accurate, probably due to their lack of understanding of Spark.
First and foremost, it would make more sense to compare against the DataFrame API of Spark, which is very Pandas like.
"Dask gives up high-level understanding to allow users to express more complex parallel algorithms."
- I don't think this is true. Their example of complex algorithms (SVD) is not that complicated, and there are even implementations of that in Spark's MLlib directly. Spark's DAG/RDD API is essentially the low level user-facing task API.
"If you have petabytes of JSON files, a simple workflow, and a thousand node cluster then you should probably use Spark. If you have 10s-1000s of gigabytes of binary or numeric data, complex algorithms, and a large multi-core workstation then you should probably use dask."
- Actually a lot of Spark users use Spark as a way to parallelize Python programs on multi-core machines.
"If you have a terabyte or less of CSV or JSON data then you should forget both Spark and Dask and use Postgres or MongoDB."
- Loading "terabyte" of JSON into Postgres seems pretty painful.
Thanks for the comments. I wrote most of that document so I'll try to explain my reasoning in line.
First and foremost, it would make more sense to compare against the DataFrame API of Spark, which is very Pandas like.
It would make more sense to me to compare dask.dataframe to spark's dataframe. This document is comparing dask to spark. dask.dataframe is a relatively small part of dask.
> "Dask gives up high-level understanding to allow users to express more complex parallel algorithms."
I don't think this is true. Their example of complex algorithms (SVD) is not that complicated, and there are even implementations of that in Spark's MLlib directly. Spark's DAG/RDD API is essentially the low level user-facing task API.
The point here is that it's quite natural for dask users to create custom graphs (here is another example matthewrocklin.com/blog/work/2015/07/23/Imperative/). Doing this in Spark requires digging more deeply into its guts. This sort of work is not idiomatic or much intended in Spark.
Loading "terabyte" of JSON into Postgres seems pretty painful
I've found that loading a terabyte of CSV into Postgres or a terabyte of JSON into Mongo to be quite pleasant actually. I'd be curious to know what problems you ran into.
Comments
Comparison of PySpark vs Dask:
http://dask.pydata.org/en/latest/spark.html
This is great work, but certain part of the comparison is not accurate, probably due to their lack of understanding of Spark.
First and foremost, it would make more sense to compare against the DataFrame API of Spark, which is very Pandas like.
"Dask gives up high-level understanding to allow users to express more complex parallel algorithms."
- I don't think this is true. Their example of complex algorithms (SVD) is not that complicated, and there are even implementations of that in Spark's MLlib directly. Spark's DAG/RDD API is essentially the low level user-facing task API.
"If you have petabytes of JSON files, a simple workflow, and a thousand node cluster then you should probably use Spark. If you have 10s-1000s of gigabytes of binary or numeric data, complex algorithms, and a large multi-core workstation then you should probably use dask."
- Actually a lot of Spark users use Spark as a way to parallelize Python programs on multi-core machines.
"If you have a terabyte or less of CSV or JSON data then you should forget both Spark and Dask and use Postgres or MongoDB."
- Loading "terabyte" of JSON into Postgres seems pretty painful.
Thanks for the comments. I wrote most of that document so I'll try to explain my reasoning in line.
It would make more sense to me to compare dask.dataframe to spark's dataframe. This document is comparing dask to spark. dask.dataframe is a relatively small part of dask.
The point here is that it's quite natural for dask users to create custom graphs (here is another example matthewrocklin.com/blog/work/2015/07/23/Imperative/). Doing this in Spark requires digging more deeply into its guts. This sort of work is not idiomatic or much intended in Spark.
I've found that loading a terabyte of CSV into Postgres or a terabyte of JSON into Mongo to be quite pleasant actually. I'd be curious to know what problems you ran into.