Spark can read from and write to HDFS, and YARN (the resource negotiator in Hadoop v 2) is one of the cluster managers supported by Spark.
but you can run a Spark cluster without YARN--eg, with Mesos--or with the built-in manager provided in the Spark distro. Likewise, your Spark cluster doesn't need to read or write to HDFS.
Comments
Spark can read from and write to HDFS, and YARN (the resource negotiator in Hadoop v 2) is one of the cluster managers supported by Spark.
but you can run a Spark cluster without YARN--eg, with Mesos--or with the built-in manager provided in the Spark distro. Likewise, your Spark cluster doesn't need to read or write to HDFS.