Comment on We decided to go for the big rewriteComments−sandGorgon6yin case you are using PySpark - a good framework to move to is Dask. https://docs.dask.org/en/latest/it is also natively integrated with K8s - https://github.com/dask/dask-kubernetes - and Yarn https://yarn.dask.org/en/latest/
Comments
in case you are using PySpark - a good framework to move to is Dask. https://docs.dask.org/en/latest/
it is also natively integrated with K8s - https://github.com/dask/dask-kubernetes - and Yarn https://yarn.dask.org/en/latest/