#Download spark 2.1.0 64 bits

Using Python 3 would be just the same, with the only difference being in terms of code and module compatibility.

#Download spark 2.1.0 64 bits

I will use Miniconda for Python 2.7 64 bits throughout. Isolated separate python (anaconda or a separate python) is harder to get working but will provide a more consistent environment where each user can have their own (and only their own) modules installed.System python is easier to make work, it's already there and shared everywhere.If it's Python 2.6.X, it's probably a good idea to use a recent build of Python 2.7 If it's Python 2.7.X, then you'll need to choose to use the system python or not. Verify your version of Python: python - version I've implemented it before and found THE JUPYTER DOCUMENTATION explains setting it up for encryption (HTTPS) and authentication to be pretty good. I'm leaving out Jupyter server mode security, which could be the topic of a future blog, potentially. Do all this using Jupyter in server mode that I access from my own laptop.Have access to modules like numpy, scipy, pandas and others.Run python code in YARN distributed mode.Run PySpark successfully from either a cluster node or an edge node.Hence my having so much trouble getting everything working to my satisfaction. It seems like it changed quite a bit since the earlier versions and so most of the information I found in blogs were pretty outdated. In fact, I've tested this to work with MapR 5.0 with MEP 1.1.2 (Spark 1.6.1) for a customer. It should work equally well for earlier releases of MapR 5.0 and 5.1.

This article targets the latest releases of MapR 5.2.1 and the MEP 3.0 version of Spark 2.1.0. Having gone through the process myself, I've documented my steps and will share my knowledge, hoping it will save some time and frustration for some of you. I'll guess that many people reading this have spent time wrestling with a configuration to get Python and Spark to play nicely.