spark-py-notebooks - Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

---

This is a collection of IPython notebook/Jupyter notebooks intended to train the reader on different Apache Spark concepts, from basic to advanced, by using the Python language. If Python is not your language, and it is R, you may want to have a look at our R on Apache Spark (SparkR) notebooks instead. Additionally, if your are interested in being introduced to some basic Data Science Engineering, you might find these series of tutorials interesting. There we explain different concepts and applications using Python and R.

spark pyspark data-analysis mllib ipython-notebook notebook ipython data-science machine-learning big-data bigdata

MMLSpark - Microsoft Machine Learning for Apache Spark

---

MMLSpark provides a number of deep learning and data science tools for Apache Spark, including seamless integration of Spark Machine Learning pipelines with Microsoft Cognitive Toolkit (CNTK) and OpenCV, enabling you to quickly create powerful, highly-scalable predictive and analytical models for large image and text datasets.MMLSpark requires Scala 2.11, Spark 2.1+, and either Python 2.7 or Python 3.5+. See the API documentation for Scala and for PySpark.

machine-learning spark cntk pyspark azure microsoft-machine-learning microsoft ml

python_mozetl - ETL jobs for Firefox Telemetry

---

This repository is a collection of ETL jobs for Firefox Telemetry.Jobs committed to python_mozet can be scheduled via airflow or ATMO. We provide a testing suite and code review, which makes your job more maintainable. Centralizing our jobs in one repository allows for code reuse and easier collaboration.

firefox-telemetry etl pyspark

data-analytics-machine-learning-big-data - Slides, code and more for my class: Data Analytics and Machine Learning on Big Data

---

If you want to install and run everything on your computer, here are the best tutorials I've found for getting Python and Spark running on your computer. In order to visualize the decision trees in Jupyter, you will need to install Graphviz as well as the Python package.

big-data machine-learning jupyter-notebook graphviz data-exploration pyspark mllib

pyspark-for-data-processing - Code for my presentation: Using PySpark to Process Boat Loads of Data

---

Download the slides from Slideshare. This code uses the Hazardous Air Pollutants dataset from Kaggle.

spark pyspark data-science machine-learning analytics

pyspark-notebook - Pyspark Notebook With Docker

---

Run your docker with docker-compose. It helps to keep your arguments/settings in a single file and run together in an isolated environment.

docker docker-image pyspark-notebook spark pyspark docker-compose notebook apache-spark bigdata python-notebook jupyter-notebook

PySparkGeoAnalysis - :globe_with_meridians: Interactive Workshop on GeoAnalysis using PySpark

---

This workshop will introduce you to Apache Spark via the exciting domain of Geospatial Analysis.

pyspark geospatial-analysis spark docker