PixieDust

An open source helper library for Python notebooks. It makes working with data simpler.

David Taieb

Passionate about data science, machine learning, and meeting developers.

Jupyter Notebooks are powerful interactive tools for fast and flexible data experimentation and analysis. They can contain live code, rich text, equations, and visualizations. However, the learning curve is steep. Coding a simple visualization requires lots of research, which works against the promise of data science notebooks as tools for easy collaboration.

This is where PixieDust comes in. It’s an open source helper library that’s designed to lower the barrier to entry for scientists and developers working in Jupyter Notebooks. PixieDust speeds the main steps of data science:

Data loading from remote files
Data visualisation with a simple display() API
Dashboard creation with minimal coding using PixieApps

With PixieDust

Without PixieDust

Both produce a chart, like so

Screenshot of PixieDust-generated scatterplot visualization in a Jupyter Notebook.

PixieDust & Spark

If you’re processing data at scale, PixieDust also works with Apache Spark™ data structures like PySpark DataFrames, DataSets, and GraphFrames. When detecting that Spark is available, PixieDust enables these extra capabilities:

PackageManager to install third-party Spark packages
Spark Progress Monitoring to see how Spark Jobs are progressing
Scala bridge to write Scala/Java code in the same Python Notebook

Getting started

Sample notebooks are available on IBM’s Data Science Experience:

If you prefer to work locally, you can start by installing PixieDust using its local installer. For more advanced applications using PixieDust, check out the projects featured here.

Projects

Not the Chart You’re Looking for? Filter and Try Again.
Medium

Filtering data with PixieDust.
- PixieDust
- Jupyter
Analyzing Data with IBM Cloud SQL Query
Medium | Notebook

Easily make SQL queries on your object storage data.
- Object Storage
- Cloud SQL Service
- PixieDust
- Jupyter Notebook
- Python
- Mapbox
Quick Access to IBM Cloud Service Credentials from a Python Notebook
Medium | Notebook

See all your orgs, spaces, and services together in one place.
- Jupyter Notebook
- PixieDust
- Python
The Visual Python Debugger for Jupyter Notebooks You’ve Always Wanted
Medium

Introducing PixieDebugger.
- PixieDust
- Jupyter Notebook
My First Pull Request for the Open Source PixieDust Project
Medium

Setting up Python test environments, collaborating on GitHub, and sharing my changes with the world.
- PixieDust
- Python
Notebooks for Spreadsheet Users
Medium

Dreading pivot tables? Reach for a Python Notebook + PixieDust instead.
- Python
- Jupyter Notebook
- PixieDust
Share Your Jupyter Notebook Charts on the Web
Medium | GitHub

Quickly turn charts into standalone web pages.
- PixieDust
Building Your First Machine Learning System
Medium | Notebook

Train your model and deploy it, Watson ML for developers (part 2).
- Watson Machine Learning
- Python
- Jupyter Notebook
- PixieDust
Deploy Your Analytics as Web Apps Using PixieDust’s 1.1 Release
Notebook | Medium

From data science, directly to the web.
- Jupyter Notebook
- Python
- PixieDust
Visualizing Data with Jupyter Notebooks, PixieDust, and Compose MongoDB
Compose Articles

Connecting to Compose MongoDB and creating rich presentations for your data inside a Jupyter notebook is made easier with PixieDust.
- Jupyter Notebook
- Compose
- MongoDB
- PixieDust
- Python
- Nodejs
Medium.com, More Stats Please
GitHub | Medium

Analyzing Medium publication stats in a Python notebook.
- Jupyter Notebook
- Python
- PixieDust
Analyzing Podcast Metadata Using Jupyter Notebooks
GitHub | Medium

Building podcast playlists based on your current mood.
- PixieDust
- Watson Natural Language Understanding
- Jupyter Notebook
Using Watson Machine Learning to Predict a Post's Potential
Notebook | Medium

Gaze into the Reddit crystal ball.
- PixieDust
- Watson Machine Learning
- Jupyter Notebook
Mapping for Data Science with PixieDust and Mapbox
Notebook | Medium

Add another layer to your Jupyter notebooks with built-in map rendering.
- Jupyter Notebook
- Apache Spark
- Mapbox
- PixieDust
pixiedust_peacetech
GitHub

Local visibility to mitigate business disruption in emerging economies.
- PixieDust
- Jupyter Notebook
- Python
Share the (PixieDust) Magic on PyPI
Medium

Packaging and distributing your PixieDust plugin as a Python module.
- PixieDust
- Python
You Too Can Make Magic (in Jupyter Notebooks with PixieDust)
Medium

Getting started with custom visualizations, simple tables & word clouds.
- Jupyter Notebook
- Python
- PixieDust
The Most Popular Search Term at SXSW, According to Our Chatbot
GitHub | Medium | Notebook

Analyzing conversation data using Spark, Jupyter, and PixieDust.
- Jupyter Notebook
- Apache Spark
- PixieDust
- Watson Conversation
- Cloudant
- Chatbots
I am Not a Data Scientist (But I play one in this blog post, thanks to PixieDust)
Medium | developerWorks

Notebooks aren't just for Data Scientists.
- Jupyter Notebook
- Apache Spark
- PixieDust
Sentiment Analysis of Twitter Hashtags With Spark
Notebook | Medium | developerWorks

Build a real-time dashboard with PixieDust & Jupyter Notebooks.
- Spark
- Watson Tone Analyzer
- PixieDust
- Python
- Scala
Predict Flight Delays with Apache Spark
Notebook | Medium

Know about delayed flights before they happen.
- Weather
- Cloudant
- Spark
- Jupyter Notebook
- PixieDust
Weather Forecast with PixieDust
GitHub | developerWorks

Make your own forecast in a Python notebook.
- Weather
- Python
- PixieDust

PixieDust

With PixieDust

Without PixieDust

Both produce a chart, like so

PixieDust & Spark

Getting started

Projects

Analyzing Data with IBM Cloud SQL Query Medium | Notebook

Quick Access to IBM Cloud Service Credentials from a Python Notebook Medium | Notebook

Share Your Jupyter Notebook Charts on the Web Medium | GitHub

Building Your First Machine Learning System Medium | Notebook

Deploy Your Analytics as Web Apps Using PixieDust’s 1.1 Release Notebook | Medium

Medium.com, More Stats Please GitHub | Medium

Analyzing Podcast Metadata Using Jupyter Notebooks GitHub | Medium

Using Watson Machine Learning to Predict a Post's Potential Notebook | Medium

Mapping for Data Science with PixieDust and Mapbox Notebook | Medium

The Most Popular Search Term at SXSW, According to Our Chatbot GitHub | Medium | Notebook

I am Not a Data Scientist (But I play one in this blog post, thanks to PixieDust) Medium | developerWorks

Sentiment Analysis of Twitter Hashtags With Spark Notebook | Medium | developerWorks

Predict Flight Delays with Apache Spark Notebook | Medium

Weather Forecast with PixieDust GitHub | developerWorks

Analyzing Data with IBM Cloud SQL Query
Medium | Notebook

Quick Access to IBM Cloud Service Credentials from a Python Notebook
Medium | Notebook

Share Your Jupyter Notebook Charts on the Web
Medium | GitHub

Building Your First Machine Learning System
Medium | Notebook

Deploy Your Analytics as Web Apps Using PixieDust’s 1.1 Release
Notebook | Medium

Medium.com, More Stats Please
GitHub | Medium

Analyzing Podcast Metadata Using Jupyter Notebooks
GitHub | Medium

Using Watson Machine Learning to Predict a Post's Potential
Notebook | Medium

Mapping for Data Science with PixieDust and Mapbox
Notebook | Medium

The Most Popular Search Term at SXSW, According to Our Chatbot
GitHub | Medium | Notebook

I am Not a Data Scientist (But I play one in this blog post, thanks to PixieDust)
Medium | developerWorks

Sentiment Analysis of Twitter Hashtags With Spark
Notebook | Medium | developerWorks

Predict Flight Delays with Apache Spark
Notebook | Medium

Weather Forecast with PixieDust
GitHub | developerWorks