PixieDust

https://pixiedust.github.io/pixiedust/
Admin
PixieDust is an open-source Python helper library for Jupyter notebooks that enhances data...
Tokens:28,281
Snippets:212
Trust Score:9.7
Update:1 month ago
Show doc for...
Tokens:
Raw
### Installer prompt for SCALA_HOME

Source: https://pixiedust.github.io/pixiedust/install.html

Example output from the installer when it prompts for the Scala location. If Scala is not installed or the version is unsupported, the installer downloads the appropriate version.

```text
Step 3: Please enter a SCALA_HOME location: /Users/USERNAME/scala
Directory /Users/USERNAME/scala does not contain a valid SCALA install
    Download Scala y/n [y]? y
SCALA_HOME will be set to /Users/USERNAME/scala/scala-2.11.8
Downloading Scala 2.11
Extracting Scala 2.11 to /Users/USERNAME/scala

```

--------------------------------

### Example output of starting PixieGateway

Source: https://pixiedust.github.io/pixiedust/install-pixiegateway.html

Example output when starting PixieGateway on port 8899. Shows kernel startup and the Jupyter Kernel Gateway URL.

```bash
dtaieb$ jupyter pixiegateway --port 8899
[PixieGatewayApp] Kernel started: b5be0b3b-a018–4ace-95d1-d94b556a0bfe
kernel client initialized
[PixieGatewayApp] Jupyter Kernel Gateway at http://127.0.0.1:8899

```

--------------------------------

### Installer prompt for SPARK_HOME

Source: https://pixiedust.github.io/pixiedust/install.html

Example output from the installer when it prompts for the Apache Spark location. If the directory is not a valid Spark install, the installer offers to download Spark for you.

```text
Step 2: Please enter a SPARK_HOME location: /Users/USERNAME/spark
Directory /Users/USERNAME/spark does not contain a valid SPARK install
    Download Spark y/n [y]? y

```

--------------------------------

### Installer prompt for PIXIEDUST_HOME

Source: https://pixiedust.github.io/pixiedust/install.html

Example output from the installer when it prompts for PIXIEDUST_HOME. You can keep the default path or enter a new one. This directory is used to track your PixieDust install and any libraries you install.

```text
Step 1: PIXIEDUST_HOME: /Users/USERNAME/pixiedust
    Keep y/n [y]? y

```

--------------------------------

### Installer prompt for kernel name

Source: https://pixiedust.github.io/pixiedust/install.html

Example output from the installer when it asks for a name for the kernel. You can keep the default name or enter a new one.

```text
Step 4: Kernel Name: Python with Pixiedust (Spark 2.1)
    Keep y/n [y]? y

```

--------------------------------

### Kubernetes on Bluemix: login and setup commands

Source: https://pixiedust.github.io/pixiedust/install-pixiegateway.html

Commands to log in to Bluemix, target your org and space, install the container-service plugin, initialize the service, and create a cluster.

```bash
bx login [-sso] -a https://api.ng.bluemix.net

```

```bash
bx target -o <YOUR_ORG> -s <YOUR_SPACE>

```

```bash
bx plugin install container-service -r Bluemix

```

```bash
bx cs init

```

```bash
bx cs cluster-create --name my-cluster

```

--------------------------------

### Downloading Apache Spark during kernel install

Source: https://pixiedust.github.io/pixiedust/install.html

Example output when the installer downloads Apache Spark. It prompts for the version, sets SPARK_HOME, and downloads and extracts Spark to the specified directory.

```text
What version would you like to download? 1.6.3, 2.0.2, 2.1.0 [2.1.0]: 2.1.0
SPARK_HOME will be set to /Users/USERNAME/spark/spark-2.1.0-bin-hadoop2.7
Downloading Spark 2.1.0
Extracting Spark 2.1.0 to /Users/USERNAME/spark

```

--------------------------------

### Install PixieDust from source

Source: https://pixiedust.github.io/pixiedust/install.html

Install PixieDust from source by first cloning the GitHub repository, then running pip with the -e flag to install from the local directory. Recommended for contributors.

```bash
git clone https://github.com/ibm-cds-labs/pixiedust

```

```bash
pip install -e ./pixiedust

```

--------------------------------

### Install a spark package with pixiedust.installPackage

Source: https://pixiedust.github.io/pixiedust/packagemanager.html

Installs a Maven package by its maven ID. The example uses graphframes:graphframes:0.1.0-spark1.6. After issuing the command, the kernel must be restarted to complete installation.

```python
import pixiedust
pixiedust.installPackage("graphframes:graphframes:0.1.0-spark1.6")

```

--------------------------------

### List all installed packages with pixiedust.printAllPackages

Source: https://pixiedust.github.io/pixiedust/packagemanager.html

Lists all packages installed on the system.

```python
import pixiedust
pixiedust.printAllPackages()

```

--------------------------------

### Install a Jupyter kernel with jupyter pixiedust install

Source: https://pixiedust.github.io/pixiedust/install.html

Run this command from a Terminal or command-line interface to start the interactive installer that configures a new Jupyter kernel using pyspark. The installer walks you through setting PIXIEDUST_HOME, installing Apache Spark and Scala, and naming the kernel.

```bash
jupyter pixiedust install

```

--------------------------------

### Install Spark samples jar with pixiedust.installPackage

Source: https://pixiedust.github.io/pixiedust/scalabridge.html

Installs the Spark samples jar into the Python kernel so that the Scala code can access the StreamingTwitter class. Run this before using the %%scala magic.

```python
import pixiedust
pixiedust.installPackage("https://github.com/ibm-cds-labs/spark.samples/raw/master/dist/streaming-twitter-assembly-1.6.jar")

```

--------------------------------

### Start PixieGateway with jupyter command

Source: https://pixiedust.github.io/pixiedust/install-pixiegateway.html

Start the PixieGateway server on a specified port. Replace <portnumber> with the desired port.

```bash
jupyter pixiegateway --port <portnumber>

```

--------------------------------

### Install latest release of a spark package with version 0

Source: https://pixiedust.github.io/pixiedust/packagemanager.html

Specify 0 as the version to fetch the latest release. The example installs graphframes:graphframes:0.

```python
import pixiedust
pixiedust.installPackage("graphframes:graphframes:0")

```

--------------------------------

### Output after successful installPackage command

Source: https://pixiedust.github.io/pixiedust/packagemanager.html

Displays the output after successfully issuing the import command. It shows the download progress and the instruction to restart the kernel to complete installation.

```text
Downloading artifact graphframes:graphframes:0.1.0-spark1.6
Downloaded 326112 of 326112 bytes (100.00%)
Downloaded artifact graphframes:graphframes:0.1.0-spark1.6 to /Users/dtaieb/data/libs/graphframes-0.1.0-spark1.6.jar
Artifact downloaded successfully graphframes:graphframes:0.1.0-spark1.6
>>>Please restart Kernel to complete installation of the new package
Successfully added artifact graphframes:graphframes:0.1.0-spark1.6

```

--------------------------------

### Install from a custom maven repository using base keyword

Source: https://pixiedust.github.io/pixiedust/packagemanager.html

Specify a custom maven repository with the base keyword argument. By default, PixieDust looks for http://repo1.maven.org/maven2 and http://dl.bintray.com/spark-packages/maven.

```python
import pixiedust
pixiedust.installPackage("groupId:artifactId:0", base="<<url to my custom maven repo>>"

```

--------------------------------

### Run a Jupyter notebook with PixieDust

Source: https://pixiedust.github.io/pixiedust/install.html

Launches Jupyter Notebook in the directory containing the notebook. Use this command to run a PixieDust sample notebook after installation. The command opens a web browser automatically; on first launch you may be prompted to select a kernel, and you should choose the kernel created during installation.

```bash
jupyter notebook directory/containing/notebook

```

--------------------------------

### Install pixiegateway package with pip

Source: https://pixiedust.github.io/pixiedust/install-pixiegateway.html

Install the pixiegateway package from PyPi. Supports Python 2.7 and 3.x.

```bash
pip install pixiegateway

```

--------------------------------

### Install PixieDust from PyPI

Source: https://pixiedust.github.io/pixiedust/install.html

Install the latest version of PixieDust from PyPI using pip. Do not include --user in the command, as that installs the Jupyter PixieDust command in the wrong directory.

```bash
pip install pixiedust

```

--------------------------------

### Sample notebook URL for Watson Studio

Source: https://pixiedust.github.io/pixiedust/use.html

Use this notebook URL to create a new notebook in IBM Watson Studio and try PixieDust online without installation.

```text
| https://github.com/ibm-cds-labs/pixiedust/raw/master/notebook/DSX/Welcome%20to%20PixieDust.ipynb

```

--------------------------------

### Output of jupyter pixiedust list

Source: https://pixiedust.github.io/pixiedust/install.html

Example output from the 'jupyter pixiedust list' command, showing the available kernels and their paths.

```text
Available kernels:
    pythonwithpixiedustspark20    /Users/USERNAME/Library/Jupyter/kernels/pythonwithpixiedustspark20
    pythonwithpixiedustspark21    /Users/USERNAME/Library/Jupyter/kernels/pythonwithpixiedustspark21

```

--------------------------------

### Install missing packages for Spark 1.6

Source: https://pixiedust.github.io/pixiedust/loaddata.html

If running Spark 1.6 and an error occurs, install these packages once manually.

```python
pixiedust.installPackage("com.databricks:spark-csv_2.10:1.5.0")
pixiedust.installPackage("org.apache.commons:commons-csv:0")

```

--------------------------------

### pd_refresh_rate attribute example

Source: https://pixiedust.github.io/pixiedust/html-attributes-pixieapp.html

Use the pd_refresh_rate attribute to repeat execution at a specified interval in milliseconds. This example polls the state of a variable and displays the result in the UI every 3000 milliseconds.

```html
<div pd_refresh_rate="3000" pd_script="print(self.get_status())"></div>

```

--------------------------------

### pd_render_onload in PixieApp class

Source: https://pixiedust.github.io/pixiedust/html-attributes-pixieapp.html

Use pd_render_onload to trigger a kernel request upon loading, not on user click. It must be used with a div element, and the output is placed as a child of that div. This example prints 'hello world rendered on load' when the app runs.

```python
from pixiedust.display.app import *
@PixieApp
class RenderOnLoad():
    @route()
    def mainScreen(self):
        return """<div pd_render_onload pd_script="print('hello world rendered on load')"></div>"""

RenderOnLoad().run()

```

--------------------------------

### Import SimpleDisplayMeta in __init__.py

Source: https://pixiedust.github.io/pixiedust/writeviz.html

Add this import line to the imports of __init__.py in the display/table directory so PixieDust recognizes the new menu item code.

```python
from .SimpleDisplayMeta import SimpleDisplayMeta
```

--------------------------------

### Invoke PixieDebugger with %%pixie_debugger magic

Source: https://pixiedust.github.io/pixiedust/1-1-8.html

Invokes the PixieDebugger magic command to break at the first executable line of cell code. The example defines a find_max function and calls it with a random sample of 10 numbers from 0 to 99. Use this in a Jupyter Notebook cell to start debugging the code with PixieDebugger.

```python
%%pixie_debugger
import random
def find_max (values):
    max = 0
    for val in values:
        if val > max:
            max = val
    return max
find_max(random.sample(range(100), 10))
```

--------------------------------

### Hello World PixieApp with two routes

Source: https://pixiedust.github.io/pixiedust/hello-world-pixieapp.html

Defines a minimal PixieApp with two routes: the default view shows a button that sets pd_options to clicked=true, and the second route handles that option and shows a button to go back. Run with HelloWorldPixieApp().run(runInDialog='false') to launch it.

```python
from pixiedust.display.app import *
@PixieApp
class HelloWorldPixieApp:
    @route()
    def main(self):
        return"""
            <input pd_options="clicked=true" type="button" value="Click Me">
        """
    @route(clicked="true")
    def _clicked(self):
        return """
            <input pd_options="clicked=false" type="button" value="You Clicked, Now Go back">
        """
#run the app
HelloWorldPixieApp().run(runInDialog='false')
```

--------------------------------

### Kubernetes on Bluemix: create deployment and service

Source: https://pixiedust.github.io/pixiedust/install-pixiegateway.html

Create the Kubernetes deployment and service for PixieGateway using the provided YAML files.

```bash
kubectl create -f https://github.com/ibm-watson-data-lab/pixiegateway/raw/master/etc/deployment.yml

```

```bash
kubectl create -f https://github.com/ibm-watson-data-lab/pixiegateway/raw/master/etc/service.yml

```

--------------------------------

### Kubernetes on Bluemix: cluster verification and config

Source: https://pixiedust.github.io/pixiedust/install-pixiegateway.html

Commands to verify cluster creation, download the cluster config, and export the KUBECONFIG environment variable.

```bash
bx cs clusters

```

```bash
bx cs cluster-config my-cluster

```

```bash
export KUBECONFIG=/Users/dtaieb/.bluemix/plugins/container-service/clusters/davidcluster/kube-config-hou02-davidcluster.yml

```

--------------------------------

### Install PixieDust in editable mode

Source: https://pixiedust.github.io/pixiedust/writeviz.html

Run this command from your terminal after shutting down Jupyter to install the updated PixieDust code, then restart Jupyter to see the new Simple Table menu option.

```bash
pip install --user -e <your directory path to pixiedust code>
```

--------------------------------

### List available sample data sets with pixiedust.sampleData()

Source: https://pixiedust.github.io/pixiedust/loaddata.html

Run this command in a notebook to list the sampling data sets included with PixieDust.

```python
pixiedust.sampleData()

```

--------------------------------

### Kubernetes on Bluemix: get public IP and delete pod

Source: https://pixiedust.github.io/pixiedust/install-pixiegateway.html

Get the public IP of the cluster workers and optionally delete a pod to update PixieGateway. Note that deleting the pod removes deployed PixieApps.

```bash
bx cs workers my-cluster

```

```bash
kubectl delete pod <name>

```

--------------------------------

### Kubernetes on Bluemix: verify pods, nodes, and services

Source: https://pixiedust.github.io/pixiedust/install-pixiegateway.html

Commands to verify the Kubernetes deployment by checking pods, nodes, and services.

```bash
kubectl get pods

```

```bash
kubectl get nodes

```

```bash
kubectl get services

```

--------------------------------

### Hello World with Data PixieApp

Source: https://pixiedust.github.io/pixiedust/hello-world-data-pixieapp.html

Defines a PixieApp that renders a Preview Data button and displays a dataframe as a table when clicked. The app is run with a Spark DataFrame created from sample sales data, and the runInDialog='false' option keeps the output in the cell area.

```python
from pixiedust.display.app import *
@PixieApp
class HelloWorldPixieAppWithData:
    @route()
    def main(self):
        return"""
        <div class="row">
           <div class="col-sm-2">
              <input pd_options="handlerId=dataframe"
                     pd_entity
                     pd_target="target{{prefix}}"
                     type="button" value="Preview Data">
           </div>
           <div class="col-sm-10" id="target{{prefix}}"></div>
        </div>
        """
#Create dataframe
df = SQLContext(sc).createDataFrame(
[(2010, 'Camping Equipment', 3, 200),(2010, 'Camping Equipment', 10, 200),(2010, 'Golf Equipment', 1, 240),
 (2010, 'Mountaineering Equipment', 1, 348),(2010, 'Outdoor Protection',2,200),(2010, 'Personal Accessories', 2, 200),
 (2011, 'Camping Equipment', 4, 489),(2011, 'Golf Equipment', 5, 234),(2011, 'Mountaineering Equipment',2, 123),
 (2011, 'Outdoor Protection', 4, 654),(2011, 'Personal Accessories', 2, 234),(2012, 'Camping Equipment', 5, 876),
 (2012, 'Golf Equipment', 5, 200),(2012, 'Mountaineering Equipment', 3, 156),(2012, 'Outdoor Protection', 5, 200),
 (2012, 'Personal Accessories', 3, 345),(2013, 'Camping Equipment', 8, 987),(2013, 'Golf Equipment', 5, 434),
 (2013, 'Mountaineering Equipment', 3, 278),(2013, 'Outdoor Protection', 8, 134),(2013,'Personal Accessories',4, 200)],
 ["year","zone","unique_customers", "revenue"])

#run the app
HelloWorldPixieAppWithData().run(df, runInDialog='false')
```

--------------------------------

### PixieApp pd_refresh example

Source: https://pixiedust.github.io/pixiedust/html-attributes-pixieapp.html

Use this PixieApp example to see how the pd_refresh attribute works. It defines a counter that increments when the button is clicked, and the div with id counter{{prefix}} is refreshed because its pd_script is executed. The pd_refresh attribute on the button specifies the target element's id.

```python
from pixiedust.display.app import *
@PixieApp
class Refresh():
    def setup(self):
        self.counter=0
    def incrCounter(self):
        self.counter+=1
        print(self.counter)
    @route()
    def mainScreen(self):
        return """
        <input type="button" pd_refresh="counter{{prefix}}" value="Refresh Counter">
        <div id="counter{{prefix}}" pd_script="self.incrCounter()"></div>
        """
Refresh().run()
```

--------------------------------

### Create SimpleDisplayMeta.py with PixiedustDisplay

Source: https://pixiedust.github.io/pixiedust/writeviz.html

Create SimpleDisplayMeta.py in the display/table directory. The getMenuInfo method adds a menu item to the Table category with the given title, icon, and unique id. The newDisplayHandler method returns the SimpleDisplay handler for data processing.

```python
from .SimpleDisplay import SimpleDisplay
from ..display import *
import pixiedust.utils.dataFrameMisc as dataFrameMisc

@PixiedustDisplay()
class SimpleDisplayMeta(DisplayHandlerMeta):
   @addId
   def getMenuInfo(self,entity,dataHandler):
        if dataFrameMisc.isPySparkDataFrame(entity) or dataFrameMisc.isPandasDataFrame(entity):
           return [
               {"categoryId": "Table", "title": "Simple Table", "icon": "fa-table", "id": "simpleTest"}
           ]
        else:
      return []
   def newDisplayHandler(self,options,entity):
       return SimpleDisplay(options,entity)
```

--------------------------------

### Check pip and Jupyter versions

Source: https://pixiedust.github.io/pixiedust/install.html

Run these commands in a Terminal or command-line interface after installing Anaconda to verify that pip and Jupyter are available and configured correctly.

```bash
pip  --version
jupyter --version

```

--------------------------------

### Create Spark DataFrame and call display()

Source: https://pixiedust.github.io/pixiedust/displayapi.html

Creates a Spark DataFrame with quarterly sales data and then calls display() on it to generate an interactive visualization widget. The first block imports the PixieDust display module and builds the DataFrame; the second block invokes display(dd) to render the chart menu.

```python
#import pixiedust display module
from pixiedust.display import *

#Create a dataframe with Quarterly sales results
sqlContext = SQLContext(sc)
dd = sqlContext.createDataFrame(
     [(2010, 'Camping Equipment', 3),
     (2010, 'Golf Equipment', 1),
     (2010, 'Mountaineering Equipment', 1),
     (2010, 'Outdoor Protection', 2),
     (2010, 'Personal Accessories', 2),
     (2011, 'Camping Equipment', 4),
     (2011, 'Golf Equipment', 5),
     (2011, 'Mountaineering Equipment',2),
     (2011, 'Outdoor Protection', 4),
     (2011, 'Personal Accessories', 2),
     (2012, 'Camping Equipment', 5),
     (2012, 'Golf Equipment', 5),
     (2012, 'Mountaineering Equipment', 3),
     (2012, 'Outdoor Protection', 5),
     (2012, 'Personal Accessories', 3),
     (2013, 'Camping Equipment', 8),
     (2013, 'Golf Equipment', 5),
     (2013, 'Mountaineering Equipment', 3),
     (2013, 'Outdoor Protection', 8),
     (2013, 'Personal Accessories', 4)],
     ["year", "zone", "unique_customers"])

```

```python
#call a simple display api to visualize the data
display(dd)

```

--------------------------------

### PixieApp Markdown rendering with @captureOutput

Source: https://pixiedust.github.io/pixiedust/1-1-5.html

Example of a PixieApp that uses the @captureOutput decorator to render Markdown content in a view. The main_screen route displays a Markdown string with raw=True, enabling Markdown syntax in PixieApps. Requires PixieDust 1.1.5 or later.

```python
from pixiedust.display.app import *
from IPython.display import display_markdown

@PixieApp
class TestMarkdown():
    @route()
    @captureOutput
    def main_screen(self):
            display_markdown("""
# Section Header
## Second *bullet*
1. item1
2. item2
3. item3
            """, raw=True)
TestMarkdown().run()
```
Tokens:
Raw