Using Jupyter Notebooks and JupyterHub on Savio

Getting Started  | Setting up IPython Clusters  | Adding New Kernels  | Support for Juypter Notebooks on Savio

The Jupyter Notebook is a web application that enables you to create and share documents (called "notebooks") that can contain a mix of live code, equations, visualizations, and explanatory text.

This is an introduction to using these notebooks on the Savio high performance computing cluster at the University of California, Berkeley.

JupyterHub and Jupyter notebooks are currently offered on Savio without warranty and with an expectation that users will be self-supporting. Before using Jupyter notebooks on the cluster, please see Support for Juypter Notebooks on Savio.

Getting Started

1. Connect in your browser to https://ln000.brc.berkeley.edu

This is a test site, whose URL will change when it goes into production.

When visiting this site - at least for the first time - your browser will display a warning about the site’s use of a self-signed SSL certificate.
As a result, you will need to follow the on-screen instructions to accept that certificate as valid, and proceed. (The method for doing this varies by browser and platform. One known issue: with Safari on a Macintosh, at the "Safari can't verify the identity ..." dialog, select "Show Certificate" and then be sure to click the "Always trust "ln000.brc.berkeley.edu" ..." checkbox.)

(A valid certificate will be provided once the site goes into production.)

2. Log in with your BRC username and one-time password (OTP).

3. After logging in, the initial Jupyter screen will be displayed. Click the "Start My Server" button on that screen.

4. On the next screen, you will see a “Select a job profile” dropdown for selecting where you want your notebook’s computational jobs to be run (“spawned”). Choose one of the following job profiles, depending on the needs of your job(s).

Job profile When to choose it
Local server When testing your code. (This is the default profile.)
Savio - 1 node When running resource-intensive computations.
Savio2 - 1 node When running resource-intensive computations.
(Runs code slightly faster, but costs modestly more for Faculty Computing Allowance users, than the “Savio - 1” node profile.)
Savio2_HTC - 1 cpu When running computations that require just one core (CPU).

When selecting the Savio, Savio2, or Savio2_HTC job profiles, your jobs are spawned onto the Savio cluster, and will be run under your default account. (For Condo participants, this will be your Condo account; otherwise, your Faculty Computing Allowance account.) Currently each of these three job profiles is limited to a single node, and to a maximum runtime of 8 hours per job.

5. Click the “Spawn” button.

6. After selecting a job profile and clicking “Spawn”, the home directory will be displayed. From the "New" dropdown menu (next to 'Upload' near the top right of the screen) select one of the following options:

JupyterHub New menu, showing options available

Select "Python 2" under “Notebooks” and you should find yourself in a notebook with full access to Python 2.7.8, as well as all of the Python packages on Savio that are part of the python/2.7.8 module tree.

Select “Terminal” to open a UNIX terminal session instead, so that you can work at the command line, rather than in a notebook. (You might select this option if you need to set up IPython Clusters, create custom configuration profiles for same, or add kernels, as described further below.)

(Note that “Python 3” is also available under “Notebooks”. However, currently, Python 3 is not as fully supported by Python packages on Savio as is Python 2.)

7. To move between your notebook and the control page that allows you to see your files and select any running notebooks, clusters and terminals, simply click on the ‘jupyter’ banner in the upper left corner of your notebook. (Or alternately, select “Control Panel” and then click “My Server”.)

8. You can have your session continue to operate in the background by selecting the “Logout” button. Note that, when doing so, you will continue to be charged for FCA use or, if using a Condo node, you will prevent others in your group from accessing the relevant resource on which the job is running.

9. To terminate a running notebook, select the “Running” tab and click the ‘Shutdown’ button for that notebook. Or alternately, select “Control Panel” and then click “Stop My Server” to terminate all running notebooks.

If you don't need to use IPython Parallel to parallelize your code, you can stop here and you should already have a fully working Jupyter environment. To start working with Jupyter notebooks, please see the Jupyter Documentation.

Advanced Topics

Note: the material that follows is more advanced and is intended to provide tips to users wishing to explore additional features.

To use the parallelization capabilities of IPython Parallel, you’ll need to first set up a cluster, as discussed next.

Setting up IPython Clusters

Basic usage of IPython Clusters

If you need to set up one or more IPython Clusters, for parallel computing tasks, a little more configuration on your part is needed.

The following is an example of setting up an IPython Cluster using the default Python 3 kernel on Savio (powered by Python 3.5.1):

1. Login to the cluster via a terminal (or in JupyterHub start a "Terminal" session via the "New" dropdown menu).

2. Enter the following command, which enables the IPython Clusters tab extension:

/global/software/sl-6.x86_64/modules/langs/python/3.5.1/bin/ipcluster nbextension enable --user

3. Now, on the JupyterHub page, you should see that the name of the "Clusters" tab has been changed to "IPython Clusters". (Note: you may need to refresh your browser page to see this change)

If this change doesn’t appear, you will need to stop the current Jupyter server. To do that, click the "Control Panel" button at the upright corner (next to the "Logout" button). From there, click "Stop My Server". Then, on the next screen, wait a few seconds, then click "My Server". Now the “Clusters” tab name should change to “IPython Clusters”. (If you click that button too early, you may encounter a "The page isn't redirecting properly" error message, but refreshing the page will fix this - it is a hit-or-miss timing issue.)

4. When you click the "IPython Clusters" tab, you will see a "default" profile which allows you to start a local IPython Cluster with a user-specified number of engines. If you are just testing the basic IPython Cluster concept, the "default" profile is sufficient. This includes running a cluster using as many cores as available on a node, assuming you’ve selected either the Savio or Savio2 profiles when spawning the cluster. (For more advanced usage, see below for information on creating your own profile.)

5. Go to a running notebook, or start a new one. For parallel computation, you should select the Savio or Savio2 job profile, as discussed in Step 4 of Getting Started.

6. You can now get started via the following Python code, using the ‘rc’ object to interact with your cluster. (Note that for Python 3, you should import ipyparallel not IPython.parallel.)

import IPython.parallel as ipp # For Python 3, use instead: import ipyparallel
rc = ipp.Client(profile='default', cluster_id='')
rc.ids # this should show the number of workers equal to the number you requested

To learn more about how to work with your new IPython Cluster, please see the IPython Parallel Documentation and/or the instructions and sample code in the August 2016 Savio Intermediate / Parallel training.

Advanced usage of IPython Clusters: creating your own profile

If you need to run your IPython Cluster with specific choices of the partition, time limit, number of nodes, account, QOS, etc., you will need to create your own configuration profile(s). In particular this allows you to modify all of the flags that you might customize when submitting a standard SLURM job via sbatch or srun. These flags will affect the SLURM job controlling the IPython Cluster (i.e., the worker processes) but have no effect on the notebook that interacts with the IPython Cluster.

Please follow these steps:

1. Login to the Savio via a terminal (or in JupyterHub start a "Terminal" session via the "New" dropdown menu), and enter the following command:

/global/software/sl-6.x86_64/modules/langs/python/3.5.1/bin/ipython profile create --parallel --profile=myprofile

(The profile name can be anything; “myprofile” is used as an example here, and in several of the steps below.)

2. Within the same terminal, enter the following command:

cd $HOME/.ipython/profile_myprofile

(In the above command, the profile name following the underscore has to exactly match the one that was just created in step 1, above.)

3. Add the following contents to the end of the "ipcontroller_config.py" file:

import netifaces
c.IPControllerApp.location = netifaces.ifaddresses('eth0')[netifaces.AF_INET][0]['addr']
c.HubFactory.ip = '*'

4. Add the following contents to the end of the "ipcluster_config.py" file:

#import uuid
#c.BaseParallelApplication.cluster_id = str(uuid.uuid4())
c.IPClusterStart.controller_launcher_class = 'SlurmControllerLauncher'
c.IPClusterEngines.engine_launcher_class = 'SlurmEngineSetLauncher'
c.IPClusterEngines.n = 12
c.SlurmLauncher.queue = 'myprofile'
c.SlurmLauncher.account = 'fc_xyz'
c.SlurmLauncher.qos = 'savio_normal'
c.SlurmLauncher.timelimit = '8:0:0'
#c.SlurmLauncher.options = '--export=ALL --mem=10g'
c.SlurmControllerLauncher.batch_template = '''#!/bin/bash -l
#SBATCH --job-name=ipcontroller-fake
#SBATCH --partition={queue}
#SBATCH --account={account}
#SBATCH --qos={qos}
#SBATCH --ntasks=1
#SBATCH --time={timelimit}
'''
c.SlurmEngineSetLauncher.batch_template = '''#!/bin/bash -l
#SBATCH --job-name=ipcluster-{cluster_id}
#SBATCH --partition={queue}
#SBATCH --account={account}
#SBATCH --qos={qos}
#SBATCH --ntasks={n}
#SBATCH --time={timelimit}

/global/software/sl-6.x86_64/modules/langs/python/3.5.1/bin/ipcontroller --profile-dir={profile_dir} --cluster-id="{cluster_id}" & sleep 10
srun /global/software/sl-6.x86_64/modules/langs/python/3.5.1/bin/ipengine --profile-dir={profile_dir} --cluster-id="{cluster_id}"
'''

Note that the commented lines above (but not the #SBATCH lines) are optional and you can choose whether to uncomment and modify them; all other lines are necessary.

5. In the "ipcluster_config.py" file, you will also need to examine and change the values of at least one or more of the following four entries, to specify your Savio scheduler account name (e.g., 'fc_something', 'co_something' ...), the queue (partition) and QoS on which you want to launch the cluster, and the wall clock time that the cluster will be active:

c.SlurmLauncher.queue =
c.SlurmLauncher.account =
c.SlurmLauncher.qos =
c.SlurmLauncher.timelimit =

6. After adding and configuring all of these various settings, via the steps above, you can go back to the JupyterHub "IPython Clusters" tab to start a new IPython Cluster using your newly created profile, with a selected number of engines.

7. Once your cluster is started, follow the instructions at step 6 under the "Basic usage of IPython Clusters" section, above, but making sure to provide the correct ‘profile’ and ‘cluster_id’ arguments when calling ‘Client()’. The value of ‘profile’ should be the name chosen in step 1. The value of ‘cluster_id’ should be the value to which you set c.BaseParallelApplication.cluster_id in step 4 of this section. If you do not set it in step 4 (as is the case above where it is commented out) then ‘cluster_id’ should be an empty string, as in the "Basic usage" section.

8. You should be able to monitor the SLURM job controlling your cluster via the squeue command, looking for the SLURM job name indicated in the “ipcluster_config.py” file.

Note that if you set your time limit to more than eight hours, your cluster will continue to run but your original notebook will stop because of the eight hour limit discussed earlier in this document. For information on how to reconnect to your running cluster, please contact us at brc-hpc-help@berkeley.edu.

Finally note that it makes sense to use the “Local server” when starting the notebook from which you control the IPython Cluster, provided all your heavy computation will occur via the IPython Cluster workers.

Adding New Kernels

If for some reason the default kernels provided do not meet your computation requirements, you can add one or more new kernels, whether for Python or any of the many other languages and environments supported by Jupyter.

You can add new kernels via configuration files. As an example, to add a new IPython kernel to your Jupyter environment, please use the following "kernel.json" configuration file as a template. You will need to put this file in your "$HOME/.ipython/kernels/mykernel" directory. "mykernel" should be a meaningful name to you if you need to install multiple kernels this way.

{
    "argv": [
        "/global/software/sl-6.x86_64/modules/langs/python/2.7.8/bin/python2",
        "-m",
        "IPython.kernel",
        "-f",
        "{connection_file}"
    ],
    "language": "python",
    "display_name": "mykernel",
    "env": {
        "PATH" : "/bin:/usr/bin:/usr/local/bin:/path/to/myexec",
        "LD_LIBRARY_PATH": "/path/to/mylib",
        "PYTHONPATH" : "/path/to/mypythonlib"
    }
}

Please review the IPython Kernel Specs for more details regarding the format and contents of this configuration file. In particular, please make sure $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, and all other environment variables that you use in the kernel are properly populated with the correct values.

Support for Jupyter Notebooks on Savio

The Berkeley Research Computing (BRC) program currently is making JupyterHub and Jupyter notebooks available on UC Berkeley's high performance computing cluster, Savio, without warranty.

Users are responsible for a basic understanding of how to create and work with Jupyter notebooks, as well as how to run batch jobs on the cluster (whether spawned from within Jupyter notebooks or run at the command line), and how to create and edit configuration files in a terminal window.

Documentation and user support for Jupyter notebooks is provided by Project Jupyter and its community, and/or other third-party resources, rather than by BRC. Please see the Project's Community Resources web page for links to documentation, community mailing lists, Question & Answer sites, and file repositories that can help you learn and use this software.

Users of the software are responsible for understanding that Jupyter technologies are rapidly emerging and changing, and that their future maintenance and development is provided by (and dependent upon) Project Jupyter and its open source community. UC Berkeley staff, students, and researchers participate in this community, but the Project is independent of both the campus and BRC.