Jupyter kernels - Manual management

Introduction

Jupyter kernels execute the code that you write. Mahuika provides a number of Python and R kernels by default, which can be selected from the Launcher.

Many packages are preinstalled in our default Python and R environments and these can be extended further as described on the Python and R support pages.

Adding a Custom Python kernel

You can configure custom Python kernels for running your Jupyter notebooks. This could be necessary and/or recommended in some situations, including:

• If you wish to load a different combination of environment modules than those we load in our default kernels
• If you would like to activate a virtual environment or conda environment before launching the kernel

The following example will create a custom kernel based on either a conda environment (using the Miniforge3 environment module) or a Python virtual environment (using a Python environment module). The same approach applies to other environment modules too.

see also

See the Jupyter kernels - Tool-assisted management page for the preferred way to register kernels, which uses the nesi-add-kernel command line tool to automate most of these manual steps.

First, change directory into the path that you would like to place your environment.

• If you would like to share this environment with other users, change directory into your project folder using cd /nesi/project/<project-code>. Do not use the path that includes 00_nesi_projects or home in the name as this causes issues.

Next, you will create your environment and a wrapper script for it. You can choose whether to use a conda environment, or a Python virtual environment:

Conda environmentPython virtual environment

Second, in a terminal run the following commands to load a Miniforge environment module:

module purge
module load Miniforge3

Now create a conda environment named "my-conda-env" using Python 3.11. The ipykernel Python package is required but you can change the names of the environment, version of Python and install other Python packages as required.

conda create --prefix ./my-conda-env python=3.11
source $(conda info --base)/etc/profile.d/conda.sh
conda activate ./my-conda-env
conda install ipykernel
# you can pip/conda install other packages here too

Third, we will create a wrapper for your conda environment. Change directory into your my-conda-env folder:

cd my-conda-env

And add the following as wrapper.sh into your my-conda-env folder:

#!/usr/bin/env bash

# load required modules here
module purge
module load Miniforge3

# activate conda environment
source $(conda info --base)/etc/profile.d/conda.sh
conda deactivate  # workaround for https://github.com/conda/conda/issues/9392
conda activate my-conda-env

# run the kernel
exec python $@

Make the wrapper script executable:

chmod +x wrapper.sh

Fourth, create a Jupyter kernel based on your new conda environment:

python -m ipykernel install --user --name my-conda-env --display-name="My Conda Env"

We must now edit the kernel to load the required environment modules before the kernel is launched. Change to the directory the kernelspec was installed to ~/.local/share/jupyter/kernels/my-conda-env, (assuming you kept --name my-conda-env in the above command):

mkdir -p ~/.local/share/jupyter/kernels/my-conda-env
cd ~/.local/share/jupyter/kernels/my-conda-env

and edit the kernel.json to change the first element of the argv list to point to the wrapper script we just created. The file should look like this:

{
 "argv": [
 "<full_path_to_your_wrapper_file>/wrapper.sh",
 "-m",
 "ipykernel_launcher",
 "-f",
 "{connection_file}"
 ],
 "display_name": "My Conda Env",
 "language": "python"
}

After refreshing JupyterLab your new kernel should show up in the Launcher as "My Conda Env".

Second, in a terminal run the following commands to load a Python environment module:

module purge
module load Python/3.14.4-foss-2026

Now create a Python virtual environment named "my-venv". The ipykernel Python package is required but you can change the name of the environment and install other Python packages as required.

python3 -m venv ./my-venv
source ./my-venv/bin/activate
pip install --upgrade pip
pip install ipykernel
# you can pip install other packages here too

Third, we will create a wrapper for your virtual environment. Change directory into your my-venv folder:

cd my-venv

And add the following as wrapper.sh into your my-venv folder:

#!/usr/bin/env bash

# load required modules here
module purge
module load Python/3.14.4-foss-2026

# activate virtual environment
source <full_path_to_your_venv>/my-venv/bin/activate

# run the kernel
exec python $@

Make the wrapper script executable:

chmod +x wrapper.sh

Fourth, create a Jupyter kernel based on your new virtual environment:

python -m ipykernel install --user --name my-venv --display-name="My Venv"

We must now edit the kernel to load the required environment modules before the kernel is launched. Change to the directory the kernelspec was installed to ~/.local/share/jupyter/kernels/my-venv, (assuming you kept --name my-venv in the above command):

mkdir -p ~/.local/share/jupyter/kernels/my-venv
cd ~/.local/share/jupyter/kernels/my-venv

and edit the kernel.json to change the first element of the argv list to point to the wrapper script we just created. The file should look like this:

{
 "argv": [
 "<full_path_to_your_wrapper_file>/wrapper.sh",
 "-m",
 "ipykernel_launcher",
 "-f",
 "{connection_file}"
 ],
 "display_name": "My Venv",
 "language": "python"
}

After refreshing JupyterLab your new kernel should show up in the Launcher as "My Venv".

Sharing your custom kernal with your project team members

You can also configure a shared Python kernel that others with access to the same project will be able to load.

• To do this, you must make sure it also exists in a shared location (other users cannot see your home directory).

First, you need to perform the steps in Adding a Custom Python kernel

Next, your team members need to set up the kernel on their side. The exact steps depend on the type of environment you created, so follow the steps in the tab that matches it:

Conda environmentPython virtual environment

Second, your team members need to run the following commands in the terminal:

# change directory into the path that contains your conda environment
cd <full_path_to_your_conda_environment>

# load Miniforge3
module purge
module load Miniforge3

# Activate your shared conda environment
source $(conda info --base)/etc/profile.d/conda.sh
conda activate ./my-conda-env

Third, get your team members to create a Jupyter kernel based on your conda environment:

python -m ipykernel install --user --name my-conda-env --display-name="My Conda Env"

Your project members must now edit the kernel in their home directories to load the required environment modules before the kernel is launched. Change to the directory the kernelspec was installed to ~/.local/share/jupyter/kernels/my-conda-env, (assuming you kept --name my-conda-env in the above command):

mkdir -p ~/.local/share/jupyter/kernels/my-conda-env
cd ~/.local/share/jupyter/kernels/my-conda-env

and edit the kernel.json to change the first element of the argv list to point to the wrapper script we just created. The file should look like this:

{
 "argv": [
 "<full_path_to_your_wrapper_file>/wrapper.sh",
 "-m",
 "ipykernel_launcher",
 "-f",
 "{connection_file}"
 ],
 "display_name": "My Conda Env",
 "language": "python"
}

After refreshing JupyterLab your new kernel should show up in the Launcher as "My Conda Env".

Second, your team members need to run the following commands in the terminal:

# load the Python environment module
module purge
module load Python/3.14.4-foss-2026

# Activate the shared virtual environment
source <full_path_to_your_venv>/my-venv/bin/activate

Third, get your team members to create a Jupyter kernel based on your virtual environment:

python -m ipykernel install --user --name my-venv --display-name="My Venv"

Your project members must now edit the kernel in their home directories to load the required environment modules before the kernel is launched. Change to the directory the kernelspec was installed to ~/.local/share/jupyter/kernels/my-venv, (assuming you kept --name my-venv in the above command):

mkdir -p ~/.local/share/jupyter/kernels/my-venv
cd ~/.local/share/jupyter/kernels/my-venv

and edit the kernel.json to change the first element of the argv list to point to the wrapper script we just created. The file should look like this:

{
 "argv": [
 "<full_path_to_your_wrapper_file>/wrapper.sh",
 "-m",
 "ipykernel_launcher",
 "-f",
 "{connection_file}"
 ],
 "display_name": "My Venv",
 "language": "python"
}

After refreshing JupyterLab your new kernel should show up in the Launcher as "My Venv".

Custom kernel in a Singularity container

An example showing setting up a custom kernel running in a Singularity container can be found on our Lambda Stack support page.

Adding a custom R kernel

You can configure custom R kernels for running your Jupyter notebooks. The following example will create a custom kernel based on the R/3.6.2-gimkl-2020a environment module and will additionally load an MPFR environment module (e.g. if you wanted to load the Rmpfr package).

In a terminal run the following commands to load the required environment modules:

module purge
module load IRkernel/1.1.1-gimkl-2020a-R-3.6.2
module load Python/3.8.2-gimkl-2020a

The IRkernel module loads the R module as a dependency and provides the R kernel for Jupyter. Python is required to install the kernel (since Jupyter is written in Python).

Now create an R Jupyter kernel based on your new conda environment:

R -e "IRkernel::installspec(name='myrwithmpfr', displayname = 'R with MPFR', user = TRUE)"

We must now to edit the kernel to load the required environment modules when the kernel is launched. Change to the directory the kernelspec was installed to (~/.local/share/jupyter/kernels/myrwithmpfr, assuming you kept --name myrwithmpfr in the above command):

cd ~/.local/share/jupyter/kernels/myrwithmpfr

Now create a wrapper script in that directory, called wrapper.sh, with the following contents:

#!/usr/bin/env bash

# load required modules here
module purge
module load MPFR/4.0.2-GCCcore-9.2.0
module load IRkernel/1.1.1-gimkl-2020a-R-3.6.2

# run the kernel
exec R $@

Make the wrapper script executable:

chmod +x wrapper.sh_
 "argv": [
 "/home/<username>/.local/share/jupyter/kernels/myrwithmpfr/wrapper.sh",
 "--slave",
 "-e",
 "IRkernel::main()",
 "--args",
 "{connection_file}"
 ],
 "display_name": "R with MPFR",
 "language": "R"
}

After refreshing JupyterLab your new R kernel should show up in the Launcher as "R with MPFR".

Quiz Progress

• Answered: 0 / 0 (0%)
• Correct: 0 / 0 (0%) Reset