How to create a new virtual environment

Although it's convenient to have a preconfigured Python environment loaded as soon as you start an instance, some users are better off managing their own virtual environments. You can do this by creating a new virtual environment in your home directory.

There are several important advantages when you create your own environment:

Any packages that you install will persist across restarts of your instance.
Your environment will be accessible from any of your instances and any of your projects.
You control all package versions.

There are two steps to setting your environment in Nightingale OS. First, create the environment and install packages. Second, add the environment to Jupyter. Let's get started.

Create your virtual environment

Currently, Python packages can only be installed via pip through the Nightingale OS PyPI mirror and not from conda repositories. This means you should use Python's built-in venv library to create and manage your environment. The Python documentation shows step-by-step examples.

⚠️ Because Nightingale OS uses a local pip mirror, the pip install command sometimes times out with 502 Bad Gateway errors while waiting for the PyPI mirror to download and cache a package before responding to your request. This is normal and expected since so many data science libraries are quite large. Generally, you can resolve this issue by running pip install again.

# Create a new Python virtual environment in HOME
$ python -m venv ~/.venv/some-project

# Activate the environment
$ source ~/.venv/some-project/bin/activate

# Install some packages
$ pip install pendulum

# Optionally, install the entire default environment.
# Get some coffee while you wait...
# Be prepared to restart pip install if you get any 502 errors.
$ pip install -r ~/default-venv-requirements.txt

After these steps, you've created a virtual environment, but you're not quite done. Next, you need to tell Jupyter to use this environment.

Add a virtual environment to Jupyter

# Activate your environment, "some-project"
source ~/.venv/some-project/bin/activate

# install the ipykernel in your venv
pip install ipykernel

#Install ipykernel for "some-project"
python -m ipykernel install --user --name=some-project

Now whenever you open the Jupyter Lab launcher—across all your instances and all your projects—you should see the virtual environment that you created. (You might need to refresh your browser.)

And when you click on this option, you should see it in the kernel selector for your notebook (top right navigation bar).

You can confirm that your custom environment is active. Suppose we have installed a custom package, pendulum .

You don't have to remember to select your environment in the launcher. After launching a notebook, you can switch back and forth using the kernel selector in the top-right corner. For example, if you change the kernel back to the default and re-run the cell above, you'll get an error because pendulum is only installed in your personal environment, "some-project".

You can fix this by switching back to "some-project":

Remove a virtual environment from Jupyter

Just as you needed to tell Jupyter about your virtual environment after you created it, if you want to delete a virtual environment, you also need to remove it from Jupyter.

# Activate your environment, "some-project"
$ source ~/.venv/some-project/bin/activate

# Uninstall the kernel, "some-project"
$ jupyter kernelspec uninstall some-project 
Kernel specs to remove: 
  some-project    /home/ngsci/.local/share/jupyter/kernels/some-project 
Remove 1 kernel specs [y/N]: y 
[RemoveKernelSpec] Removed /home/ngsci/.local/share/jupyter/kernels/some-project

You will no longer see this environment listed as an option in your Jupyter notebooks.