Local Installation - Collaborative and Reproducible Data Science

These notes will help to setup your local machine with the JupyterLab we are using in the Stat 159/259 Hub. Even if you already have conda installed in your local machine, it’s not a bad idea to reset the conda installation. The idea is to have a very simplistic base environment from which you can launch JupyterLab and connect it to the kernels associated to other environments you create.

Useful links:

Part 1: Installation and configuration of miniconda¶

Instead of installing conda with the full package management system and many Python libraries, we are going to install Miniconda, a minimalist version of conda that included the package management tool and a small number of useful packages. If you already have conda installed in your machine, you may don’t need to do this. An easy way to check if conda is installed in your computes is by enter conda --help in the terminal. You are also welcome of installing conda from zero by removing all conda directories and installing miniconda.

Note

We recommend you to work from shell with Bash. It may be the case that your default is zsh. You can change this by entering

chsh -s /bin/bash

Then, the next time you open a terminal it will open it with Bash.

You can install the last version of miniconda depending on you operating system from the Miniconda website. For unix machines, a more simple way of doing this is by using the wget command followed by the installer link (silent installation mode). For example, for a MacOSX machine (without M1 chip) you can use

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O ~/miniconda.sh
mkdir -p $HOME/local/conda
bash ~/miniconda.sh -b -p $HOME/local/conda

and for a Linux machine

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
mkdir -p $HOME/local/conda
bash ~/miniconda.sh -b -p $HOME/local/conda

If you don’t have wget installed, you can do it using brew with brew install wget; or download the .sh file directly from the miniconda installation website and replace ~/miniconda.sh by the name of the downloaded file (don’t change the name of the file and be sure that the direction to the file is correct, either by adding the full path to the file or by moving the sh file to your home directory).

Note for Windows users

If you are working from a Windows machine, you can directly install Miniconda using the link provided in the website. Most cloud-hosted computing is based on a Unix foundation, which both MacOS and Linux provide out of the box, so some of our instructions fit that workflow a little more easily. But for Windows platforms, Microsoft now officially supports something very similar, which is excellent and we highly encourage you to play with and learn about: the Windows Subsystem for Linux (WSL). It provides top-notch Unix functionality next to your regular Windows tools and workflow, and is probably the ideal setup for modern scientific and research computing on Windows. Discussing it in detail is beyond the scope of this course, but we highly encourage you to explore this tool on your own (and let us know how it goes!).

In order to specify the path where conda is installed, you need to add the following line to your .bashrc file

# add path to conda
export PATH="$HOME/local/conda/bin:$PATH"

and execute it by source .bashrc. After doing this, you will see that now conda is recognized. You can fix this changes by now running conda init, so you don’t need to worry again about PATH. You can see where conda has been installed by typing which conda in your terminal.

You can change the configuration of conda inside .condarc manually or directly though the shell by using the following commands. In this case, we specify the channel used by conda/mamba to download the libraries.

conda config --add channels conda-forge
conda config --set channel_priority strict

Once you have installed conda, you can use it to install mamba:

conda install --yes mamba

Part 2: Setting the Hub environment in local¶

Once you had installed miniconda, you can create a virtual environments from a yml file just as we did in the JupyterHub in the Environment lectures. As a first step, we are going to create a virtual environment with the same configuration we have in the JupyterHub. The configuration file environment.yml for the Hub is available at the site repository for the course. Before using environment.yml to create a new environment, take a look to the text file and try to identify parts of the file that are not necessary or irrelevant to create the Hub environment; you will have to remove the last lines of the file that include the pip instructions for the requirements of the JupyterBook we use for the website, which are not necessary for the creation of the virtual environment.

Then, you can create a the virtual environment with

mamba env create -f environmnent.yml

You can always use conda env create -f <yml file>, but mamba is much faster than conda. It will take a few minutes until all the dependencies get installed in your machine. It could be the case that one specific library produces conflict in certain machines. If conda prints a error message associated to a specific library included in the environment file, try to comment such line in environment.yml and try again.

Now you can activate the environment from the terminal using conda activate <environment name>. From there, you can lunch the classical notebook by tipping jupyter notebook but you also have installed the JupyterLab, which you can lunch with jupyter lab.

In order to reproduce the same environment we have in the Hub, we are going to install JupyterLab in our base environment and then create a Python kernel with the environment of the Hub. In your base environment (conda activate base), be sure to have installed both JupyterLab and ipykernel. If not, you can install then using conda:

mamba install -c conda-forge jupyterlab
mamba install ipykernel

From here, the story is pretty much the same of what we did in the Hub to create a kernel with a new virtual environment (see Using an environmnet in your notebooks). Moving back to the new environment we just created (conda activate <environment name>) we run

python -m ipykernel install --user --name <environment name> --display-name "IPython - STAT 159"

Now, if we comeback to base and lunch Jupyter Lab with jupyter lab, we should see the option creating a new notebook with the IPython - STAT 159 kernel.

Additional Comments¶

If you had made it to this point, you can continue the configuration of your local machine so it also includes some of the other tools we were using in the Hub.

Dotfiles: You can clone the bare repository you used to configure your bash, git and more setups in the hub. Remember to clone the repository you had forked in your personal GH account.
Midnight Commander: You can install the midnight commander on your local computer using brew, an installation tool for Mac computers.