Create a conda environment for deeplabcut on the ICL cluster

Published on Author Giorgio Gilestro

This guide helps you creating a split deeplabcut environment.

  1. One half will run on your local office computer (no GPU needed) and will be used to annotate images for the training of the NN
  2. The other half will run on the ICL cluster, so that it will take full advantage of the higher performance during training

Install locally (1)

Local installation will still require tensorflow (not the GPU version) and Python 2.69
This guide refers to how to install this using miniconda on archlinux

Install miniconda and pip using the package manager

yay -S miniconda python-pip

source miniconda 

sudo ln -s /opt/miniconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh 
source /etc/profile.d/conda.sh

create a new conda environment running python 2.6 and activate it. We also install tensorflow, wxpython and psutil in this way because for some reason psutil will not work through pip.

conda create --name deeplabcut python=2.6 conda-forge psutil wxpython tensorflow=1.12
conda activate deeplabcut

Install deeplabcut via pip

pip install deeplabcut

Make sure your DLClight variable is set to false and launch the project manager

export DLClight=False
python -m deeplabcut

Mount RDS storage locally. This is smb share so you can either mount using smbfs in fstab or using nautilus. ICL instructions are provided here.

Install on the cluster (2)

SSH into the cluster

ssh -xy username@login.hpc.ic.ac.uk

If you need to start from a fresh conda installation, remove your current conda directory and reinstall conda.

rm -rf ~/anaconda3
anaconda-setup

Create a new conda environment called deeplabcut with a downgraded version of Python (3.6.9)

#this will install python 3.6.9 in a new conda env
conda create -n deeplabcut python=3.6
conda activate deeplabcut

Install the older version of tensorflow-gpu (deeplabcut does not support version 2.0.0 as of now). 1.12 is the one reccomended in the Nature Protocols paper.

conda install tensorflow-gpu==1.12

install deeplabcut using PIP

pip install deeplabcut

optional but reccomended: set an environmental variable called DLClight. For this change to be permanent, you will have to create some new files in the anaconda3 environment specific directory. Follow instructions here, using the following env variable export line

export DLClight=True

if you decide not to do this, you will have to set the env variable from jupyter using the following

import os
os.environ['DLClight']='True'

Install the ipykernel conda package

conda install ipykernel

Install python kernel for Jupyter:

python -m ipykernel install --user --name deeplabcut --display-name "Python (deeplabcut)"

Now it should be all ready to go. Connect to an ICL Jupyter instance and select Python (deeplabcut) as kernel.

Useful links:
ICL Conda on Jupyter