Install script for MXNet
New here? Learn about Bountify and follow @bountify to get notified of new bounties! x

I need a bash script that will install MXNet (with gpu support and python bindings) and all of the necessary prerequisites (for a g2 instance in AWS) running ubuntu, this includes:

  • cuda
  • cudnn
  • opencv
  • mxnet itself
awarded to 5osxcwbf
Tags
python
aws

Crowdsource coding tasks.

1 Solution

Winning solution

You didn't specify exactly which ubuntu version and you didn't specify a AWS ami so i couldn't test it on an exact copy of your environment, so some paths/environments variables might need to be updated. (i could do this for you if you increase your bounty; g2 instances & my time don't come for free).

Also if you want it to be more automated, host your own version of cuDNN & OpenCV since the nvidia site requires a dev account (which results in an install script requiring manual intervention)

Edit4;

#!/bin/bash -e
clear
echo "============================================"
echo "Install script v4 for MXNet on AWS & Ubuntu 14.04"
echo "Runs on Community AMI > ami-1117a87a [Ubuntu 14.04 - NVIDIA CUDA base image (g2)]  + g2.2xlarge" 
echo "============================================"
echo "ssh into your instance; ssh -i 'your_key.pem' ubuntu@[code].compute-1.amazonaws.com"
echo "Check if cuda is running; run: nvidia-smi"
echo "Check if kernel module and devices are present; run: lsmod | grep -i nvidia"
echo "if it looks okay execute this install script"
echo "============================================"

echo "============================================"
echo "Installing some requirements (including opencv libraries etc)" 
echo "============================================"

sudo apt-get update;sudo apt-get install -y build-essential git libblas-dev libatlas-base-dev libopencv-dev

echo "============================================"
echo "Installing CUDNN" 
echo "============================================"

wget https://s3.amazonaws.com/filelocker/public_uploads/cudnn-7.0-linux-x64-v4.0-prod.tar
tar xvf cudnn-7.0-linux-x64-v4.0-prod.tar
rm cudnn-7.0-linux-x64-v4.0-prod.tar
cd cuda
sudo cp lib64/* /usr/local/cuda/lib64/
sudo cp include/cudnn.h /usr/local/cuda/include/

cd ~/

echo "============================================"
echo "Installing mxnet" 
echo "============================================"

git clone --recursive https://github.com/dmlc/mxnet
cd mxnet; cp make/config.mk .

#configure CUDA options
echo "USE_CUDA=1" >>config.mk
echo "USE_CUDA_PATH=/usr/local/cuda" >>config.mk 
echo "USE_CUDNN=1" >>config.mk
echo "USE_BLAS=atlas" >> config.mk
echo "USE_DIST_KVSTORE = 1" >>config.mk
echo "USE_S3=1" >>config.mk

make -j8

echo "============================================"
echo "Installation completed!, adding some new paths & configuring python bindings" 
echo "============================================"

#Second path can be found with the command; sudo find / -name  "libcudnn.so"
echo "export LD_LIBRARY_PATH=/home/ubuntu/mxnet/lib:/usr/local/cuda-7.0/lib64" >> ~/.bashrc
source ~/.bashrc
exec bash

sudo apt-get install -y python-pip
cd /home/ubuntu/mxnet/python
python setup.py install --user
sudo ln /dev/null /dev/raw1394
cd ~/

echo "============================================"
echo "Everything should be working now. you can run the following tests" 
echo "============================================"
echo "Open a python interpreter and type 'import mxnet'"
echo "Or test it by running the following mxnet demo; "
echo "python /home/ubuntu/mxnet/example/image-classification/train_mnist.py --network lenet --gpus 0"
echo "You can also use for example; --gpus 0,1,2,3 to use (4x GPUs) on a g2.8xlarge instance"
echo "============================================" 
echo "To monitor cuda, while code is running open another ssh instance and execute 'nvidia-smi' to check GPU emory usage etc" 
echo "============================================" 
I'm using Ubuntu Server 14.04 LTS (HVM), SSD Volume Type
mr142 over 2 years ago
This is the output: ... Reading state information... Done E: Unable to locate package cuda
mr142 over 2 years ago
If you want to test it, I'm using just a blank ami to start: Ubuntu Server 14.04 LTS (HVM), SSD Volume Type - ami-2d39803a
mr142 over 2 years ago
What instance type did this work on for you?
mr142 over 2 years ago
Whoops, sorry, i made a small mistake, but i corrected it, please try again. (I downloaded the cuda-repo-ubuntu14047.5-18amd64.deb manually but forgot to include it in the installer) Will look into getting it working with your specific ami version.
5osxcwbf over 2 years ago
It dies in this part: "Retrieving the CUDA repository package & install the CUDA Toolkit"
mr142 over 2 years ago
I did ask you asked, but it has the same error. The actual filename I have is cuda-repo-ubuntu14047.5-18amd64.deb
mr142 over 2 years ago
Did you successfully download cuda-repo-ubuntu14047.5-18amd64.deb from https://developer.nvidia.com/cuda-downloads and manually copy it to the home directory of your AWS instance and press a key afterwards? The nvidia site requires dev accounts to download some stuff. I could host the deb files somewhere myself and i could modify the install script so it doesn't require any manual intervention. (so it runs automatically). But this would take me more time than i'm willing to spend on this problem for 10$.
5osxcwbf over 2 years ago
Yes I downloaded it. Here's the output: Installing CUDA Toolkit toolkit (Reading database ... 188097 files and directories currently installed.) Preparing to unpack cuda-repo-ubuntu14047.5-18amd64.deb ... ... Building dependency tree Reading state information... Done E: Unable to locate package cuda
mr142 over 2 years ago
I just updated the bounty so you can fix/test the script. Thanks!
mr142 over 2 years ago
Amazing, will take another look in just a moment. Will make sure that i create a completely automated working script for; Ubuntu Server 14.04 LTS (HVM), SSD Volume Type - ami-2d39803a - g2 instance; Just sit back and relax! :]
5osxcwbf over 2 years ago
Thanks again, I'm relaxing now!
mr142 over 2 years ago
Any luck with getting this thing to work
mr142 over 2 years ago
I have it working! I picked community AMI ami-1117a87a as a base image since that way the install procedure is more robust (less stuff to download install). There are three Amazon S3 files in the script currently hosted by me; Make sure you download these files & host them yourself, i will be removing them in a week. (will try to upload the files to a public host somewhere and update the script once more) (but i can't guarantee files on a public host will remain available for ever) But if you just snapshot your instance after the installation is completed there shouldn't be any problem! :D
5osxcwbf over 2 years ago
Ok it seems to install, but if I open a python interpreter and type 'import mxnet' I get this: File "/usr/lib/python2.7/ctypes/init.py", line 365, in init self.handle = _dlopen(self.name, mode) OSError: libcudnn.so.4: cannot open shared object file: No such file or directory
mr142 over 2 years ago
Fixed it! its working!1!1 Check my (hopefully final) update! :D It was a simple error, basically some path was messed up due to me making a mistake with CUDA versions, i used the spare computing time i had to remove some redundancies from the script (this is why its better to start of with a fresh instance & run the updated script again) The mxnet/example/image-classification/train_mnist.py example executed successfully within a couple seconds for me (as explained in the comments on the bottom of the script) & starting a python interpreter and typing 'import mxnet' also works fine. Hope it works flawlessly and you are happy :]
5osxcwbf over 2 years ago
Did it work for you? :D
5osxcwbf over 2 years ago
Testing it now...
mr142 over 2 years ago
So did it work for you? if it did please accept this answer so bountify can give me the bounty :] ( You can still ask me questions or ask for small changes, no worries )
5osxcwbf over 2 years ago
I had to add 'export PYTHONPATH=$PYTHONPATH:/home/ubuntu/mxnet/python' at the end but other than that it works well, thanks so much!
mr142 over 2 years ago
Hey, so I'm going to post another question like this one for another libraries, I'd I'll to add it to this script so the box will have mxnet but also has a few new others. If you can answer I'll be sure to give you the tip.
mr142 over 2 years ago
Ok great, lets continue discussing @ the new question/bounty, i will try to create a script for the new requirements :]
5osxcwbf over 2 years ago