Too Long; Did not Read:
Update heroku to use this buildpack URL: https://github.com/slobdell/heroku-buildpack-python-opencv
True Story Follows:
Recently I’ve been playing around with Heroku, and I’ve been trying to create a web application that uses OpenCV on the backend. Initially there were quite a few problems to even get this working at all, but I did find a custom buildpack that worked quite well. Sort of.
This buildpack worked by offloading pre-compiled OpenCV binaries to heroku’s environment. This had to hapen because OpenCV takes forever to compile and install, and a 15 minute build timeout on Heroku will prevent an easy solution to getting OpenCV deployed (and that’s also a really long time anyway). However, using the buildpack mentioned above, I ended up having a few random problems. For instance, I couldn’t get libmemcached installed (which is otherwise trivial in Heroku’s default buildpack), and I couldn’t include any python module in my requirements.txt file that used bz2 compression. This was all acceptable to work with until I got to the point where I actually tried to do some stuff with OpenCV…I was playing around with videos so I needed ffmpeg support, but the version of OpenCV from the buildpack was not compiled with that support enabled. So, at this point I had to learn and figure out how to replicate the work to make a custom buildpack. And this is how it felt:
Context: Minimal understanding of integrating the buildback
A custom buildpack can be specified for Heroku using:
In the default buildpacks, there are a lot of extra files and options because of automated and dynamic configurations, but if you’re using a custom buildpack, then there’s not really a need to be dynamic or infer any options because you’re specifying exactly what they are. When you deploy to Heroku, a virtual machine will be created using the specifications from the above Github URL. You can learn more about Heroku custom buildbacks here.
Getting Started: Create an empty Virtual Machine
The nice thing about this tutorial is that presumably, if you copy and paste from start to finish you should end up with the exact same results given that the startpoint should be identical. I grabbed an empty Vagrant template from https://github.com/ejholmes/vagrant-heroku. So for these steps:
- Download VirtualBox (in order to run virtual machines)
- Download Vagrant (in order for VirtualBox to open the heroku box machine below)
- Create a file somewhere called “VagrantFile” with the following lines:
- Note that the URL I have in the configuration is different from the one in the repo’s README. At the time of this writing I copied the file to Amazon so that this tutorial couldn’t change over time.
- Run “vagrant up”
- Open VirtualBox and there should now be a new machine running 64 bit Ubuntu 10.04 that you can start
Open VirtualBox and there should now be a new machine. Before you start, you’ll need to up the available RAM in the machine, otherwise the OpenCV compilation will end up failing. I went ahead and changed it from 512 to 2045 MB (settings->system). Start the machine and login with credentials:
Also note that the shell wasn’t visible at first, so like most problems, you can fix by turning it off and turning it back on
Install some stuff before OpenCV
Ubuntu 10.04’s version of python is 2.6, but we probably want python 2.7. Without it, you can’t do list comprehensions with dictionaries, and there are some things left out of the collections module. So unless you like to go to the gym and put on a couple quarters instead of stacking 45’s, you shouldn’t skip these steps:
(lesson learned from using the other buildpack; without it you might have issues with pip for certain modules)
- Download Python 2.7.8 from here
(IMPORTANT! The extra options to compile with a shared object are needed for compiling OpenCV with ffmpeg support)
(I had to run this because running python threw an error about not being able to find libpython2.7.so.1.0)
(should give the correct version of 2.7.8 now)
Now if we want to install anything via pip, we also need to verify that it’s installing for Python2.7 and not Python2.6.
should output something about how it’s configured for 2.6. To fix this:
Honestly I’m not sure if the next steps are entirely necessary. I’m pretty sure you can just wait and include these in your requirements.txt file for when you deploy to Heroku, but I went ahead and installed Numpy now. It’s used in the OpenCV compilation, and using “sudo apt-get install python-numpy” will install for Python 2.6:
There are a few tutorials on the internets about how to install OpenCV with ffmpeg support, but remember to keep in mind that you are using 64 bit Ubuntu so compilation will end up failing if you don’t add a few options when you’re downloading some of the dependencies. The tutorial I used that worked is here: A Comprehensive Guide to Installing and Configuring OpenCV. If you don’t want to click the link though, here’s a list of all the commands you need to run:
With the above setup, you’ll get a compile error from line 50 of cl2cpp.cmake. After some googling around for this problem, I just deleted line 50 and continued the install. It worked fine.
Veryify everything works:
If there were no errors you’re good so far.
Tar the pre-compiled stuff to export it to Heroku
At this point I just used the existing template from the initial buildpack I went off of. Don’t judge. When I untarred the file that the author initially created I found that the contents of the file were basically a few of the contents from /usr/local moved to .heroku. So I simply created a temporary folder and copied the contents of bin, include, lib, man, and share from /usr/local. I also found that I needed to copy /usr/lib into the tar file as well because of some shared objects that OpenCV depended on. I’m not sure if all of these were actually necessary, but I’m not going to overcomplicate something that already works. Don’t judge. So what I did:
Tar the contents of .heroku, transfer it to some external URL, and use that URL in the buildpack. At this point, I had to fiddle with some of the settings in the buildpack to get everything working properly. You can checkout the resulting repository on git for details.