DISCLAIMER: This post is a summary of my short experience with the tools mentioned. There is no flash news or revolutionary recipes. If you are familiar with them, you will probably find it boring. If you are not, this post might make your introduction less painful.
I stumbled upon this post following my frustration with setuptools after virtualenvising telvee repository. I have played with buildout to create repeatable deployments before. Don’t get me wrong, buildout is superb. But I have a few minor issues with it:
- Buildout can create an interpreter but the isolation from the host system is not a good as virtualenv. (Please correct me if I’m wrong)
- Buildout is a dependency1 whereas virtualenv is a system-wide tool.
- It is, naturally, more work to set it up than a couple of ad-hoc bash scripts.
None of these issues make buildout less awesome, I am just not comfortable enough with it. I am considering other alternatives and their strengths and weaknesses.
Virtualenv
I am quite comfortable with virtualenv. It creates an isolated Python environment for you. For instance the following command will create an environment inside test directory that doesn’t have access to packages installed system-wide:
~/$ virtualenv --no-site-packages test
That is of course when you activate the virtual environment with:
~/$ cd test
~/test/$ source bin/activate
(test)~/test/$
Note the (test) prefix to your prompt. Any packages you install within this environment will be installed only for itself and won’t be available system-wide. If you have omitted --no-site-packages argument you could have access to globally installed packages too. When you are done with this virtual environment you can issue deactivate command to return to your normal shell. That’s basically what virtualenv does.
You can script virtualenv to a certain extend:
#!/usr/bin/env bash
virtualenv $1
cd $1
source bin/activate
echo $PATH
echo $PYTHONPATH
Or you can create bootstrap scripts with virtualenv. Virtualenv doesn’t provide a clean and powerful enough API here, just a callback and two methods to modify commandline arguments.
Virtualenv is great at what it does. But I think setuptools as a package installer (and it’s just an installer, not a manager) cripples virtualenv. Good news is newer versions have a commandline argument --distribute that appereantly substitutes distribute for setuptools.
Pip
My experiments with pip went just fine. Except I bumped into this problem. Globally installed pip was seeing system-wide packages when used on a virtual environment created with --no-site-packages. Again, good news is it is fixed in the trunk.
I will play with pip more once figure out how to best integrate the these applications.
Fabric
I see fabric as the glue that binds everything together. I have known about it long before, but I never had the chance to experiment. I have read most of the documentation2 and played with it a little. Basically the following script is an attempt to create a virtual environment and install pip and django on it, much like the shell script above:
from __future__ import with_statement
import os
from fabric.api import *
from fabric import context_managers
def _get_virtualenv_location():
location = prompt('New location: ', default='../test')
env.envdir, env.envname = os.path.split(os.path.abspath(location))
env.envpath = os.path.join(env.envdir, env.envname)
print('using "%(envname)s" at "%(envdir)s" as virtualenv' % env)
def _virtualenv(command):
with context_managers.cd(env.envpath):
result = local('. bin/activate && ' + command)
return result
def clone():
_get_virtualenv_location()
with context_managers.cd(env.envdir):
local('virtualenv --no-site-packages --clear %(envname)s' % env)
print _virtualenv('echo $PATH')
_virtualenv('easy_install pip')
print _virtualenv('pip install django==1.1.1')
It took me a while to figure out how I can issue commands within a virtual environment. Since fabric commands don’t share state sourcing activate has no effect on subsequent commands. This SO entry helped me to write _virtualenv() function. It is kind of ugly making all functions but fab commands private. I think if fab used __all__ or something similar it would be more explicit. Also a contrib module for virtualenv would be nice3.
Fabric has cool features such as failure handling and code/config editing. It is a great tool to create repeatable deployments. It feels great to be coding in python (well, to some extend). Perhaps the resulting code is a little too complex for local operations. I wish I could write a fab command that handles both local and remote deployments. I think it is not unusual to deploy on the same machine in another location. But having to SSH into localhost is weird, don’t you think?
I am sure there are better ways to accomplish the goal of the script above. Maybe there is an entirely different way to integrage fabric, virtualenv and pip. So, comments and suggestions are welcome as usual.
Maybe I’ll revisit buildout again as well.
1: You need to bootstrap (install) buildout with each development/deployment site.
2: About one third of it I think.
3: I would happily attempt one, once I learn fabric a little better.
Related posts:
- How to Install MySQL with Fabric
- Sad State of Web Development Industry in Türkiye
- django-formfieldset
- A Civilized Way Display Lots Of Data
- Web Site Performance Optimizations
Tags: deployment, fabric, pip, python, virtualenv
Rather than using a context manager to begin every command with sourcing activate, just use binaries in the venv/bin directory. Activating does nothing but set your $PATH to use those binaries, it’s not necessary.
I’m fairly certain you can set up buildout as a system-wide tool, too, such that when you bootstrap your project, it will use the system-wide install. Of course, when you deploy, you don’t have to use buildout, just like with virtualenv you don’t. In this aspect, I don’t see much difference between the two.
One thing that I like better about virtualenv is that you can use the interpreters it creates in, e.g., PyDev, and your project’s dependencies can automatically be loaded for auto-completion, etc. With buildout, the “interpreter” is just a shell script that “calls” the actual interpreter after modifying sys.path, and I haven’t been able to get this to automagically work with PyDev–I have to manually add a project’s dependencies in PyDev’s config.
Yep, you can install zc.buildout system-wide and then run ‘buildout’ in your source trees, skipping the bootstrap step.
Buildout is nice in that it is declarative, while with virtualenv is more REPL-like. As a programmer I like my sources to live in text files. I’m usually afraid that I can forget what packages I’ve easy_installed (or pip installed) and why. When I’m using virtualenv, I usually have a develop.sh that rm -rf’s my local virtualenv, recreates it, and installs the required dependencies/dev tools into it. Buildout is typically faster than this kind of full-reinstall, since it remembers what it had and compares that with the current version of buildout.cfg before applying changes. On the other hand easy_installing a single ad-hoc development tool is faster when you’ve got a virtualenv, since adding a single package to a buildout.cfg and running bin/buildout can easily cause it to burn CPU for 10-20 seconds before it decides what it needs to change. Also, writing a buildout.cfg is a bit of a black art where cargo-culting works best — mostly because anything interesting is done by plugins, with varying quality of docs, and the added difficulty of finding a plugin to do what you need.
1. “substitute X with Y” is to replace Y with X
2. I always laugh seeing non-hyperlinked footnotes on the web. What’s the point?
Thanks
Friend, whoever you are. I didn’t read your comment as anything but constructive criticism. I have fixed the substitute part. Linked footnotes are a little more difficult, markdown doesn’t even have footnotes.I hope you use your real name next time, so I can thank you properly.
@Marius: you can use pip freeze to capture your installed packages. I also tend to keep a “requirements.pip” file in my projects and whenever I install a new package I add it to this file. You can even pin packages/modules to a specific commit/revision if you’re installing from code repositories. Then cloning the env. elsewhere is as easy as “pip install -r requirements.pip”
Cheers -Brian