Why Python Virtual Environments Aren’t

Python’s virtual environments are not real virtual environments.  They copy the binary executables but not the python environment.  Here’s what I mean:

➤ virtualenv -p /usr/bin/python3.3 /tmp/foobar
Running virtualenv with interpreter /usr/bin/python3.3
Using base prefix '/usr'
New python executable in /tmp/foobar/bin/python3.3
Also creating executable in /tmp/foobar/bin/python
Installing setuptools, pip...done.

➤ cd /tmp/foobar

➤ ls -l
total 12
drwxrwxr-x 2 wrighroy wrighroy 4096 Nov  7 11:05 bin
drwxrwxr-x 2 wrighroy wrighroy 4096 Nov  7 11:05 include
drwxrwxr-x 3 wrighroy wrighroy 4096 Nov  7 11:05 lib

➤ ls -l bin
total 3512
-rw-rw-r-- 1 wrighroy wrighroy    2192 Nov  7 11:05 activate
-rw-rw-r-- 1 wrighroy wrighroy    1248 Nov  7 11:05 activate.csh
-rw-rw-r-- 1 wrighroy wrighroy    2461 Nov  7 11:05 activate.fish
-rw-rw-r-- 1 wrighroy wrighroy    1129 Nov  7 11:05 activate_this.py
-rwxrwxr-x 1 wrighroy wrighroy     242 Nov  7 11:05 easy_install
-rwxrwxr-x 1 wrighroy wrighroy     242 Nov  7 11:05 easy_install-3.3
-rwxrwxr-x 1 wrighroy wrighroy     214 Nov  7 11:05 pip
-rwxrwxr-x 1 wrighroy wrighroy     214 Nov  7 11:05 pip3
-rwxrwxr-x 1 wrighroy wrighroy     214 Nov  7 11:05 pip3.3
lrwxrwxrwx 1 wrighroy wrighroy       9 Nov  7 11:05 python -> python3.3
lrwxrwxrwx 1 wrighroy wrighroy       9 Nov  7 11:05 python3 -> python3.3
-rwxrwxr-x 1 wrighroy wrighroy 3558736 Nov  7 11:05 python3.3

➤ ls -l include
total 0
lrwxrwxrwx 1 wrighroy wrighroy 23 Nov  7 11:05 python3.3m -> /usr/include/python3.3m

➤ ls -l lib
total 4
drwxrwxr-x 4 wrighroy wrighroy 4096 Nov  7 11:05 python3.3

➤ ls -l lib/python3.3/
total 40
lrwxrwxrwx 1 wrighroy wrighroy    25 Nov  7 11:05 abc.py -> /usr/lib/python3.3/abc.py
lrwxrwxrwx 1 wrighroy wrighroy    28 Nov  7 11:05 base64.py -> /usr/lib/python3.3/base64.py
lrwxrwxrwx 1 wrighroy wrighroy    28 Nov  7 11:05 bisect.py -> /usr/lib/python3.3/bisect.py

Virtualenv decouples the python, pip, and easy_install executables by copying the python executable to the virtual environments’ bin directory then writes wrappers for pip and easy_install that use the copied python. Cool.

The include sub-directory contains a symbolic link to /usr/include/python3.3m which is owned by:

➤ dpkg -S /usr/include/python3.3m
python3.3-minimal, python3.3-dev: /usr/include/python3.3m

and lib/python3.3 is owned by:

➤ dpkg -S /usr/bin/python3.3
python3.3-minimal: /usr/bin/python3.3

Herein lies the gotcha. Notice the versioning is only major.minor, but python releases using three part versioning: major.minor.patch. So if you upgrade, say from python 3.3.5 to python 3.3.6 (with for example, an apt-get dist-upgrade), your virtualenv is now using binaries from python 3.3.5 with the support files from python 3.3.6. Needless to say dragons galore!

The work-around is to upgrade the python in your virtualenv by using the non-intuitive pattern of creating a new virtualenv over the existing virtualenv:

➤ virtualenv -p /usr/bin/python3.3 /tmp/foobar

Which just happens to be the same command we created the initial environment with.

Unfortunately this does not guarantee that you have slayed all the dragons, as any site-packages you have installed may need to be reinstalled. YMMV

So when your virtualenv and/or python app(s) break after a package upgrade (ex: apt-get dist-upgrade), it may be time to cleanup the pythonidae scat!


Using Bundler with Subversion

Yes, git is the coolest version control system (VCS), but sometimes you get stuck having to use another VCS.  Been there, done that, here’s a quick how to on using bundler with subversion.

This example uses Bundler version 1.1.5 and svn version 1.6.18.

First we create a gem project using bundler and rvm.

➤ bundle gem svn-bundler-demo
create  svn-bundler-demo/Gemfile
create  svn-bundler-demo/Rakefile
create  svn-bundler-demo/LICENSE
create  svn-bundler-demo/README.md
create  svn-bundler-demo/.gitignore
create  svn-bundler-demo/svn-bundler-demo.gemspec
create  svn-bundler-demo/lib/svn-bundler-demo.rb
create  svn-bundler-demo/lib/svn-bundler-demo/version.rb
Initializating git repo in /Users/roy/Projects/svnroot/svn-bundler-demo
➤ cd svn-bundler-demo
➤ rvm use --rvmrc --create 1.9.3@svn-bundler-demo
Using /Users/roy/.rvm/gems/ruby-1.9.3-p286 with gemset svn-bundler-demo
➤ git ls-files

Next we convert the project to use subversion instead of git:

➤ rm -rf .git
➤ cd ..
➤ svn add svn-bundler-demo
A         svn-bundler-demo
A         svn-bundler-demo/.gitignore
A         svn-bundler-demo/.rvmrc
A         svn-bundler-demo/Gemfile
A         svn-bundler-demo/lib
A         svn-bundler-demo/lib/svn-bundler-demo
A         svn-bundler-demo/lib/svn-bundler-demo/version.rb
A         svn-bundler-demo/lib/svn-bundler-demo.rb
A         svn-bundler-demo/LICENSE
A         svn-bundler-demo/Rakefile
A         svn-bundler-demo/README.md
A         svn-bundler-demo/svn-bundler-demo.gemspec
➤ svn commit -m 'initial gem'
Adding         svn-bundler-demo
Adding         svn-bundler-demo/.gitignore
Adding         svn-bundler-demo/.rvmrc
Adding         svn-bundler-demo/Gemfile
Adding         svn-bundler-demo/LICENSE
Adding         svn-bundler-demo/README.md
Adding         svn-bundler-demo/Rakefile
Adding         svn-bundler-demo/lib
Adding         svn-bundler-demo/lib/svn-bundler-demo
Adding         svn-bundler-demo/lib/svn-bundler-demo/version.rb
Adding         svn-bundler-demo/lib/svn-bundler-demo.rb
Adding         svn-bundler-demo/svn-bundler-demo.gemspec
Transmitting file data .........
Committed revision 6.
➤ svn update
At revision 6.
➤ cd svn-bundler-demo
= NOTICE                                                                           =
= RVM has encountered a new or modified .rvmrc file in the current directory       =
= This is a shell script and therefore may contain any shell commands.             =
=                                                                                  =
= Examine the contents of this file carefully to be sure the contents are          =
= safe before trusting it! ( Choose v[iew] below to view the contents )            =
Do you wish to trust this .rvmrc file? (/Users/roy/Projects/svnroot/svn-bundler-demo/.rvmrc)
y[es], n[o], v[iew], c[ancel]> y
Using: /Users/roy/.rvm/gems/ruby-1.9.3-p286@svn-bundler-demo
➤ svn ls -R

Notice the differences between the “git ls-files” and “svn ls -R”.  The git ls-files shows just the files while the svn ls shows directories and files.

And now the trick.  Edit the gemspec file (svn-bundler-demo.gemspec) and locate the following line:

gem.files         = `git ls-files`.split($\)

replace with:

gem.files         = `svn ls -R`.split($\).reject{|fn| fn =~ /\/$/}

Note that bundler requires a .gitignore file so you will need to keep it.

Have fun!

A Manager for DRb

The other day I wanted to parallelize one of my applications so dusted off DRb (included in ruby’s standard library). While DRb is pretty easy to use, the pain has always been setting up the servers and keeping their code current. This time I wanted the application to handle everything.

So here are my basic requirements:

  • Support using multiple DRb objects
  • Support local and remote hosting of the DRb server(s)
  • Application to install current code on host machine(s)
  • No application code to have to be manually installed on host machine(s)
  • Minimal software requirements on host machine(s)
  • Application to control the DRb server(s) (start/stop)
  • Linux and Mac support (don’t care about windows)

The result is Drbman.
Continue reading

Code generators and git-rebase

The classic example of using git-rebase is when a developer is working on their own branch while commits are made to the master branch, then the developer wants to sync up with the master branch, maybe as a precursor to merging the changes back into the master branch. Git’s rebase handles this elegantly by pushing the changes on the developers branch, moving the point where the developers branch is forked from the master branch, then apply the saved changes to the new developers branch.
Continue reading

Specifications to Generated Merb Project


While in the early requirements phase of a new open source project, I was wanting to encourage discussion on the database schema and web API. So I wrote some rspecs to capture the models, their attributes, and their relationships. As we played around with the schema, it was difficult to manually keep the specs accurate, particularly with the relationships. So I wrote a script to generate a merb project from the rspecs to catch relationship problems. This was good and also let me generate model and controller diagrams using railroad_xing.
Continue reading