Python Virtual Environments

Overview

Many of us work on multiple Python projects at the same time. Multiple projects may depend on different versions of the same library. This is a problem. Even if you work with a single project and you deploy it to production, you may run into trouble, because the system's Python on your production server might change due to OS upgrade or security patch, and your application might fail as a result. In general, you want full control over the Python environment of your projects. Enter virtual environments...

The basic idea of a virtual environment is to have a Python interpreter and its site-packages separate from the system one. Also, you can have many of them. That solves both problems. You can assign a separate virtual environment with its own dependencies for each project and forget about conflicts with other projects and the system's Python.

In this tutorial, you'll learn the concepts behind virtual environments and how to create and use them, and you'll discover various alternatives for special situations.

Virtualenv

The virtualenv package supports this concept. You can install virtualenv using pip install virtualenv.

Once virtualenv is installed, you can start creating virtual environments. Let's create two environments called "venv_1" and "venv_2".

1

~ > virtualenv ~/venv_1

2

3

Using real prefix '/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7'

4

5

New python executable in /Users/gigi/venv_1/bin/python2.7

6

7

Also creating executable in /Users/gigi/venv_1/bin/python

8

9

Installing setuptools, pip, wheel...done.

10

11

12

13

~ > virtualenv ~/venv_2

14

15

Using real prefix '/usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7'

16

17

New python executable in /Users/gigi/venv_2/bin/python2.7

18

19

Also creating executable in /Users/gigi/venv_2/bin/python

20

21

Installing setuptools, pip, wheel...done.

Let's see what happened.

1

~ > ls -la ~/venv_1

2

3

total 16

4

5

drwxr-xr-x    7 gigi  staff   238 Mar 29 23:12 .

6

7

drwxr-xr-x+ 254 gigi  staff  8636 Mar 29 23:12 ..

8

9

lrwxr-xr-x    1 gigi  staff    79 Mar 29 23:12 .Python -> /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/Python

10

11

drwxr-xr-x   16 gigi  staff   544 Mar 29 23:12 bin

12

13

drwxr-xr-x    3 gigi  staff   102 Mar 29 23:12 include

14

15

drwxr-xr-x    3 gigi  staff   102 Mar 29 23:12 lib

16

17

-rw-r--r--    1 gigi  staff    60 Mar 29 23:12 pip-selfcheck.json

Inside the "bin" sub-directory, you'll find a few executables and symlinks. Those include the Python interpreter itself, pip, easy_install, and most importantly a few activate scripts.

1

~ > ls -la ~/venv_1/bin

2

3

total 136

4

5

drwxr-xr-x  16 gigi  staff    544 Mar 29 23:12 .

6

7

drwxr-xr-x   7 gigi  staff    238 Mar 29 23:12 ..

8

9

-rw-r--r--   1 gigi  staff   2077 Mar 29 23:12 activate

10

11

-rw-r--r--   1 gigi  staff   1019 Mar 29 23:12 activate.csh

12

13

-rw-r--r--   1 gigi  staff   2217 Mar 29 23:12 activate.fish

14

15

-rw-r--r--   1 gigi  staff   1137 Mar 29 23:12 activate_this.py

16

17

-rwxr-xr-x   1 gigi  staff    249 Mar 29 23:12 easy_install

18

19

-rwxr-xr-x   1 gigi  staff    249 Mar 29 23:12 easy_install-2.7

20

21

-rwxr-xr-x   1 gigi  staff    221 Mar 29 23:12 pip

22

23

-rwxr-xr-x   1 gigi  staff    221 Mar 29 23:12 pip2

24

25

-rwxr-xr-x   1 gigi  staff    221 Mar 29 23:12 pip2.7

26

27

lrwxr-xr-x   1 gigi  staff      9 Mar 29 23:12 python -> python2.7

28

29

-rwxr-xr-x   1 gigi  staff   2336 Mar 29 23:12 python-config

30

31

lrwxr-xr-x   1 gigi  staff      9 Mar 29 23:12 python2 -> python2.7

32

33

-rwxr-xr-x   1 gigi  staff  12744 Mar 29 23:12 python2.7

34

35

-rwxr-xr-x   1 gigi  staff    228 Mar 29 23:12 wheel

The activate script is the key. In order to activate a specific virtual environment, you source the activate script, as in: source ~/venv_1/bin_activate. The effect is setting a bunch of environment variables and changing the prompt to the name of the activated virtual environment. It also adds a deactivate() shell function that will reset everything. Once a virtual environment is activated, typing python will launch its Python with its dependencies.

1

~ > source ~/venv_1/bin/activate

2

3

(venv_1) ~ > which python

4

5

/Users/gigi/venv_1/bin/python

6

7

(venv_1) ~ >

Let's deactivate:

1

(venv_1) ~ > deactivate

2

3

~ > which python

4

5

/usr/local/bin/python

If you have multiple Python interpreters installed on your systems, you can specify which one to use for your virtual environment using the -p option. Here is a Python 3 virtual environment:

1

~ > virtualenv ~/venv_3 -p /usr/local/bin/python3

2

3

Running virtualenv with interpreter /usr/local/bin/python3

4

5

Using base prefix '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5'

6

7

New python executable in /Users/gigi/venv_3/bin/python3.5

8

9

Also creating executable in /Users/gigi/venv_3/bin/python

10

11

Installing setuptools, pip...done.

12

13

14

15

~ > source ~/venv_3/bin/activate

16

17

(venv_3)~ > python

18

19

Python 3.5.1 (default, Jan  9 2016, 19:28:52)

20

21

[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin

22

23

Type "help", "copyright", "credits" or "license" for more information.

24

25

>>>

Virtualenv works even on pypy.

1

~ > virtualenv ~/venv_pypy -p `which pypy`

2

3

Running virtualenv with interpreter /usr/local/bin/pypy

4

5

New pypy executable in /Users/gigi/venv_pypy/bin/pypy

6

7

Installing setuptools, pip...done.

8

9

~ > source ~/venv_pypy/bin/activate

10

11

(venv_pypy)~ > python

12

13

Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Feb 11 2016, 20:39:39)

14

15

[PyPy 4.0.1 with GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin

16

17

Type "help", "copyright", "credits" or "license" for more information.

18

19

>>>>

You can switch directly from one environment to the other without deactivating first:

1

(venv_3)~ > source ~/venv_2/bin/activate

2

3

(venv_2) ~ > which python

4

5

/Users/gigi/venv_2/bin/python

OK. Let's see how to use two different versions of the same package in two different virtual environments. This is as simple as activating each environment and installing the desired version. The environments are totally independent, so the fact that there is a different version in another environment is a non-issue.

Let's install the sh package version 1.0.0 to "venv_1".

1

(venv_1) ~ > pip install sh==1.0

2

3

Collecting sh==1.0.0

4

5

  Downloading sh-1.0.tar.gz

6

7

Building wheels for collected packages: sh

8

9

  Running setup.py bdist_wheel for sh ... done

10

11

  Stored in directory: /Users/gigi/Library/Caches/pip/wheels/f9/fb/a1/383f6dc2834b319a788a006d3ab7cc014ee852485f00b9e8c3

12

13

Successfully built sh

14

15

Installing collected packages: sh

16

17

Successfully installed sh-1.0

18

19

20

21

(venv_1) ~ > pip freeze | grep sh

22

23

sh==1.0

Let's switch to "venv_2" and install version 1.11.

1

(venv_1) ~ > source ~/venv_2/bin/activate

2

3

(venv_2) ~ > pip install sh==1.11

4

5

Collecting sh==1.11

6

7

  Downloading sh-1.11.tar.gz

8

9

Building wheels for collected packages: sh

10

11

  Running setup.py bdist_wheel for sh ... done

12

13

  Stored in directory: /Users/gigi/Library/Caches/pip/wheels/ba/4f/a5/ec77d662c3bf38f564b5ab16f1f3dbb9575922826fe810961c

14

15

Successfully built sh

16

17

Installing collected packages: sh

18

19

Successfully installed sh-1.11

20

21

(venv_2) ~ > pip freeze | grep sh

22

23

sh==1.11

Now, let's switch back to "venv_1" and verify that its version of the sh package is still 1.0.

1

(venv_2) ~ > source ~/venv_1/bin/activate

2

3

(venv_1) ~ > pip freeze | grep sh

4

5

sh==1.0

6

7

(venv_1) ~ >

Virtualenvwrapper

All that activating, deactivating and switching can get old after a while. If you manage a lot of virtual environments, it can get out of control. That's where virtualenvwrapper comes in. Virtualenvwrapper lets you list, create, delete and copy virtual environments. It also lets you switch environments easily. 

Here are all the commands:

1

~ > virtualenvwrapper

2

3

4

5

virtualenvwrapper is a set of extensions to Ian Bicking's virtualenv

6

7

tool.  The extensions include wrappers for creating and deleting

8

9

virtual environments and otherwise managing your development workflow,

10

11

making it easier to work on more than one project at a time without

12

13

introducing conflicts in their dependencies.

14

15

16

17

For more information please refer to the documentation:

18

19

20

21

    http://virtualenvwrapper.readthedocs.org/en/latest/command_ref.html

22

23

24

25

Commands available:

26

27

28

29

  add2virtualenv: add directory to the import path

30

31

32

33

  allvirtualenv: run a command in all virtualenvs

34

35

36

37

  cdproject: change directory to the active project

38

39

40

41

  cdsitepackages: change to the site-packages directory

42

43

44

45

  cdvirtualenv: change to the $VIRTUAL_ENV directory

46

47

48

49

  cpvirtualenv: duplicate the named virtualenv to make a new one

50

51

52

53

  lssitepackages: list contents of the site-packages directory

54

55

56

57

  lsvirtualenv: list virtualenvs

58

59

60

61

  mkproject: create a new project directory and its associated virtualenv

62

63

64

65

  mktmpenv: create a temporary virtualenv

66

67

68

69

  mkvirtualenv: Create a new virtualenv in $WORKON_HOME

70

71

72

73

  rmvirtualenv: Remove a virtualenv

74

75

76

77

  setvirtualenvproject: associate a project directory with a virtualenv

78

79

80

81

  showvirtualenv: show details of a single virtualenv

82

83

84

85

  toggleglobalsitepackages: turn access to global site-packages on/off

86

87

88

89

  virtualenvwrapper: show this help message

90

91

92

93

  wipeenv: remove all packages installed in the current virtualenv

94

95

96

97

  workon: list or change working virtualenvs

I use pretty much two commands: mkvirtualenv and workon. All the virtual environments are created under ~/.virtualenvironments.

Here is how to create a new virtual environment:

1

~ > mkvirtualenv venv

2

3

New python executable in venv/bin/python2.7

4

5

Also creating executable in venv/bin/python

6

7

Installing setuptools, pip...done.

8

9

10

11

(venv)~ >

This is similar to virtualenv, but you don't specify a directory, just a name. Your new environment is here:

1

(venv)~ > tree -L 2 ~/.virtualenvs/venv/

2

3

/Users/gigi/.virtualenvs/venv/

4

5

├── bin

6

7

│   ├── activate

8

9

│   ├── activate.csh

10

11

│   ├── activate.fish

12

13

│   ├── activate_this.py

14

15

│   ├── easy_install

16

17

│   ├── easy_install-2.7

18

19

│   ├── get_env_details

20

21

│   ├── pip

22

23

│   ├── pip2

24

25

│   ├── pip2.7

26

27

│   ├── postactivate

28

29

│   ├── postdeactivate

30

31

│   ├── preactivate

32

33

│   ├── predeactivate

34

35

│   ├── python -> python2.7

36

37

│   ├── python2 -> python2.7

38

39

│   └── python2.7

40

41

├── include

42

43

│   └── python2.7 -> /usr/local/Cellar/python/2.7.10/Frameworks/Python.framework/Versions/2.7/include/python2.7

44

45

└── lib

46

47

    └── python2.7

To switch between environments, you use the workon command, which without arguments just lists all the virtual environments. I have quite a few:

1

(venv)~ > workon

2

3

acme_server

4

5

conman

6

7

curr-gen

8

9

nupic

10

11

over-achiever

12

13

pandas

14

15

prime_hunter

16

17

pypy

18

19

quote-service

20

21

venv

22

23

work

24

25

26

27

(venv)~ > workon conman

28

29

(conman) ~ >

Virtualenv-Burrito

Virtualenv-Burrito is a project to install both virtualenv and virtualenvwrapper in one command.

Python 3 Venv

The venv module was added to Python 3.3 and provides built-in virtual environment creation and management just like virtualenv. The command to create virtual environments is pyenv. Otherwise it is pretty similar to virtualenv.

Conda

Virtual environments are very useful for isolating dependencies between different projects. But that doesn't extend to native libraries. Many C extensions depend on particular versions of native libraries, and those can't be virtual environment specific. 

Conda can address this problem. It is a package management system that handles all dependencies, not just Python dependencies. It can even be used for packaging non-Python software.

Alternatives to Virtual Environments

Do you have to use virtual environments? Not really. If for some reason you're not fond of the magic of virtual environments, there are other options.

Manually Vendorize

The functionality of a virtual environment is pretty simple. If you need total control, you can just do it yourself and copy exactly the subset of tools and packages you want into a target directory structure, set up some environment variables, and you're good to go.

VM or Dockerized System Python

If you bake your applications into a docker container or a cloud image then there will be just one project, and you may not need a virtual environment in the middle. You can just build on top of the system Python, being sure it will not be changed.

Tox

If all you care about is testing your code under different interpreters and environments then Tox can do all the heavy lifting for you. It will still use virtual environments under the covers, but you don't have to deal with them.

Best Practices

There are some packaging and virtual environment best practices that have emerged over time for robust Python systems.

Pin Versions in Your Requirements Files

Pinning means specifying precisely the versions of your dependencies. If a new version comes out and you install your application on a new server, you'll still use the version you tested against and that works, and not the latest and greatest. There is a downside here—you'll have to explicitly upgrade versions if you want to keep up with the progress made by your dependencies—but it is usually worth it.

Never Use the System Python

Relying on the system version is bad practice because there are other tools that rely on it, and if you start upgrading system packages, you may break them. You may also be affected by security updates that modify system packages, or in general if you want to upgrade your OS you may find that the system Python is now different.

Use a Private Package Index When Baking Images

PyPI may be down. If you need to bake a new image and can't access PyPI, you're in trouble. Devpi is a good option to avoid frustration.

Conclusion

There are many options for managing multiple Python projects on the same machine without conflicts. Figure out which option is best for you and use it. It is fast and easy to create virtual environments. Don't hesitate to take advantage of this useful tool. If you have special requirements, there are many solutions too.