Virtual Environments

This document contains information about how to use Python virtual environments with mod_wsgi. You can use a Python virtual environment created using virtualenv and virtualenvwrapper, or if using Python 3, the pyvenv or python -m venv commands.

The purpose of a Python virtual environments is to allow one to create multiple distinct Python environments for the same version of Python, but with different sets of Python modules and packages installed. It is recommended that you always use Python virtual environments and not install additional Python packages direct into your Python installation.

A Python virtual environment is also required where it is necessary to run multiple WSGI applications which have conflicting requirements as to what version of a Python module or package needs to be installed. They can also be used when distinct mod_wsgi daemon process groups are used to host WSGI applications for different users and each user needs to be able to separately install their own Python modules and packages.

How you configure mod_wsgi or setup your WSGI application script file for a Python virtual environment will depend on your specific requirements. The more common scenarios are explained below.

Location of the Virtual Environment

Whichever method you use to create a Python virtual environment, before you use it with mod_wsgi, you should validate what the location of the Python virtual environment is. If using virtualenvwrapper this may be a non obvious directory hidden away under your home directory.

The way to determine the location of the Python virtual environment is to activate the Python virtual environment from an interactive shell so it is being used, and then run the command:

python -c 'import sys; print(sys.prefix)'

This will output the directory path you will use when setting up mod_wsgi to use the Python virtual environment. For the purposes of the examples below, it is assumed the location of any Python virtual environments are under the /usr/local/venvs directory. A specific Python virtual environment may thus return for sys.prefix:

/usr/local/venvs/example

Note that this should be the root directory of the Python virtual environment, which in turn contains the bin and lib directories for the Python virtual environment. It is a common mistake when setting up a Python virtual environment with mod_wsgi to use the full path to the python executable instead of the root directory. That will not work, so do not use the path for the python executable as the location of the Python virtual environment, it has to be the root directory.

Do be aware that the user that Apache runs your code as will need to be able to access the Python virtual environment. On some Linux distributions, the home directory of a user account is not accessible to other users. Rather than change the permissions on your home directory, it might be better to consider locating your WSGI application code and any Python virtual environment outside of your home directory.

Virtual Environment and Python Version

When using a Python virtual environment with mod_wsgi, it is very important that it has been created using the same Python installation that mod_wsgi was originally compiled for. It is not possible to use a Python virtual environment to force mod_wsgi to use a different Python version, or even a different Python installation.

You cannot for example force mod_wsgi to use a Python virtual environment created using Python 3.5 when mod_wsgi was originally compiled for Python 2.7. This is because the Python library for the Python installation it was originally compiled against is linked directly into the mod_wsgi module. In other words, Python is embedded within mod_wsgi. When mod_wsgi is used it does not run the command line python program to run the interpreter and thus why you can’t force it to use a different Python installation.

The problem in trying to force mod_wsgi to use a different Python installation than what it was compiled for, even where it is the same Python version, is that the Python installation may itself not have been compiled with the same options. This is especially a problem when it comes to issues around how Python stores Unicode characters in memory.

The end result is that if you want to use a different Python installation or version than what mod_wsgi was originally compiled for, you would need to re-install mod_wsgi such that it is compiled for the Python installation or version you do want to use. Do not try and use a Python virtual environment from one Python installation or version with mod_wsgi, when mod_wsgi was compiled for a different one.

Daemon Mode (Single Application)

The preferred way of setting up mod_wsgi is to run each WSGI application in its own daemon process group. This is called daemon mode. A typical configuration for running a WSGI application in daemon mode would be:

WSGIDaemonProcess myapp

WSGIProcessGroup myapp
WSGIApplicationGroup %{GLOBAL}

WSGIScriptAlias / /some/path/project/myapp.wsgi

<Directory /some/path/project>
    Require all granted
</Directory>

The WSGIDaemonProcess directive defines the daemon process group. The WSGIProcessGroup directive indicates that the WSGI application should be run within the defined daemon process group.

As only the single application is being run within the daemon process group, the WSGIApplicationGroup directive is also being used. When this is used with the %{GLOBAL} value, it forces the WSGI application to run in the main Python interpreter context of each process. This is preferred in this scenario as some third party packages for Python which include C extensions will not run in the Python sub interpreter contexts which mod_wsgi would use by default. By using the main Python interpreter context you eliminate the possibility of such third party packages for Python causing problems.

To modify the configuration for this scenario to use a Python virtual environment, all you need to do is add the python-home option to the WSGIDaemonProcess directive resulting in:

WSGIDaemonProcess myapp python-home=/usr/local/venvs/myapp

All the additonal Python packages and modules would then be installed into that Python virtual environment.

Daemon Mode (Multiple Applications)

If instead of running each WSGI application in a separate daemon process group as is the recommended practice, you are running multiple WSGI applications in one daemon process group, a different approach to using Python virtual environments is required.

For this scenario there are various ways the configuration could be set up. If mounting each WSGI application explicitly you might be using:

WSGIDaemonProcess myapps

WSGIProcessGroup myapps

WSGIScriptAlias /myapp3 /some/path/project/myapp3.wsgi
WSGIScriptAlias /myapp2 /some/path/project/myapp2.wsgi

WSGIScriptAlias / /some/path/project/myapp1.wsgi

<Directory /some/path/project>
    Require all granted
</Directory>

If instead the directory containing the WSGI application script files is being mounted, you might be using:

WSGIDaemonProcess myapps

WSGIProcessGroup myapps

WSGIScriptAlias / /some/path/project/

<Directory /some/path/project>
    Require all granted
</Directory>

The use of the WSGIDaemonProcess and WSGIProcessGroup is the same as before, however the WSGIApplicationGroup directive is not being used.

When the WSGIApplicationGroup directive isn’t being used to override which Python interpreter context is being used, each WSGI application will be run in its own Python sub interpreter context of the processes. This is necessary as often WSGI application frameworks (Django being a prime example), do not support running more than one instance of a WSGI application using the framework, in the same Python interpreter context at the same time.

In this scenario of running multiple WSGI applications in the same daemon process group, more than one change is possibly required. The changes required depend on whether or not all WSGI applications should share the same Python virtual environment.

If all of the WSGI applications should share the same Python virtual environment, then the same change as was performed above for the single application case would be made. That is, add the python-home option to the WSGIDaemonProcess directive:

WSGIDaemonProcess myapp python-home=/usr/local/venvs/myapps

All the additonal Python packages and modules that any of the WSGI applications required would then be installed into that Python virtual environment. Because it is a shared environment, they must all use the same version of any specific Python package or module.

If instead of all WSGI applications using the same Python virtual environment each needed their own, then a change will instead need to be made in each of the WSGI script files for the applications.

How this is done will depend on how the Python virtual environment is created.

If the Python virtual environment is created using virtualenv or virtualenvwrapper, the WSGI script for each application should be modified to include code of the following form:

python_home = '/usr/local/envs/myapp1'

activate_this = python_home + '/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

Because each WSGI application is to use a separate Python virtual environment, the value of the python_home variable would be set differently for each WSGI script file, with it referring to the root directory of the respective Python virtual environments.

This code should be placed in the WSGI script file before any other module imports in the WSGI script file, with the exception of from __future__ imports used to enable Python feature flags.

Important to note is that when the Python virtual environment is activated from within the WSGI script, what happens is a bit different to when the python-home option to WSGIDaemonProcess is used.

When activating the Python virtual environment from within the WSGI script file, only the site-packages directory from the Python virtual environment is being used. This directory will be added to the Python module search path, along with any additional directories related to the site-packages directory registered using .pth files present in the site-packages directory. This will be placed at the start of the existing sys.path.

The consequence of this is that the Python virtual environment isn’t completely overriding the original Python installation the Python virtual environment was created from. This means that if the main Python installation had additional Python packages installed they will also potentially be visible to the WSGI application.

That this occurs could cause confusion as you might for example think you had all the packages you require listed in your requirements.txt file for pip, but didn’t and so a package may not have been installed. If that package was installed in the main Python installation, it would be picked up from there, but it might be the wrong version and have dependencies on versions of other packages for which you have different versions installed in your Python virtual environment and which are found instead of those in the main Python installation.

To avoid such problems, when activating the Python virtual environment from within the WSGI script file, it is necessary to still set the python-home option of the WSGIDaemonProcess directive, but set it to an empty Python virtual environment which has had no additional packages installed:

WSGIDaemonProcess myapp python-home=/usr/local/venvs/empty

By doing this, the main Python installation will not be consulted and instead it will fallback to the empty Python virtual environment. This Python virtual environment should remain empty and you should not install additional Python packages or modules into it, or you will cause the same sort of conflicts that can arise with the main Python installation when it was being used.

When needing to activate the Python virtual environment from within the WSGI script file as described, it is preferred that you be using the either virtualenv or virtualenvwrapper to create the Python virtual environment. This is because they both provide the activate_this.py script file which does all the work of setting up sys.path. When you use either pyvenv or python -m venv with Python 3, no such activation script is provided.

So use virtualenv or virtualenvwrapper if you can. If you cannot for some reason and are stuck with pyvenv or python -m venv, you can instead use the following code in the WSGI script file:

python_home = '/usr/local/envs/myapp1'

import sys
import site

# Calculate path to site-packages directory.

python_version = '.'.join(map(str, sys.version_info[:2]))
site_packages = python_home + '/lib/python%s/site-packages' % python_version

# Add the site-packages directory.

site.addsitedir(site_packages)

As before this code should be placed in the WSGI script file before any other module imports in the WSGI script file, with the exception of from __future__ imports used to enable Python feature flags.

When using this method, do be aware that the additions to the Python module search path are made at the end of sys.path. For that reason, you must set the python-home option to WSGIDaemonProcess to the location of an empty Python virtual environment. If you do not do this, any additional Python package installed in the main Python installation will hide those in the Python virtual environment for the application.

There is extra code you could add which would reorder sys.path to make it work in an equivalent way to the activate_this.py script provided when you use virtualenv or virtualenvwrapper but it is messy and more trouble than it is worth:

python_home = '/usr/local/envs/myapp1'

import sys
import site

# Calculate path to site-packages directory.

python_version = '.'.join(map(str, sys.version_info[:2]))
site_packages = python_home + '/lib/python%s/site-packages' % python_version
site.addsitedir(site_packages)

# Remember original sys.path.

prev_sys_path = list(sys.path)

# Add the site-packages directory.

site.addsitedir(site_packages)

# Reorder sys.path so new directories at the front.

new_sys_path = []

for item in list(sys.path):
    if item not in prev_sys_path:
        new_sys_path.append(item)
        sys.path.remove(item)

sys.path[:0] = new_sys_path

It is better to avoid needing to manually activate the Python virtual environment from inside of a WSGI script by using a separate daemon process group per WSGI application. At the minimum, at least avoid pyvenv and python -m venv.

Embedded Mode (Single Application)

The situation for running a single WSGI application in embedded mode is not much different to running a single WSGI application in daemon mode. In the case of embedded mode, there is though no WSGIDaemonProcess directive.

The typical configuration when running a single WSGI application in embedded module might be:

WSGIScriptAlias / /some/path/project/myapp.wsgi

WSGIApplicationGroup %{GLOBAL}

<Directory /some/path/project>
    Require all granted
</Directory>

The WSGIDaemonProcess and WSGIProcessGroup directives are gone, but the WSGIApplicationGroup directive is still used to force the WSGI application to run in the main Python interpreter context of each of the Apache worker processes. This is to avoid those issues with some third party packages for Python with C extensions as mentioned before.

In this scenario, to set the location of the Python virtual environment to be used, the WSGIPythonHome directive is used:

WSGIPythonHome /usr/local/envs/myapp

Note that if the WSGI application is being setup within the context of an Apache VirtualHost, the WSGIPythonHome cannot be placed inside of the VirtualHost. Instead it must be placed outside of all VirtualHost definitions. This is because it applies to the whole Apache instance and not just the single VirtualHost.

Embedded Mode (Multiple Applications)

Running multiple applications in embedded mode is also similar to when running multiple WSGI applications in one daemon process group. You still need to ensure each WSGI application runs in its own Python sub interpreter context to avoid potential issues with Python web frameworks that don’t allow more than one WSGI application to be using it at the same time in a Python interpreter context.

If mounting each WSGI application explicitly you might be using:

WSGIScriptAlias /myapp3 /some/path/project/myapp3.wsgi
WSGIScriptAlias /myapp2 /some/path/project/myapp2.wsgi

WSGIScriptAlias / /some/path/project/myapp1.wsgi

<Directory /some/path/project>
    Require all granted
</Directory>

If instead the directory containing the WSGI application script files is being mounted, you might be using:

WSGIScriptAlias / /some/path/project/

<Directory /some/path/project>
    Require all granted
</Directory>

In this scenario, to set the location of the Python virtual environment to be used by all WSGI application, the WSGIPythonHome directive is used:

WSGIPythonHome /usr/local/envs/myapps

If the WSGI application is being setup within the context of an Apache VirtualHost, the WSGIPythonHome cannot be placed inside of the VirtualHost. Instead it must be placed outside of all VirtualHost definitions. This is because it applies to the whole Apache instance and not just the single VirtualHost.

If each WSGI application needs its own Python virtual environment, then activation of the Python virtual environment needs to be performed in the WSGI script itself as explained previously for the case of daemon mode being used. The WSGIPythonHome directive should be used to refer to an empty Python virtual environment if needed to ensure that any additional Python packages in the main Python installation don’t interfere with what packages are installed in the Python virtual environment for each WSGI application.

Adding Additional Module Directories

The python-home option to WSGIDaemonProcess and the WSGIPythonHome directive are the preferred way of specifying the location of the Python virtual environment to be used. If necessary, activation of the Python virtual environment can also be performed from the WSGI script file itself.

If you need to add additional directories to search for Python packages or modules this can also be done. You may want to do this where you need to specify where the actual WSGI application is located, where a WSGI script file needs to import application specific modules.

If you are using daemon mode and want to add additional directories to the Python module search path, you can use the python-path option to WSGIDaemonProcess:

WSGIDaemonProcess myapp python-path=/some/path/project

This option would be in addition to the python-home option used to specify where the Python virtual environment is located.

If you are using embedded mode, you can use the WSGIPythonPath directive:

WSGIPythonPath /some/path/project

This directive is in addition to the WSGIPythonHome directive used to specify where the Python virtual environment is located.

In either case, if you need to specify more than one directory, they can be separated using a ‘:’ character.

If you are having to activate the Python virtual enviromment from within a WSGI script and need to add additional directories to the Python module search path, you should modify sys.path directly from the WSGI script file.

Note that prior practice was that these ways of setting the Python module search path were used to specify the location of the Python virtual environment. Specifically, they were used to add the site-packages directory of the Python virtual environment. You should not do that.

The better way to specify the location of the Python virtual environment is using the python-home option of the WSGIDaemonProcess directive for daemon mode, or the WSGIPythonHome directive for embedded mode. These ways of specifying the Python virtual environment have been available since mod_wsgi 3.0 and Linux distributions have not shipped such an old version of mod_wsgi for quite some time. If you are using the older way, please update your configurations.