Installation Issues

Although mod_wsgi is not a large package in itself, it depends on both Apache and Python to get it compiled and installed. Because Apache and Python are complicated systems in their own right, various problems can come up during installation of mod_wsgi. These problems can arise for various reasons, including an incomplete or suboptimal Python installation or presence of multiple Python versions.

The purpose of this document is to capture all the known problems that can arise regarding installation, including workarounds if available.

If you are having a problem which doesn’t seem to be covered by this document, also make sure you see Configuration Issues and Application Issues.

Missing Python Header Files

In order to compile mod_wsgi from source code you must have installed the full Python distribution, including header files. On a Linux distribution where binary Python packages are split into a runtime package and a developer package, the developer package is often not installed by default. This means that you will be missing the header files required to compile mod_wsgi from source code. An example of the error messages you will see if the developer package is not installed are:

mod_wsgi.c:113:20: error: Python.h: No such file or directory
mod_wsgi.c:114:21: error: compile.h: No such file or directory
mod_wsgi.c:115:18: error: node.h: No such file or directory
mod_wsgi.c:116:20: error: osdefs.h: No such file or directory
mod_wsgi.c:119:2: error: #error Sorry, mod_wsgi requires at least Python 2.3.0.
mod_wsgi.c:123:2: error: #error Sorry, mod_wsgi requires that Python supporting thread.

To remedy the problem, install the developer package for Python corresponding to the Python runtime package you have installed. What the name of the developer package is can vary from one Linux distribution to another. Normally it has the same name as the Python runtime package with -dev appended to the package name. You will need to lookup up list of available packages in your packaging system to determine actual name of package to install.

Lack Of Python Shared Library

In the optimal case, when mod_wsgi is compiled the resulting Apache module should be less than 250 Kbytes in size. If this is not the case and the module is over 1MB in size, it indicates that the version of Python being used was not originally configured so as to produce a Python shared library and is instead only producing a static library.

Although the existance of only a static library for Python doesn’t normally cause compilation of mod_wsgi to fail, it does mean that when ‘libtool’ is used to generate the mod_wsgi Apache module, that it has to embed the actual static library objects into the Apache module instead of it being used as a shared library.

The consequences of this are that when the mod_wsgi Apache module is loaded by Apache, the operating system dynamic linker has to perform address relocations on the Python library component of the mod_wsgi Apache module. Because these relocations require memory to be modified, the full Python library then becomes private memory to the process and not shared.

On a Linux system this need to perform the address relocations at runtime will immediately cause each Apache child process to bloat out in size by between 1 and 2MB. On a Solaris system, depending on which compiler is being used and which options, the amount of additional memory used can be 5MB or more.

To determine whether the compiled mod_wsgi module is making use of a shared library for Python, many UNIX systems provide the ‘ldd’ program. The output from running this on the ‘mod_wsgi.so’ file would be something like:

$ ldd mod_wsgi.so
 linux-vdso.so.1 =>  (0x00007fffeb3fe000)
 libpython2.5.so.1.0 => /usr/local/lib/libpython2.5.so.1.0 (0x00002adebf94d000)
 libpthread.so.0 => /lib/libpthread.so.0 (0x00002adebfcba000)
 libdl.so.2 => /lib/libdl.so.2 (0x00002adebfed6000)
 libutil.so.1 => /lib/libutil.so.1 (0x00002adec00da000)
 libc.so.6 => /lib/libc.so.6 (0x00002adec02dd000)
 libm.so.6 => /lib/libm.so.6 (0x00002adec0635000)
 /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

Note how there is a dependency listed on the ‘.so’ file for Python. If this is not present then mod_wsgi is using a static Python library.

Although mod_wsgi will still work when compiled against a version of Python which only provides a static library, you are highly encouraged to ensure that your Python installation has been configured and compiled with the --enable-shared option to enable the production and use of a shared library for Python.

If rebuilding Python to generate a shared library, do make sure that the Python shared library, or a symlink to it appears in the Python ‘config’ directory of your Python installation. If the shared library doesn’t appear here next to the static version of the library, ‘libtool’ will not be able to find it and will still use the static version of the library. It is understood that the Python build process may not actually do this, so you may have to do it by hand.

To check, go to the Python ‘config’ directory of your Python installation and do a directory listing:

$ ls -las

   4 drwxr-sr-x  2 root staff    4096 2007-11-29 23:26 .
  20 drwxr-sr-x 21 root staff   20480 2007-11-29 23:26 ..
   4 -rw-r--r--  1 root staff    2078 2007-11-29 23:26 config.c
   4 -rw-r--r--  1 root staff    1446 2007-11-29 23:26 config.c.in
   8 -rwxr-xr-x  1 root staff    7122 2007-11-29 23:26 install-sh
7664 -rw-r--r--  1 root staff 7833936 2007-11-29 23:26 libpython2.5.a
  40 -rw-r--r--  1 root staff   38327 2007-11-29 23:26 Makefile
   8 -rwxr-xr-x  1 root staff    7430 2007-11-29 23:26 makesetup
   8 -rw-r--r--  1 root staff    6456 2007-11-29 23:26 python.o
  20 -rw-r--r--  1 root staff   17862 2007-11-29 23:26 Setup
   4 -rw-r--r--  1 root staff     368 2007-11-29 23:26 Setup.config
   4 -rw-r--r--  1 root staff      41 2007-11-29 23:26 Setup.local

If you only see a ‘.a’ file for Python library, then either Python wasn’t installed with the shared library, or the shared library was placed elsewhere. What appears to normally happen is that the shared library is actually placed in the ‘lib’ directory two levels above the Python ‘config’ directory. In that case you need to create a symlink in the ‘config’ directory to where the shared library is actually installed:

$ ln -s ../../libpython2.5.so .

Apart from the additional memory consumption when using a static library, it is also preferable that a shared library be used where it is possible that you will upgrade your Python installation to a newer patch revision. This is because if you upgrade Python to a newer patch revision but do not recompile mod_wsgi, mod_wsgi will still incorporate the older static Python library and will not pick up any changes from the newer version of Python. This will result in undefined behaviour as the Python library code may not match up with the Python code modules or external modules in the Python installation. If a Python shared library is used, this will not be a problem.

Multiple Python Versions

Where there are multiple versions of Python installed on a system and it is necessary to ensure that a specific version is used, the --with-python option can be supplied to ‘configure’ when installing mod_wsgi:

./configure --with-python=/usr/local/bin/python2.5

This may be necessary where for example the default Python version supplied with the system is an older version of Python. More specifically, it would be required where it isn’t possible to replace the older version of Python outright due to operating system management scripts being dependent on the older version of Python and not working with newer versions of Python.

Where multiple versions of Python are present and are installed under the same directory, this should generally be all that is required. If however the newer version of Python you wish to use is in a different location, for example under ‘/usr/local’, it is possible that when Apache is started that it will not be able find the Python library files for the version of Python you wish to use.

This can occur because the Python library when initialised determines where the Python installation resides by looking through directories specified in the ‘PATH’ environment variable for the ‘python’ executable and using that as base location for calculating installation prefix. Specifically, the directory above the directory containing the ‘python’ executable is taken as being the installation prefix.

When the Python which should be used is installed in a non standard location, then that ‘bin’ directory is unlikely to be in the ‘PATH’ used by Apache when it is started. As such, rather than find ‘/usr/local/bin/python’ it would instead find ‘/usr/bin/python’ and so use ‘/usr’ rather than the directory ‘/usr/local/’ as the installation prefix.

When this occurs, if under ‘/usr’ there was no Python installation of the same version number as Python which should be used, then normally:

'import site' failed; use -v for traceback

would appear in the Apache error log file when Python is first being initialised within Apache. Any attempt to make a request against a WSGI application would also result in errors as no modules at all except for inbuilt modules, would be able to be found when an attempt is made to import them.

Alternatively, if there was a Python installation of the same version, albeit not the desired installation, then there may be no obvious issues on startup, but at run time you may find modules cannot be found when being imported as they are installed into a different location than that which was being used. Even if equivalent module is found, it could fail at run time in subtle ways if the two Python installations are of same version but at the different locations are compiled in different ways, or if it is a third party module and they are different versions and so API is different.

In this situation it will be necessary to explicitly tell mod_wsgi where the Python executable for the version of Python which should be used, is located. This can be done using the WSGIPythonHome directive:

WSGIPythonHome /usr/local

The value given to the WSGIPythonHome directive should be a normalised path corresponding to that defined by the Python {{{sys.prefix}}} variable for the version of Python being used and passed to the --with-python option when configuring mod_wsgi:

>>> import sys
>>> sys.prefix
'/usr/local'

An alternative, although less desirable way of achieving this is to set the ‘PATH’ environment variable in the startup scripts for Apache. For a standard Apache installation using ASF structure, this can be done by editing the ‘envvars’ file in same directory as the Apache executable and adding the alternate bin directory to the head of the ‘PATH’:

PATH=/usr/local/bin:$PATH
export PATH

If there are any concerns over what Python installation directory is being used and you want to verify what it is, then use a small test WSGI script which outputs the values of ‘sys.prefix’ and ‘sys.path’. For example:

import sys

def application(environ, start_response):
    status = '200 OK'
    output = b'Hello World!'

    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)

    print >> sys.stderr, 'sys.prefix = %s' % repr(sys.prefix)
    print >> sys.stderr, 'sys.path = %s' % repr(sys.path)

    return [output]

Using ModPython and ModWsgi

Using mod_python and mod_wsgi together is no longer supported and recent versions of mod_wsgi will cause the startup of Apache to be aborted if both are loaded at the same time.

Python Patch Level Mismatch

If the Python package is upgraded to a newer patch level revision, one will likely see the following warning messages in the Apache error log when Python is being initialised:

mod_wsgi: Compiled for Python/2.4.1.
mod_wsgi: Runtime using Python/2.4.2.

The warning is indicating that a newer version of Python is now being used than what mod_wsgi was originally compiled for.

This would generally not be a problem provided that both versions of Python were originally installed with the --enable-shared option supplied to ‘configure’. If this option is used then the Python library will be linked in dynamically at runtime and so an upgrade to the Python version will be automatically used.

If --enable-shared was however not used and the Python library is therefore embedded into the actual mod_wsgi Apache module, then there is a risk of undefined behaviour. This is because the version of the Python library embedded into the mod_wsgi Apache module will be older than the corresponding Python code modules and extension modules being used from the Python library directory.

Thus, if a shared library is not being used for Python it will be necessary to rebuild mod_wsgi against the newer patch level revision of mod_wsgi and reinstall it.

Mixing 32 Bit And 64 Bit Packages

When attempting to compile mod_wsgi on a Linux system using an X86 64 bit processor, the following error message can arise:

/bin/sh /usr/lib64/apr/build/libtool --silent --mode=link gcc -o \
  mod_wsgi.la -I/usr/local/include/python2.4 -DNDEBUG  -rpath \
  /usr/lib64/httpd/modules -module -avoid-version mod_wsgi.lo \
  -L/usr/local/lib/python2.4/config -lpython2.4 -lpthread -ldl -lutil
/usr/bin/ld: /usr/local/lib/python2.4/config/
libpython2.4.a(abstract.o): relocation R_X86_64_32 against `a local
symbol' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/python2.4/config/libpython2.4.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
apxs:Error: Command failed with rc=65536
.
make: *** [mod_wsgi.la] Error 1

This error is believed to be result of the version of Python being used having been originally compiled for the generic X86 32 bit architecture whereas mod_wsgi is being compiled for X86 64 bit architecture. The actual error arises in this case because ‘libtool’ would appear to be unable to generate a dynamically loadable module for the X86 64 bit architecture from a X86 32 bit static library. Alternatively, the problem is due to ‘libtool’ on this platform not being able to create a loadable module from a X86 64 bit static library in all cases.

If the first issue, the only solution to this problem is to recompile Python for the X86 64 bit architecture. When doing this, it is preferable, and may actually be necessary, to ensure that the --enable-shared option is provided to the ‘configure’ script for Python when it is being compiled and installed.

If rebuilding Python to generate a shared library, do make sure that the Python shared library, or a symlink to it appears in the Python ‘config’ directory of your Python installation. If the shared library doesn’t appear here next to the static version of the library, ‘libtool’ will not be able to find it and will still use the static version of the library. It is understood that the Python build process may not actually do this, so you may have to do it by hand.

If the version of Python being used was compiled for X86 64 bit architecture and a shared library does exist, but not in the ‘config’ directory, then adding the missing symlink may be all that is required.

Unable To Find Python Shared Library

When mod_wsgi is built against a version of Python providing a shared library, the Python shared library must be in a directory which is searched for libraries at runtime by Apache. If this isn’t the case the Python shared library will not be able to be found when loading the mod_wsgi module in to Apache. The error in this situation will be similar to:

error while loading shared libraries: libpython2.4.so.1.0: \
 cannot open shared object file: No such file or directory

A number of alternatives exist for resolving this problem. The preferred solution would be to copy the Python shared library into a directory which is searched for dynamic libraries at run time. Directories which would generally always be searched are ‘/lib’ and ‘/usr/lib’.

For some systems the directory ‘/usr/local/lib’ may also be searched, but this may depend on the directory having been explicitly added to the approrpiate system file listing the directories to be searched. The name and location of this configuration file differs between platforms. On Linux systems it is often called ‘/etc/ld.so.conf’. If changes are made to the file on Linux systems the ‘ldconfig’ command also needs to be run. See the manual page for ‘ldconfig’ for further details.

Rather than changing the system wide list of directories to search for shared libraries, additional search directories can be specified just for Apache. On Linux this would entail setting the ‘LD_LIBRARY_PATH’ environment variable to include the directory where the Python shared library is installed.

The setting and exporting of the environment variable would be placed in the Apache ‘envvars’ file, for a standard Apache installation, located in the same directory as the Apache web server executable. If using a customised Apache installation, such as on Red Hat, the ‘envvars’ file may not exist. In this case you would need to add this into the actual startup script for Apache. For Red Hat this is ‘/etc/sysconfig/httpd’.

A final alternative on some systems is to embed the directory to search for the Python shared library into the mod_wsgi Apache module itself. On Linux systems this can be done by setting the environment variable ‘LD_RUN_PATH’ to the directory containing the Python shared library when initially building the mod_wsgi source code.

GNU C Stack Smashing Extensions

Various Linux distributions are starting to ship with a version of the GNU C compiler which incorporates an extension which implements protection for stack-smashing. In some instances where such a compiler is used to build mod_wsgi, the module is unable to then be loaded by Apache. The specific problem is that the symbol __stack_chk_fail_local is being flagged as undefined:

$ invoke-rc.d apache2 reload
apache2: Syntax error on line 190 of /etc/apache2/apache2.conf: \
 Cannot load /usr/lib/apache2/modules/mod_wsgi.so into server: \
 /usr/lib/apache2/modules/mod_wsgi.so: \
 undefined symbol: __stack_chk_fail_local failed!
invoke-rc.d: initscript apache2, action "reload" failed.

The exact reason for this is not known but it is speculated to be caused when the system libraries or Apache itself has not been compiled with a version of the GNU C compiler incorporating the extension.

To workaround the problem, modify the ‘Makefile’ for mod_wsgi and change the value of ‘CFLAGS’ to:

CFLAGS = -Wc,-fno-stack-protector

Perform a ‘clean’ in the directory and then rebuild and reinstall the mod_wsgi module.

Undefined ‘forkpty’ On Fedora 7

On Fedora 7, the provided binary version of Apache is not linked against the ‘libutil’ system library. This causes problems when Python is initialised and the ‘posix’ module imported for the first time. This is because the ‘posix’ module requires functions from ‘libutil’ but they will not be present. The error encountered would be similar to:

httpd: Syntax error on line 54 of /etc/httpd/conf/httpd.conf: Cannot \
 load /etc/httpd/modules/mod_wsgi.so into server: \
 /etc/httpd/modules/mod_wsgi.so: undefined symbol: forkpty

This problem can be fixed by adding -lutil to the list of libraries to link mod_wsgi against when it is being built. This can be done by adding -lutil to the ‘LDLIBS’ variable in the mod_wsgi ‘Makefile’ after having run ‘configure’.

An alternative method which may work is to edit the ‘envvars’ file, if it exists and is used, located in the same directory as the Apache ‘httpd’ executable, or the Apache startup script, and add:

LD_PRELOAD=/usr/lib/libutil.so
export LD_PRELOAD

Missing Include Files On SUSE

SUSE Linux follows a slightly different convention to other Linux distributions and has split their Apache “dev” packages in a way as to allow packages for different Apache MPMs to be installed at the same time. Although the resultant mod_wsgi module isn’t strictly MPM specific, it does indirectly include the MPM specific header file “mpm.h”. Because the header file is MPM specific, when configuring mod_wsgi, it is necessary to reference the version of “apxs” from the MPM specific “dev” package else the “mpm.h” header file will not be found at compile time. These errors are:

In file included from mod_wsgi.c:4882: /usr/include/apache2/mpm_common.h:46:17: error: mpm.h: No such file or directory
...
mod_wsgi.c: In function 'wsgi_set_accept_mutex':
mod_wsgi.c:5200: error: 'ap_accept_lock_mech' undeclared (first use in this function)
mod_wsgi.c:5200: error: (Each undeclared identifier is reported only once
mod_wsgi.c:5200: error: for each function it appears in.)
apxs:Error: Command failed with rc=65536

To avoid this problem, when configuring mod_wsgi, it is necessary to use the --with-apxs option to designate that either “apxs2-worker” or “apxs2-prefork” should be used. Thus:

./configure --with-apxs=/usr/sbin/apxs2-worker

or:

./configure --with-apxs=/usr/sbin/apxs2-prefork

Although which is used is not important, since mod_wsgi when compiled isn’t specific to either, best to use that which corresponds to the version of Apache being used.

Apache Maintainer Mode

When building mod_wsgi from source code, on UNIX systems there should be minimal if no compiler warnings. If you see a lot of warnings, especially complaints about ap_strstr, then your Apache installation has been configured for maintainer mode:

mod_wsgi.c: In function 'wsgi_process_group':
mod_wsgi.c:722: warning: passing argument 1 of 'ap_strstr' discards
qualifiers from pointer target type
mod_wsgi.c:740: warning: passing argument 1 of 'ap_strstr' discards
qualifiers from pointer target type

Specifically, whoever built the version of Apache being used supplied the option --enable-maintainer-mode when configuring Apache prior to installation. You would be able to tell at the time of compiling mod_wsgi if this has been done as the option -DAP_DEBUG would be supplied to the compiler when mod_wsgi source code is compiled.

These warnings can be ignored, but in general you shouldn’t run Apache in maintainer mode.

A further reason for not running Apache in maintainer mode is that certain situations can cause Apache to fail an internal assertion check when using mod_wsgi. The specific error message is:

[crit] file http_filters.c, line 346, assertion "readbytes > 0" failed
[notice] child pid 18551 exit signal Aborted (6)

This occurs because the Apache code has an overly agressive assertion check, which is arguably incorrect. This particular assertion check will fail when a zero length read is perform on the Apache ‘HTTP_IN’ input filter.

This scenario can arise in mod_wsgi due to a workaround in place to get around a bug in Apache related to generation of ‘100-continue’ response. The Apache bug is described in:

The scenario can also be triggered as a result of a WSGI application performing a zero length read on ‘wsgi.input’.

Changes to mod_wsgi are being investigated to see if zero length reads can be ignored, but due to the workaround for the bug, this would only be able to be done for Apache 2.2.8 or later.

The prefered solution is simply not to use Apache with maintainer mode enabled for systems where you are running real code. Unfortunately, it looks like some Linux distributions, eg. SUSE, accidentally released Apache binary packages with this mode enabled by default. You should update to a Apache binary package that doesn’t have the mode enabled, or compile from source code.