File Wrapper Extension

The WSGI specification supports an optional feature that can be implemented by WSGI adapters for platform specific file handling.

What this allows is for a WSGI application to return a special object type which wraps a Python file like object. If that file like object statisfies certain conditions as dictated by a specific platform, then the WSGI adapter is allowed to return the content of that file in an optimised manner.

The intent of this is to provide better performance for serving up static file content than a pure Python WSGI application may itself be able to achieve.

Do note however that for the best performance, static files should always be served by a web server. In the case of mod_wsgi this means by Apache itself rather than mod_wsgi or the WSGI application. Using the web server may not always be possible however, such as for files generated on demand.

Example Of Wrapper Usage

A WSGI adapter implementing this extension needs to supply a special callable object under the key ‘wsgi.file_wrapper’ in the ‘environ’ dictionary passed to the WSGI application.

What this callable does will be specific to a WSGI adapter, but it must be a callable that accepts one required positional parameter, and one optional positional parameter. The first parameter is the file like object to be sent, and the second parameter is an optional block size. If the block size is not supplied then the WSGI adapter would choose a value which would be most appropriate for the specific hosting mechanism.

Whatever the WSGI adapter does, the result of the callable must be an iterable object which can be used directly as the response from the WSGI application or for passing into any WSGI middleware. Provided the response content isn’t consumed by any WSGI middleware and the iterable object gets passed through the WSGI adapter, the WSGI adapter should recognise the special iterable object and trigger any special handling to return the response in a more efficient way.

Because the support of this platform specific file handling is optional for any specific WSGI adapter, any user code should be coded so as to be able to cope with it not existing.

Using the snippet as described in the WSGI specification as guide, the WSGI application would be written as follows:

def application(environ, start_response):
    status = '200 OK'
    response_headers = [('Content-type', 'text/plain')]
    start_response(status, response_headers)

    filelike = file('usr/share/dict/words', 'rb')
    block_size = 4096

    if 'wsgi.file_wrapper' in environ:
            return environ['wsgi.file_wrapper'](filelike, block_size)
    else:
        return iter(lambda: filelike.read(block_size), '')

Note that the file must always be opened in binary mode. If this isn’t done then on platforms which do CR/LF translation automatically then the original content will not be returned but the translated form. As well as it not being the original content this can cause problems with calculated content lengths if the ‘Content-Length’ response header is returned by the WSGI application and it has been generated by looking at the actual file size rather than the translated content.

Addition Of Content Length

The WSGI specification does not say anything specific about whether a WSGI adapter should generate a ‘Content-Length’ response header when the ‘wsgi.file_wrapper’ extension is used and the WSGI application does not return one itself.

For mod_wsgi at least, if the WSGI application doesn’t provide a ‘Content-Length’ response header it will calculate the response content length automatically as being from the current file position to the end of the file. A ‘Content-Length’ header will then be added to the response for that value.

As far as is known, only mod_wsgi automatically supplies a ‘Content-Length’ response header in this way. If consistent behaviour is required on all platforms, the WSGI application should always calculate the length and add the header itself.

Existing Content Length

Where a ‘Content-Length’ is specified by the WSGI application, mod_wsgi will honour that content length. That is, mod_wsgi will only return as many bytes of the file as specified by the ‘Content-Length’ header.

This is not a requirement of the WSGI specification, but then this is one area of the WSGI specification which is arguably broken. This manifests in the WSGI specification where it says:

“””transmission should begin at the current position within the “file” at the time that transmission begins, and continue until the end is reached”””

If this interpretation is used, where a WSGI application supplies a ‘Content-Length’ header and the number of bytes listed is less than the number of bytes remaining in the file from the current position, then more bytes than specified by the ‘Content-Length’ header would be returned.

To do this would technically be in violation of HTTP specifications which should dictate that the number of bytes returned be the same as that specified by the ‘Content-Length’ response header if supplied.

Not only is this statement in the WSGI specification arguably wrong, the example snippet of code which shows how to implement a fallback where the ‘wsgi.file_wrapper’ is not present, ie.:

if 'wsgi.file_wrapper' in environ:
    return environ['wsgi.file_wrapper'](filelike, block_size)
else:
    return iter(lambda: filelike.read(block_size), '')

is also wrong. This is because it doesn’t restrict the amount of bytes returned to that specified by ‘Content-Length’.

Although mod_wsgi for normal iterable content would also discard any bytes in excess of the specified ‘Content-Length’, many other WSGI adapters are not known to do this and would just pass back all content regardless. The result of returning excessive content above the specified ‘Content-Length’ would be the failure of subsequent connections were the connection using keep alive and was pipe lining requests.

This problem is also compounded by the WSGI specification not placing any requirement on WSGI middleware to respect the ‘Content-Length’ response header when processing response content. Thus WSGI middleware could also in general generate incorrect response content by virtue of not honouring the ‘Content-Length’ response header.

Overall, although mod_wsgi does what is the logical and right thing to do, if you need to write code which is portable to other WSGI hosting mechanisms, you should never produce a ‘Content-Length’ response header which lists a number of bytes different to that which would be yielded from an iterable object such as a file like object. Thus it would be impossible to use any platform specific file handling features to return a range of bytes from a file.

Restrictions On Optimisations

Although mod_wsgi always supplies the ‘wsgi.file_wrapper’ callable object as part of the WSGI ‘environ’ dictionary, optimised methods of returning the file contents as the response are not always used.

A general restriction is that the file like object must supply both a ‘fileno()’ and ‘tell()’ method. This is necessary in order to get access to the underlying file descriptor and to determine the current position within the file.

The file descriptor is needed so as to be able to use the ‘sendfile()’ function to return file contents in a more optimal manner. The ‘tell()’ method is needed to be able to calculate response ‘Content-Length’ and to validate that where the WSGI application supplies its own ‘Content-Length’ header that there is sufficient bytes in the file.

Because the ‘sendfile()’ function is used by Apache to return file contents in a more optimal manner and because on Windows a Python file object only provides a Windows file handle and not a file descriptor, no optimisations are available on the Windows platform.

The optimisations are also not able to be used if using Apache 1.3. This is because Apache doesn’t provide access to a mechanism for optimised sending of file contents to a content handler under Apache 1.3.

Finally, optimisations are not used where the WSGI application is running in daemon mode. This is currently disabled because some UNIX platforms do not appear to support use of the ‘sendfile()’ function over UNIX sockets and only support INET sockets. This situation may possibly have changed with recent versions of Linux at least but this has yet to be investigated properly.

Whether or not optimisations are supported, the mod_wsgi ‘wsgi.file_wrapper’ extension generally still performs better than if a pure Python iterable object was used to yield the file contents.

Note that this all presumes that the iterable object returned by ‘wsgi.file_wrapper’ is actually passed back to mod_wsgi and is not consumed by a WSGI middleware. For example, a WSGI middleware which compresses the response content would consume the response content and modify it with a different iterable object being returned. In this case there is no chance for optimisations to be used for returning the file contents.

This problem isn’t restricted though to just where the response content is modified in some way and also extends to any WSGI middleware that wants to replace the ‘close()’ method to perform some cleanup actions at the end of a request.

This is because in order to interject the cleanup actions triggered on the ‘close()’ method of the iterable object it has to replace the existing iterable object with another which wraps the first, with the outer providing its own ‘close()’ method. An example of a middleware which replaces the ‘close()’ method in this way can be found in Registering Cleanup Code.

It is thus quite easy for a WSGI application stack to inadvertantly defeat completely any attempts to return file contents in an optimised way using the ‘wsgi.file_wrapper’ extension of WSGI. As such, attempts should always be used instead to make use of a real web server, whether that be a separate web server, or in the case of mod_wsgi the underlying Apache web server.

Where necessary, features of web servers or proxies such as ‘X-Accel-Redirect’, ‘X-Sendfile’ or other special purpose headers could be used. If using mod_wsgi daemon mode and using mod_wsgi version 3.0 or later, the ‘Location’ response header can also be used.