How to add STATIC files

You will most likely add STATIC files to your source code repository. As they are likely to be hardcoded in your code and templates it is a good idea to keep those and your STATIC files in sync.

Your STATIC files are collected (found and copied or symlinked) with the help of finders. This collection process might be a little confusing for you. Basically you don’t need to worry about where your files are copied, you just need to maintain the files in the locations I mention below. Collection process consolidates all your STATIC files for you, and it does it automatically.

If you are developing a reusable app place your STATIC files under the static directory. Your app directory should look like this:

$ ls -1
models.py
static/
templates/
views.py

If you are developing a project absolute filesystem paths listed in your STATICFILES_DIRS setting will be collected:

>>> from django.conf import settings
>>> settings.STATICFILES_DIRS
('/opt/myproject/src/project/static',)

One important thing to note, as I mentioned in the first part, is to set your STATIC_ROOT outside of your project directory. This is the location where all the STATIC files found will be consolidated. Making this directory independent from your project installation enabled its reuse.

How to upload MEDIA files

MEDIA files are typically uploaded by users when the project is online and a reference is stored in a model field. While this is true most of the time, MEDIA files can be generated by code and/or a reference can be provided in some other way that doesn’t involve models. But these edge cases are out of the scope of this post.

The easiest way to allow users upload their files is to use a FileField or ImageField on a model and derive a form from it:

from django.db import models
from django.forms.models import modelform_factory


class MediaModel(models.Model):
    media_file = models.FileField(upload_to='user_media')


MediaForm = modelform_factory(MediaModel)

You can then use this form to provide upload functionality:

def media_create(request):
    if request.method == "POST":
        form = MediaForm(request.POST, request.FILES)
        if form.is_valid():
            form.save()
            return HttpResponseRedirect(reverse('media-list'))
    else:
        form = MediaForm()
    return render_to_response('usermedia/create/html', {'form': form})

Of course it is better to use a generic CreateView now we have them with Django 1.3. I wanted to emphasize one point and avoid the complexities of a class based view; you must pass request.FILES to the form’s constructor. Actually this is a good practice whether or not your form has an file field.

As in the case with STATIC files, MEDIA_ROOT should be in a location seperate from your project files. If you delete your project directory you would also lose MEDIA files otherwise.

Conclusion

This concludes Working with files in Django. I hope these posts are helpful to you. Once you are comfortable working with files I strongly recommend you to take a look at Django Compressor

If you enjoyed this post please tell me a little bit about yourself.

Related posts:

1. Working with files in Django – Part 1
2. Working with files in Django – Part 2
3. My PyCon APAC 2011 Presentation: Optimizing Media Performance with django_compressor

How to setup file serving for development server

As I mentioned earlier, it is best to serve files on a fast HTTP server. But this setup is overkill for development environments, you can safely let Django handle your files. The snippet below relies on the assumption, behind the scenes, that DEBUG is always True in development environment and it is always False in production environment. It should be fine for the purposes of this post:

from django.conf.urls.static import static
from django.contrib.staticfiles.urls import staticfiles_urlpatterns

urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
urlpatterns += staticfiles_urlpatterns()

It is quite straightforward. Nothing gets added to urlpatterns unless DEBUG is True and prefix (MEDIA_URL and STATIC_URL) is not a fully qualified URL.

How to reference STATIC files in templates

There are two ways to reference any file. You can use a file’s location in your file storage (your filesystem for example) for internal use. Or you can build a URL for the file so that clients can access it. This post will cover the latter, as the former is a simple matter of applying os.path.join() to STATIC_ROOT and the particular file’s path.

To build a URL in your templates you can simply concetenate STATIC_URL with the file’s relative location to it:

<img src="{{ STATIC_URL }}myapp/img/logo.png" />

However, to have STATIC_URL available within your template context you need to make sure of two things:

• The template needs to be rendered with a RequestContext.
• django.core.context_processors.static needs to be included in TEMPLATE_CONTEXT_PROCESSORS setting.

I can’t think of a reason why, but if for some reason you are not using RequestContext, you can provide STATIC_URL using getstaticprefix tag:

{% load static %}
{% get_static_prefix as STATIC_URL %}

<img src="{{ STATIC_URL }}myapp/img/logo.png" />

How to reference MEDIA files in templates

Dealing with MEDIA files is much simpler. The FieldFile object returned by ImageField and FileField has path and url properties and you don’t need to do concetenation yourself:

<img src="{{ some_model.some_image_field.url }}" />

Next part deals with adding files to your project.

If you enjoyed this post please tell me a little bit about yourself.

Related posts:

1. Working with files in Django – Part 1
2. Working with files in Django – Part 3
3. My PyCon APAC 2011 Presentation: Optimizing Media Performance with django_compressor

• Files that are hard coded in templates or in code. We will call them STATIC files.
• Files that are referenced in the code but only known in the run time. We will call them MEDIA files.

Now of course all these files are static and most of them can be classified as media, but I chose those terms because that more or less how they are referred to in Django documentation.

Since files are stored on disk or some other storage backend it is best to serve them on a fast HTTP server and let Django handle only dynamic content.

How to configure STATIC/MEDIA related settings

Let’s look at some code first and we can comment on it afterwards. I use the following setup in my projects:

import os

_PATH = os.path.abspath(os.path.dirname(__file__))

MEDIA_ROOT = os.path.join(_PATH, 'files', 'media')
MEDIA_URL = '/media/'

STATIC_ROOT = os.path.join(_PATH, 'files', 'static')
STATIC_URL = '/static/'
STATICFILES_DIRS = (
    os.path.join(_PATH, 'static'),
)
STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
)

ADMIN_MEDIA_PREFIX = '/static/admin/'

_PATH should be obvious. But the way I use it is a little tricky here. Basically I am using a files directory that contains media, for MEDIA files, and static, for STATIC files, directories. This seperation is essential. You will pull your hair if you don’t seperate those two from the beginning. This will allow you to reuse them easily if you deploy a new version to a different location.

But why is files under the project directory? Because we need to check-in STATIC files? NO! It is just a convenience for the development environments. I strongly suggest overriding MEDIA_ROOT and STATIC_ROOT in your deployment settings and basically move the files directory outside of the project path. STATIC files that we check-in resides in <_PATH>/static. django.contrib.staticfiles app collects your STATIC files from your project specific directories and within your apps and copies everything in the right places inside STATIC_ROOT.

To sum up the points above:

• Check-in STATIC files in a folder within your project directory (see STATICFILES_DIRS & FileSystemFinder). If you are developing an app AppDirectoriesFinder will collect static files within the static directory in your app directory.
• Seperate the directories, where your MEDIA files are saved and where your STATIC files are collected (see STATIC_ROOT & MEDIA_ROOT).
• Keep your STATIC_ROOT and MEDIA_ROOT outside of your project directory in production environment.

One more thing to note is STATIC_URL and MEDIA_URL should both have a trailing slash. This will save you a lot of trouble later.

Next part deals with serving files in development environment and referencing them in code and templates.

If you enjoyed this post please tell me a little bit about yourself.

Related posts:

1. Working with files in Django – Part 2
2. Working with files in Django – Part 3
3. My PyCon APAC 2011 Presentation: Optimizing Media Performance with django_compressor

Related posts:

1. Working with files in Django – Part 1
2. Working with files in Django – Part 2
3. Working with files in Django – Part 3

I needed to draw gradients for the background of my little learning project. First I tried the naive approach: iterating over each pixel and setting the correct value. This takes forever as you could have guessed. I continued my research to find out the close connection between PyGame and NumPy. Yes, fast array operations. My gradient generating code using linspace and tile is below:

def generate_gradient(fromcolor, tocolor, height, width):
    channels = []
    for channel in range(3):
        fromvalue, tovalue = fromcolor[channel], tocolor[channel]
        channels.append(numpy.tile(numpy.linspace(fromvalue, tovalue, width),
                                                                  [height, 1]))
    return numpy.dstack(channels)

I create three 1D arrays for each channel and then tile them to get a 2D array. Then I stack those 2D arrays and get combined (r, g, b) values. Each 1D array contain values linearly sampled between given maximum and minimum values by linspace. I can then blip this array on a surface like this:

gradient = generate_gradient((63, 95, 127), (195, 195, 255), 1024, 600)
pygame.surfarray.blit_array(some_surface, pygame.surfarray.map_array(some_surface, gradient))

This works quite well. But I was curious whether there is a better method, so I did some googling. First I stumbled upon James Tauber’s gradient code. Arrays! I was a bit ashamed not remembering standard library had array module. James Tauber’s code builds an array with the correct values and then writes the image as a PNG file. It also has the ability to generate multi-gradients.

Then I found gradients package for PyGame. This package does much more than linear gradients. It creates a PyGame surface of 1 pixel thickness, fills it with the gradient values and then resizes it.

I made a small test to see which method runs faster. I had to strip file and compression related code from James Tauber’s gradient module. I run a simple method tha just generates a gradient image/array in memory and returns it 10 time, below is the average times:

• 320×240:
  • numpy: 0.01469659805
  • array: 3.6310561180
  • pygame: 0.0069756984711
• 800×600:
  • numpy: 0.1323119163
  • array: 19.9719331265
  • pygame: 0.0192141056061

Here numpy is my method above, array is the stripped version of James Tauber’s code and pygame is the gradiends package. If my benchmarking method doesn’t have dramatic errors fastest way to generate linear gradients is to fill a strip with correct values and then to resize it.

Now I’ll go back to work on my learning project.

Related posts:

1. My PyCon APAC 2011 Presentation: Optimizing Media Performance with django_compressor
2. Working with files in Django – Part 1
3. Working with files in Django – Part 2

First let’s have a look at the supporting code:

from __future__ import with_statement
from fabric.api import *
from fabric.utils import warn


def apt_get(*packages):
    sudo('apt-get -y --no-upgrade install %s' % ' '.join(packages), shell=False)

Our apt_get fabric command issues apt_get install that answers yes to all questions and does not upgrade if the requested package is already installed. Now let’s take a look at the main command that installs mysql-server:

def install_mysql():
    with settings(hide('warnings', 'stderr'), warn_only=True):
        result = sudo('dpkg-query --show mysql-server')
    if result.failed is False:
        warn('MySQL is already installed')
        return
    mysql_password = prompt('Please enter MySQL root password:')
    sudo('echo "mysql-server-5.0 mysql-server/root_password password ' \
                              '%s" | debconf-set-selections' % mysql_password)
    sudo('echo "mysql-server-5.0 mysql-server/root_password_again password ' \
                              '%s" | debconf-set-selections' % mysql_password)
    apt_get('mysql-server')

First we want to make sure mysql-server is not already installed. For this reason we issue dpkg-query --show mysql-server. Note that if this command fails then mysql-server is not installed and we can proceed otherwise we want to return. We hide the ugly error messages by running our dpkg-query command within a settings context:

with settings(hide('warnings', 'stderr'), warn_only=True):
    result = sudo('dpkg-query --show mysql-server')
if result.failed is False:
    warn('MySQL is already installed')
    return

The rest of the code is pretty straightforward. We prompt to the user running fabric for the root password and seed its value twice into the debconf database.

sudo('echo "mysql-server-5.0 mysql-server/root_password password ' \
                            '%s" | debconf-set-selections' % mysql_password)
sudo('echo "mysql-server-5.0 mysql-server/root_password_again password ' \
                            '%s" | debconf-set-selections' % mysql_password)
apt_get('mysql-server')

Finally, having finished our little dance, we install mysql-server. I hope this helps some automated deployment believer out there.

Related posts:

1. Drawing Gradients with PyGame
2. My PyCon APAC 2011 Presentation: Optimizing Media Performance with django_compressor
3. Working with files in Django – Part 1

Let me reiterate the last sentence. Having a Django-based blog just because Django is fashinable is a little dumb in my opinion. If Django-based X blogging engine suits you better than anything else, use it. Why not? But my personal choice of blogging engine is WordPress¹. The value of a Django blogging app, for me, is in adding a blog to a Django project. And different projects might have different requirements. So my idea of a Django blogging app is one that is highly configurable and highly extendable.

On the other hand I don’t need the convenience of clicking a checkbox on a polished UI. I can write a function. Or I don’t necessarily need it to, say, provide a navigation menu. There are apps that do that. Even if there wasn’t it shouldn’t be the blog app’s job. So I am not looking for an instant-blog. I have a Django app in my mind, nothing more.

What Should Be Left Out

Basically any feature that can be provided by another reusable app should be left out. Why should we re-implement something that is already done… and reviewed by others… and tested. Of course this doesn’t necessarily mean providing no convenience functions.

• No admin. Because we already have one.
• No theming. For the love of Flying Spaghetti Monster, you don’t need any theming other than what django.template offers. Pre-built themes are for turn-key solutions.
• No comments or contact forms. (See django.contrib.comments and django-contact-form)
• No official markup format (or formats). This can be handled in the templates without difficulty. But, maybe, pluggable content filters is a good idea. I haven’t made up my mind on this one entirely. It won’t use any markup format by default, that is for sure.

What Should Be Included

Remember, every project has a different set of needed features for its blog. Some need catagories, some need tags and some others need both. But it would end up as a disaster if we implemented each one of those features into a single app. Instead I think it should consist of many small apps that work together. But I wouldn’t want to end up having huge spaghetti of apps that all depend on one another, like Pinax does. A minimal amount of core apps² and then everything else should be optional. By optional I mean you don’t have to install packages you won’t need.

I think the components (apps) should be activated via adding to INSTALLED_APPS and configured with settings. I can’t think of any parameter that needs to be changed dynamically, so why not use the established way of doing configuration in Django.

Two must have features for such a blogging app are previews and scheduled publishing. It is possible that you sometimes write a post quickly and publish it immediately. But I suppose nobody will say they don’t care about these two features.

Built-in feeds and sitemaps are also nice to have.

Multiple instances of this blogging app running on the same project? À la admin. I can’t make my mind on this one. Sure it would be a nice feature. But it could complicate the code. Peehaps too much for a not so common case.

What do you think about the general idea? Are there any other must-have features? Would you be willing to learn a new app when you are already comfortable with another blogging app?

1: Even though it’s written in the abomination called PHP. But since there are plugins for everything I don’t have to touch the code.

2: One sounds like a good number, if possible.

Related posts:

1. Working with files in Django – Part 1
2. Working with files in Django – Part 2
3. Working with files in Django – Part 3

New Example Project

There is a new and improved example project. It is designed to be some sort of documentation at the same time. When you run the example project and visit different pages you will see, for each examle, Python code, template code, text of rendered result and finally the result embedded.

If you have Pygments installed all the code will be nicely highlighted.

Fieldset & FieldsetMixin Improvements

Fieldset definitions are validated now. An exception will be raised if all of your fields are not included exactly once.

Template strings that are used by as_table, as_p and as_ul methods are now class attributes. You can simply override them instead of writing your own as_* method.

FieldsetMixin provides a fieldset_dict attribute. This dictionary has slugified fieldset names as keys and Fieldset instances as values. Your fieldset declarations can still be accessed from fielsets attribute.

Rendering Improvements

There are two rendering related improvements: individual fieldset rendering and renderform template filter.

Fieldset objects have as_table, as_p, as_ul methods just like forms. Errors from hidden fields are handled correctly, but you still need to call non_field_errors() to get the top level errors. Also it is template author’s responsibility to make sure all the fieldsets are rendered.

If as_* methods are not enough for you, with renderform filter you can render your forms or Fieldsets through a custom template. It works like this:

{{ form.fieldset_dict.mytitlerenderform:"myapp/mytitle_fieldset.html" }}

If you call it without an argument formfieldset/form.html template will be used.

Related posts:

1. Working with files in Django – Part 1
2. Working with files in Django – Part 2
3. Working with files in Django – Part 3

Taking advantage of signals doesn’t necessarily mean providing no sane defaults. You can send signals and provide default event handling. Here is a couple of ideas how this can be done:

• Your app can check if there are any listeners and connect the default handlers if there is none.
• You can ship an auxiliary app that connects default handlers when added to INSTALLED_APPS.

I personally prefer the second approach since it’s simpler and more explicit. I’m sure there are other ways to implement default handlers for signals.

Dispatch_uid

Don’t forget to assign a unique dispatch_uid for each connect() call. Otherwise your handler can get connected twice. I would also suggest you to use both module path and your handler function’s name in your dispatch_uid:

"%s.%s" % (os.path.splitext(__file__)[0].replace(os.sep, '.')[1:],
           handler_name)

Now I should take my own advice and replace hardcoded User.message_set.create()s with signals in django-simple-friends.

Related posts:

1. Working with files in Django – Part 1
2. Working with files in Django – Part 2
3. Working with files in Django – Part 3

I follow (and read with great interest) Steve Souders’s blog High Performance Web Sites. I must admit I was sceptical about it at first; spriting images, individual different loading behaviours of browsers… I thought they were premature optimizations. But I realized I was wrong as I continued to read. Steve Souders is an expert on high performance web sites and what he preaches are realistic techniques, backed by test results most of the time. If you are not following, I suggest you add it to your RSS reader.

Optimization Techniques

We can explore the techniques in two main categories:

1. Techniques to reduce the data to be transferred.
   • Minifying: Minification is removing comments and unneeded whitespace in CSS and JavaScript files. Compilers like YUI Compressor and Closure modifies JavaScript code to compress even further, without any changes to the functionality.
   • Gzipping: Web browsers accept gzip encoded content for a long time. I have just compressed a 13 KB text file down to 4 KB. A two thirds compression ratio is not bad at all.
2. Techniques to reduce the number of HTTP requests.
   • Combining: CSS and JavaScript files can be combined together into a single file and therefore a single HTTP request. Gzip may be more efficient on these files. In a similar way background images can be merged into a sprite and then reconstructed again using their coordinates on that sprite.
   • Data URI’s: Images (or other file types) can be embedded into CSS (or JavaScript or HTML) using data: URI‘s. Extra HTTP requests for those resources can be avoided this way.

You might think it’s fine to perform all these optimizations, but what happens when I want to make some changes to my combined, minified JavaScript file? Instead of applying these techniques blindly, it’s best to follow a sensible plan for implementing these optimizations:

• First of all everything that can be automated should be automated. Regarding the example above script files should stay uncombined and uncompressed in the development environment and optimizations should be applied when the application is published. Taking it a step further we can have the application detect changes in those files and update optimized versions automatically. (django_compressor works this way)
• I was worried that spriting would complicate managing the design. But I have seen, on the contrary; if images each sprite contain are choosen carefully it makes the process easier. Start combining images that belong to the same design element. Avoid complex arrangements, stick with horizontal or vertical stacking as much as possible. Don’t forget to leave transparent spaces between items and sprite border. Try to combine images that are loaded on the same page, avoid loading a sprite for only half of it’s elements. Don’t force yourself to combine all images, if you follow the guidelines I have mentione they won’t.
• When performing optimizations don’t forget to use easily available tools. You can use YSlow for general analysis, SpriteMe for image combining tips, CompressorRater to compare different compilers’ performance on your scripts. I would like to note that Steve Souders is the developer of first two.

Telvee Results

I didn’t think about performance at all when I started developing telvee. Too many CSS files and too many images were being loaded. Here is what it looked like before optimizations:

Then I have installed and configured django_compressor. I used YUI Compressor for both JavaScript and CSS. I have created sprites and modified CSS files manually¹. Then I deployed these changes and measured again:

In the load column of Homepage, the number in parens is the actual load. But the design of homepage is changed with this upgrade and a new 90 KB image is being loaded now. So I have accepted 70 KB in my calculations. The result of optimizations are as follows:

Django_compressor and Data URI’s

Django_compressor, developed by Christian Metts, helps you to apply optmizations I have mentioned above easily to your Django projects. You can see my fork here where I have merged some other branches and added a little bit of code myself.

Using compressor.filters.datauri.CssDataUriFilter in data-uri branch of this repository, you can embed linked files within your CSS files. It will only embed files less than or equal to 1024 Bytes (1 KB) by default. You can change this limit by setting COMPRESS_DATA_URI_MIN_SIZE in your settings.py.

There are a couple of things to pay attention when you convert your references to data: URIs. Firstly file contents are base64 encoded which means approximately one third increase in size. It’s up to you to balance between increased bandwidth and reduced request counts². Another thing to watch for is multiple references to the same file will end up embedding the same data many times. The solution to this problem is to reduce all references to one³ but this might break your CSS arrangement strategy.

Please test django_compressor’s data: URI support and tell me what you think. If you haven’t applied optimizations I mentioned above, you should. Thanks to django_compressor they are quite easy to implement on Django projects.

1: I would like to add automatic sprite building/linking support to django_compressor sometime.

2: With Today’s modern connections 1~2 KB increase is a good price for 1 less HTTP request..

3: http://meiert.com/en/blog/20090401/why-css-needs-no-variables/

Related posts:

1. My PyCon APAC 2011 Presentation: Optimizing Media Performance with django_compressor
2. Working with files in Django – Part 1
3. Working with files in Django – Part 2