Posts Tagged ‘python’

« Older Entries

Developing Reusable Django Apps: Signals

Thursday, March 4th, 2010

I wrote “signals provide a great way to propagate the events generated from your app” earlier. I think reusable apps should avoid hardcoding any kind of event handling and send signals instead. App consumers might prefer an email over an on-screen notification. They may even choose to ignore the event silently. A reusable app should give this choice to the consumer.

Taking advantage of signals doesn’t necessarily mean providing no sane defaults. You can send signals and provide default event handling. Here is a couple of ideas how this can be done:

  • Your app can check if there are any listeners and connect the default handlers if there is none.
  • You can ship an auxiliary app that connects default handlers when added to INSTALLED_APPS.

I personally prefer the second approach since it’s simpler and more explicit. I’m sure there are other ways to implement default handlers for signals.

Dispatch_uid

Don’t forget to assign a unique dispatch_uid for each connect() call. Otherwise your handler can get connected twice. I would also suggest you to use both module path and your handler function’s name in your dispatch_uid:

"%s.%s" % (os.path.splitext(__file__)[0].replace(os.sep, '.')[1:],
           handler_name)

Now I should take my own advice and replace hardcoded User.message_set.create()s with signals in django-simple-friends.

Bookmark and Share

Tags: , , ,
Posted in Programming | 5 Comments »

Web Site Performance Optimizations

Monday, February 22nd, 2010

Recently I have done some optimizations to make telvee a little faster using django_compressor and making sprites for background images. Good news is substantial changes to development environment and the design wasn’t required. I’ll get into details below. But first I’d like to write about the theory a little bit.

I follow (and read with great interest) Steve Souders’s blog High Performance Web Sites. I must admit I was sceptical about it at first; spriting images, individual different loading behaviours of browsers… I thought they were premature optimizations. But I realized I was wrong as I continued to read. Steve Souders is an expert on high performance web sites and what he preaches are realistic techniques, backed by test results most of the time. If you are not following, I suggest you add it to your RSS reader.

Optimization Techniques

We can explore the techniques in two main categories:

  1. Techniques to reduce the data to be transferred.
    • Minifying: Minification is removing comments and unneeded whitespace in CSS and JavaScript files. Compilers like YUI Compressor and Closure modifies JavaScript code to compress even further, without any changes to the functionality.
    • Gzipping: Web browsers accept gzip encoded content for a long time. I have just compressed a 13 KB text file down to 4 KB. A two thirds compression ratio is not bad at all.
  2. Techniques to reduce the number of HTTP requests.
    • Combining: CSS and JavaScript files can be combined together into a single file and therefore a single HTTP request. Gzip may be more efficient on these files. In a similar way background images can be merged into a sprite and then reconstructed again using their coordinates on that sprite.
    • Data URI’s: Images (or other file types) can be embedded into CSS (or JavaScript or HTML) using data: URI’s. Extra HTTP requests for those resources can be avoided this way.

You might think it’s fine to perform all these optimizations, but what happens when I want to make some changes to my combined, minified JavaScript file? Instead of applying these techniques blindly, it’s best to follow a sensible plan for implementing these optimizations:

  • First of all everything that can be automated should be automated. Regarding the example above script files should stay uncombined and uncompressed in the development environment and optimizations should be applied when the application is published. Taking it a step further we can have the application detect changes in those files and update optimized versions automatically. (django_compressor works this way)
  • I was worried that spriting would complicate managing the design. But I have seen, on the contrary; if images each sprite contain are choosen carefully it makes the process easier. Start combining images that belong to the same design element. Avoid complex arrangements, stick with horizontal or vertical stacking as much as possible. Don’t forget to leave transparent spaces between items and sprite border. Try to combine images that are loaded on the same page, avoid loading a sprite for only half of it’s elements. Don’t force yourself to combine all images, if you follow the guidelines I have mentione they won’t.
  • When performing optimizations don’t forget to use easily available tools. You can use YSlow for general analysis, SpriteMe for image combining tips, CompressorRater to compare different compilers’ performance on your scripts. I would like to note that Steve Souders is the developer of first two.

Telvee Results

I didn’t think about performance at all when I started developing telvee. Too many CSS files and too many images were being loaded. Here is what it looked like before optimizations:

 # of requestsload (KB)
Homepage25~85
Cup Detail48~80

Then I have installed and configured django_compressor. I used YUI Compressor for both JavaScript and CSS. I have created sprites and modified CSS files manually1. Then I deployed these changes and measured again:

 # of requestsload (KB)
Homepage12~70 (~160)
Cup Detail14~64

In the load column of Homepage, the number in parens is the actual load. But the design of homepage is changed with this upgrade and a new 90 KB image is being loaded now. So I have accepted 70 KB in my calculations. The result of optimizations are as follows:

 # of requestsload (KB)
Homepage52%17%
Cup Detail70%19%

Django_compressor and Data URI’s

Django_compressor, developed by Christian Metts, helps you to apply optmizations I have mentioned above easily to your Django projects. You can see my fork here where I have merged some other branches and added a little bit of code myself.

Using compressor.filters.datauri.CssDataUriFilter in data-uri branch of this repository, you can embed linked files within your CSS files. It will only embed files less than or equal to 1024 Bytes (1 KB) by default. You can change this limit by setting COMPRESS_DATA_URI_MIN_SIZE in your settings.py.

There are a couple of things to pay attention when you convert your references to data: URIs. Firstly file contents are base64 encoded which means approximately one third increase in size. It’s up to you to balance between increased bandwidth and reduced request counts2. Another thing to watch for is multiple references to the same file will end up embedding the same data many times. The solution to this problem is to reduce all references to one3 but this might break your CSS arrangement strategy.

Please test django_compressor’s data: URI support and tell me what you think. If you haven’t applied optimizations I mentioned above, you should. Thanks to django_compressor they are quite easy to implement on Django projects.

1: I would like to add automatic sprite building/linking support to django_compressor sometime.

2: With Today’s modern connections 1~2 KB increase is a good price for 1 less HTTP request..

3: http://meiert.com/en/blog/20090401/why-css-needs-no-variables/

Bookmark and Share

Tags: , , , ,
Posted in Internet, Programming | 3 Comments »

Top 5 Untrends According To Me

Tuesday, February 9th, 2010

My dear friend Ochronus posted an article titled Top 5 trends and technologies in software development that got me thinking. My thoughts below. Go check Ochronus’s blog if you haven’t, he is the lead developer at Arukereso.hu.

I agree with the suggestions from the original article. Yet, I would like to change the order a little bit; DVCS and then agile (with lowercase a) and then the rest. None of my points below are cool trends, in fact I can guarantee most of you will find them boring. But I think they are all important. OK, I hope you are all psyched now. Here we go:

1. Be Careful With The Buzz

Trends are cool. What could be wrong about following cutting edge stuff? We all want to be up to date, no? I think it’s good to follow the trends if you have the experience and the ability to filter the BS. I know a young developer who was constantly going back and forth between Rails/Ruby and Django/Python. I haven’t heard from him for a while, but he is probably still doing that same dance. Why? Because his considerations were solely based on buzz, not on simple requirements analysis or technical comparisons or personal experience.

2. Learn And Use An Old-Fashioned Modern Low-Level Scripting Language

To all the scripting people, like me, out there: you need to have an understanding of what’s happening under the hood. At the least to appreciate our high-level environments, at the most to become genuinely good programmers. Being a Python person myself, I think the best low-level language to be proficient for me is C. Many other high-level languages have C interfaces. So investing the time to learn C should pay off one way or the other.

3. Do Less Web Programming

Aren’t we doing a lot of web programming these days? Actually I think doing X development exclusively is bad for your programming muscles. Web programming, enterprise work or system scripting, it doesn’t matter. But web programming happens more than anything else. Maybe some of you have only been playing with it, but there are a huge number of us doing nothing but web programming. This is so sad; both in an individual level and for the community at large.

4. Learn How To Educate Yourself

What is a noob? Here is a definition and disambiguation (from newbie):

Newbs are those who are new to some task and are very beginner at it, possibly a little overconfident about it, but they are willing to learn and fix their errors to move out of that stage. n00bs, on the other hand, know little and have no will to learn any more. They expect people to do the work for them and then expect to get praised about it, and make up a unique species of their own.

Make an active effort not to be a noob. Learn how to ask smart questions, how to communicate others and seek help. Being polite is good but actually improving and being a valuable member of the community is much, much better.

5. Open Source Properly

It’s great to open source your project. But please do it properly. There are already too many unmaintained, undocumented projects out there that noone seem to care. Do you really have to add to that? As is argument doesn’t make much sense today. But if you really have to make an open source dead drop, please at least document the status of your project and your intentions clearly.

I wouldn’t be surprised if some you think they all are obvious. But if they are so obvious then why are they widely being ignored? Is it because they are under-retweeted, under-reddited and therefore not trendy.

Bookmark and Share

Tags: , , , , ,
Posted in Programming | 9 Comments »

Developing Reusable Django Apps: App Settings

Tuesday, January 26th, 2010

Eventually your app will need some sort of configuration. Supplying many parameters to customise your views, template tags and filters to allow template authors easily harness the power of your app and implementing registration pattern are all sensible things to do. But at some point you will need configuration for your app at project level. Because app level configuration is stupidnot reusable. Consumers of your app should never have to change its source code. And what better place to put our configuration statements than settings.py? Remember; we want to make things easier for our app’s consumers, not harder. There is no need to add a new file to the project for a few lines of settings1.

We already know we shouldn’t import settings.py directly. Instead we import the settings object (much like a singleton) of django.conf module:

from django.conf import settings

Now you can access different configuration options as attributes on this object. But I suggest you to use getattr() in order to avoid getting AttributeErrors. Also, notice how we didn’t hardcode the name of the attribute in the second method below:

# This is too verbose
try:
    some_setting = settings.SOME_SETTING
except AttributeError:
    some_setting = DEFAULT_VALUE

# Plain and simple
some_setting = getattr(settings, 'SOME_SETTING', DEFAULT_VALUE)

Instead of requiring consumers to define all your app settings it is better to supply sensible defaults. Also I find it useful to prefix names of app settings within settings.py.

# in settings.py
MYAPP_FOO_CHOICES = [('bar', u'Bar'), ('baz', u'Baz')]


# in myapp/models.py
from django.db import models
from django.conf import settings


FOO_CHOICES = getattr(settings, 'MYAPP_FOO_CHOICES', [('quux', u'Quux')])


class FooRecord(models.Model):
    foo = models.CharField(max_length=10, choices=FOO_CHOICES)

This works fine for simple apps with fewer settings. But it can easily get out of hand when your app grows. An app_settings.py module would help keeping track of configuration by keeping all configuration options in one place:

# in myapp/app_settings.py
from django.conf import settings


FOO_CHOICES = getattr(settings, 'MYAPP_FOO_CHOICES', [('quux', u'Quux')])


# in myapp/models.py
from django.db import models
from app_settings import FOO_CHOICES


class FooRecord(models.Model):
    foo = models.CharField(max_length=10, choices=FOO_CHOICES)

To summarize the points above:

  • Import settings from django.conf
  • Use getattr()
  • Always supply a default value.
  • Prefix settings you made up in settings.py
  • Use app_settings.py if you have many

1: If your configuration is long, say more than 100 lines, you should step back and reconsider. Perhaps you should prefer a strategy similar to django.contrib.sitemaps or django.contrib.syndication.

Bookmark and Share

Tags: , , , ,
Posted in Programming | 1 Comment »

Quest For Ultimate Development/Deployment Toolset: Fabric, Pip & Virtualenv

Thursday, January 21st, 2010

DISCLAIMER: This post is a summary of my short experience with the tools mentioned. There is no flash news or revolutionary recipes. If you are familiar with them, you will probably find it boring. If you are not, this post might make your introduction less painful.

I stumbled upon this post following my frustration with setuptools after virtualenvising telvee repository. I have played with buildout to create repeatable deployments before. Don’t get me wrong, buildout is superb. But I have a few minor issues with it:

  • Buildout can create an interpreter but the isolation from the host system is not a good as virtualenv. (Please correct me if I’m wrong)
  • Buildout is a dependency1 whereas virtualenv is a system-wide tool.
  • It is, naturally, more work to set it up than a couple of ad-hoc bash scripts.

None of these issues make buildout less awesome, I am just not comfortable enough with it. I am considering other alternatives and their strengths and weaknesses.

Virtualenv

I am quite comfortable with virtualenv. It creates an isolated Python environment for you. For instance the following command will create an environment inside test directory that doesn’t have access to packages installed system-wide:

~/$ virtualenv --no-site-packages test

That is of course when you activate the virtual environment with:

~/$ cd test
~/test/$ source bin/activate
(test)~/test/$

Note the (test) prefix to your prompt. Any packages you install within this environment will be installed only for itself and won’t be available system-wide. If you have omitted --no-site-packages argument you could have access to globally installed packages too. When you are done with this virtual environment you can issue deactivate command to return to your normal shell. That’s basically what virtualenv does.

You can script virtualenv to a certain extend:

#!/usr/bin/env bash

virtualenv $1
cd $1
source bin/activate

echo $PATH
echo $PYTHONPATH

Or you can create bootstrap scripts with virtualenv. Virtualenv doesn’t provide a clean and powerful enough API here, just a callback and two methods to modify commandline arguments.

Virtualenv is great at what it does. But I think setuptools as a package installer (and it’s just an installer, not a manager) cripples virtualenv. Good news is newer versions have a commandline argument --distribute that appereantly substitutes distribute for setuptools.

Pip

My experiments with pip went just fine. Except I bumped into this problem. Globally installed pip was seeing system-wide packages when used on a virtual environment created with --no-site-packages. Again, good news is it is fixed in the trunk.

I will play with pip more once figure out how to best integrate the these applications.

Fabric

I see fabric as the glue that binds everything together. I have known about it long before, but I never had the chance to experiment. I have read most of the documentation2 and played with it a little. Basically the following script is an attempt to create a virtual environment and install pip and django on it, much like the shell script above:

from __future__ import with_statement
import os
from fabric.api import *
from fabric import context_managers


def _get_virtualenv_location():
    location = prompt('New location: ', default='../test')
    env.envdir, env.envname = os.path.split(os.path.abspath(location))
    env.envpath = os.path.join(env.envdir, env.envname)
    print('using "%(envname)s" at "%(envdir)s" as virtualenv' % env)


def _virtualenv(command):
    with context_managers.cd(env.envpath):
        result = local('. bin/activate && ' + command)
    return result


def clone():
    _get_virtualenv_location()
    with context_managers.cd(env.envdir):
        local('virtualenv --no-site-packages --clear %(envname)s' % env)
    print _virtualenv('echo $PATH')
    _virtualenv('easy_install pip')
    print _virtualenv('pip install django==1.1.1')

It took me a while to figure out how I can issue commands within a virtual environment. Since fabric commands don’t share state sourcing activate has no effect on subsequent commands. This SO entry helped me to write _virtualenv() function. It is kind of ugly making all functions but fab commands private. I think if fab used __all__ or something similar it would be more explicit. Also a contrib module for virtualenv would be nice3.

Fabric has cool features such as failure handling and code/config editing. It is a great tool to create repeatable deployments. It feels great to be coding in python (well, to some extend). Perhaps the resulting code is a little too complex for local operations. I wish I could write a fab command that handles both local and remote deployments. I think it is not unusual to deploy on the same machine in another location. But having to SSH into localhost is weird, don’t you think?

I am sure there are better ways to accomplish the goal of the script above. Maybe there is an entirely different way to integrage fabric, virtualenv and pip. So, comments and suggestions are welcome as usual.

Maybe I’ll revisit buildout again as well.

1: You need to bootstrap (install) buildout with each development/deployment site.

2: About one third of it I think.

3: I would happily attempt one, once I learn fabric a little better.

Bookmark and Share

Tags: , , , ,
Posted in Programming | 6 Comments »

« Older Entries