Archive for September, 2009

Django Fixtures

Thursday, September 24th, 2009

A fixture is basically a data dump in a specific format. There is no restriction of which models or how much data a fixture can contain. Fixtures are portable. They work the same with all the database backends and operating systems supported by Django. This means that you can use fixtures as a simple data migration tool1. Django supports a number of formats by default. JSON format is widely used in the community. But you can easily create serializers for other formats. Although it is not a requirement, all built-in serializers produce human-readable output. Therefore fixtures are an ideal way to keep your data and code together without tightly coupling them.

One limitation you should be aware of is that fixtures are deserialized with the absolute values of primary keys. This makes it difficult to store permissions in fixtures. Also, say, you have two different fixtures for the same model with overlapping pk values. If you load both, instances with overlapping pk‘s will be overwritten by the second load.

Test Fixtures

Fixtures are most useful during testing. They are an easy, reliable way to test your app with some data. If you prefer unittests like I do, here’s how to attach fixtures to your TestCase:

   1 from django.test import TestCase
   2 
   3 
   4 class MyTestCase(TestCase):
   5     fixtures = ['test_users', 'test_foos.json', 'test_bars.xml']
   6 
   7     # rest of your class

Here is a tip for preparing fixtures quickly:

  1. Enable admin for your app. Just registering your models should be enough.
  2. Create and/or edit test data via admin.
  3. You can also use Django shell2 for test data creation.
  4. Dump your data as intermediary fixtures3.
  5. Manually perform any final editing if necessary and save.

This method might look like a lot of work. But it is actually very practical to prepare your test fixtures, or fixtures of any kind this way. Most of the work is removing unneeded data and tweaking the primary keys in the final step. But it is still faster than manually writing the whole thing.

Initial Data

If you want to load some data right after creating your database tables, you provide initial_data fixtures. At the end of syncdb (and flush), Django automatically loads fixtures in your installed apps4 named initial_data regardless of their format.

This is good if your app require some data to function correctly. But there is a pitfall; as stated in Django documentation your initial_data fixture will be loaded every time you run syncdb. And if you edit these entries they will be overwritten next time you run syncdb. Therefore initial_data is only good for data immutable in nature, such as units of length.

If you want to supply initial data for your project it is better to create one or more bootstrap fixtures and load them manually. Here is my minimal list of things I put in my bootstrap fixture:

  • An auth.User as my superuser. It is much easier to load a superuser from fixtures than to create it interactively. I always run syncdb with --noinput and just change my superuser’s password once when I deploy.
  • A profile for my superuser.
  • A sites.site object.

And of course if all of the following fixtures will be loaded when you issue manage.py loaddata bootstrap command:

  • project_dir/myapp1/fixtures/bootstrap.json
  • project_dir/myapp1/fixtures/bootstrap.xml
  • project_dir/myapp2/fixtures/bootstrap.json
  • project_dir/myapp3/fixtures/bootstrap.yml

I think it would be nice if re-usable app authors agreed upon a convention like naming initial data fixtures bootstrap.


1: For very little amount of data this works fine. But serialization/deserialization becomes unreasonably slow for larger data sets. Then it’s better to use native tools for your database, modifying the input if necessarily.

2: Use manage.py shell command to enter Django shell.

3: Use manage.py dumpdata <app_name> > <out_file> command to dump your app data as a fixture. By default Django uses JSON format. If you add --indent=2 it will make the output much easier to read and edit.

4: Apps that are in your settings.INSTALLED_APPS.

Bookmark and Share

Piracy Tax

Tuesday, September 15th, 2009

There is a discussion going on about founding The Pirate Party in TürkiyeTurkish. Serdar Kuzuloğlu recently said what if there was a tax on sharing?Turkish. The idea is basically you pay a fixed amount of tax for the content you share, regardless of the amount or if you share at all.

This might sound nice at first. If you are pirating music, movies or whatever, and feel guilty, this way you can say I'm paying my tax, therefore my conscience is clear. Pirating is suddenly legit, content is finally free as in free speech. Moreover there is a good sum of money in the Piracy Tax pool now. Yay! We saved the music industry (or content producers in general)! No, you didn’t. You just eased your conscience.

Let’s get real. Suppose you passed the bill of Piracy Tax at the parliament and a good number of people are paying this tax. That’s just one part of the equation. How will this money be distributed? What will determine the relative share between artists? Do we need distributors anymore? Do we even need producers anymore? And most importantly when you cap the total amount of earnings, will this have any negative effects on creativity? And finally which problem exactly is this Piracy Tax solving?

In the comments of Serdar Kuzuloğlu’s post, several people stated that it should be trivial to gather reliable statistics on downloads. This is not true at all. Even if there was a single distributor (I hope we agree this is undesirable) there will be statistical errors. No system is 100% reliable. But HTTP, i.e. web sites are especially not suitable for transactional operations. Aside from this fact, there will likely be a dozen of services. Are you just planning on adding the figures to calculate the final share? How about the differences in these services’ reach? Don’t you think simply adding the figures makes the system easy to game?

Something everybody seems to forget is there will be always piracy a.k.a P2P1. It would take you to the wrong conclusion to assume everybody is the same. Not everybody is using Windows, not everybody has Flash, there are even people who doesn’t have a graphical browser. Do you even know what a text browser is?2 It is not wise to ignore these people because they are the minority. These people are Internet residents, like it or not you are just tourists. They have practically invented the Internet. So please don’t make the mistake of ignoring the possible effects of technologies like tor and the culture of FLOSS.

In any case, statistical errors are unavoidable. I would suggest 3% is an acceptable error margin. Then what happens to the independent artist or the small production company that should have 0.25% and ends up having 0.025% share? Is this fair now? The pirate tax pool would likely benefit the big players and crush the independents. Big players have already been employing portfolio management strategies, independents just can’t do that. It is naive to think Piracy Tax could sweep so called low-quality productions away and elevate finer artists. It will be exactly the opposite.

Virtue consists, not in abstaining from vice, but in not desiring it. (George Bernard Shaw)

Piracy Tax will severe the already severed bonds between the artists (the producers of art) and us (the consumers) even more. When we buy a DVD or CD it is not just bread on the tables. It is also sending the message “I like what you do. Here, I even choose to give my hard earned money willingly to support your art”. Emphasis on willingly and supporting art. It’s not a passive action like listening to the radio. You (consumer) take a conscious step. Your contributions are small materially, but it should mean something to the artist even at the individual level. We need art. It is not just listening to music or whatever. But we need art to compete in the neverending race of civilization. No art, no culture. No culture, no civilization. Lack of civilization is inevitable slavery. So, we need to learn to support our artists willingly and directly. Piracy Tax is an obstacle for this social goal.

What about opting-out? If I opted out will I still be able to use legal download services? If I am not allowed to opt out but I continue pirating3 your download statistics will be further skewed. If this is being done to free the content then why do we have to give up our freedom4. If this is being done to increase the profits of content industry… Well, then it makes sense.

If the free market is a bad idea, why don’t we shift the whole economy to a controlled market? With pirate tax in effect, content industry’s income is not only centralized but falls under government control. The government controls the art. I don’t like that idea. Art should be free. In short I don’t think Piracy Tax is the way to go.

What Is My Proposal Then?

I have been thinking hard about Pirate Party and the problem of piracy. The question is not what should we do about piracy? Unless you want to treat the symptoms. The question is how will content industry adapt to the information age? We adapted to information age quite well, didn’t we? Why can’t they do the same? I don’t want to believe they are so stupid that they don’t really know how. As far as I can understand they don’t want to change. Because change has a price. And they don’t want to pay. They want us to pay the price for them stagnating like that. Let’s stop beating around the bush and see it as it is.

Smart artists are already making the move. They let you download their movies and songs for free, and find other channels of monetization. Lots of concerts for example. Not necessarily huge stadium concerts. If you take all those middlemen out of the equation you don’t have to be all that popular. All in all, it is much better than continuously whine about piracy.

My proposal is not to engage ourselves with Piracy Tax5. There has to be a better solution. At least a solution that doesn’t necessarily make big players more powerful than they are. Let’s support far-sighted artists who take the steps to adapt new conditions for now. And continue to discuss alternatives.


1: I know piracy doesn’t equal to P2P. It is possible to share content legally via P2P networks. Also P2P is not the only channel you can distribute pirate content. I just think they are interchangeable in the context of this post.

2: It does matter. Let’s not go to the extreme. It would be a waste of time and energy for me to even subscribe to a web based service. Torrents are much easier and flexible for me. And I have a graphical browser with Flash capability.

3: See 2 above.

4: Money can buy freedom, any objections?

5: I would like to remind those who would suggest Piracy Tax as a temporary solution; laws might end up being in effect for too long.

Bookmark and Share

Django: Testing Views

Wednesday, September 9th, 2009

Although I usually begin writing tests for models, if I had to pick only one component to test I would pick The View. That is because testing views will give you the best coverage for a typical Django application.

If your apps are not project specific you need to decouple tests from project particulars. While it is safe to expect a project will provide database access, assuming too much might lead to your tests failing undesirebly1. You may even end up losing data. Ideally, a reusable app should be testable under any project.

Your tests will probably assume a certain URL configuration and rendered bits of templates. If developers of the project change the URL configuration or install their own templates your tests will fail. They did nothing wrong though, it is called customization.

Specifying URLConf And Templates For Tests

The first step of isolating your view related tests is to supply a specific URL configuration for your tests. django.contrib.auth does this. It also uses a specific set of templates. This is important because an innocent {% url some_project_specific_view %} in your base template could prevent tests to run properly.

   1 class AuthViewsTestCase(TestCase):
   2     ...
   3     urls = 'django.contrib.auth.urls'
   4 
   5     def setUp(self):
   6         ...
   7         self.old_TEMPLATE_DIRS = settings.TEMPLATE_DIRS
   8         settings.TEMPLATE_DIRS = (
   9             os.path.join(
  10                 os.path.dirname(__file__),
  11                 'templates'
  12             )
  13         ,)
  14 
  15     def tearDown(self):
  16         ...
  17         settings.TEMPLATE_DIRS = self.old_TEMPLATE_DIRS

How To Reset URLConf

If your URL configuration is generated dynamically and varies on a setting, you might need to re-evaluate it during testing. One of my apps generate some URLs (effectively overriding another apps URLs) only when a certain setting is True. When testing this specific feature I would need to first set this value then reset URLConf and finally test if everything is OK.

Resetting URLConf is actually as easy as reloading related modules and calling django.core.urlresolvers.clear_url_caches:

   1 from django.core.urlresolvers import clear_url_caches
   2 from django.utils.importlib import import_module
   3 from test_settings import SettingsTestCase
   4 
   5 class FooTestCase(SettingsTestCase):
   6     urls = 'foo.test_urls'
   7 
   8     def reset_urlconf(self):
   9         reload(import_module('foo.test_urls'))
  10         clear_url_caches()
  11 
  12     def test_bar_true(self):
  13         self.settings_manager.set(BAR=True)
  14         self.reset_urlconf()
  15         # Rest of the test
  16         ...

I’m using TestSettingsManager by carljm here.

Room For Improvement

I am sure more can be done here. If you know other techniques to make tests2 more reliable and expressive please let me know.


1: OK, we never desire our tests to fail. But it is undesirable to have them failing when our code is actually valid.

2: For Django apps that is.

Bookmark and Share

Alain De Botton Talks About A Kinder, Gentler Philosophy Of Success

Friday, September 4th, 2009

Where I live, people judge you largely based on how expensive your cell phone is. It is such a sad fact that your job or how wealthy you are matters even for those who would expect no material favors from you. Not so surprisingly most of those people would speak the opposite of what they are doing.

My reaction to all this vulgarness is that I don’t buy one of those new and expensive phones. I use one until it breaks apart and then get a new one that allows only calling and SMS. Once in a while, rarely, you meet someone who doesn’t care about your accessories.

Bookmark and Share