ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

Anatomy of a Large Django site

?
?
Andy McKay
Andy McKay

This document summarizes Andy McKay's presentation on the anatomy of Mozilla's large Django site. It discusses performance optimizations including caching, reducing SQL queries, async tasks with Celery, and localization. It also covers code reuse through libraries like Bleach and JavaScript testing. Testing, linting, and continuous integration with Hudson were also mentioned.

1 of 80
Downloaded 37 times
Anatomy of a large Django site Andy McKay Mozilla mozilla
Vancouver mozilla
Python Zope and Plone ...now at Mozilla mozilla
Using Django http://www.djangoproject.com Credit: http://www.?ickr.com/photos/abiavati/3110357974/ mozilla
1. About the site 2. Performance 3. Localisation 4. Reuse mozilla
1. About the site mozilla
mozilla
All code is open: https://github.com/jbalogh/zamboni mozilla
All* bugs are open: https://bugzilla.mozilla.org mozilla
Convert from CakePHP (remora) to Django (zamboni) mozilla
Credit: http://www.?ickr.com/photos/improbcat/4177702580/ mozilla
Changing one URL at a time from CakePHP to Django General trend to move away from PHP and do more Python and Django mozilla
How large? 250k+ addons 150 mn views month 500 mn API hits day mozilla
Lines of code PHP 40k Python 18k mozilla
Lines of code PHP 40k Python 18k Unit tests 15k mozilla
running both php and python side by side. a few issues on that No pages go out until they are faster mozilla
3 zeus load balancers 24 django (and php) 1 mysql + 4 slaves 3 memcached 3 sphinx 1 rabbitmq + 2 celeryd 1 redis master + 1 slave Credit: http://www.?ickr.com/photos/tbridge/15300843/ mozilla
2. Performance mozilla
As usual, database bottleneck mozilla
Cache machine http://bit.ly/cache-machine Credit: http://www.?ickr.com/photos/mwichary/4063534688/ mozilla
from django.db import models import caching.base class Addon(caching.base.CachingMixin, models.Model): ... status = models.IntegerField() objects = caching.base.CachingManager() available as a mixin need to addin the custom manager mozilla
>>> Addon.objects.filter(status=public) >>> len(connection.queries) 13 mozilla
>>> Addon.objects.filter(status=public) >>> len(connection.queries) 13 >>> Addon.objects.filter(status=public) >>> len(connection.queries) 13 mozilla
Invalidation mozilla
md5(¡®select... a¡¯) [addon 3615] mozilla
md5(¡®select... a¡¯) [addon 3615] addon 3615 md5(¡®select... a¡¯) mozilla
md5(¡®select... a¡¯) [addon 3615] md5(¡®select... b¡¯) [addon 3615, addon 1685] md5(¡®select... a¡¯) addon 3615 md5(¡®select... b¡¯) mozilla
Memcached rules = cache.get(3615) rules.add('select...') cache.set(3615, rules) mozilla
Redis redis.SADD(3615, ¡®select...¡¯) mozilla
Home page 20+ addons 400+ sql queries mozilla
add-on version version version ?les standard answer in django is select-related ?les ?les mozilla
django: select_related() http://bit.ly/select-related mozilla
simonw Transformer http://bit.ly/queryset-transform mozilla
@staticmethod def transformer(addons): addon_dict = dict((a.id, a) for a in addons) vs = filter(None, (a.current_version_id for a in addons) versions = list(Version.objects.filter(id__in=vs)) for version in versions: addon_dict[version.addon_id].current_version = version slightly outdated example mozilla
big SQL statements... :( 14313 character Home page 20+ addons ~14 sql queries mozilla
Update Called on startup about:con?g extensions.update.url mozilla
Incoming 8,000 req/sec Uncached 1,600 req/sec Im used to Plone in the bad old days mozilla
Incoming 8,000 req/sec Uncached 1,600 req/sec PHP 550 req/sec Im used to Plone in the bad old days mozilla
v1 Plain Django PHP Python mozilla
v2 Min. SQL queries PHP Python mozilla
oh god mozilla
v3 Django and raw SQL max-requests 200, actually we hit 210 PHP Python mozilla
v4 WSGI no Django PHP Python mozilla
v5 Pooling, optimised queries Thats 700 req/sec PHP Python which translates into mozilla
Reducing the SQL queries... doesn¡¯t always help mozilla
mySQL query cache is fast mozilla
Celery http://celeryproject.org Credit: http://www.?ickr.com/photos/chiotsrun/3843988392/ mozilla
Push things async email image processing add-on validation specifically fixing data changes bet ween php and python mozilla
from celeryutils import task @task def update_tag(tag, **kw): tag.update_stat() mozilla
from celeryutils import task @task(rate_limit='60/h') def update_tag(tag, **kw): tag.update_stat() mozilla
from tasks import update_tag update_tag.delay(tag) mozilla
Measurement mozilla
Timing Middleware http://bit.ly/timing-ware Credit: http://www.?ickr.com/photos/wwarby/3297205226/ mozilla
mozilla
3. Localization mozilla
show site in arabic? 40+ languages including rtl mozilla
content translated and templates Database strings mozilla
class Addon(caching.base.CachingMixin, models.Model): ... name = models.ForeignKey(Translation) mozilla
addon.name = 'name' addon.save() mozilla
addon.name = 'name' addon.save() addon.name = {'fr': 'la nomme'} addon.save() mozilla
Templates mozilla
Django {% blocktrans with app=request.APP %} Add-ons for {{ app }} {% endblocktrans %} mozilla
Jinja2 http://jinja.pocoo.org/ {{ _('Add-ons for {0}')|f(request.APP) }} mozilla
Python Unicode hell UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 16: ordinal not in range(128) mozilla
4. Reuse mozilla
Bleach http://pypi.python.org/pypi/ Credit: http://www.?ickr.com/photos/maisonbisson/3350954463/ mozilla
>>> bleach.clean('an <script>evil()</script> example') 'an &lt;script&gt;evil()&lt;/ script&gt; example' mozilla
>>> bleach.linkify('an http:// ex.com url') 'an <a href="http://ex.com" rel="nofollow">http://ex.com</ a> url' mozilla
Javascript tests mozilla
django-qunit http://bit.ly/django-qunit kumar mozilla
test('English', function() { ??? z.refreshL10n('en-us'); ??? equals($('textarea:visible', this.sandbox).text().trim(), ?????????? 'Firebug integrates with Firefox to put ' + 'a wealth of development tools...'); }); mozilla
test('Japanese', function() { ??? z.refreshL10n('ja'); ??? equals($('textarea:visible', this.sandbox).text().trim(), ?????????? 'Firebug Web ' + ' Firefox' + ?????????? ' '); }); mozilla
So we use hudson for CI, but haven¡¯t got the automated tests in yet Hoping to do this via jstestnet Use HudsonJenkins http://bit.ly/jstestnet mozilla
pep 8 py flakes MacCabe Flake8 http://bit.ly/?ake8 Credit: http://www.?ickr.com/photos/nebarnix/357779131/ mozilla
~/sandboxes/zamboni(632719) $ flake8 apps/editors/tasks.py apps/editors/tasks.py:1: 'datetime' imported but unused apps/editors/tasks.py:3: 'stat' imported but unused mozilla
Playdoh http://bit.ly/mozilla-playdoh fred wenzel Credit: http://www.?ickr.com/photos/ahmee/97960570/ mozilla
Basis for new projects mozilla
Celery support Jinja2 support Simple migrations By default: SHA-512 password hashing X-Frame-Options: DENY secure and httponly ?ags on cookies fred wenzel mozilla
Take inspiration from... but not the best for you for example jinja2 which makes integration with lots of django addons possible, but a bit harder mozilla
Questions? @andymckay andym@mozilla.com andym on irc.freenode.net, irc.mozilla.org mozilla

More Related Content

Anatomy of a Large Django site