ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

Pycon2012

?
?
John Moylan
John Moylan

My PyCon Ireland 2012 talk with an analysis and description of some of the various in production parts of our stack. IE, Linux, MySQL, Memcache, Django, Apache, Solr, Django-sentry, Buildout/Fabric, statsd etc. Some discussion also of alternative choices, up and coming technology, things to watch.

1 of 20
Download to read offline
Lessons Learnt. An In Production, Scaleable Django Stack PyCon ireland - Dublin, Oct 13th-14th 2012 john.moylan@rte.ie / john@8t8.eu / @redsnapper88
Brief Intro to RTE.ie Website of Ireland's national broadcaster
Problem Mostly W.O.R.M, but some stuff can't be cached :( Lot's of content (100's GB's of HTML) 10's TB of Audio and Video Lots of strands (News, Sport, Business, Ent., TV, Radio, Teletext, Archives, Young Peoples, Performing Groups...) Small team. Try to keep things as simple and efficient as possible. ...massive growth
Traffic Growth Oct 2010 V's Oct 2011 Traffic nearly doubled.
Flashmob Traffic 36Gb/s of Traffic. 41K Concurrent streams. Playlists were not cacheable........ But what a great fight.
Static? Legacy system published to static files using XML/XSL Publisher. Multiple versions per type of view. SSI's inside SSI's inside SSI's Republishing a strand could take days.
Dynamic Requirement Evaluated some systems: Wicket, Rails, Shale....Django Hmmm... nice All in package, sensible defaults. Similar problem domain.
Try Before you buy 2008 Wrote prototype video upload sharing site... (Django 0.96) and then decided to make it live. http://web.archive.org/web/20090318064215/http://www.rte.ie/tv/theden/ugc/red/video/
Try Before You Buy 2009 Original RT? Player - Written in Django. 2011 Let's use Django to make the whole site dynamic!
..continued 2012 Sport, About, RT? Player v.2, Radio, News?, Archives, playlists, lifestyle, SSO......and on and on 2013.. More
Django Delivery Linux Apache with Event MPM and mod_wsgi Mysql and MHA (tip Innodb plugin) Solr for search and 'NoSQL' Varnish (used to use Squid) LVS-NAT and LVS-DR Load-balancers. Memcache Deploying Glusterfs
Code Management Git Redmine Gitosis
Deployment Cobbler Fabric Buildout
Monitoring Monitor & Measure everything! Zenoss RRDTool MRTG Graphite Statsd - mozilla's Django-statsd Sentry Lot's of custom monitoring
In House Monitoring projects Pyvarnish https://github.com/redsnapper8t8/pyvarnish
In House Monitoring Projects Digital Dash - (open sourced soon)
Benchmarking Tool agnostic but seem to use these alot. Bonnie++ Netperf #HTTP Httperf Siege
Performance Tips 1. Memory, memory, memory 2. Caching - in memory 3. Does it scale horizontally? 4. KISS
Lessons Learnt. Read from memory whenever possible Cache, learned this one a long time ago but whenever we have issues it can usually be tracked down bad design and lack of caching. Don't hit your web servers unless you have to. Memcache, Varnish Long live Rest - GET's should not have side effects. Varnish KISS - and no resume driven development.
Thank you Questions? Also, did I mention that we are hiring?

More Related Content

Pycon2012

  • 1. Lessons Learnt. An In Production, Scaleable Django Stack PyCon ireland - Dublin, Oct 13th-14th 2012 john.moylan@rte.ie / john@8t8.eu / @redsnapper88
  • 2. Brief Intro to RTE.ie Website of Ireland's national broadcaster
  • 3. Problem Mostly W.O.R.M, but some stuff can't be cached :( Lot's of content (100's GB's of HTML) 10's TB of Audio and Video Lots of strands (News, Sport, Business, Ent., TV, Radio, Teletext, Archives, Young Peoples, Performing Groups...) Small team. Try to keep things as simple and efficient as possible. ...massive growth
  • 4. Traffic Growth Oct 2010 V's Oct 2011 Traffic nearly doubled.
  • 5. Flashmob Traffic 36Gb/s of Traffic. 41K Concurrent streams. Playlists were not cacheable........ But what a great fight.
  • 6. Static? Legacy system published to static files using XML/XSL Publisher. Multiple versions per type of view. SSI's inside SSI's inside SSI's Republishing a strand could take days.
  • 7. Dynamic Requirement Evaluated some systems: Wicket, Rails, Shale....Django Hmmm... nice All in package, sensible defaults. Similar problem domain.
  • 8. Try Before you buy 2008 Wrote prototype video upload sharing site... (Django 0.96) and then decided to make it live. http://web.archive.org/web/20090318064215/http://www.rte.ie/tv/theden/ugc/red/video/
  • 9. Try Before You Buy 2009 Original RT? Player - Written in Django. 2011 Let's use Django to make the whole site dynamic!
  • 10. ..continued 2012 Sport, About, RT? Player v.2, Radio, News?, Archives, playlists, lifestyle, SSO......and on and on 2013.. More
  • 11. Django Delivery Linux Apache with Event MPM and mod_wsgi Mysql and MHA (tip Innodb plugin) Solr for search and 'NoSQL' Varnish (used to use Squid) LVS-NAT and LVS-DR Load-balancers. Memcache Deploying Glusterfs
  • 14. Monitoring Monitor & Measure everything! Zenoss RRDTool MRTG Graphite Statsd - mozilla's Django-statsd Sentry Lot's of custom monitoring
  • 15. In House Monitoring projects Pyvarnish https://github.com/redsnapper8t8/pyvarnish
  • 16. In House Monitoring Projects Digital Dash - (open sourced soon)
  • 17. Benchmarking Tool agnostic but seem to use these alot. Bonnie++ Netperf #HTTP Httperf Siege
  • 18. Performance Tips 1. Memory, memory, memory 2. Caching - in memory 3. Does it scale horizontally? 4. KISS
  • 19. Lessons Learnt. Read from memory whenever possible Cache, learned this one a long time ago but whenever we have issues it can usually be tracked down bad design and lack of caching. Don't hit your web servers unless you have to. Memcache, Varnish Long live Rest - GET's should not have side effects. Varnish KISS - and no resume driven development.
  • 20. Thank you Questions? Also, did I mention that we are hiring?