際際滷

際際滷Share a Scribd company logo

Pycon India 12

Lakshman Prasad
Lakshman Prasad

This document discusses various stages in developing and deploying a Python web application, including development, testing, deploying, scaling, hosting, and monitoring. It covers topics like unit testing, feature testing, deployment automation, separating databases and static files from application servers, load balancing, and monitoring resources and performance. It also provides some recommendations for getting started in production and names several cloud hosting providers, monitoring tools, and resources for further reading.

1 of 43
Download to read offline
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Development, Testing, Deploying, Hosting, Monitoring of your Python Web App. Lakshman Prasad (@becomingGuru) September 18, 2011
Development is but, one part of the Application Growth Cycle
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Common non-Development Tasks Distributed log storage and analysis Backups and Snapshots Graphing, Instrumentation and Monitoring HTTP Caching, Memory Caching Failover, Node addition/removal Auto-scaling for cloud resources Data Retention/Archival, Data Model Changes, Database sharding CDN Management API Metering, Rate Limiting Handling Multiple Environments, Multiple Versions, Rollbacks
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites The Application Growth Cycle Develop Test Design Production Environment Deploy Monitor Tune
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites
17444 Pypi packages
More open source projects than you think ...
... one for every size and style
So-Starving: Same app in 19 Frameworks
So you wrote a web app.
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Jacob on Testing Code without tests is broken by design
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Unit Testing from d j a n g o . u t i l s import u n i t t e s t from myapp . m o d e l s import Animal c l a s s AnimalTestCase ( u n i t t e s t . TestCase ) : def s e t U p ( s e l f ) : s e l f . l i o n = Animal . o b j e c t s . c r e a t e ( name= l i o n , sound= r o a r ) s e l f . c a t = Animal . o b j e c t s . c r e a t e ( name= c a t , sound=meow ) def t e s t S p e a k i n g ( s e l f ) : s e l f . a s s e r t E q u a l ( s e l f . l i o n . speak ( ) , The l i o n s a y s r o a r ) s e l f . a s s e r t E q u a l ( s e l f . cat . speak ( ) , The c a t s a y s meow )
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Titus Brown on TDD and BDD I dont do test-driven development; I do stupidity-driven testing. When I do something stupid, I write a test to make sure I dont do it again.
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Feature Testing >>> from d j a n g o . t e s t . c l i e n t import C l i e n t >>> c = C l i e n t ( ) >>> r e s p o n s e = c . p o s t ( / l o g i n / , { username : j o h n , password : smith }) >>> r e s p o n s e . s t a t u s c o d e 200 >>> r e s p o n s e = c . g e t ( / c u s t o m e r / d e t a i l s / ) >>> r e s p o n s e . c o n t e n t <!DOCTYPE html . . .
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites In Browser Testing Selenium Twill
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites An idempotent deployment Automated - Fabric, Puppet, Capistrano, Buildout Isolated - virtualenv, Buildout Repeatable - pip, easyinstall Dependency Management - Yum, Deb, pip
Real World Production Environment
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites To start with
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites DB Prop
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Seperate DB machine (with pooling)
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Seperate static server
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Together called a web server
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Then, a LB and all set to Scale
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Then, scale into clusters
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Many click-hosting providers
The (now) Premium Hosting service
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Custom APIs
The BigQuery API
Prediction API
By far, the simplest possible Django Hosting
Enterprise level SLAs
Standard practices hosting
Kitchen Sink included, on EC2
Balance of Features and Flexibility
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Monitoring Nagios Monit PingDom PagerDuty, ServiceUptime, ...
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Monitor Resources, generate pretty graphs
Fire UDP Packets per each action
Pycon India 12
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites For PyCon India attendees epio invite code: pyconindia82731 django-zoom priority invites!
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites About Me Active Djangonaut and active in Python world Part of a few popular open source django applications github.com/becomingGuru, stackover鍖ow.com/users/55562 Co-Authored an ebook django-design-patterns Architect and develop django applications at InMobi Earlier, Consulting and Development via Agiliq Solutions Developed several custom proprietory django applications twitter.com/becomingGuru http://becomingguru.com
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Resources So s t a r v i n g : h t t p s : / / g i t h u b . com/ a g i l i q / sos t a r v i n g Scaling : C a l H e n d e r s o n : B u i l d i n g S c a l a b l e Web S i t e s H i g h s c a l a b i l i t y . com , K i t c h e n s o a p . com Performance : S t e v e S o u d e r s : Even F a s t e r W e b s i t e s Cloud H o s t i n g : Ken C o c h r a n e h t t p : / / k e n c o c h r a n e . n e t / b l o g / Jacob Kaplan Moss : Django i n t h e R e a l World
Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Image Attributions h t t p : / /www . f l i c k r . com/ p h o t o s / t e j e d o r o d e l u z /3157690060/ h t t p : / /www . f l i c k r . com/ p h o t o s /23820645 @N05 /4287681570/ h t t p : / /www . f l i c k r . com/ p h o t o s / a i d a n j o n e s /3575000735/ http :// j a c o b i a n . org / h t t p : / / s a n j u a n c o l l e g e . edu / l i b / i m a g e s / p h i l o s o p h y b r a i n . j p g h t t p : / /www . f l i c k r . com/ p h o t o s / uhop /105062059/ h t t p : / / s 3 . amazonaws . com/memebox/ u p l o a d s /136/ e x p o n e n t i a l g r a p h 2 . j p g h t t p : / / g e e k a n d p o k e . t y p e p a d . com/ g e e k a n d p o k e / i m a g e s /2008/06/03/ s e x p l 1 8 . j p g h t t p : / /www . f l i c k r . com/ p h o t o s / go /253819/ h t t p : / / a r o u n d t h e s p h e r e . f i l e s . w o r d p r e s s . com /2009/05/ s w i s s armyk n i f e . j p g h t t p : / /www . f r e e f o t o . com/ i m a g e s /41/04/41 0 4 9 KeepL e f t w e b . j p g h t t p : / /www . f l i c k r . com/ p h o t o s / o r i n r o b e r t j o h n /114430223/
@becomingGuru, hello@lakshmanprasad.com

More Related Content

Pycon India 12

  • 1. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Development, Testing, Deploying, Hosting, Monitoring of your Python Web App. Lakshman Prasad (@becomingGuru) September 18, 2011
  • 2. Development is but, one part of the Application Growth Cycle
  • 3. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Common non-Development Tasks Distributed log storage and analysis Backups and Snapshots Graphing, Instrumentation and Monitoring HTTP Caching, Memory Caching Failover, Node addition/removal Auto-scaling for cloud resources Data Retention/Archival, Data Model Changes, Database sharding CDN Management API Metering, Rate Limiting Handling Multiple Environments, Multiple Versions, Rollbacks
  • 4. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites The Application Growth Cycle Develop Test Design Production Environment Deploy Monitor Tune
  • 5. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites
  • 7. More open source projects than you think ...
  • 8. ... one for every size and style
  • 9. So-Starving: Same app in 19 Frameworks
  • 10. So you wrote a web app.
  • 11. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Jacob on Testing Code without tests is broken by design
  • 12. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Unit Testing from d j a n g o . u t i l s import u n i t t e s t from myapp . m o d e l s import Animal c l a s s AnimalTestCase ( u n i t t e s t . TestCase ) : def s e t U p ( s e l f ) : s e l f . l i o n = Animal . o b j e c t s . c r e a t e ( name= l i o n , sound= r o a r ) s e l f . c a t = Animal . o b j e c t s . c r e a t e ( name= c a t , sound=meow ) def t e s t S p e a k i n g ( s e l f ) : s e l f . a s s e r t E q u a l ( s e l f . l i o n . speak ( ) , The l i o n s a y s r o a r ) s e l f . a s s e r t E q u a l ( s e l f . cat . speak ( ) , The c a t s a y s meow )
  • 13. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Titus Brown on TDD and BDD I dont do test-driven development; I do stupidity-driven testing. When I do something stupid, I write a test to make sure I dont do it again.
  • 14. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Feature Testing >>> from d j a n g o . t e s t . c l i e n t import C l i e n t >>> c = C l i e n t ( ) >>> r e s p o n s e = c . p o s t ( / l o g i n / , { username : j o h n , password : smith }) >>> r e s p o n s e . s t a t u s c o d e 200 >>> r e s p o n s e = c . g e t ( / c u s t o m e r / d e t a i l s / ) >>> r e s p o n s e . c o n t e n t <!DOCTYPE html . . .
  • 15. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites In Browser Testing Selenium Twill
  • 16. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites An idempotent deployment Automated - Fabric, Puppet, Capistrano, Buildout Isolated - virtualenv, Buildout Repeatable - pip, easyinstall Dependency Management - Yum, Deb, pip
  • 17. Real World Production Environment
  • 18. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites To start with
  • 19. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites DB Prop
  • 20. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Seperate DB machine (with pooling)
  • 21. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Seperate static server
  • 22. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Together called a web server
  • 23. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Then, a LB and all set to Scale
  • 24. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Then, scale into clusters
  • 25. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Many click-hosting providers
  • 26. The (now) Premium Hosting service
  • 27. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Custom APIs
  • 30. By far, the simplest possible Django Hosting
  • 34. Balance of Features and Flexibility
  • 35. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Monitoring Nagios Monit PingDom PagerDuty, ServiceUptime, ...
  • 36. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Monitor Resources, generate pretty graphs
  • 37. Fire UDP Packets per each action
  • 39. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites For PyCon India attendees epio invite code: pyconindia82731 django-zoom priority invites!
  • 40. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites About Me Active Djangonaut and active in Python world Part of a few popular open source django applications github.com/becomingGuru, stackover鍖ow.com/users/55562 Co-Authored an ebook django-design-patterns Architect and develop django applications at InMobi Earlier, Consulting and Development via Agiliq Solutions Developed several custom proprietory django applications twitter.com/becomingGuru http://becomingguru.com
  • 41. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Resources So s t a r v i n g : h t t p s : / / g i t h u b . com/ a g i l i q / sos t a r v i n g Scaling : C a l H e n d e r s o n : B u i l d i n g S c a l a b l e Web S i t e s H i g h s c a l a b i l i t y . com , K i t c h e n s o a p . com Performance : S t e v e S o u d e r s : Even F a s t e r W e b s i t e s Cloud H o s t i n g : Ken C o c h r a n e h t t p : / / k e n c o c h r a n e . n e t / b l o g / Jacob Kaplan Moss : Django i n t h e R e a l World
  • 42. Introduction Development Testing Deploying Scaling and Performance Cloud Hosting Providers Monitoring Beta Invites Image Attributions h t t p : / /www . f l i c k r . com/ p h o t o s / t e j e d o r o d e l u z /3157690060/ h t t p : / /www . f l i c k r . com/ p h o t o s /23820645 @N05 /4287681570/ h t t p : / /www . f l i c k r . com/ p h o t o s / a i d a n j o n e s /3575000735/ http :// j a c o b i a n . org / h t t p : / / s a n j u a n c o l l e g e . edu / l i b / i m a g e s / p h i l o s o p h y b r a i n . j p g h t t p : / /www . f l i c k r . com/ p h o t o s / uhop /105062059/ h t t p : / / s 3 . amazonaws . com/memebox/ u p l o a d s /136/ e x p o n e n t i a l g r a p h 2 . j p g h t t p : / / g e e k a n d p o k e . t y p e p a d . com/ g e e k a n d p o k e / i m a g e s /2008/06/03/ s e x p l 1 8 . j p g h t t p : / /www . f l i c k r . com/ p h o t o s / go /253819/ h t t p : / / a r o u n d t h e s p h e r e . f i l e s . w o r d p r e s s . com /2009/05/ s w i s s armyk n i f e . j p g h t t p : / /www . f r e e f o t o . com/ i m a g e s /41/04/41 0 4 9 KeepL e f t w e b . j p g h t t p : / /www . f l i c k r . com/ p h o t o s / o r i n r o b e r t j o h n /114430223/