ݺߣ

ݺߣShare a Scribd company logo

Yatoto-technical

•
•
Ivailo Bardarov
Ivailo Bardarov

This document summarizes the Yatoto social crowdfunding platform. It provides details on its business model, which incorporates elements from lottery, Facebook, eBay, charities and Kickstarter. It discusses user and financial metrics from 2013-2016. It also describes the technical architecture including the Ruby on Rails backend, use of Docker for deployment, monitoring with Zabbix and error tracking with Errbit. Development practices like pair programming, code reviews and CI/CD are outlined.

1 of 27
Download to read offline
YATOTO SOCIAL CROWD FUNDING PLATFORM THE TECHNICAL PART You can find the most recent version online here http://yatoto.gudaso .com
BUSINESS Lottery - Different kind of rewards Facebook - Pages, Events, Friends, Chat, Main/private newsfeed Ebay - Bidding on offers Charities - Donations Kickstater - Projects Gamification - A lot of tricks to engage the users
NUMBERS (JUL 2013 - SEP 2016) More than 250 000EUR for rewards More than 100 000 Users Watched 9 Millon ads Launched in Bulgaria and the United States 4 Designs changed 3 Devs, 1 frontend, graphic designer 1.3M Publications, 90M Stream Entries 800 Accomplished Missions for ~ 1 month
BACKEND Easy to use backend interface.
Google Analytics on steroids Play with A/B Testing Smart Feature flapping Track every event on the website KNOWING OUR USERS
Ruby On Rails, PostgreSQL, Elasticsearch, Redis, Memcached, Message bus implementation on Sidekiq Ansible provisioning with multistage deployments via Capistrano/Docker Monitoring via Zabbix + Uptime Robot Zero downtime deployment within our CI/CD process TECHNOLOGY
THE APPLICATION We started with Ruby On Rails with PostgreSQL, Elasticsearch, Redis, Memcached. At first we use QEMU Virtual Machine with 9GB of RAM and it works great with +100k users. Living on a virtual machine allow us to know every CPU cycle and every disk IO. We expect 1m of users, thats why we scale on 3 servers with dockers. We have 50Milion user stream entries. For our Newsfeed stream processing we use a pipeline over Sidekiq. We have to send a lot of notifications in almost real time fashion.
THE FRONTEND For the frontend we use Twitter bootsrap, and for the backend we used Zurb Foundation. Both were good, +1 for Zurb for its usability on mobile devices. We monitor the usage of every html element on the website with google analytics, because of this we know how the users interact with the website. We decided to convert Yatoto to SPA with React. We made a small feature with React, then we remove it. The performance, and the SEO were poor, and we don't have human power to maintain it. We kept the real time chat.
THE MOBILE APP With Ionic 2, we shipped simple Yatoto application.
DEVELOPMENT
Always stay on the latest versions of everything. With Rails we start at 4.0.0.rc2 and end at 5.0.1. Adopt new technology like Nodejs/React Learn to delete code and write it from scratch Low Technical Depth THE DEVELOPER HAPPINESS
2 weeks project Yatoto a er 3+ years YES IT IS MONOLITH We have to be fast because we deliver entire new functionality every week or 1/3/6 months. We change the product a lot! It is faster to refactor than micro services architecture. We wrote Facebook, EBay, Quora, Kickstarter, GA, Games and Meme generator functionality sadly we forgot to double LinkedIn. Compared to the we done it easer and faster.Biggest Monolith
METHODOLOGY Scrum with a lot of XP The team do pair programming, have coding standard. Collective code ownership and simple design. Code reviews were more effective than writing tests. BDD, CI/CD, Small releases
SPECIFICATION Each functionality is made in mock and polished there before it enters the project
PROCESS 3400+ Tickets Sometimes we were lucky to work with bigger than one week milestones.
The tests 50min = 36min rspec + 13min cucumber 165 rubygems, (330 with dependencies) external services: geoip, epay/banking, sending sms we publish open source gems for most of our external services. The Commits from 9 developers and 3 others THE CODE Comits Developer 3781 I. B. 2303 D. N. 1721 R. B. 969 A. D. 923 A. Y. 879 I. T. 463 V. S. 323 V. N. 309 N. 65 The CEO :) 8 M. 0 K.
DEPLOYMENT
Initially was hosted on one virtual machine, 4CPU, 3GB RAM, serving > 100+K users every ActiveRecord N+1 was noticeable For the USA we launch 3 servers, expecting a lot of traffic no one came... ORCHESTRATION WITH ANSIBLE
Network Host (Security, Accounts, ntp, basic packages) Mail Docker Webserver Zabbix/ZabbixAgents Errbit Memcached Redis Elasticsearch Postgres Samba EACH WEBSITE IS ORCHESTRATED WITH ANSIBLE WITH THOSE PLAYBOOKS A er the initial effort of setup the Ansible provisioning there were no need for Sysadmin, the "configuration" was documented in the playbooks. We are stable and fast.
OUR SYSADMIN WELCOME BROCHURE When he saw it, he run away - true story! Here is the title part of the brochure.
generates rails_best_practices, brakeman, rdoc, states machines diagrams once a day tried jenkins, gitlabci, cruisecontrol.rb, drone A er successful build the CI deploys on testing. fails if we introduce library with strange license. We deploy with the Vincent Driessen's branching model. The development team uses test environments, and the business has stable staging environment. CI/CD
SPEED & AVAILABILITY
DEVELOPERS KNOW THE SPEED OF THE WEBSITE IN TEST/PRODUCTION Web pages above 100ms are considered to be slow and those urls are redirected to separate application server dedicated to process them without hurting the overall performance. When the pages are fixed they are returned to the fast application servers. We are faster in discovering problems than Newrelic services.
ERRBIT Monitor the errors from staging and testing environment so we know when someone hits the error. The errors are redirected to rocket chat so that instantly knew about them
MONITORING Zabbix installed on another data center is monitoring the website availability, the speed of the web pages and alerts via push notifications/chat the developers. UptimeRobot is monitoring Zabbix Logcheck is reporting everthing unusual
OPTIMIZATIONS Assets - Use separate IP (assets.yatoto.com) for faster loading Cache - Heavy caching on memcached and Nginx Return HTML ASAP - later process the request in the background Fight SPAM - Process the mail logs to find fake email accounts Dragonfly behind Nginx with big cache page preloading
SATISFACTION The project is huge, and we have been able to maintain and develop it during the years. There were no noticeable downtimes. Maintaining the project was with very little effort because of the automation and the tests. We have played with a lot of technologies

More Related Content

Yatoto-technical

  • 1. YATOTO SOCIAL CROWD FUNDING PLATFORM THE TECHNICAL PART You can find the most recent version online here http://yatoto.gudaso .com
  • 2. BUSINESS Lottery - Different kind of rewards Facebook - Pages, Events, Friends, Chat, Main/private newsfeed Ebay - Bidding on offers Charities - Donations Kickstater - Projects Gamification - A lot of tricks to engage the users
  • 3. NUMBERS (JUL 2013 - SEP 2016) More than 250 000EUR for rewards More than 100 000 Users Watched 9 Millon ads Launched in Bulgaria and the United States 4 Designs changed 3 Devs, 1 frontend, graphic designer 1.3M Publications, 90M Stream Entries 800 Accomplished Missions for ~ 1 month
  • 4. BACKEND Easy to use backend interface.
  • 5. Google Analytics on steroids Play with A/B Testing Smart Feature flapping Track every event on the website KNOWING OUR USERS
  • 6. Ruby On Rails, PostgreSQL, Elasticsearch, Redis, Memcached, Message bus implementation on Sidekiq Ansible provisioning with multistage deployments via Capistrano/Docker Monitoring via Zabbix + Uptime Robot Zero downtime deployment within our CI/CD process TECHNOLOGY
  • 7. THE APPLICATION We started with Ruby On Rails with PostgreSQL, Elasticsearch, Redis, Memcached. At first we use QEMU Virtual Machine with 9GB of RAM and it works great with +100k users. Living on a virtual machine allow us to know every CPU cycle and every disk IO. We expect 1m of users, thats why we scale on 3 servers with dockers. We have 50Milion user stream entries. For our Newsfeed stream processing we use a pipeline over Sidekiq. We have to send a lot of notifications in almost real time fashion.
  • 8. THE FRONTEND For the frontend we use Twitter bootsrap, and for the backend we used Zurb Foundation. Both were good, +1 for Zurb for its usability on mobile devices. We monitor the usage of every html element on the website with google analytics, because of this we know how the users interact with the website. We decided to convert Yatoto to SPA with React. We made a small feature with React, then we remove it. The performance, and the SEO were poor, and we don't have human power to maintain it. We kept the real time chat.
  • 9. THE MOBILE APP With Ionic 2, we shipped simple Yatoto application.
  • 11. Always stay on the latest versions of everything. With Rails we start at 4.0.0.rc2 and end at 5.0.1. Adopt new technology like Nodejs/React Learn to delete code and write it from scratch Low Technical Depth THE DEVELOPER HAPPINESS
  • 12. 2 weeks project Yatoto a er 3+ years YES IT IS MONOLITH We have to be fast because we deliver entire new functionality every week or 1/3/6 months. We change the product a lot! It is faster to refactor than micro services architecture. We wrote Facebook, EBay, Quora, Kickstarter, GA, Games and Meme generator functionality sadly we forgot to double LinkedIn. Compared to the we done it easer and faster.Biggest Monolith
  • 13. METHODOLOGY Scrum with a lot of XP The team do pair programming, have coding standard. Collective code ownership and simple design. Code reviews were more effective than writing tests. BDD, CI/CD, Small releases
  • 14. SPECIFICATION Each functionality is made in mock and polished there before it enters the project
  • 15. PROCESS 3400+ Tickets Sometimes we were lucky to work with bigger than one week milestones.
  • 16. The tests 50min = 36min rspec + 13min cucumber 165 rubygems, (330 with dependencies) external services: geoip, epay/banking, sending sms we publish open source gems for most of our external services. The Commits from 9 developers and 3 others THE CODE Comits Developer 3781 I. B. 2303 D. N. 1721 R. B. 969 A. D. 923 A. Y. 879 I. T. 463 V. S. 323 V. N. 309 N. 65 The CEO :) 8 M. 0 K.
  • 18. Initially was hosted on one virtual machine, 4CPU, 3GB RAM, serving > 100+K users every ActiveRecord N+1 was noticeable For the USA we launch 3 servers, expecting a lot of traffic no one came... ORCHESTRATION WITH ANSIBLE
  • 19. Network Host (Security, Accounts, ntp, basic packages) Mail Docker Webserver Zabbix/ZabbixAgents Errbit Memcached Redis Elasticsearch Postgres Samba EACH WEBSITE IS ORCHESTRATED WITH ANSIBLE WITH THOSE PLAYBOOKS A er the initial effort of setup the Ansible provisioning there were no need for Sysadmin, the "configuration" was documented in the playbooks. We are stable and fast.
  • 20. OUR SYSADMIN WELCOME BROCHURE When he saw it, he run away - true story! Here is the title part of the brochure.
  • 21. generates rails_best_practices, brakeman, rdoc, states machines diagrams once a day tried jenkins, gitlabci, cruisecontrol.rb, drone A er successful build the CI deploys on testing. fails if we introduce library with strange license. We deploy with the Vincent Driessen's branching model. The development team uses test environments, and the business has stable staging environment. CI/CD
  • 23. DEVELOPERS KNOW THE SPEED OF THE WEBSITE IN TEST/PRODUCTION Web pages above 100ms are considered to be slow and those urls are redirected to separate application server dedicated to process them without hurting the overall performance. When the pages are fixed they are returned to the fast application servers. We are faster in discovering problems than Newrelic services.
  • 24. ERRBIT Monitor the errors from staging and testing environment so we know when someone hits the error. The errors are redirected to rocket chat so that instantly knew about them
  • 25. MONITORING Zabbix installed on another data center is monitoring the website availability, the speed of the web pages and alerts via push notifications/chat the developers. UptimeRobot is monitoring Zabbix Logcheck is reporting everthing unusual
  • 26. OPTIMIZATIONS Assets - Use separate IP (assets.yatoto.com) for faster loading Cache - Heavy caching on memcached and Nginx Return HTML ASAP - later process the request in the background Fight SPAM - Process the mail logs to find fake email accounts Dragonfly behind Nginx with big cache page preloading
  • 27. SATISFACTION The project is huge, and we have been able to maintain and develop it during the years. There were no noticeable downtimes. Maintaining the project was with very little effort because of the automation and the tests. We have played with a lot of technologies