際際滷

際際滷Share a Scribd company logo
When crontab must die
Francesco Laurita
ItalianCoders 19 June 2013, Milan - Italy
Building Scalable,
Distributed Job
Queues with Redis
(and Ruby)
Wednesday, June 19, 13
Playing with your backend
A Front End request is supposed to return a response really fast
There are certain actions that may take longer and cant be executed live
* Processing an order (fraud detection, external service call)
* Converting a video
* Resizing an Image
* In general, Not customer-facing operations that might take longer than few
milliseconds (even delivering an email)
Wednesday, June 19, 13
Processing Jobs: The wrong way (crontab)
 It loads your database
 Almost impossible to scale
 Painful errors recovery
 Waste of time between each schedule
Wednesday, June 19, 13
Processing Jobs: The wrong way (crontab)
 It loads your database
 Almost impossible to scale
 Painful errors recovery
 Waste of time between each schedule
Wednesday, June 19, 13
Processing Jobs like a pro: Message Queue
 A producer pushes a message into a jobs queue
 One ore more consumer consume messages from a job queue
LPUSH RPOP
RPUSHLPOP
Wednesday, June 19, 13
Processing Jobs like a pro: Message Queue
Wednesday, June 19, 13
Our choice: Redis
What is it?
* Is a NoSQL DB
* Is an advanced Key/Value Store
* Is a caching server
* Is a lot of things...
Wednesday, June 19, 13
Our choice: Redis
What is it?
* It is a way to share Memory over TCP/IP
Can share memory (data structure) between di鍖erent processes and
languages
* List (LinkedList) --> queue.pop, queue.push
* Hash --> {}
* Set --> Set
* SortedSet --> SortedSet.new
* ....
Wednesday, June 19, 13
A real use case
Q1
Q2
Q4
Q5
S1
S2
Q3 Q6
Qi = queue i | Sj = service j
S3
Wednesday, June 19, 13
Simple Queue vs Reliable Queue
Simple queue:
1) pushing values into a queue
2) a consumer waits for these values using polling or a blocking operation
* Is not reliable: a message can be lost
- Network problems
- Consumer crashes while consuming message
Wednesday, June 19, 13
Simple Queue vs Reliable Queue
Reliable queue:
1) pushing values into a queue
2) a consumer fetches a message and at the same time pushes it into a
processing list
3) remove the message from the processing list once message has been
processed
Wednesday, June 19, 13
Simple Queue vs Reliable Queue
Messages queueProducer Consumer
Processing list
/dev/null
push
push
pop
lrem
Wednesday, June 19, 13
Reliable Queue in Ruby
https://github.com/taganaka/redis-queue
gem install redis-queue
Wednesday, June 19, 13
Reliable Queue in Ruby
S1 S2 S3 S4 Sx
Wednesday, June 19, 13
Everything seems perfect....but
Keep in mind that
 Redis is an in-memory database
 All of the operations are done atomically in memory
But
 RAM cannot be expanded indefinitely
 RAM is expensive
 A queue of 1M of messages is about 370MB occupied on Redis (about 400
chars for each entry)
Wednesday, June 19, 13
Thats all Folks...Questions?
twitter.com/鍖aurita
github.com/taganaka
francesco@gild.com
facebook.com/francesco.laurita
www.gild.com
We are hiring!
Wednesday, June 19, 13

More Related Content

Building Scalable, Distributed Job Queues with Redis and Ruby

  • 1. When crontab must die Francesco Laurita ItalianCoders 19 June 2013, Milan - Italy Building Scalable, Distributed Job Queues with Redis (and Ruby) Wednesday, June 19, 13
  • 2. Playing with your backend A Front End request is supposed to return a response really fast There are certain actions that may take longer and cant be executed live * Processing an order (fraud detection, external service call) * Converting a video * Resizing an Image * In general, Not customer-facing operations that might take longer than few milliseconds (even delivering an email) Wednesday, June 19, 13
  • 3. Processing Jobs: The wrong way (crontab) It loads your database Almost impossible to scale Painful errors recovery Waste of time between each schedule Wednesday, June 19, 13
  • 4. Processing Jobs: The wrong way (crontab) It loads your database Almost impossible to scale Painful errors recovery Waste of time between each schedule Wednesday, June 19, 13
  • 5. Processing Jobs like a pro: Message Queue A producer pushes a message into a jobs queue One ore more consumer consume messages from a job queue LPUSH RPOP RPUSHLPOP Wednesday, June 19, 13
  • 6. Processing Jobs like a pro: Message Queue Wednesday, June 19, 13
  • 7. Our choice: Redis What is it? * Is a NoSQL DB * Is an advanced Key/Value Store * Is a caching server * Is a lot of things... Wednesday, June 19, 13
  • 8. Our choice: Redis What is it? * It is a way to share Memory over TCP/IP Can share memory (data structure) between di鍖erent processes and languages * List (LinkedList) --> queue.pop, queue.push * Hash --> {} * Set --> Set * SortedSet --> SortedSet.new * .... Wednesday, June 19, 13
  • 9. A real use case Q1 Q2 Q4 Q5 S1 S2 Q3 Q6 Qi = queue i | Sj = service j S3 Wednesday, June 19, 13
  • 10. Simple Queue vs Reliable Queue Simple queue: 1) pushing values into a queue 2) a consumer waits for these values using polling or a blocking operation * Is not reliable: a message can be lost - Network problems - Consumer crashes while consuming message Wednesday, June 19, 13
  • 11. Simple Queue vs Reliable Queue Reliable queue: 1) pushing values into a queue 2) a consumer fetches a message and at the same time pushes it into a processing list 3) remove the message from the processing list once message has been processed Wednesday, June 19, 13
  • 12. Simple Queue vs Reliable Queue Messages queueProducer Consumer Processing list /dev/null push push pop lrem Wednesday, June 19, 13
  • 13. Reliable Queue in Ruby https://github.com/taganaka/redis-queue gem install redis-queue Wednesday, June 19, 13
  • 14. Reliable Queue in Ruby S1 S2 S3 S4 Sx Wednesday, June 19, 13
  • 15. Everything seems perfect....but Keep in mind that Redis is an in-memory database All of the operations are done atomically in memory But RAM cannot be expanded indefinitely RAM is expensive A queue of 1M of messages is about 370MB occupied on Redis (about 400 chars for each entry) Wednesday, June 19, 13