This document discusses using message queues like Redis as a more scalable and reliable alternative to cron jobs for processing asynchronous backend tasks. It describes how message queues allow producers to push messages into queues for consumers to process later, avoiding loading databases and allowing for easier scaling. The document recommends using Redis specifically due to its ability to share in-memory data structures like lists between processes. It also describes how to implement a reliable queue in Ruby using the redis-queue gem to push and pop messages from Redis while moving processed messages to a separate list. Finally, it notes some limitations of using Redis, such as its reliance on memory that cannot scale indefinitely.
1 of 16
Downloaded 19 times
More Related Content
Building Scalable, Distributed Job Queues with Redis and Ruby
1. When crontab must die
Francesco Laurita
ItalianCoders 19 June 2013, Milan - Italy
Building Scalable,
Distributed Job
Queues with Redis
(and Ruby)
Wednesday, June 19, 13
2. Playing with your backend
A Front End request is supposed to return a response really fast
There are certain actions that may take longer and cant be executed live
* Processing an order (fraud detection, external service call)
* Converting a video
* Resizing an Image
* In general, Not customer-facing operations that might take longer than few
milliseconds (even delivering an email)
Wednesday, June 19, 13
3. Processing Jobs: The wrong way (crontab)
It loads your database
Almost impossible to scale
Painful errors recovery
Waste of time between each schedule
Wednesday, June 19, 13
4. Processing Jobs: The wrong way (crontab)
It loads your database
Almost impossible to scale
Painful errors recovery
Waste of time between each schedule
Wednesday, June 19, 13
5. Processing Jobs like a pro: Message Queue
A producer pushes a message into a jobs queue
One ore more consumer consume messages from a job queue
LPUSH RPOP
RPUSHLPOP
Wednesday, June 19, 13
7. Our choice: Redis
What is it?
* Is a NoSQL DB
* Is an advanced Key/Value Store
* Is a caching server
* Is a lot of things...
Wednesday, June 19, 13
8. Our choice: Redis
What is it?
* It is a way to share Memory over TCP/IP
Can share memory (data structure) between di鍖erent processes and
languages
* List (LinkedList) --> queue.pop, queue.push
* Hash --> {}
* Set --> Set
* SortedSet --> SortedSet.new
* ....
Wednesday, June 19, 13
9. A real use case
Q1
Q2
Q4
Q5
S1
S2
Q3 Q6
Qi = queue i | Sj = service j
S3
Wednesday, June 19, 13
10. Simple Queue vs Reliable Queue
Simple queue:
1) pushing values into a queue
2) a consumer waits for these values using polling or a blocking operation
* Is not reliable: a message can be lost
- Network problems
- Consumer crashes while consuming message
Wednesday, June 19, 13
11. Simple Queue vs Reliable Queue
Reliable queue:
1) pushing values into a queue
2) a consumer fetches a message and at the same time pushes it into a
processing list
3) remove the message from the processing list once message has been
processed
Wednesday, June 19, 13
12. Simple Queue vs Reliable Queue
Messages queueProducer Consumer
Processing list
/dev/null
push
push
pop
lrem
Wednesday, June 19, 13
13. Reliable Queue in Ruby
https://github.com/taganaka/redis-queue
gem install redis-queue
Wednesday, June 19, 13
15. Everything seems perfect....but
Keep in mind that
Redis is an in-memory database
All of the operations are done atomically in memory
But
RAM cannot be expanded indefinitely
RAM is expensive
A queue of 1M of messages is about 370MB occupied on Redis (about 400
chars for each entry)
Wednesday, June 19, 13