Scala Days 2011 - Rogue: A Type-Safe DSL for MongoDB
Download as KEY, PDF9 likes11,650 views
This document summarizes a presentation about Rogue, a Scala DSL for MongoDB. Some key points: - Rogue allows type-safe querying of MongoDB with features like filters, pagination, and awareness of indexes. - It uses phantom types to prevent issues like multiple selects or limits. - Queries can be logged and validated. - Future goals include iteratees for cursors, compile-time checking, and generating JavaScript for map-reduce.
![mongodb: bson example
{
¡°_id¡± :
{ ¡°oid¡± : ¡°4ddbd194686148110d5c1ccc¡± },
¡°venuename¡± : ¡°Starbucks¡±,
¡°mayorid¡± : 464,
¡°tags¡± : [¡°coffee¡±, ¡°wifi¡±, ¡°snacks¡±],
¡°latlng¡± : [39.0, -74.0]
}](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-8-320.jpg)
![mongodb: query example
{
¡°mayorid¡± : { ¡°$lte¡± : 100 },
¡°venuename¡± : ¡°Starbucks¡±,
¡°tags¡± : ¡°wifi¡±,
¡°latlng¡± : { ¡°$near¡± : [39.0, -74.0] }
}
{ ¡°_id¡± : -1 }](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-9-320.jpg)
![rogue: schema example
class Venue extends MongoRecord[Venue] {
object _id extends ObjectIdField(this)
object venuename extends StringField(this)
object mayorid extends LongField(this)
object tags extends ListField[String](this)
object latlng extends LatLngField(this)
}
object Venue extends Venue
with MongoMetaRecord[Venue]](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-12-320.jpg)
![rogue: code example
val vs: List[Venue] =
(Venue where (_.mayorid <= 100)
and (_.venuename eqs ¡°Starbucks¡±)
and (_.tags contains ¡°wifi¡±)
and (_.latlng near
(39.0, -74.0, Degrees(0.2))
orderDesc (_._id)
fetch (5))](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-13-320.jpg)
![rogue: BaseQuery
class BaseQuery[M <: MongoRecord[M], R,
Ord, Sel, Lim, Sk](...) {
def where[F](clause: M => QueryClause[F]): ...
...
}](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-14-320.jpg)
![rogue: QueryField
class QueryField[V, M <: MongoRecord[M]]
(val field: Field[V, M]) {
def eqs(v: V) = new EqClause(...)
def neqs(v: V) = ...
def in[L <% List[V]](vs: L) = ...
def nin[L <% List[V]](vs: L) = ...
def exists(b: Boolean) = ...
}](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-15-320.jpg)
![rogue: implicits
Field[F, M] => QueryField[F, M]
Field[LatLong, M] => GeoQueryField[M]
Field[List[F], M] => ListQueryField[F, M]
Field[String, M] => StringQueryField[M]
Field[Int, M] => NumericQueryField[Int, M]
Field[Double, M] => NumericQueryField[Double, M]
...](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-16-320.jpg)
![rogue: select
val vs: List[(Long, Int)] =
(Venue where (_.venuename eqs ¡°Starbucks¡±)
select (_.mayorid, _.mayorCheckins)
fetch ())](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-17-320.jpg)
![rogue: select
class BaseQuery[M <: MongoRecord[M], R,
Ord, Sel, Lim, Sk](...) {
def select[F1, F2](
f1: M => Field[F1, M],
f2: M => Field[F2, M]):
BaseQuery[M, (F1, F2), ...] = ...
...
}](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-18-320.jpg)
![rogue: select problems
val vs: List[???] =
(Venue where (_.venuename eqs ¡°Starbucks¡±)
select (_.mayorid, _.mayorCheckins)
select (_.venuename)
fetch ())](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-19-320.jpg)
![rogue: limit problems
val vs: List[Venue] =
(Venue where (_.venuename eqs ¡°Starbucks¡±)
limit (5)
limit (100) // (???)
fetch ())](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-20-320.jpg)
![rogue: initializing phantoms
class BaseQuery[M <: MongoRecord[M], R,
Ord, Sel, Lim, Sk](...) {
...
}
def dfltQuery[M <: MongoRecord[M]](rec: M) =
new BaseQuery[M, M,
Unordered,
Unselected,
Unlimited,
Unskipped](...)](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-24-320.jpg)
![rogue: select phantom
class BaseQuery[M <: MongoRecord[M], R,
Ord, Sel, Lim, Sk](...) {
def select[F1, F2](
f1: M => Field[F1, M],
f2: M => Field[F2, M])
(implicit ev: Sel =:= Unselected):
BaseQuery[M, (F1, F2),
Ord, Selected, Lim, Sk] = ...
...
}](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-25-320.jpg)
![rogue: =:=
sealed abstract class =:=[From, To] ...
implicit def tpEquals[A]: A =:= A = ...](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-26-320.jpg)
![rogue: select solutions
val vs: List[???] =
(Venue where (_.venuename eqs ¡°Starbucks¡±)
select (_.mayorid, _.mayorCheckins)
select (_.venuename)
fetch ())
// won¡¯t compile!](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-27-320.jpg)
![rogue: index-aware
val vs: List[Checkin] =
(Checkin
where (_.userid eqs 646)
and (_.venueid eqs vid)
fetch ())](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-29-320.jpg)
![rogue: index-aware
val vs: List[Checkin] =
(Checkin
where (_.userid eqs 646)
and (_.venueid eqs vid) // hidden scan!
fetch ())](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-30-320.jpg)
![rogue: index-aware
val vs: List[Checkin] =
(Checkin
where (_.userid eqs 646) // known index
scan (_.venueid eqs vid) // known scan
fetch ())](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-31-320.jpg)
![rogue: pagination
val query: Query[Venue] = ...
val vs: List[Venue] =
(query
.countPerPage(20)
.setPage(5)
.fetch())](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-33-320.jpg)
![rogue: cursors
val query: Query[Checkin] = ...
for (checkin <- query) {
... f(checkin) ...
}](https://image.slidesharecdn.com/scaladays2011-110603093737-phpapp02/85/Scala-Days-2011-Rogue-A-Type-Safe-DSL-for-MongoDB-34-320.jpg)
![[131]??? ???? ????](https://cdn.slidesharecdn.com/ss_thumbnails/131deview2018hackerperspectivesubmit-181010154431-thumbnail.jpg?width=560&fit=bounds)
More Related Content
What's hot (20)
Similar to Scala Days 2011 - Rogue: A Type-Safe DSL for MongoDB (20)
Scala Days 2011 - Rogue: A Type-Safe DSL for MongoDB
- 1. rogue: a scala dsl for mongodb ScalaDays 2011 - 6/2/2011 Jason Liszka (@jliszka) Jorge Ortiz (@jorgeortiz85)
- 2. what is foursquare? location-based social network - ¡°check-in¡± to bars, restaurants, museums, parks, etc friend-finder (where are my friends right now?) virtual game (badges, points, mayorships) city guide (local, personalized recommendations) location diary + stats engine (where was I a year ago?) specials (get rewards at your favorite restaurant)
- 3. foursquare: the numbers >9M users ~3M checkins/day >15M venues >400k merchants >60 employees
- 4. foursquare: the tech Nginx, HAProxy Scala, Lift MongoDB, PostgreSQL (legacy) (Kestrel, Munin, Ganglia, Python, Memcache, ...) All on EC2
- 5. what is mongodb? fast, schema-less document store indexes & rich queries on any field sharding, auto-balancing replication geo-indexes
- 6. mongodb: our numbers 8 clusters some sharded, some not some master/slave, some replica set ~40 machines (68.4GB, m2.4xl on EC2) 2.3 billion records ~15k QPS
- 7. mongodb: bson json++ binary wire format more built-in types: objectid date regex bina
- 8. mongodb: bson example { ¡°_id¡± : { ¡°oid¡± : ¡°4ddbd194686148110d5c1ccc¡± }, ¡°venuename¡± : ¡°Starbucks¡±, ¡°mayorid¡± : 464, ¡°tags¡± : [¡°coffee¡±, ¡°wifi¡±, ¡°snacks¡±], ¡°latlng¡± : [39.0, -74.0] }
- 9. mongodb: query example { ¡°mayorid¡± : { ¡°$lte¡± : 100 }, ¡°venuename¡± : ¡°Starbucks¡±, ¡°tags¡± : ¡°wifi¡±, ¡°latlng¡± : { ¡°$near¡± : [39.0, -74.0] } } { ¡°_id¡± : -1 }
- 10. mongodb: query example val query = (BasicDBOBjectBuilder .start .push(¡°mayorid¡±) .add(¡°$lte¡±, 100) .pop .add(¡°veneuname¡±, ¡°Starbucks¡±) ... .get)
- 11. rogue: a scala dsl for mongo type-safe all mongo query features logging & validation hooks pagination index-aware cursors http://github.com/foursquare/rogue
- 12. rogue: schema example class Venue extends MongoRecord[Venue] { object _id extends ObjectIdField(this) object venuename extends StringField(this) object mayorid extends LongField(this) object tags extends ListField[String](this) object latlng extends LatLngField(this) } object Venue extends Venue with MongoMetaRecord[Venue]
- 13. rogue: code example val vs: List[Venue] = (Venue where (_.mayorid <= 100) and (_.venuename eqs ¡°Starbucks¡±) and (_.tags contains ¡°wifi¡±) and (_.latlng near (39.0, -74.0, Degrees(0.2)) orderDesc (_._id) fetch (5))
- 14. rogue: BaseQuery class BaseQuery[M <: MongoRecord[M], R, Ord, Sel, Lim, Sk](...) { def where[F](clause: M => QueryClause[F]): ... ... }
- 15. rogue: QueryField class QueryField[V, M <: MongoRecord[M]] (val field: Field[V, M]) { def eqs(v: V) = new EqClause(...) def neqs(v: V) = ... def in[L <% List[V]](vs: L) = ... def nin[L <% List[V]](vs: L) = ... def exists(b: Boolean) = ... }
- 16. rogue: implicits Field[F, M] => QueryField[F, M] Field[LatLong, M] => GeoQueryField[M] Field[List[F], M] => ListQueryField[F, M] Field[String, M] => StringQueryField[M] Field[Int, M] => NumericQueryField[Int, M] Field[Double, M] => NumericQueryField[Double, M] ...
- 17. rogue: select val vs: List[(Long, Int)] = (Venue where (_.venuename eqs ¡°Starbucks¡±) select (_.mayorid, _.mayorCheckins) fetch ())
- 18. rogue: select class BaseQuery[M <: MongoRecord[M], R, Ord, Sel, Lim, Sk](...) { def select[F1, F2]( f1: M => Field[F1, M], f2: M => Field[F2, M]): BaseQuery[M, (F1, F2), ...] = ... ... }
- 19. rogue: select problems val vs: List[???] = (Venue where (_.venuename eqs ¡°Starbucks¡±) select (_.mayorid, _.mayorCheckins) select (_.venuename) fetch ())
- 20. rogue: limit problems val vs: List[Venue] = (Venue where (_.venuename eqs ¡°Starbucks¡±) limit (5) limit (100) // (???) fetch ())
- 21. rogue: more limit problems (Venue where (_.venuename eqs ¡°Starbucks¡±) orderAsc (_.total_checkins) limit (5) bulkDelete_!!) // How many Venues got deleted?
- 22. rogue: phantom types Sel =:= Selected, Unselected Ord =:= Ordered, Unordered Lim =:= Limited, Unlimited Sk =:= Skipped, Unskipped http://james-iry.blogspot.com/2010/10/phantom-types-in-haskell-and-scala.html
- 23. rogue: phantom types abstract sealed class Ordered abstract sealed class Unordered abstract sealed class Selected abstract sealed class Unselected abstract sealed class Limited abstract sealed class Unlimited abstract sealed class Skipped abstract sealed class Unskipped
- 24. rogue: initializing phantoms class BaseQuery[M <: MongoRecord[M], R, Ord, Sel, Lim, Sk](...) { ... } def dfltQuery[M <: MongoRecord[M]](rec: M) = new BaseQuery[M, M, Unordered, Unselected, Unlimited, Unskipped](...)
- 25. rogue: select phantom class BaseQuery[M <: MongoRecord[M], R, Ord, Sel, Lim, Sk](...) { def select[F1, F2]( f1: M => Field[F1, M], f2: M => Field[F2, M]) (implicit ev: Sel =:= Unselected): BaseQuery[M, (F1, F2), Ord, Selected, Lim, Sk] = ... ... }
- 26. rogue: =:= sealed abstract class =:=[From, To] ... implicit def tpEquals[A]: A =:= A = ...
- 27. rogue: select solutions val vs: List[???] = (Venue where (_.venuename eqs ¡°Starbucks¡±) select (_.mayorid, _.mayorCheckins) select (_.venuename) fetch ()) // won¡¯t compile!
- 28. rogue: limit solutions (Venue where (_.venuename eqs ¡°Starbucks¡±) orderAsc (_.total_checkins) limit (5) bulkDelete_!!) // won¡¯t compile!
- 29. rogue: index-aware val vs: List[Checkin] = (Checkin where (_.userid eqs 646) and (_.venueid eqs vid) fetch ())
- 30. rogue: index-aware val vs: List[Checkin] = (Checkin where (_.userid eqs 646) and (_.venueid eqs vid) // hidden scan! fetch ())
- 31. rogue: index-aware val vs: List[Checkin] = (Checkin where (_.userid eqs 646) // known index scan (_.venueid eqs vid) // known scan fetch ())
- 32. rogue: logging & validation logging: slf4j Tracer validation: radius, $in size index checks
- 33. rogue: pagination val query: Query[Venue] = ... val vs: List[Venue] = (query .countPerPage(20) .setPage(5) .fetch())
- 34. rogue: cursors val query: Query[Checkin] = ... for (checkin <- query) { ... f(checkin) ... }
- 35. rogue: future directions iteratees for cursors compile-time index checking select partial objects generate full javascript for mapreduce
- 36. we¡¯re hiring (nyc & sf) http://foursquare.com/jobs @jliszka, @jorgeortiz85