Cтрах и ненависть в MongoDB
•Download as PPTX, PDF•
0 likes•602 views
Cтрах и ненависть в MongoDB
![MongoDB - за 5ть минут
> db.unicorns.find({gender: 'm', weight: {$gt: 700}})
>
>
> db.unicorns.insert({name: 'Horny', dob: new Date(1992,2,13,7,47), loves:
['carrot','papaya'], weight: 600, gender: 'm', vampires: 63});
>
>
> db.unicorns.update({weight: 590}, {$set: {name: 'Roooooodles', dob: new
Date(1979, 7, 18, 18, 44), loves: ['apple'], gender: 'm', vampires: 99}})
>
>
>db.unicorns.findAndModify({query: { name: "Andy" }, sort: { rating: 1 }, update: {
$inc: { score: 1 } }, upsert: true})
>](https://image.slidesharecdn.com/mongodb-141211073802-conversion-gate02/85/C-MongoDB-4-320.jpg)
![Почему не все так просто
db1 db2 db3
writer.py reader.py
# touch /etc/supervisord.d/mongo.conf
[program:mongo]
directory=/mnt/mongo
command=mongod --dbpath /mnt/mongo/ --logappend --
logpath /mnt/mongo/log --port 27017 --replSet asd
# mongo --port 27017
> rs.initiate({
_id: 'asd',
members: [
{_id: 0, host:'db1:27017'},
{_id: 1, host:'db2:27017'},
{_id: 2, host:'db3:27017'}
]
})](https://image.slidesharecdn.com/mongodb-141211073802-conversion-gate02/85/C-MongoDB-12-320.jpg)
![# cat write.py
import datetime, random, time, pymongo
con = pymongo.MongoReplicaSetClient('db1:27017,db2:27017,db3:27017', replicaSet='asd')
cl = con.test.entities
while True:
time.sleep(1)
try:
res = cl.insert({
'time': time.time(),
'value': random.random(),
'title': random.choice(['python', 'php', 'ruby', 'java', 'cpp', 'javascript', 'go', 'erlang']),
'type': random.randint(1, 5)
})
print '[', datetime.datetime.utcnow(), ']', 'wrote:', res
except pymongo.errors.AutoReconnect, e:
print '[', datetime.datetime.utcnow(), ']', 'autoreconnect error:', e
except Exception, e:
print '[', datetime.datetime.utcnow(), ']', 'error:', e](https://image.slidesharecdn.com/mongodb-141211073802-conversion-gate02/85/C-MongoDB-13-320.jpg)
![# cat read.py
import datetime, time, random, pymongo
con = pymongo.MongoReplicaSetClient('db1:27017,db2:27017,db3:27017',
replicaSet='asd')
cl = con.test.entities
while True:
time.sleep(1)
try:
res = cl.find_one({'type': random.randint(1, 5)}, sort=[("time",
pymongo.DESCENDING)])
print '[', datetime.datetime.utcnow(), ']', 'read:', res
except Exception, e:
print '[', datetime.datetime.utcnow(), ']', 'error'
print e](https://image.slidesharecdn.com/mongodb-141211073802-conversion-gate02/85/C-MongoDB-14-320.jpg)
![[ 2014-12-07 18:33:41.221093 ] wrote: 54849d85824dbc173442246c
[ 2014-12-07 18:33:47.228148 ] autoreconnect error: timed out
[ 2014-12-07 18:33:57.259425 ] autoreconnect error: No primary available
[ 2014-12-07 18:34:07.274283 ] autoreconnect error: No primary available
[ 2014-12-07 18:34:17.303580 ] autoreconnect error: No primary available
[ 2014-12-07 18:34:18.306756 ] wrote: 54849daa824dbc1734422471
^CTraceback (most recent call last):
File "write.py", line 8, in <module>
time.sleep(1)
KeyboardInterrupt
http://docs.mongodb.org/manual/faq/replica-sets/#how-long-does-replica-set-failover-take
It varies, but a replica set will select a new primary within a minute.
It may take 10-30 seconds for the members of a replica set to declare a primary
inaccessible. This triggers an election. During the election, the cluster is unavailable for
writes.
The election itself may take another 10-30 seconds.](https://image.slidesharecdn.com/mongodb-141211073802-conversion-gate02/85/C-MongoDB-18-320.jpg)
![Почему драйвер pymongo сам не повторяет
запрос при ошибке AutoReconnect?
https://jira.mongodb.org/browse/PYTHON-197
http://emptysqua.re/blog/save-the-monkey-reliably-writing-to-mongodb/
from pymongo.objectid import ObjectId
….
while True:
time.sleep(1)
data = { '_id': ObjectId(), …. }
for i in range(60):
try:
res = cl.insert(data)
print '[', datetime.datetime.utcnow(), ']', 'wrote:', res
break
except pymongo.errors.AutoReconnect, e:
print '[', datetime.datetime.utcnow(), ']', 'autoreconnect error:', e
time.sleep(5)
except pymongo.errors.DuplicateKeyError:
break](https://image.slidesharecdn.com/mongodb-141211073802-conversion-gate02/85/C-MongoDB-19-320.jpg)
Cтрах и ненависть в MongoDB
- 1. Страх и ненависть в MongoDB (Дмитрий Висков, ASD Technologies)
- 2. NoSQL vs SQL
- 3. MongoDB - за 5ть минут Collection 1 { _id: ObjectId(‘123’) title: ‘some’, value: 1 } ... Collection 2 { BSON } { BSON } { BSON } .... Collection 3 { BSON } { BSON } { BSON } .... Collection 4 { BSON } { BSON } { BSON } ....
- 4. MongoDB - за 5ть минут > db.unicorns.find({gender: 'm', weight: {$gt: 700}}) > > > db.unicorns.insert({name: 'Horny', dob: new Date(1992,2,13,7,47), loves: ['carrot','papaya'], weight: 600, gender: 'm', vampires: 63}); > > > db.unicorns.update({weight: 590}, {$set: {name: 'Roooooodles', dob: new Date(1979, 7, 18, 18, 44), loves: ['apple'], gender: 'm', vampires: 99}}) > > >db.unicorns.findAndModify({query: { name: "Andy" }, sort: { rating: 1 }, update: { $inc: { score: 1 } }, upsert: true}) >
- 5. Особенности 1.Shema less 2.Нет транзакций. Отсутствие транзакций можно компенсировать при помощи двухфазового коммита 3.Скоростная разработка и багфикс: MongoDB 2.4.3 — 23 апреля 2013 MongoDB 2.4.0 — 13 марта 2013 Postgresql 8.4.13 — 17 марта 2012 Postgresql 8.4.0 — 9 сентября 2009 1.Шардинг и репликация из коробки 2.Отсутствие JOIN-ов 3.MapReduce + Aggregation framework 4.Глобальная блокировка на запись на уровне БД: 5.MongoDB 2.0 - блокировка на уровне процесса mongod, все очень плохо MongoDB 2.2 - добавили lock yielding и сделали блокировку на уровне DB MongoDB 2.4 - усовершенствовали lock yielding - стало немного полегче MongoDB 2.7 - обещают добавить блокировку на уровене коллекций MongoDB 2.8 - долгожданная блокировка на уровне документов
- 6. Почему MongoDB нужно всегда обновлять до последней версии или история одного бага > db.entites.find({path: “/home/foo/bar/”, owner_id: 42}).pretty() { "_id" : ObjectId("54837b27bf0a898b0cbb3b78"), "path" : "/home/foo/bar/", "owner_id" : 42, "some_value_1" : 1, "some_value_2" : 2, "some_value_3" : 3 } > db.entites.ensureIndex({"path": 1, “owner_id”: 1})
- 7. MongoDB 2.4: > db.entites.insert({path: new Array(1025).join('x'), owner_id: 42, "some_value_1" : 1, "some_value_2" : 2, "some_value_3" : 3}) > db.entites.find({path: new Array(1025).join('x'), owner_id: 42}).count() 0 >db.entites.find({path: new Array(1025).join('x'), owner_id: 42}).hint({_id: 1}).pretty() { "_id" : ObjectId("54838f3434b20bf44b0cdd1f"), "path" : "aaaaaaaaaa...", "owner_id" : 42, "some_value_1" : 1, "some_value_2": 2, "some_value_3": 3 }
- 8. MongoDB 2.6: > db.entites.insert({path: new Array(1025).join('x'), owner_id: 42, "some_value_1" : 1, "some_value_2" : 2, "some_value_3" : 3}) WriteResult({ "nInserted" : 0, "writeError" : { "code" : 17280, "errmsg" : "insertDocument :: caused by :: 17280 Btree::insert: key too large to index, failing test.entites.$path_1_owner_id_1 1037 { : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx..." }" } }) http://docs.mongodb.org/manual/reference/limits/#Index-Key-Limit The total size of an index entry, which can include structural overhead depending on the BSON type, must be less than 1024 bytes.
- 9. Аналогичная ситуация в MySQL INSERT INTO entites VALUES(NULL, REPEAT('x', 1025), 'test2', 'test3'); 1 row(s) affected, 1 warning(s) Warning Code : 1265 Data truncated for column 'a' at row 1
- 10. Аналогичная ситуация в MySQL import MySQLdb import warnings warnings.filterwarnings('error', category=MySQLdb.Warning) try: cursor.execute(some_statement) except MySQLdb.Warning, e: # handle warnings, if the cursor you're using raises them except Warning, e: # handle warnings, if the cursor you're using raises them
- 11. Шардинг и репликация из коробки
- 12. Почему не все так просто db1 db2 db3 writer.py reader.py # touch /etc/supervisord.d/mongo.conf [program:mongo] directory=/mnt/mongo command=mongod --dbpath /mnt/mongo/ --logappend -- logpath /mnt/mongo/log --port 27017 --replSet asd # mongo --port 27017 > rs.initiate({ _id: 'asd', members: [ {_id: 0, host:'db1:27017'}, {_id: 1, host:'db2:27017'}, {_id: 2, host:'db3:27017'} ] })
- 13. # cat write.py import datetime, random, time, pymongo con = pymongo.MongoReplicaSetClient('db1:27017,db2:27017,db3:27017', replicaSet='asd') cl = con.test.entities while True: time.sleep(1) try: res = cl.insert({ 'time': time.time(), 'value': random.random(), 'title': random.choice(['python', 'php', 'ruby', 'java', 'cpp', 'javascript', 'go', 'erlang']), 'type': random.randint(1, 5) }) print '[', datetime.datetime.utcnow(), ']', 'wrote:', res except pymongo.errors.AutoReconnect, e: print '[', datetime.datetime.utcnow(), ']', 'autoreconnect error:', e except Exception, e: print '[', datetime.datetime.utcnow(), ']', 'error:', e
- 14. # cat read.py import datetime, time, random, pymongo con = pymongo.MongoReplicaSetClient('db1:27017,db2:27017,db3:27017', replicaSet='asd') cl = con.test.entities while True: time.sleep(1) try: res = cl.find_one({'type': random.randint(1, 5)}, sort=[("time", pymongo.DESCENDING)]) print '[', datetime.datetime.utcnow(), ']', 'read:', res except Exception, e: print '[', datetime.datetime.utcnow(), ']', 'error' print e
- 16. wrote: 54848c29824dbc0410687a43 wrote: 54848c2a824dbc0410687a44 wrote: 54848c2b824dbc0410687a45 ... wrote: 54848c30824dbc0410687a4a ^CTraceback (most recent call last): File "write.py", line 13, in <module> 'type': random.randint(1, 5) File "/usr/lib64/python2.6/site-packages/pymongo/collection.py", line 409, in insert gen(), check_keys, self.uuid_subtype, client) File "/usr/lib64/python2.6/site-packages/pymongo/message.py", line 393, in _do_batched_write_command results.append((idx_offset, send_message())) File "/usr/lib64/python2.6/site-packages/pymongo/message.py", line 345, in send_message command=True) File "/usr/lib64/python2.6/site-packages/pymongo/mongo_replica_set_client.py", line 1511, in _send_message response = self.__recv_msg(1, rqst_id, sock_info) File "/usr/lib64/python2.6/site-packages/pymongo/mongo_replica_set_client.py", line 1444, in __recv_msg header = self.__recv_data(16, sock) File "/usr/lib64/python2.6/site-packages/pymongo/mongo_replica_set_client.py", line 1432, in __recv_data chunk = sock_info.sock.recv(length) KeyboardInterrupt
- 17. con = pymongo.MongoReplicaSetClient('db1:27017,db2:27017,db3:27017', replicaSet='asd') -> con = pymongo.MongoReplicaSetClient('db1:27017,db2:27017,db3:27017', replicaSet='asd', connectTimeoutMS=5000, socketTimeoutMS=5000) -> con = pymongo.MongoReplicaSetClient('db1:27017,db2:27017,db3:27017', replicaSet='asd', connectTimeoutMS=5000, socketTimeoutMS=5000, read_preference=pymongo.ReadPreference.SECONDARY_PREFERRED)
- 18. [ 2014-12-07 18:33:41.221093 ] wrote: 54849d85824dbc173442246c [ 2014-12-07 18:33:47.228148 ] autoreconnect error: timed out [ 2014-12-07 18:33:57.259425 ] autoreconnect error: No primary available [ 2014-12-07 18:34:07.274283 ] autoreconnect error: No primary available [ 2014-12-07 18:34:17.303580 ] autoreconnect error: No primary available [ 2014-12-07 18:34:18.306756 ] wrote: 54849daa824dbc1734422471 ^CTraceback (most recent call last): File "write.py", line 8, in <module> time.sleep(1) KeyboardInterrupt http://docs.mongodb.org/manual/faq/replica-sets/#how-long-does-replica-set-failover-take It varies, but a replica set will select a new primary within a minute. It may take 10-30 seconds for the members of a replica set to declare a primary inaccessible. This triggers an election. During the election, the cluster is unavailable for writes. The election itself may take another 10-30 seconds.
- 19. Почему драйвер pymongo сам не повторяет запрос при ошибке AutoReconnect? https://jira.mongodb.org/browse/PYTHON-197 http://emptysqua.re/blog/save-the-monkey-reliably-writing-to-mongodb/ from pymongo.objectid import ObjectId …. while True: time.sleep(1) data = { '_id': ObjectId(), …. } for i in range(60): try: res = cl.insert(data) print '[', datetime.datetime.utcnow(), ']', 'wrote:', res break except pymongo.errors.AutoReconnect, e: print '[', datetime.datetime.utcnow(), ']', 'autoreconnect error:', e time.sleep(5) except pymongo.errors.DuplicateKeyError: break
- 20. Just another benchmark https://github.com/StraNNiKK/mongodb-performance-tests Рассмотрим 1 000 пользователей ●У каждого пользователя 1 000 документов. Каждый документ имеет некоторое булевое состояние ●Единовременно в несколько конкурентных процессов (N = 1..20) обновляем все 1 000 состояний для каждого юзера ●В качестве альтернативы рассмотрим аналогичную задачу на MySQL ●Две тестовые машины: ●1. MongoDB 2.4.14, Intel Core i7 3100 МГц, 8GB DDR3, HDD 5400 об/мин ●2. MongoDB 2.6.3, Intel Core i5 2500 МГц, 8GB DDR3, HDD 7200 об/мин
- 21. А вот есть такая база из эпохи динозавров, MySQL вроде бы... $ cat /etc/mysql/my.cnf .... key_buffer = 32M query_cache_limit = 16M query_cache_size = 512M innodb_buffer_pool_size = 4096M innodb_log_file_size = 512M innodb_thread_concurrency=16 key_buffer_size=3072M thread_cache=32 ( Кстати, MongoDB практически не тюнится )
- 22. mongostat
- 25. MongoDB
- 26. MySQL
- 27. Так как же хранить данные в Mongo? Схема БД идеально укладывается в идеологию жирных, слабосвязных документов (без массовых апдейтов, транзакции за счет атомарности операций над документами) 1.Желательно, чтобы приложение не удаляло много данных (т.е. удаление данных не удаляет данные с диска)
- 28. Особенности 1.Курсорная модель (не рекомендуется извлекать большой набор данных) 2.Есть небольшие проблемы в ситуациях split brain: http://aphyr.com/posts/284-call-me-maybe-mongodb 3.Сложности с сортировкой по порядку записи документов в коллекцию 4.Есть проблемы с pymongo и greenlet-ами: 5.http://emptysqua.re/blog/it-seemed-like-a-good-idea-at-the- time-pymongo-use-greenlets/ 6. --noprealloc --smallfiles
- 29. Спасибо за внимание! Вопросы? http://webenterprise.ru http://strannikk.habrahabr.ru