際際滷

際際滷Share a Scribd company logo
Harder, better, faster, stronger.
HA with RelStorage and Postgres
abstract @ PLOG 2014simone.deponti@abstract.it /
HA?
From the not-very-realiable Wikipedia page
about it:
A = Ut / Tt
Where Ut is the uptime and Tt the total time
Three rules of HA
1. Eliminate single points of failure
2. Have a reliable failover
3. Detect failures as they occur
Its a short way to HA
Two elephants in a cluster
1. Use PostgreSQLs native Streaming
Replication
2. Means explicit master/slave roles at any
given time
3. Manual failover (slave promotion) or use
repmgr
repmgr
1. Developed by phoenicians 2ndQuadrant
2. Works in addition to streaming replication
3. Acts as watchdog and can take automatic
actions (run bash scripts)
4. You run it on the slave node
(https://github.com/2ndQuadrant/repmgr)
How it works
1. Continuously and compulsively checks
twitter that the master is alive
2. If the master is unreachable for more than
N seconds, runs a bash script
It also offers convenient command line tools
to check status of cluster, promote nodes,
syncronize.
repmgrs gotchas
1. Create a database to store replication info
(do not follow the bad example of the
documentation)
2. The suggested wal_keep_segments
setting is too high, will use up to 78GB,
can be reduced with -w option
3. Use custom promote and follow scripts
4. Launch daemon with with --monitoring-
history
Fail scenarios
1. One node has a catastrophic failure
(easy)
2. There is a total network outage (when the
network goes up again, you have a split
brain)
3. There is network partitioning (similar to
above, can be worst)
repmgr and streaming replication do no
perform too well in cases 2 and 3
Our work is never over
1. Always notify when a failure is detected
2. Investigate ASAP, even if automatic
action was taken
3. Have the slave try to exclude the master
upon promotion
Example of #3:
The slave upon promotion contacts all the
clients and tells them to avoid talking to master
RelStorage and PostgreSQL
It has a smal issue with connections IDLE IN
TRANSACTION.
Check with:
SELECT datname, usename,
query_start, state_change, state,
query
FROM pg_stat_activity;
Its not fatal, but might result in locks during
backups.
Simone Deponti
simone.deponti@abstract.it
Ad

Recommended

TDD a piccoli passi
TDD a piccoli passi
Ferdinando Santacroce
vbench: lightweight performance testing for Python
vbench: lightweight performance testing for Python
Wes McKinney
161027
161027
robo_lab
Atlas camp
Atlas camp
James Winters
Atlas camp 1
Atlas camp 1
James Winters
Implementation of Multilayer Perceptron from Scratch
Implementation of Multilayer Perceptron from Scratch
Eng Teong Cheah
Coursera Machine Learning朱 蠍郁 覦一郁鍵 : week5
Coursera Machine Learning朱 蠍郁 覦一郁鍵 : week5
Kwangsik Lee
Automated dependency updates
Automated dependency updates
Jamie Magee
On The Building Of A PostgreSQL Cluster
On The Building Of A PostgreSQL Cluster
Srihari Sriraman
Streaming Replication Made Easy in v9.3
Streaming Replication Made Easy in v9.3
Sameer Kumar
PostgreSQL HA
PostgreSQL HA
haroonm
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRest
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRest
PGDay.Amsterdam
Out of the box replication in postgres 9.4(pg confus)
Out of the box replication in postgres 9.4(pg confus)
Denish Patel
Out of the Box Replication in Postgres 9.4(PgConfUS)
Out of the Box Replication in Postgres 9.4(PgConfUS)
Denish Patel
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Command Prompt., Inc
Out of the Box Replication in Postgres 9.4(pgconfsf)
Out of the Box Replication in Postgres 9.4(pgconfsf)
Denish Patel
Built in physical and logical replication in postgresql-Firat Gulec
Built in physical and logical replication in postgresql-Firat Gulec
FIRAT GULEC
PostgreSQL Sharding and HA: Theory and Practice (PGConf.ASIA 2017)
PostgreSQL Sharding and HA: Theory and Practice (PGConf.ASIA 2017)
Aleksander Alekseev
PostgreSQL High Availability in a Containerized World
PostgreSQL High Availability in a Containerized World
Jignesh Shah
How to Replicate PostgreSQL Database
How to Replicate PostgreSQL Database
SangJin Kang
ProstgreSQLFailoverConfiguration
ProstgreSQLFailoverConfiguration
Suyog Shirgaonkar
Fail over fail_back
Fail over fail_back
PostgreSQL Experts, Inc.
High Availability PostgreSQL with Zalando Patroni
High Availability PostgreSQL with Zalando Patroni
Zalando Technology
PostgreSQL Streaming Replication Cheatsheet
PostgreSQL Streaming Replication Cheatsheet
Alexey Lesovsky
Patroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easy
Alexander Kukushkin
Out of the box replication in postgres 9.4
Out of the box replication in postgres 9.4
Denish Patel
Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)
Denish Patel
Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)
Denish Patel
Milwaukee Marketo User Group June 2025 - Optimize and Enhance Efficiency - Sm...
Milwaukee Marketo User Group June 2025 - Optimize and Enhance Efficiency - Sm...
BradBedford3
Best MLM Compensation Plans for Network Marketing Success in 2025
Best MLM Compensation Plans for Network Marketing Success in 2025
LETSCMS Pvt. Ltd.

More Related Content

Similar to HA with RelStorage and Postgres (20)

On The Building Of A PostgreSQL Cluster
On The Building Of A PostgreSQL Cluster
Srihari Sriraman
Streaming Replication Made Easy in v9.3
Streaming Replication Made Easy in v9.3
Sameer Kumar
PostgreSQL HA
PostgreSQL HA
haroonm
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRest
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRest
PGDay.Amsterdam
Out of the box replication in postgres 9.4(pg confus)
Out of the box replication in postgres 9.4(pg confus)
Denish Patel
Out of the Box Replication in Postgres 9.4(PgConfUS)
Out of the Box Replication in Postgres 9.4(PgConfUS)
Denish Patel
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Command Prompt., Inc
Out of the Box Replication in Postgres 9.4(pgconfsf)
Out of the Box Replication in Postgres 9.4(pgconfsf)
Denish Patel
Built in physical and logical replication in postgresql-Firat Gulec
Built in physical and logical replication in postgresql-Firat Gulec
FIRAT GULEC
PostgreSQL Sharding and HA: Theory and Practice (PGConf.ASIA 2017)
PostgreSQL Sharding and HA: Theory and Practice (PGConf.ASIA 2017)
Aleksander Alekseev
PostgreSQL High Availability in a Containerized World
PostgreSQL High Availability in a Containerized World
Jignesh Shah
How to Replicate PostgreSQL Database
How to Replicate PostgreSQL Database
SangJin Kang
ProstgreSQLFailoverConfiguration
ProstgreSQLFailoverConfiguration
Suyog Shirgaonkar
Fail over fail_back
Fail over fail_back
PostgreSQL Experts, Inc.
High Availability PostgreSQL with Zalando Patroni
High Availability PostgreSQL with Zalando Patroni
Zalando Technology
PostgreSQL Streaming Replication Cheatsheet
PostgreSQL Streaming Replication Cheatsheet
Alexey Lesovsky
Patroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easy
Alexander Kukushkin
Out of the box replication in postgres 9.4
Out of the box replication in postgres 9.4
Denish Patel
Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)
Denish Patel
Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)
Denish Patel
On The Building Of A PostgreSQL Cluster
On The Building Of A PostgreSQL Cluster
Srihari Sriraman
Streaming Replication Made Easy in v9.3
Streaming Replication Made Easy in v9.3
Sameer Kumar
PostgreSQL HA
PostgreSQL HA
haroonm
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRest
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRest
PGDay.Amsterdam
Out of the box replication in postgres 9.4(pg confus)
Out of the box replication in postgres 9.4(pg confus)
Denish Patel
Out of the Box Replication in Postgres 9.4(PgConfUS)
Out of the Box Replication in Postgres 9.4(PgConfUS)
Denish Patel
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...
Command Prompt., Inc
Out of the Box Replication in Postgres 9.4(pgconfsf)
Out of the Box Replication in Postgres 9.4(pgconfsf)
Denish Patel
Built in physical and logical replication in postgresql-Firat Gulec
Built in physical and logical replication in postgresql-Firat Gulec
FIRAT GULEC
PostgreSQL Sharding and HA: Theory and Practice (PGConf.ASIA 2017)
PostgreSQL Sharding and HA: Theory and Practice (PGConf.ASIA 2017)
Aleksander Alekseev
PostgreSQL High Availability in a Containerized World
PostgreSQL High Availability in a Containerized World
Jignesh Shah
How to Replicate PostgreSQL Database
How to Replicate PostgreSQL Database
SangJin Kang
ProstgreSQLFailoverConfiguration
ProstgreSQLFailoverConfiguration
Suyog Shirgaonkar
High Availability PostgreSQL with Zalando Patroni
High Availability PostgreSQL with Zalando Patroni
Zalando Technology
PostgreSQL Streaming Replication Cheatsheet
PostgreSQL Streaming Replication Cheatsheet
Alexey Lesovsky
Patroni - HA PostgreSQL made easy
Patroni - HA PostgreSQL made easy
Alexander Kukushkin
Out of the box replication in postgres 9.4
Out of the box replication in postgres 9.4
Denish Patel
Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)
Denish Patel
Out of the Box Replication in Postgres 9.4(PgCon)
Out of the Box Replication in Postgres 9.4(PgCon)
Denish Patel

Recently uploaded (20)

Milwaukee Marketo User Group June 2025 - Optimize and Enhance Efficiency - Sm...
Milwaukee Marketo User Group June 2025 - Optimize and Enhance Efficiency - Sm...
BradBedford3
Best MLM Compensation Plans for Network Marketing Success in 2025
Best MLM Compensation Plans for Network Marketing Success in 2025
LETSCMS Pvt. Ltd.
OpenChain Webinar - AboutCode - Practical Compliance in One Stack Licensing...
OpenChain Webinar - AboutCode - Practical Compliance in One Stack Licensing...
Shane Coughlan
Foundations of Marketo Engage - Programs, Campaigns & Beyond - June 2025
Foundations of Marketo Engage - Programs, Campaigns & Beyond - June 2025
BradBedford3
Sysinfo OST to PST Converter Infographic
Sysinfo OST to PST Converter Infographic
SysInfo Tools
Canva Pro Crack Free Download 2025-FREE LATEST
Canva Pro Crack Free Download 2025-FREE LATEST
grete1122g
Folding Cheat Sheet # 9 - List Unfolding as the Computational Dual of ...
Folding Cheat Sheet # 9 - List Unfolding as the Computational Dual of ...
Philip Schwarz
Making significant Software Architecture decisions
Making significant Software Architecture decisions
Bert Jan Schrijver
SAP PM Module Level-IV Training Complete.ppt
SAP PM Module Level-IV Training Complete.ppt
MuhammadShaheryar36
Who will create the languages of the future?
Who will create the languages of the future?
Jordi Cabot
openSAP_s4h27_Week_1_2025_All_際際滷s.pdf
openSAP_s4h27_Week_1_2025_All_際際滷s.pdf
Thomas Qiao
Porting Qt 5 QML Modules to Qt 6 Webinar
Porting Qt 5 QML Modules to Qt 6 Webinar
ICS
Threat Modeling a Batch Job Framework - Teri Radichel - AWS re:Inforce 2025
Threat Modeling a Batch Job Framework - Teri Radichel - AWS re:Inforce 2025
2nd Sight Lab
Emvigo Capability Deck 2025: Accelerating Innovation Through Intelligent Soft...
Emvigo Capability Deck 2025: Accelerating Innovation Through Intelligent Soft...
Emvigo Technologies
Decipher SEO Solutions for your startup needs.
Decipher SEO Solutions for your startup needs.
mathai2
Download Adobe Illustrator Crack free for Windows 2025?
Download Adobe Illustrator Crack free for Windows 2025?
grete1122g
Heat Treatment Process Automation in India
Heat Treatment Process Automation in India
Reckers Mechatronics
Y - Recursion The Hard Way GopherCon EU 2025
Y - Recursion The Hard Way GopherCon EU 2025
Eleanor McHugh
Women in Tech: Marketo Engage User Group - June 2025 - AJO with AWS
Women in Tech: Marketo Engage User Group - June 2025 - AJO with AWS
BradBedford3
arctitecture application system design os dsa
arctitecture application system design os dsa
za241967
Milwaukee Marketo User Group June 2025 - Optimize and Enhance Efficiency - Sm...
Milwaukee Marketo User Group June 2025 - Optimize and Enhance Efficiency - Sm...
BradBedford3
Best MLM Compensation Plans for Network Marketing Success in 2025
Best MLM Compensation Plans for Network Marketing Success in 2025
LETSCMS Pvt. Ltd.
OpenChain Webinar - AboutCode - Practical Compliance in One Stack Licensing...
OpenChain Webinar - AboutCode - Practical Compliance in One Stack Licensing...
Shane Coughlan
Foundations of Marketo Engage - Programs, Campaigns & Beyond - June 2025
Foundations of Marketo Engage - Programs, Campaigns & Beyond - June 2025
BradBedford3
Sysinfo OST to PST Converter Infographic
Sysinfo OST to PST Converter Infographic
SysInfo Tools
Canva Pro Crack Free Download 2025-FREE LATEST
Canva Pro Crack Free Download 2025-FREE LATEST
grete1122g
Folding Cheat Sheet # 9 - List Unfolding as the Computational Dual of ...
Folding Cheat Sheet # 9 - List Unfolding as the Computational Dual of ...
Philip Schwarz
Making significant Software Architecture decisions
Making significant Software Architecture decisions
Bert Jan Schrijver
SAP PM Module Level-IV Training Complete.ppt
SAP PM Module Level-IV Training Complete.ppt
MuhammadShaheryar36
Who will create the languages of the future?
Who will create the languages of the future?
Jordi Cabot
openSAP_s4h27_Week_1_2025_All_際際滷s.pdf
openSAP_s4h27_Week_1_2025_All_際際滷s.pdf
Thomas Qiao
Porting Qt 5 QML Modules to Qt 6 Webinar
Porting Qt 5 QML Modules to Qt 6 Webinar
ICS
Threat Modeling a Batch Job Framework - Teri Radichel - AWS re:Inforce 2025
Threat Modeling a Batch Job Framework - Teri Radichel - AWS re:Inforce 2025
2nd Sight Lab
Emvigo Capability Deck 2025: Accelerating Innovation Through Intelligent Soft...
Emvigo Capability Deck 2025: Accelerating Innovation Through Intelligent Soft...
Emvigo Technologies
Decipher SEO Solutions for your startup needs.
Decipher SEO Solutions for your startup needs.
mathai2
Download Adobe Illustrator Crack free for Windows 2025?
Download Adobe Illustrator Crack free for Windows 2025?
grete1122g
Heat Treatment Process Automation in India
Heat Treatment Process Automation in India
Reckers Mechatronics
Y - Recursion The Hard Way GopherCon EU 2025
Y - Recursion The Hard Way GopherCon EU 2025
Eleanor McHugh
Women in Tech: Marketo Engage User Group - June 2025 - AJO with AWS
Women in Tech: Marketo Engage User Group - June 2025 - AJO with AWS
BradBedford3
arctitecture application system design os dsa
arctitecture application system design os dsa
za241967
Ad

HA with RelStorage and Postgres

  • 1. Harder, better, faster, stronger. HA with RelStorage and Postgres abstract @ PLOG 2014simone.deponti@abstract.it /
  • 2. HA? From the not-very-realiable Wikipedia page about it: A = Ut / Tt Where Ut is the uptime and Tt the total time
  • 3. Three rules of HA 1. Eliminate single points of failure 2. Have a reliable failover 3. Detect failures as they occur
  • 4. Its a short way to HA
  • 5. Two elephants in a cluster 1. Use PostgreSQLs native Streaming Replication 2. Means explicit master/slave roles at any given time 3. Manual failover (slave promotion) or use repmgr
  • 6. repmgr 1. Developed by phoenicians 2ndQuadrant 2. Works in addition to streaming replication 3. Acts as watchdog and can take automatic actions (run bash scripts) 4. You run it on the slave node (https://github.com/2ndQuadrant/repmgr)
  • 7. How it works 1. Continuously and compulsively checks twitter that the master is alive 2. If the master is unreachable for more than N seconds, runs a bash script It also offers convenient command line tools to check status of cluster, promote nodes, syncronize.
  • 8. repmgrs gotchas 1. Create a database to store replication info (do not follow the bad example of the documentation) 2. The suggested wal_keep_segments setting is too high, will use up to 78GB, can be reduced with -w option 3. Use custom promote and follow scripts 4. Launch daemon with with --monitoring- history
  • 9. Fail scenarios 1. One node has a catastrophic failure (easy) 2. There is a total network outage (when the network goes up again, you have a split brain) 3. There is network partitioning (similar to above, can be worst) repmgr and streaming replication do no perform too well in cases 2 and 3
  • 10. Our work is never over 1. Always notify when a failure is detected 2. Investigate ASAP, even if automatic action was taken 3. Have the slave try to exclude the master upon promotion Example of #3: The slave upon promotion contacts all the clients and tells them to avoid talking to master
  • 11. RelStorage and PostgreSQL It has a smal issue with connections IDLE IN TRANSACTION. Check with: SELECT datname, usename, query_start, state_change, state, query FROM pg_stat_activity; Its not fatal, but might result in locks during backups.