PostgreSQL 10 Administration Cookbook Over 165 effective recipes for database management and maintenance in PostgreSQL 10 Simon Riggs Gianni Ciolli BIRMINGHAM - MUMBAI PostgreSQL 10 Administration Cookbook Copyright © 2018 Packt Publishing All rights reserved No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews Every effort has been made in the preparation of this book to ensure the accuracy of the information presented However, the information contained in this book is sold without warranty, either express or implied Neither the author(s), nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals However, Packt Publishing cannot guarantee the accuracy of this information Commissioning Editor: Amey Varangaonkar Acquisition Editor: Namrata Patil Content Development Editor: Amrita Noronha Technical Editor: Sneha Hanchate Copy Editor: Safis Project Coordinator: Shweta Birwatkar Proofreader: Safis Editing Indexer: Pratik Shirodkar Graphics: Jisha Chirayil Production Coordinator: Shraddha Falebhai First published: April 2017 Production reference: 1070518 Published by Packt Publishing Ltd Livery Place 35 Livery Street Birmingham B3 2PB, UK ISBN 978-1-78847-492-4 www.packtpub.com mapt.io Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career For more information, please visit our website Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Mapt is fully searchable Copy and paste, print, and bookmark content PacktPub.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy Get in touch with us at service@packtpub.com for more details At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks Contributors About the authors Simon Riggs is the CTO of 2ndQuadrant, having contributed to PostgreSQL as a major developer and committer for 14 years He has written and designed features for replication, performance, BI, management, and security Under his guidance, 2ndQuadrant is now a leading developer of open source PostgreSQL, serving hundreds of clients in USA, Europe, and worldwide Simon is a frequent speaker at many conferences on PostgreSQL Futures He has worked as a database architect for 30 years Gianni Ciolli is the head of professional services at 2ndQuadrant PostgreSQL consultant, trainer, and speaker at many PostgreSQL conferences in Europe and abroad since the last 10 years He has a PhD in Mathematics from the University of Florence He has worked with free and open source software since the 1990s and is active in the community (Prato Linux User Group and Italian PostgreSQL Users Group) He lives in London with his son His other interests include music, drama, poetry, and athletics See also If you want to browse the content of the WAL files, you'll need the pg_waldump program, which is an additional server-side utility Upgrading minor releases Minor release upgrades are released regularly by all software developers, and PostgreSQL has its share of corrections When a minor release occurs, we bump the last number, usually by one So the first release of a major release 10 is 10.0 The first set of bug fixes is 10.1, then 10.2, and so on This recipe is about moving from a minor release to minor release Getting ready First, get hold of the new release, by downloading either the source or fresh binaries How to it… In most cases, PostgreSQL aims for minor releases to be simple upgrades We make great efforts to keep the on-disk format the same for both data/index files and transaction log (WAL) files, but this isn't always the case; some files can change sometimes The upgrade process is as follows: Read the release notes to see whether there are any special actions that need to be taken for this particular release Make sure you consider steps required by all extensions that you have installed If you have professional support, talk to your support vendor to see whether additional safety checks over and above the upgrade instructions are required or recommended Also, verify that the target release is fully supported by your vendor on your hardware, OS, and OS release level; it may not be, yet Apply any special actions or checks; for example, if the WAL format has changed, then you may need to reconfigure log-based replication following the upgrade You may need to scan tables, rebuild indexes, or perform some other actions Not every release has such actions, but watch closely for them, because if they exist, then they are important If you are using replication, test the upgrade by disconnecting one of your standby servers from the master Follow the instructions for your OS distribution and binary packager to complete the upgrade These can vary considerably Start up the database server being used for this test, apply any post-upgrade special actions, and check that things are working for you Repeat steps to for other standby servers Repeat steps to for the primary server How it works… Minor upgrades mostly affect the binary executable files, so it should be a simple matter of replacing those files and restarting But check Major upgrades in-place PostgreSQL provides an additional supplied program, called pg_upgrade, which allows you to migrate between major releases, such as from 9.2 to 9.6; or you can upgrade straight to the latest server version These upgrades are performed in-place, meaning that we upgrade your database without moving to a new system That does sound good, but pg_upgrade has a few things that you may wish to consider as potential negatives, which are as follows: The database server must be shut down while the upgrade takes place Your system must be large enough to hold two copies of the database server: old and new copies If it's not, then you have to use the link option of pg_upgrade, or use the Major upgrades online recipe later If you use the link option on pg_upgrade, then there is no pg_downgrade utility The only option in that case is a restore from backup, and that means extended unavailability while you restore If you copy the database, then the upgrade time will be proportional to the size of the database The pg_upgrade utility does not validate all your additional add-in modules, so you will need to set up a test server and confirm that these work, ahead of performing the main upgrade The pg_upgrade utility supports versions from PostgreSQL 8.4 onwards and allows you to go straight from your current release to the latest release in one hop Getting ready Find out the size of your database (using the How much disk space does a database use? recipe in Chapter 2, Exploring the Database) If the database is large or you have an important requirement for availability, you should consider doing the major upgrade using replication tools as well Then, check out the next recipe How to it… Read the release notes for the new server version to which you are migrating, including all of the intervening releases Pay attention to the incompatibilities section carefully; PostgreSQL does change from release to release Assume this will take some hours Set up a test server with the old software release on it Restore one of your backups on it Upgrade that system to the new release to verify that there are no conflicts from software dependencies Test your application Make sure you identify and test each add-in PostgreSQL module you were using to confirm that it still works at the new release level Back up your production server Prepare for the worst; hope for the best! Most importantly, work out who you will call if things go badly, and exactly how to restore from that backup you just took Install new versions of all the required software on the production server, and create a new database server Don't disable security during the upgrade Your security team will backflips if they hear about this Keep your job! Now, go and that backup Don't skip this step; it isn't optional Check whether the backup is actually readable, accessible, and complete Shut down the database servers Run pg_upgrade -v and then run any required post-upgrade scripts Make sure you check whether any were required 10 Start up the new database server and immediately run a server-wide ANALYZE operation using vacuumdb -analyze-in-stages 11 Run through your tests to check whether it worked or you need to start performing the contingency plan 12 If all is OK, re-enable wide access to the database server Restart the applications 13 Don't delete your old server directory if you used the link method The old data directory still contains the data for the new database server Confusing! So don't get caught by this How it works… The pg_upgrade utility works by creating a new set of database catalog tables, and then creating the old objects again in the new tables using the same identifiers as before The pg_upgrade utility works easily because the data block format hasn't changed between some releases Since we can't (always) see the future, make sure you read the release notes Major upgrades online Upgrading between major releases is hard, and it should be deferred until you have some good reasons and sufficient time to get it right You can use replication tools to minimize the downtime required for an upgrade, so we refer to this recipe as an online upgrade How to it The following general steps should be followed, allowing at least a month for the complete process to ensure that everything is tested and everybody understands the implications: Set up a new release of the software on a new test system Take a standalone backup from the main system and copy it to the test system Test the applications extensively against the new release on the test system When everything works and performs correctly, then the following: Set up a connection pooler to the main database (it may be there already) Set up pglogical for all tables from old to new database servers Make sure you wait until all the initial copy tasks have completed for all tables Retest the application extensively against the new release on live data, then when ready for the final cut-over, we can the following: Pause the connection pool Switch the config of the pool over to the new system, reload Resume the connection pool (so it now accesses a new server) Downtime for the application is the length of time to these last three steps How it works The preceding recipe allows online upgrades with zero data loss because of the use of the clean switchover process There's no need for lengthy downtime during the upgrade, and there's much reduced risk in comparison with an in-place upgrade It works best with new hardware, and is a good way to upgrade the hardware or change the disk layout at the same time This procedure is also very useful for those cases where binary compatibility is not possible, such as changing server encoding, or migrating the database to a different operating system or architecture, where on-disk format will change as a result of low-level differences, such as endianness and alignment Other Books You May Enjoy If you enjoyed this book, you may be interested in these other books by Packt: Mastering PostgreSQL 10 Hans-Jürgen Schönig ISBN: 978-1-78847-229-6 Get to grips with the advanced features of PostgreSQL 10 and handle advanced SQL Make use of the indexing features in PostgreSQL and fine-tune the performance of your queries Work with stored procedures and manage backup and recovery Master replication and failover techniques Troubleshoot your PostgreSQL instance for solutions to common and not-so-common problems Learn how to migrate your database from MySQL and Oracle to PostgreSQL without any hassle Learning PostgreSQL 10 - Second Edition Salahaldin Juba, Andrey Volkov ISBN: 978-1-78839-201-3 Understand the fundamentals of relational databases, relational algebra, and data modeling Install a PostgreSQL cluster, create a database, and implement your data model Create tables and views, define indexes, and implement triggers, stored procedures, and other schema objects Use the Structured Query Language (SQL) to manipulate data in the database Implement business logic on the server side with triggers and stored procedures using PL/pgSQL Make use of advanced data types supported by PostgreSQL 10: Arrays, hstore, JSONB, and others Develop OLAP database solutions using the most recent features of PostgreSQL 10 Connect your Python applications to a PostgreSQL database and work with the data efficiently Test your database code, find bottlenecks, improve performance, and enhance the reliability of the database applications Leave a review - let other readers know what you think Please share your thoughts on this book with others by leaving a review on the site that you bought it from If you purchased the book from Amazon, please leave us an honest review on this book's Amazon page This is vital so that other potential readers can see and use your unbiased opinion to make purchasing decisions, we can understand what our customers think about our products, and our authors can see your feedback on the title that they have worked with Packt to create It will only take a few minutes of your time, but is valuable to other potential customers, our authors, and Packt Thank you! ...PostgreSQL 10 Administration Cookbook Over 165 effective recipes for database management and maintenance in PostgreSQL 10 Simon Riggs Gianni Ciolli BIRMINGHAM - MUMBAI PostgreSQL 10 Administration Cookbook. .. interested in data gathering, management, and mining; maps and mapping; business intelligence; and application of data analysis He is currently focusing on the development of data management and mining... and Diagnosis Providing PostgreSQL information to monitoring tools Finding more information about generic monitoring tools Real-time viewing using pgAdmin or OmniDB Getting ready How to it…