Gentoo : update PostgreSQL

Published: 22-02-2015

Updated: 20-06-2016

By: Maxime de Roucy

tags: emerge gentoo postgresql

Détails des migrations de PostgreSQL 9.4 à 9.5 et de 9.3 à 9.4. La migration de 9.3 à 9.4 a initialement été ratée (depclean trop rapide), j’explique le travail de récupération qui a suivit.

9.4 → 9.5

Sources :

Lors que cette mise à jour, j’utilisait systemd comme système d’init.

J’ai fais cette migration le 13/03/2016.

Je fais d’abord un backup de toutes mes bases, via un service personalisé.

root@server # rm /var/backup/postgresql/postgres-2016-03-13.sql
root@server # systemctl start postgresql-backup.service
root@server # ls -l /var/backup/postgresql
total 1,4M
-rw-r--r-- 1 postgres postgres 1,4M 13 mars  14:54 postgres-2016-03-13.sql

Je stop le service postgres 9.4. Le service 9.5 est déjà stoppé mais je lance la commande par acquis de conscience.

root@server # systemctl stop postgresql-9.4.service
root@server # systemctl stop postgresql-9.5.service

Je met à jours mes services et timers systemd pour dépendre de postgresql-9.5 et non plus de postgresql-9.4.

root@server # ls -l /etc/systemd/system
…
-rw-r--r-- 1 root root 280 13 mars  00:44 postgresql-backup.service
-rw-r--r-- 1 root root 148 13 mars  00:07 postgresql-maintenance.target
drwxr-xr-x 1 root root 160 13 mars  00:12 postgresql-maintenance.target.wants/
-rw-r--r-- 1 root root 150 13 mars  00:31 postgresql-maintenance.timer
-rw-r--r-- 1 root root 198 13 mars  00:44 postgresql-reindexdb.service
-rw-r--r-- 1 root root 168 13 mars  00:44 postgresql-vacuumdb.service
…
root@server # vim /etc/systemd/system/postgresql-*
6 files to edit
root@server # cat /etc/systemd/system/postgresql-maintenance.target
[Unit]
Description=postgresql maintenance
Requisite=postgresql-9.5.service
After=postgresql-9.5.service
StopWhenUnneeded=yes
root@server # systemctl daemon-reload

Je met à jour la configuration qui servira à faire le initdb.

root@server # vimdiff /etc/conf.d/postgresql-9.{4,5}

Off topic : personnelement j’utilise la ligne suivante, de manière à avoir le trie, les dates… au format français, mais les messages d’erreurs en anglais. C’est beaucoup plus pratique pour déboguer.

PG_INITDB_OPTS="--encoding=UTF8 --locale=fr_FR.UTF-8 --lc-messages=en_US.UTF-8"

Je lance la configuration initial de postgres 9.5 (faisant notamment le initdb).

root@server # emerge --config '=dev-db/postgresql-9.5.1'
	
Configuring pkg...

 * You can modify the paths and options passed to initdb by editing:
 *     /etc/conf.d/postgresql-9.5
 *
 * Information on options that can be passed to initdb are found at:
 *     https://www.postgresql.org/docs/9.5/static/creating-cluster.html
 *     https://www.postgresql.org/docs/9.5/static/app-initdb.html
 *
 * PG_INITDB_OPTS is currently set to:
 *     --encoding=UTF8 --locale=fr_FR.UTF-8 --lc-messages=en_US.UTF-8
 *
 * Configuration files will be installed to:
 *     /etc/postgresql-9.5/
 *
 * The database cluster will be created in:
 *     /var/lib/postgresql/9.5/data
 *
 * Are you ready to continue? (y/n)
y
 * Creating the data directory ...
 * Initializing the database ...
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locales
  COLLATE:  fr_FR.utf8
  CTYPE:    fr_FR.utf8
  MESSAGES: en_US.utf8
  MONETARY: fr_FR.utf8
  NUMERIC:  fr_FR.utf8
  TIME:     fr_FR.utf8
The default text search configuration will be set to "french".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/9.5/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/9.5/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /usr/lib64/postgresql-9.5/bin/pg_ctl -D /var/lib/postgresql/9.5/data -l logfile start

 * The autovacuum function, which was in contrib, has been moved to the main
 * PostgreSQL functions starting with 8.1, and starting with 8.4 is now enabled
 * by default. You can disable it in the cluster's:
 *     /etc/postgresql-9.5/postgresql.conf
 *
 * The PostgreSQL server, by default, will log events to:
 *     /var/lib/postgresql/9.5/data/postmaster.log
 *
 * You should use the '/etc/init.d/postgresql-9.5' script to run PostgreSQL
 * instead of 'pg_ctl'.

Je mets à jours la configuration du serveur postgres 9.5.

root@server # diff -rq /etc/postgresql-9.*
Files /etc/postgresql-9.4/pg_hba.conf and /etc/postgresql-9.5/pg_hba.conf differ
Files /etc/postgresql-9.4/postgresql.conf and /etc/postgresql-9.5/postgresql.conf differ
root@server # vimdiff /etc/postgresql-9.*/pg_hba.conf
root@server # vimdiff /etc/postgresql-9.*/postgresql.conf

Off topic : voici les modifications que j’ai apporté au fichier de configuration.

root@server # diff -rq postgresql-9.*
Files postgresql-9.4/postgresql.conf and postgresql-9.5/postgresql.conf differ
root@server # diff --ignore-space-change --ignore-matching-lines='^\s*#' --ignore-blank-lines postgresql-9.*/postgresql.conf
64c64
< max_connections = 50                  # (change requires restart)
---
> max_connections = 10                  # (change requires restart)

J’utilise la méthode pam dans mon fichier pg_hba.conf pour certain certaines connection. Je dois donc mettre à jours ma configuration PAM.

root@server # vimdiff /etc/pam.d/postgresql-9.{4,5}

J’utilise le type hostssl dans mon fichier pg_hba.conf pour certain certaines connection. Je dois créer les liens symbolique nécessaire (les certificats) dans le dossier de données de postgres 9.5.

root@server # ls -l /var/lib/postgresql/9.4/data
…
lrwxrwxrwx 1 root     root       30 17 déc.  21:49 server.crt -> /etc/ssl/postgresql/server.crt
lrwxrwxrwx 1 root     root       30 17 déc.  21:49 server.key -> /etc/ssl/postgresql/server.key
…
root@server # cp -a /var/lib/postgresql/9.{4,5}/data/server.crt
root@server # cp -a /var/lib/postgresql/9.{4,5}/data/server.key

J’indique à postgres d’utiliser par défault les binaires postgres 9.5.

root@server # eselect postgresql list
Available PostgreSQL Slots
  9.4 *                      9.4.5-r2
  9.5                        9.5.1
root@server # eselect postgresql set 9.5
Setting 9.5 as the default installation...
        Removing old links...done.
        Generating new links...done.
Setting 9.5 as default was successful!
root@server # eselect postgresql list
Available PostgreSQL Slots
  9.4                        9.4.5-r2
  9.5 *                      9.5.1
root@server # sudo -u postgres pg_upgrade --version
pg_upgrade (PostgreSQL) 9.5.1

Je prépare l’environnement nécessaire à pg_upgrade.

root@server # sudo -u postgres mkdir /tmp/postgres-upgrade
root@server # cd /tmp/postgres-upgrade

Je vérifie que pg_upgrade est bien capable de faire la migration.

root@server # sudo -u postgres mkdir /tmp/postgres-upgrade
root@server # cd /tmp/postgres-upgrade
root@server # sudo -u postgres pg_upgrade --check --old-bindir=/usr/lib64/postgresql-9.4/bin --old-datadir=/var/lib/postgresql/9.4/data --new-bindir=/usr/lib64/postgresql-9.5/bin --new-datadir=/var/lib/postgresql/9.5/data
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* system OID user data types                ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

*Clusters are compatible*

Je lance la migration.

root@server # sudo -u postgres pg_upgrade --old-bindir=/usr/lib64/postgresql-9.4/bin --old-datadir=/var/lib/postgresql/9.4/data --new-bindir=/usr/lib64/postgresql-9.5/bin --new-datadir=/var/lib/postgresql/9.5/data
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* system OID user data types                ok
Checking for contrib/isn with bigint-passing mismatch       ok
Creating dump of global objects                             ok
Creating dump of database schemas
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows on the new cluster                        ok
Deleting files from new pg_clog                             ok
Copying old pg_clog to new server                           ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
                                                            ok
Creating newly-required TOAST tables                        ok
Copying user relation files
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster                      ok
Creating script to delete old cluster                       ok

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh

Je démarre le service postgres 9.5, et je l’active au démarrage.

root@server # systemctl start postgresql-9.5.service
root@server # systemctl enable postgresql-9.5.service
Created symlink from /etc/systemd/system/multi-user.target.wants/postgresql-9.5.service to /usr/lib64/systemd/system/postgresql-9.5.service.

Je génère les statistiques de mes bases sur postgres 9.5.

root@server # cat ./analyze_new_cluster.sh
#!/bin/sh

echo 'This script will generate minimal optimizer statistics rapidly'
echo 'so your system is usable, and then gather statistics twice more'
echo 'with increasing accuracy.  When it is done, your system will'
echo 'have the default level of optimizer statistics.'
echo

echo 'If you have used ALTER TABLE to modify the statistics target for'
echo 'any tables, you might want to remove them and restore them after'
echo 'running this script because they will delay fast statistics generation.'
echo

echo 'If you would like default statistics as quickly as possible, cancel'
echo 'this script and run:'
echo '    "/usr/lib64/postgresql-9.5/bin/vacuumdb" --all --analyze-only'
echo

"/usr/lib64/postgresql-9.5/bin/vacuumdb" --all --analyze-in-stages
echo

echo 'Done'

root@server # sudo -u postgres ./analyze_new_cluster.sh
…
If you would like default statistics as quickly as possible, cancel
this script and run:
    "/usr/lib64/postgresql-9.5/bin/vacuumdb" --all --analyze-only

vacuumdb: processing database "collectionbd": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "davical": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "postgres": Generating minimal optimizer statistics (1 target)
…

Done

Je nettoie le système de postgres 9.4.

root@server # emerge --depclean

 * Always study the list of packages to be cleaned for any obvious
 * mistakes. Packages that are part of the world set will always
 * be kept.  They can be manually added to this set with
 * `emerge --noreplace <atom>`.  Packages that are listed in
 * package.provided (see portage(5)) will be removed by
 * depclean, even if they are part of the world set.
 *
 * As a safety measure, depclean will not remove any packages
 * unless *all* required dependencies have been resolved.  As a
 * consequence of this, it often becomes necessary to run
 * `emerge --update --newuse --deep @world` prior to depclean.

Calculating dependencies... done!
>>> Calculating removal order...

 dev-db/postgresql
    selected: 9.4.5-r2
   protected: none
     omitted: 9.5.1
…

root@server # rm -r /var/lib/postgresql/9.4
root@server # rm -r /etc/postgresql-9.4
root@server # rm /etc/pam.d/postgresql-9.4

9.3 → 9.4

Mes notes concernant ma migration ratée d’un PostgreSQL 9.3 à un PostgreSQL 9.4. À l’époque j’utilisé OpenRC.

Je suis bien passé d’une 9.3 à une 9.4 mais je n’ai pas vraiment fait de migration. J’ai été trop vite dans ma procédure de mise à jour et j’ai lancé emerge --depclean avant d’avoir migré PostgreSQL. Mes données de la 9.3 été donc certainnement corrompue et j’ai préféré repartir d’un backup de la nuit précédente.

La procédure de mise à jour de système est classique. Attention à ne pas faire comme moi ! Il ne faut pas lancer emerge --depclean avant d’avoir migré !

Par défaut je n’autorise pas l’utilisateur « postgres » à se logger. Ça pose problème lors de la migration, je rétabli donc le comportement par défaut.

root@server # grep postgres /etc/passwd
postgres:…:/sbin/nologin
root@server # vim /etc/passwd
root@server # grep postgres /etc/passwd
postgres:…:/bin/bash

On va d’abord suivre la procédure d’installation de PostgreSQL, c’est nécessaire avant de pouvoir faire la migration. On adapte la pre-configuration de PostgreSQL 9.4 par rapport à celle de la 9.3.

root@server # vimdiff /etc/conf.d/postgresql-9.{3,4}

Off topic : personnelement j’utilise la ligne suivante, de manière à avoir le trie, les dates… au format français, mais les messages d’erreurs en anglais. C’est beaucoup plus pratique pour déboguer.

PG_INITDB_OPTS="--encoding=UTF8 --locale=fr_FR.UTF-8 --lc-messages=en_US.UTF-8"

On lance la configuration et l’initialition de la base.

root@server # emerge --config '=dev-db/postgresql-9.4.1'

On adapte la configuration par rapport à ce qu’on avait avant.

root@server # cd /etc
root@server # diff -rq postgresql-9.*
Files postgresql-9.3/pg_hba.conf and postgresql-9.4/pg_hba.conf differ
Files postgresql-9.3/postgresql.conf and postgresql-9.4/postgresql.conf differ
Only in postgresql-9.4: psqlrc

/etc/postgresql-9.4/psqlrc ne contient que des commentaires.

root@server # vimdiff postgresql-9.{3,4}/pg_hba.conf
root@server # vimdiff postgresql-9.{3,4}/postgresql.conf

On teste le démarrage du serveur PostgreSQL 9.4.

root@server # /etc/init.d/postgresql-9.4 start
…
start-stop-daemon: failed to start `/usr/lib/postgresql-9.4/bin/pg_ctl'
…

Un petit tour dans les logs. J’ai oublier de migrer les certificats du serveur.

root@server # find /var/lib/postgresql/9.3/data/ -type l
/var/lib/postgresql/9.3/data/server.crt
/var/lib/postgresql/9.3/data/server.key

root@server # find /var/lib/postgresql/9.4/data/ -type l
/var/lib/postgresql/9.4/data/pg_ident.conf
/var/lib/postgresql/9.4/data/postgresql.conf
/var/lib/postgresql/9.4/data/pg_hba.conf

root@server # cp -d /var/lib/postgresql/9.{3,4}/data/server.crt
root@server # cp -d /var/lib/postgresql/9.{3,4}/data/server.key

L’option -d permet de copier les liens en non les fichiers vers lesquels ils pointent.

J’utilise PAM pour m’authentifier sur certaines bases. Je dois copier ma configuration PAM spécifique à PostgreSQL.

root@server # cp /etc/pam.d/postgresql-9.{3,4}

On teste le démarrage du serveur PostgreSQL 9.4.

root@server # /etc/init.d/postgresql-9.4 start
 * /run/postgresql: correcting mode
 * Starting PostgreSQL ... 

Comme dit plus haut, j’ai merdé lors de la mise à jour et j’ai supprimé PostgreSQL 9.3 avant de faire la migration. Du coup, j’ai préféré restauré un backup de la nuit dernière plutôt que de faire la procédure de migration.

root@server # su - postgres
$ psql -f /home/postgres_backup/2015-02-21.sql > /tmp/psql_stdout
psql:/home/postgres_backup/2015-02-21.sql:28: ERROR:  current user cannot be dropped
psql:/home/postgres_backup/2015-02-21.sql:41: ERROR:  role "postgres" already exists

Ces deux erreurs sont normal et peuvent être ignorées. J’aurai pu utiliser pg_restore à la place de psql.

Comme préconisé dans la documentation, lancement d’un analyze global pour que tout soit propre.

$ vacuumdb -a -z

Une fois qu’on a vérifié que tous fonctionne correctement on peut supprimé/sauvegardé l’ancienne installation.

root@server # oldpostgres=/home/max/oldpostgres.tar
root@server # updatedb
root@server # locate -r 'postgresql-9\.3$'
root@server # locate -r '/9\.3$'
root@server # tar cvf $oldpostgres …
root@server # rm -r …