pgstef

pgstef

PostgreSQL DBA

Member Since 4 years ago

Belgium

Experience Points
17
follower
Lessons Completed
20
follow
Lessons Completed
15
stars
Best Reply Awards
7
repos

146 contributions in the last year

Pinned
⚡ pgBackRest backup check plugin for Nagios
⚡ SELECT * from pgstef https://pgstef.github.io/about
⚡ Create a patroni cluster using Vagrant
⚡ Reliable PostgreSQL Backup & Restore
⚡ PostgreSQL documentation translated in french
⚡ Ansible code for deploying EDB Postgres database clusters and related products.
Activity
Dec
8
13 hours ago
pull request

pgstef merge to pgbackrest/pgbackrest

pgstef
pgstef

Check that clusters are alive and correctly configured during a backup.

Fail the backup if a cluster stops or the standby is promoted. Previously, shutting down the primary would cause an error but it was not detected until the end of the backup. Now the error will happen sooner and a promotion on the standby will also cause an error.

pgstef
pgstef

Hi,

I ran my usual test suite against this branch and didn't found any regression. I tried to play a little bit with it and easily encountered the error ERROR: [058]: standby is no longer is recovery by promoting the standby during the backup. So, that's another nice safety-net I think.

Didn't noticed anything special about the code itself. So, this PR looks good to me :+1:

Kind Regards

Dec
7
1 day ago
Activity icon
issue

pgstef issue comment pgbackrest/pgbackrest

pgstef
pgstef

PITR should fail and exit when the requested time target cannot be reached

Please provide the following information when submitting an issue (feature requests or general comments can skip this):

  1. pgBackRest version: 2.36

  2. PostgreSQL version: 13

  3. Operating system/version - if you have more than one server (for example, a database server, a repository host server, one or more standbys), please specify each: CentOS8

  4. Did you install pgBackRest from source or from a package? rpm

  5. Describe the issue:

When attempting a time-based PITR and the requested target time cannot be found in the repo (WARN: unable to find backup set with stop time less than XXX) it currently proceeds and does a PITR to the latest available point in time. This seems contradictory to the user's stated intent. Instead, the WARN should be an ERROR and the restore should not proceed (if I'm trying to restore to this past Monday there's a reason for that and I do not want it to restore to today).

2021-12-02 17:11:58.237 P00   INFO: restore command begin 2.36: --archive-mode=off --delta --exec-id=6218-312e6915 --log-level-console=info --log-level-file=info --pg1-path=/var/lib/pgsql/13/eppricing/data --process-max=3 --repo1-host=192.168.122.14 --repo1-host-user=postgres --spool-path=/var/lib/pgsql/13/eppricing/spool --stanza=eppricing --target="2021-12-02 11:30:00.00000+00" --target-action=promote --type=time
2021-12-02 17:11:58.704 P00   INFO: repo1: restore backup set 20211202-151223F, recovery will start at 2021-12-02 15:12:23
2021-12-02 17:12:01.027 P00   INFO: write updated /var/lib/pgsql/13/eppricing/data/postgresql.auto.conf
2021-12-02 17:12:01.044 P00   INFO: restore global/pg_control (performed last to ensure aborted restores cannot be started)
2021-12-02 17:12:01.047 P00   INFO: restore size = 31.5MB, file total = 1252
2021-12-02 17:12:01.050 P00   INFO: restore command end: completed successfully (2814ms) | stderr: WARN: --delta or --force specified but unable to find 'PG_VERSION' or 'backup.manifest' in '/var/lib/pgsql/13/eppricing/data' to confirm that this is a valid $PGDATA directory.  --delta and --force have been disabled and if any files exist in the destination directories the restore will be aborted.
WARN: unable to find backup set with stop time less than '2021-12-02 11:30:00.00000+00', repo1: latest backup set will be used
pgstef
pgstef

It has been almost two years, though, so I'm not against making a change. Thoughts @sfrost, @pgstef, @jreidthompson?

There's a change about this in PG 13 too if I remember correctly: Generate an error if recovery does not reach the specified recovery target. I would find consistent with PG to not perform the restore if we know beforehand that the target-time is too old.

I've always felt this being a bit counterintuitive too. And if we document this change correctly enough in the release notes, "old" users won't be that affected imo. So I would be in favor of making that change :+1:

Dec
6
2 days ago
Activity icon
issue

pgstef issue comment pgstef/check_pgbackrest

pgstef
pgstef

archives service will not work if the WAL files are not compressed

When running check_pgbackrest with the archives services on an uncompressed WAL (--compress-type=none), the WAL files will not be found.

This could be fixed by replacing:

my $suffix = "\.(gz|lz4|zst|xz|bz2)";

By:

my $suffix = "(\.(gz|lz4|zst|xz|bz2))?";

pgstef
pgstef

Hi,

Closing this issue since this feature has been released in version 2.2.

Kind Regards

Activity icon
issue

pgstef issue pgstef/check_pgbackrest

pgstef
pgstef

archives service will not work if the WAL files are not compressed

When running check_pgbackrest with the archives services on an uncompressed WAL (--compress-type=none), the WAL files will not be found.

This could be fixed by replacing:

my $suffix = "\.(gz|lz4|zst|xz|bz2)";

By:

my $suffix = "(\.(gz|lz4|zst|xz|bz2))?";

Activity icon
issue

pgstef issue pgstef/check_pgbackrest

pgstef
pgstef

Option for check of minimum age of oldest full backup

It would be nice to have an option similar to --retention-age-to-full for defining the minimum age of the oldest full backup.

This way one could check that a required time period for PITR is met.

Example: One full backup is created every week and the retention in pgBackRest is set to 4 weeks. Currently I can only set a limit for the age of the newest full backup. If I want to make sure, that at least 4 weeks of PITR are covered, I also need to check that the oldest full backup is at least 4 weeks old.

Activity icon
issue

pgstef issue comment pgstef/check_pgbackrest

pgstef
pgstef

Option for check of minimum age of oldest full backup

It would be nice to have an option similar to --retention-age-to-full for defining the minimum age of the oldest full backup.

This way one could check that a required time period for PITR is met.

Example: One full backup is created every week and the retention in pgBackRest is set to 4 weeks. Currently I can only set a limit for the age of the newest full backup. If I want to make sure, that at least 4 weeks of PITR are covered, I also need to check that the oldest full backup is at least 4 weeks old.

pgstef
pgstef

Hi,

Closing this issue since this feature has been released in version 2.2.

Kind Regards

Activity icon
published release check_pgbackrest 2.2

pgstef in pgstef/check_pgbackrest create published release check_pgbackrest 2.2

createdAt 2 days ago
Activity icon
created tag
createdAt 2 days ago
open pull request

pgstef wants to merge pgbackrest/pgbackrest

pgstef
pgstef

Add timeline and checkpoint checks to backup.

Add the following checks:

  1. Checkpoint is updated in pg_control after pg_start_backup(). This helps ensure that PostgreSQL and pgBackRest have a consistent view of the storage and that PGDATA paths match.
  2. Timeline of backup start WAL file matches pg_control. Hard to see how this one could get hit, but we have the power...
  3. Standby is on the same timeline as the primary. If not, this standby is not following the primary.
  4. Last standby checkpoint is not greater than the backup checkpoint. If so, this standby is not following the primary.

This also requires some additional plumbing to read/write timeline/checkpoint from pg_control and parse timelines from WAL filenames. There were some changes in the backup tests caused by the fact that pg_control now has different contents for each backup.

The check to ensure that the required checkpoint was reached on the standby should also be updated to use pg_control (it currently uses pg_control_checkpoint()), but that requires non-trivial changes to the test harness and will need to wait.

pull request

pgstef merge to pgbackrest/pgbackrest

pgstef
pgstef

Add timeline and checkpoint checks to backup.

Add the following checks:

  1. Checkpoint is updated in pg_control after pg_start_backup(). This helps ensure that PostgreSQL and pgBackRest have a consistent view of the storage and that PGDATA paths match.
  2. Timeline of backup start WAL file matches pg_control. Hard to see how this one could get hit, but we have the power...
  3. Standby is on the same timeline as the primary. If not, this standby is not following the primary.
  4. Last standby checkpoint is not greater than the backup checkpoint. If so, this standby is not following the primary.

This also requires some additional plumbing to read/write timeline/checkpoint from pg_control and parse timelines from WAL filenames. There were some changes in the backup tests caused by the fact that pg_control now has different contents for each backup.

The check to ensure that the required checkpoint was reached on the standby should also be updated to use pg_control (it currently uses pg_control_checkpoint()), but that requires non-trivial changes to the test harness and will need to wait.

Dec
3
5 days ago
Activity icon
issue

pgstef issue comment pgstef/check_pgbackrest

pgstef
pgstef

Option for check of minimum age of oldest full backup

It would be nice to have an option similar to --retention-age-to-full for defining the minimum age of the oldest full backup.

This way one could check that a required time period for PITR is met.

Example: One full backup is created every week and the retention in pgBackRest is set to 4 weeks. Currently I can only set a limit for the age of the newest full backup. If I want to make sure, that at least 4 weeks of PITR are covered, I also need to check that the oldest full backup is at least 4 weeks old.

pgstef
pgstef

Hi,

Sorry about the late answer.

The retention service has first been created to make sure pgBR expires enough old backups but the option you suggest also seems interesting.

I've added it in 69a2edf. Feel free to try it to make sure it match your expectation before the next release.

Have a good day, Kind Regards, Stefan

push

pgstef push pgstef/check_pgbackrest

pgstef
pgstef

Add retention-age-to-oldest option

commit sha: 69a2edf8e347f16d661c62b20d3611472d4800e0

push time in 5 days ago
Activity icon
issue

pgstef issue comment pgstef/check_pgbackrest

pgstef
pgstef

add retention-incr option

Add --retention-incr option

pgstef
pgstef

Hi,

I took the time to think about your suggestion. Even if I don't see the interest of those 2 options (given it will be hard to set), the code to support it is pretty small and if it could be useful to someone, it worth adding it.

I then added the retention-diff option and some regression tests to support both options. It has then been merged.

I added you as devopstales in the release notes. Let me know if you wish to change that.

Have a good day, Kind Regards

push

pgstef push pgstef/check_pgbackrest

pgstef
pgstef

Add retention-diff/incr options (#27)

  • Add retention-diff and retention-incr options
  • Refactor options per service validation

commit sha: dd5381a55ae073f19f7a077ab50a5d90b3e319ef

push time in 5 days ago
pull request

pgstef pull request pgstef/check_pgbackrest

pgstef
pgstef

add retention-incr option

Add --retention-incr option

push

pgstef push devopstales/check_pgbackrest

pgstef
pgstef

Test suite:

  • add edb-staging and pgdg apt test repos
  • support EPAS14
  • initial support steps for rockylinux
pgstef
pgstef

Test suite: PG14 on debian-like

pgstef
pgstef

Test suite: update for pgbr >2.36

pgstef
pgstef

Merge remote-tracking branch 'upstream/main' into main

pgstef
pgstef

Add retention-diff and regression tests

commit sha: 0afa036dda3be9695c8bb5b64e8070a0154569da

push time in 5 days ago
push

pgstef push pgstef/check_pgbackrest

pgstef
pgstef

Test suite: update for pgbr >2.36

commit sha: d7b9df60a52ccbb68ca5f2c125f3c5c04a327b55

push time in 5 days ago
Dec
2
6 days ago
open pull request

pgstef wants to merge pgbackrest/pgbackrest

pgstef
pgstef

Add timeline and checkpoint checks to backup.

Add the following checks:

  1. Checkpoint is updated in pg_control after pg_start_backup(). This helps ensure that PostgreSQL and pgBackRest have a consistent view of the storage and that PGDATA paths match.
  2. Timeline of backup start WAL file matches pg_control. Hard to see how this one could get hit, but we have the power...
  3. Standby is on the same timeline as the primary. If not, this standby is not following the primary.
  4. Last standby checkpoint is not greater than the backup checkpoint. If so, this standby is not following the primary.

This also requires some additional plumbing to read/write timeline/checkpoint from pg_control and parse timelines from WAL filenames. There were some changes in the backup tests caused by the fact that pg_control now has different contents for each backup.

The check to ensure that the required checkpoint was reached on the standby should also be updated to use pg_control (it currently uses pg_control_checkpoint()), but that requires non-trivial changes to the test harness and will need to wait.

pgstef
pgstef
pull request

pgstef merge to pgbackrest/pgbackrest

pgstef
pgstef

Add timeline and checkpoint checks to backup.

Add the following checks:

  1. Checkpoint is updated in pg_control after pg_start_backup(). This helps ensure that PostgreSQL and pgBackRest have a consistent view of the storage and that PGDATA paths match.
  2. Timeline of backup start WAL file matches pg_control. Hard to see how this one could get hit, but we have the power...
  3. Standby is on the same timeline as the primary. If not, this standby is not following the primary.
  4. Last standby checkpoint is not greater than the backup checkpoint. If so, this standby is not following the primary.

This also requires some additional plumbing to read/write timeline/checkpoint from pg_control and parse timelines from WAL filenames. There were some changes in the backup tests caused by the fact that pg_control now has different contents for each backup.

The check to ensure that the required checkpoint was reached on the standby should also be updated to use pg_control (it currently uses pg_control_checkpoint()), but that requires non-trivial changes to the test harness and will need to wait.

pgstef
pgstef

Hi,

Except the little question about the error message, I don't have any specific remark about the code itself.

I agree with the interest of this PR, so +1 :-)

I ran my usual test suite against this branch and didn't found any regression. I tried to play a little bit with it and successfully encountered the error ERROR: [058]: standby is on timeline 3 but expected 2 on a 3-node cluster with cascaded replication when the 1st standby is promoted. So, that's a nice safety-net I think. Not sure how to simulate any problem regarding pg_control not being updated by the checkpoint requested by pg_start_backup though...

Anyway, this PR looks good to me :-)

pull request

pgstef merge to pgbackrest/pgbackrest

pgstef
pgstef

Add timeline and checkpoint checks to backup.

Add the following checks:

  1. Checkpoint is updated in pg_control after pg_start_backup(). This helps ensure that PostgreSQL and pgBackRest have a consistent view of the storage and that PGDATA paths match.
  2. Timeline of backup start WAL file matches pg_control. Hard to see how this one could get hit, but we have the power...
  3. Standby is on the same timeline as the primary. If not, this standby is not following the primary.
  4. Last standby checkpoint is not greater than the backup checkpoint. If so, this standby is not following the primary.

This also requires some additional plumbing to read/write timeline/checkpoint from pg_control and parse timelines from WAL filenames. There were some changes in the backup tests caused by the fact that pg_control now has different contents for each backup.

The check to ensure that the required checkpoint was reached on the standby should also be updated to use pg_control (it currently uses pg_control_checkpoint()), but that requires non-trivial changes to the test harness and will need to wait.

pgstef
pgstef

Hi,

Except the little question about the error message, I don't have any specific remark about the code itself.

I agree with the interest of this PR, so +1 :-)

I ran my usual test suite against this branch and didn't found any regression. I tried to play a little bit with it and successfully encountered the error ERROR: [058]: standby is on timeline 3 but expected 2 on a 3-node cluster with cascaded replication when the 1st standby is promoted. So, that's a nice safety-net I think. Not sure how to simulate any problem regarding pg_control not being updated by the checkpoint requested by pg_start_backup though...

Anyway, this PR looks good to me :-)

Activity icon
created branch

pgstef in pgstef/check_pgbackrest create branch wip/tests-ppc64le

createdAt 6 days ago
Dec
1
1 week ago
Activity icon
issue

pgstef issue comment pgbackrest/pgbackrest

pgstef
pgstef

How to restore only differential backup?

Please provide the following information when submitting an issue (feature requests or general comments can skip this):

  1. pgBackRest version: 2.35

  2. PostgreSQL version: 12.9

  3. Operating system/version - if you have more than one server (for example, a database server, a repository host server, one or more standbys), please specify each: Ubuntu/20.04

  4. Did you install pgBackRest from source or from a package? Package

We are successfully configured the pgbackrest and we are able to take backup as well as able to restore. But now we are performing some test cases like how we can rollback to a particular differential backup.

Below are the list of backup which we have at our end:

# sudo -u postgres pgbackrest --stanza=test --log-level-console=info info
stanza: test
    status: ok
    cipher: none

    db (current)
        wal archive min/max (12): 000000120000D485000000E5/000000240000D485000000EC

        full backup: 20211130-060533F
            timestamp start/stop: 2021-11-30 06:05:33 / 2021-11-30 06:54:39
            wal start/stop: 000000240000D485000000E7 / 000000240000D485000000E7
            database size: 850.2GB, database backup size: 850.2GB
            repo1: backup set size: 149.9GB, backup size: 149.9GB

        diff backup: 20211130-060533F_20211130-065952D
            timestamp start/stop: 2021-11-30 06:59:52 / 2021-11-30 06:59:58
            wal start/stop: 000000240000D485000000E9 / 000000240000D485000000E9
            database size: 850.2GB, database backup size: 8MB
            repo1: backup set size: 149.9GB, backup size: 885KB
            backup reference list: 20211130-060533F

        diff backup: 20211130-060533F_20211130-070159D
            timestamp start/stop: 2021-11-30 07:01:59 / 2021-11-30 07:02:04
            wal start/stop: 000000240000D485000000EB / 000000240000D485000000EB
            database size: 850.2GB, database backup size: 8MB
            repo1: backup set size: 149.9GB, backup size: 885KB
            backup reference list: 20211130-060533F

Now we are trying to rollback on backup id 20211130-060533F_20211130-065952D by without doing full backup restoration.

sudo -u postgres pgbackrest --stanza=test --delta --set=20211130-060533F_20211130-065952D --target-action=promote --type=immediate restore --log-level-console=detail

But from logs it seems that its restoring the full backup first.

We are doing this test case to find out the RTO if any user by mistake drop/delete any table.

Thank You

pgstef
pgstef

Hi,

And it took 19 minutes to restore the differential backup which is quite large time period and it will affect our RTO.

Again, you're not very specific. 19 minutes for what exactly ? Only the restore command or restore + PG wal replay ? Restore a complete empty cluster or simply delta over a few days ? How much data are saved/restored ? There could be a lot of time wasted everywhere.

I believe I have to explain what a diff backup is again. If you don't have any activity, there won't be much changes, so diff backup will only copy modified files. 2 seconds seems like you only have a few files modified. It is not possible ton restore a "diff" backup only. Restoring a "diff" backup means restoring the diff + all files needed from the full to have a replay starting point for PG and it's WAL archives. It's totally not the same amount of data saved/restored.

What I mean is that you're not comparing equal situations. Take a full backup, take a diff backup. Restore the full backup and then restore the diff backup on top of it using the --delta option and you'll have the time to fetch the files "only coming from the diff backup". Restoring a entire backup set (targeting an empty pg-data) will always be slower than a diff backup.

pgBackRest can't fetch data faster than what your storage/network allows and AWS S3 is well-known to be pretty slow at fetching data. I'm not sure there's much more we can do to help you here :confused:

Kind Regards

Nov
30
1 week ago
Activity icon
issue

pgstef issue comment pgbackrest/pgbackrest

pgstef
pgstef

Per-repo configuration (compression, process-max...) ?

Please provide the following information when submitting an issue (feature requests or general comments can skip this):

  1. pgBackRest version: 2.35
  2. PostgreSQL version: 14
  3. Operating system/version : Ubuntu 18/Debian 10
  4. Did you install pgBackRest from source or from a package? Package
  5. Describe the issue:

Is it possible to configure the compression, or other options, per repo, in the configuration file ? If yes, how? I didn't find it in the documentation.

Use case examples:

  • compress quickly on a local repo, and at maximum compression on another repo with a high retention on a slow link
  • use different process-max according to the repo

I know it is possible to override on the command line, which may be the most practical after all.

pgstef
pgstef

Hi,

Atm, compression or process-max (i.e.) are general options, not configurable per repo (when using multi-repos in the same config file).

However, command-line options override environment options which override config file options. So, for commands where it's possible to specify on which repo to operate (like the backup command) you could change the process-max option in the command itself indeed.

Another example would be archive-push. The general options will apply to the command itself which will operate on all repos defined. So, there, it won't be possible to change the behavior per repo unfortunately.

Hope it helps, Kind Regards,

Activity icon
issue

pgstef issue comment pgbackrest/pgbackrest

pgstef
pgstef

is it possible to re catalog a backup accidentally expired and removed with expire command with a S3 versioned storage compatible ?

Please provide the following information when submitting an issue (feature requests or general comments can skip this):

  1. pgBackRest version:

pgBackRest 2.35

  1. PostgreSQL version:

psql (PostgreSQL) 13.4

  1. Operating system/version - if you have more than one server (for example, a database server, a repository host server, one or more standbys), please specify each:

Red Hat Enterprise Linux release 8.4

  1. Did you install pgBackRest from source or from a package?

from package

  1. Describe the issue:

Hi there, I have a question concerning the possibility to re catalog a backup when using a S3 versioned compatible storage.

If, for example, we do a mistake by expiring a backup, with a S3 versioned storage it is possible to retrieve the files after deleting the delete markers for the specific files involved in the expired backup.

But how can i recatalog the files that i just "retrieve" ?

Is it safe to restore a version of backup.info and backup.info.copy with the expired backup description and add it back to the current backup.info and backup.info.copy ?

Is the pgbackrest entry in backup.info used to verify integrity of the backup.info file ?

Thanks

pgstef
pgstef

Hi,

Imho, it's best to avoid messing with manifests manually. There is a infoBackupLoadFileReconstruct function used in the backup and expire.

I think simply copying the full backup directory to the backup repo and the execute the expire command (with the good parameters) again should allow to reconstruct the backup info.

Example:

$ pgbackrest --stanza=c7pg info
stanza: c7pg
    status: ok
    cipher: aes-256-cbc

    db (current)
        wal archive min/max (14): 00000001000000000000000D/000000010000000000000010

        full backup: 20211130-143359F
            timestamp start/stop: 2021-11-30 14:33:59 / 2021-11-30 14:34:03
            wal start/stop: 00000001000000000000000D / 00000001000000000000000D
            database size: 25MB, database backup size: 25MB
            repo1: backup set size: 3.2MB, backup size: 3.2MB

--> copying the 20211130-125309F old backup directory back to the repo

$ pgbackrest --stanza=c7pg --repo1-retention-full=2 expire
...
WARN: backup '20211130-125309F' found in repository added to backup.info
...

$ pgbackrest --stanza=c7pg info
stanza: c7pg
    status: ok
    cipher: aes-256-cbc

    db (current)
        wal archive min/max (14): 00000001000000000000000D/000000010000000000000010

        full backup: 20211130-125309F
            timestamp start/stop: 2021-11-30 12:53:09 / 2021-11-30 12:53:12
            wal start/stop: 000000010000000000000005 / 000000010000000000000005
            database size: 25MB, database backup size: 25MB
            repo1: backup set size: 3.2MB, backup size: 3.2MB

        full backup: 20211130-143359F
            timestamp start/stop: 2021-11-30 14:33:59 / 2021-11-30 14:34:03
            wal start/stop: 00000001000000000000000D / 00000001000000000000000D
            database size: 25MB, database backup size: 25MB
            repo1: backup set size: 3.2MB, backup size: 3.2MB

Kind Regards

Activity icon
issue

pgstef issue comment pgbackrest/pgbackrest

pgstef
pgstef

Info status: error (missing stanza path) and check from S3

Hello

  1. pgBackRest version:2.36 2. PostgreSQL version:13.3 3. Operating system/version - one server: Red Hat 8.5 (Ootpa)

  2. Did you install pgBackRest from source or from a package: package

  3. Please attach the following as applicable:

  • pgbackrest.conf file(s):
[main]
pg1-path=/var/lib/pgsql/data

[global]
repo1-path=/repo4
repo1-type=s3
repo1-s3-host=serv-name-bl1.xxxx.ru
repo1-s3-endpoint=xxxx.ru:9006
repo1-s3-bucket=serv-name-bl1
repo1-s3-verify-tls=n
repo1-s3-key=ххххх
repo1-s3-key-secret=хххх
repo1-s3-region=us-east-1
repo1-retention-full=14
process-max=6
compress=n
start-fast=y

[global: archive-push]
compress-level=0
compress=n
  • postgresql.conf (archive_command = 'pgbackrest --log-level-console=info --stanza=main archive-push %p' , archive_mode= 'on' , listen_addresses='*', max_wal_senders='10', wal_level='replica', port='5432')

errors in the postgresql log file before or during the time you experienced the issue

pgbackrest --stanza=main --log-level-console=info stanza-create
2021-11-30 01:55:45.804 P00   INFO: stanza-create command end: completed successfully (692ms)

pgbackrest --stanza=main --log-level-console=info backup
2021-11-30 01:58:29.685 P00   INFO: execute non-exclusive pg_start_backup(): backup begins after the requested immediate checkpoint completes
2021-11-30 01:58:30.086 P00   INFO: backup start archive = 00000001000000000000000C, lsn = 0/C000028
2021-11-30 01:58:35.076 P00   INFO: execute non-exclusive pg_stop_backup() and wait for all WAL segments to archive
2021-11-30 01:58:35.277 P00   INFO: backup stop archive = 00000001000000000000000C, lsn = 0/C000138
2021-11-30 01:58:35.321 P00   INFO: check archive for segment(s) 00000001000000000000000C:00000001000000000000000C
ERROR: [082]: WAL segment 00000001000000000000000C was not archived before the 60000ms timeout
    HINT: check the archive_command to ensure that all options are correct (especially --stanza).
    HINT: check the PostgreSQL server log for errors.
    HINT: run the 'start' command if the stanza was previously stopped.
2021-11-30 01:59:35.399 P00   INFO: backup command end: aborted with exception [082]

--no-archive-check - backup successfully

pgbackrest --stanza=main --log-level-console=info info

stanza: main
    status: error (missing stanza path)

S3 errors log:

2021-11-29T11:16:39:000 [204 No Content] s3.DeleteObject xxxxxx:9006/repo4/backup/main/latest xxxxxx      16.602ms     ↑ 77 B ↓ 327 B
2021-11-29T11:18:25:000 [404 Not Found] s3.GetBucketPolicy xxxxxx:9006/repo4/?policy=  xxxxxx     3.312ms      ↑ 98 B ↓ 674 B
2021-11-29T11:30:30:000 [404 Not Found] s3.HeadObject xxxxxx:9006/repo4/archive/main/13-1/00000002.history xxxxxx      3.736ms      ↑ 77 B ↓ 326 B
...
2021-11-29T11:38:32:000 [400 Bad Request] s3.GetObjectLegalHold xxxxxx:9006/repo4/archive/main/13-1/0000000100000000/000000010000000000000036.00000028.backup?legal-hold=  xxxxxx     368µs       ↑ 98 B ↓ 842 B
2021-11-29T11:38:32:000 [400 Bad Request] s3.GetObjectRetention xxxxxx:9006/repo4/archive/main/13-1/0000000100000000/000000010000000000000036.00000028.backup?retention=  xxxxxx     383µs       ↑ 98 B ↓ 842 B
2021-11-29T11:38:32:000 [400 Bad Request] s3.GetObjectLegalHold xxxxxx:9006/repo4/archive/main/13-1/0000000100000000/000000010000000000000036.00000028.backup?legal-hold=&versionId=null  xxxxxx     361µs       ↑ 98 B ↓ 842 B
2021-11-29T11:38:32:000 [400 Bad Request] s3.GetObjectRetention xxxxxx:9006/repo4/archive/main/13-1/0000000100000000/000000010000000000000036.00000028.backup?retention=&versionId=null  xxxxxx     211µs       ↑ 98 B ↓ 842 B
2021-11-29T11:44:30:000 [404 Not Found] s3.HeadObject xxxxx9006/repo4/archive/main/13-1/00000002.history  xxxxx     10.001ms     ↑ 77 B ↓ 326 B
...
2021-11-29T11:50:49:000 [404 Not Found] s3.HeadObject xxxxxx:9006/repo4/archive/main/archive.info xxxxxx      5.132ms      ↑ 77 B ↓ 326 B
2021-11-29T11:50:49:000 [404 Not Found] s3.HeadObject xxxxxx:9006/repo4/archive/main/archive.info.copy xxxxxx      3.87ms       ↑ 77 B ↓ 326 B

postgresql log:

archive-push command begin 2.36: [pg_wal/00000001000000000000000C.00000028.backup] 
2021-11-30 01:58:35.759 P00   INFO: pushed WAL file '00000001000000000000000C.00000028.backup' to the archive
2021-11-30 01:58:35.759 P00   INFO: archive-push command end: completed successfully (37ms)
  1. Describe the issue:

Hello, please help me figure out the problem. The archives go to S3. The archive-get and archive-push commands works separately. When requesting information, as well as when checking the required archives in a backup copy, we receive these errors. status: error (missing stanza path)

pgstef
pgstef

Hi,

Both errors seems consistent with your S3 logs. Apparently the commands don't succeed to "see" (read) anything from the repo. The config looks correct but doesn't seem to rely on AWS S3. What S3 provider are you using ? Imho, the issue might come from that there :thinking:

Kind regards

Activity icon
issue

pgstef issue comment pgbackrest/pgbackrest

pgstef
pgstef

How to restore only differential backup?

Please provide the following information when submitting an issue (feature requests or general comments can skip this):

  1. pgBackRest version: 2.35

  2. PostgreSQL version: 12.9

  3. Operating system/version - if you have more than one server (for example, a database server, a repository host server, one or more standbys), please specify each: Ubuntu/20.04

  4. Did you install pgBackRest from source or from a package? Package

We are successfully configured the pgbackrest and we are able to take backup as well as able to restore. But now we are performing some test cases like how we can rollback to a particular differential backup.

Below are the list of backup which we have at our end:

# sudo -u postgres pgbackrest --stanza=test --log-level-console=info info
stanza: test
    status: ok
    cipher: none

    db (current)
        wal archive min/max (12): 000000120000D485000000E5/000000240000D485000000EC

        full backup: 20211130-060533F
            timestamp start/stop: 2021-11-30 06:05:33 / 2021-11-30 06:54:39
            wal start/stop: 000000240000D485000000E7 / 000000240000D485000000E7
            database size: 850.2GB, database backup size: 850.2GB
            repo1: backup set size: 149.9GB, backup size: 149.9GB

        diff backup: 20211130-060533F_20211130-065952D
            timestamp start/stop: 2021-11-30 06:59:52 / 2021-11-30 06:59:58
            wal start/stop: 000000240000D485000000E9 / 000000240000D485000000E9
            database size: 850.2GB, database backup size: 8MB
            repo1: backup set size: 149.9GB, backup size: 885KB
            backup reference list: 20211130-060533F

        diff backup: 20211130-060533F_20211130-070159D
            timestamp start/stop: 2021-11-30 07:01:59 / 2021-11-30 07:02:04
            wal start/stop: 000000240000D485000000EB / 000000240000D485000000EB
            database size: 850.2GB, database backup size: 8MB
            repo1: backup set size: 149.9GB, backup size: 885KB
            backup reference list: 20211130-060533F

Now we are trying to rollback on backup id 20211130-060533F_20211130-065952D by without doing full backup restoration.

sudo -u postgres pgbackrest --stanza=test --delta --set=20211130-060533F_20211130-065952D --target-action=promote --type=immediate restore --log-level-console=detail

But from logs it seems that its restoring the full backup first.

We are doing this test case to find out the RTO if any user by mistake drop/delete any table.

Thank You

pgstef
pgstef

Hi,

Could you be more specific about But from logs it seems that its restoring the full backup first.? Please provide the logs you are referring to.

As its name states, diff is only differential. So only files modified since the last full backup are copied. That means in the restore, some files could come from the full backup since they are probably not all in the diff backup.

With the --delta options, only the files that need to be re-synced from the diff (or originally from the full) will be fetched by the restore command.

As I see it, the difference between restoring from a full or a diff is not really in the "files fetching" part but more in how many WAL archives do PG need to replay, because the starting point won't be the same. Given you're using --type=immediate, there shouldn't be any big difference probably...

Hope this help clearing what diff backups are, Kind Regards

Nov
25
1 week ago
push

pgstef push pgstef/check_pgbackrest

pgstef
pgstef

Test suite:

  • add edb-staging and pgdg apt test repos
  • support EPAS14
  • initial support steps for rockylinux

commit sha: d7b221330062f225979972e6fe6e748fe5d48372

push time in 1 week ago
Nov
24
2 weeks ago
open pull request

pgstef wants to merge pgbackrest/pgbackrest

pgstef
pgstef

Add checkpoint_timeout warning

In the backup command, add a warning if start-fast is disabled and PG checkpoint_timeout is greater than db-timeout. In such cases, we might timeout before the checkpoint occurs and the backup really starts.

Tests does seem to run fine now:

2021-11-10 10:15:59.309 P00   INFO: P1-T1/1 - vm=f33, module=db, test=db (11.63s)
2021-11-10 10:15:59.790 P00   INFO: tested modules have full coverage
2021-11-10 10:15:59.790 P00   INFO: writing C coverage report
2021-11-10 10:15:59.901 P00   INFO: TESTS COMPLETED SUCCESSFULLY (26s)

2021-11-10 10:16:53.501 P00   INFO: P1-T1/1 - vm=f33, module=command, test=backup (33.39s)
2021-11-10 10:16:53.981 P00   INFO: tested modules have full coverage
2021-11-10 10:16:53.981 P00   INFO: writing C coverage report
2021-11-10 10:16:54.196 P00   INFO: TESTS COMPLETED SUCCESSFULLY (35s)
pgstef
pgstef

Oh, that's great, thanks :-)

Previous