Re: [pgadmin-support] Issue with a hanging apply process on the replica db after vacuum works on primary

> 19 марта 2015 г., в 20:30, Sergey Shchukin <shchukin(dot)s(dot)a(at)gmail(dot)com> написал(а):
>
> 17.03.2015 13:22, Sergey Shchukin пишет:
>> 05.03.2015 11:25, Jim Nasby пишет:
>>> On 2/27/15 5:11 AM, Sergey Shchukin wrote:
>>>>
>>>> show max_standby_streaming_delay;
>>>> max_standby_streaming_delay
>>>> -----------------------------
>>>> 30s
>>>
>>> We both need to be more clear about which server we're talking about (master or replica).
>>>
>>> What are max_standby_streaming_delay and max_standby_archive_delay set to *on the replica*?
>>>
>>> My hope is that one or both of those is set to somewhere around 8 minutes on the replica. That would explain everything.
>>>
>>> If that's not the case then I suspect what's happening is there's something running on the replica that isn't checking for interrupts frequently enough. That would also explain it.
>>>
>>> When replication hangs, is the replication process using a lot of CPU? Or is it just sitting there? What's the process status for the replay process show?
>>>
>>> Can you get a trace of the replay process on the replica when this is happening to see where it's spending all it's time?
>>>
>>> How are you generating these log lines?
>>> Tue Feb 24 15:05:07 MSK 2015 Stream: MASTER-masterdb:79607161592048 SLAVE:79607161550576 Replay:79607160986064 :: REPLAY 592 KBytes (00:00:00.398376 seconds)
>>>
>>> Do you see the confl_* fields in pg_stat_database_conflicts on the *replica* increasing?
>>
>> Hi Jim,
>>
>> max_standby_streaming_delay and max_standby_archive_delay both are 30s on master and replica dbs
>>
>> I don't see any specific or heavy workload during this issue with a hanging apply process. Just a normal queries as usual.
>>
>> But I see an increased disk activity during the time when the apply issue is ongoing
>>
>> DSK | sdc | | busy 61% | read 11511 | | write 4534 | KiB/r 46 | | KiB/w 4 | MBr/s 52.78 | | MBw/s 1.88 | avq 1.45 | | avio 0.38 ms |
>> DSK | sde | | busy 60% | read 11457 | | write 4398 | KiB/r 46 | | KiB/w 4 | MBr/s 51.97 | | MBw/s 1.83 | avq 1.47 | | avio 0.38 ms |
>> DSK | sdd | | busy 60% | read 9673 | | write 4538 | KiB/r 61 | | KiB/w 4 | MBr/s 58.24 | | MBw/s 1.88 | avq 1.47 | | avio 0.42 ms |
>> DSK | sdj | | busy 59% | read 9576 | | write 4177 | KiB/r 63 | | KiB/w 4 | MBr/s 59.30 | | MBw/s 1.75 | avq 1.48 | | avio 0.43 ms |
>> DSK | sdh | | busy 59% | read 9615 | | write 4305 | KiB/r 63 | | KiB/w 4 | MBr/s 59.23 | | MBw/s 1.80 | avq 1.48 | | avio 0.42 ms |
>> DSK | sdf | | busy 59% | read 9483 | | write 4404 | KiB/r 63 | | KiB/w 4 | MBr/s 59.11 | | MBw/s 1.83 | avq 1.47 | | avio 0.42 ms |
>> DSK | sdi | | busy 59% | read 11273 | | write 4173 | KiB/r 46 | | KiB/w 4 | MBr/s 51.50 | | MBw/s 1.75 | avq 1.43 | | avio 0.38 ms |
>> DSK | sdg | | busy 59% | read 11406 | | write 4297 | KiB/r 46 | | KiB/w 4 | MBr/s 51.66 | | MBw/s 1.80 | avq 1.46 | | avio 0.37 ms |
>>
>> Although it's not seems to be an upper IO limit.
>>
>> Normally disks are busy at 20-45%
>>
>> DSK | sde | | busy 29% | read 6524 | | write 14426 | KiB/r 26 | | KiB/w 5 | MBr/s 17.08 | | MBw/s 7.78 | avq 10.46 | | avio 0.14 ms |
>> DSK | sdi | | busy 29% | read 6590 | | write 14391 | KiB/r 26 | | KiB/w 5 | MBr/s 17.19 | | MBw/s 7.76 | avq 8.75 | | avio 0.14 ms |
>> DSK | sdg | | busy 29% | read 6547 | | write 14401 | KiB/r 26 | | KiB/w 5 | MBr/s 16.94 | | MBw/s 7.60 | avq 7.28 | | avio 0.14 ms |
>> DSK | sdc | | busy 29% | read 6835 | | write 14283 | KiB/r 27 | | KiB/w 5 | MBr/s 18.08 | | MBw/s 7.74 | avq 8.77 | | avio 0.14 ms |
>> DSK | sdf | | busy 23% | read 3808 | | write 14391 | KiB/r 36 | | KiB/w 5 | MBr/s 13.49 | | MBw/s 7.78 | avq 12.88 | | avio 0.13 ms |
>> DSK | sdd | | busy 23% | read 3747 | | write 14229 | KiB/r 33 | | KiB/w 5 | MBr/s 12.32 | | MBw/s 7.74 | avq 10.07 | | avio 0.13 ms |
>> DSK | sdj | | busy 23% | read 3737 | | write 14336 | KiB/r 36 | | KiB/w 5 | MBr/s 13.16 | | MBw/s 7.76 | avq 10.48 | | avio 0.13 ms |
>> DSK | sdh | | busy 23% | read 3793 | | write 14362 | KiB/r 35 | | KiB/w 5 | MBr/s 13.29 | | MBw/s 7.60 | avq 8.61 | | avio 0.13 ms |
>>
>>
>> Also during the issue perf shows [k] copy_user_generic_string on the top positions
>> 14.09% postmaster postgres [.] 0x00000000001b4569
>> 10.25% postmaster [kernel.kallsyms] [k] copy_user_generic_string
>> 4.15% postmaster postgres [.] hash_search_with_hash_value
>> 2.08% postmaster postgres [.] SearchCatCache
>> 1.79% postmaster postgres [.] LWLockAcquire
>> 1.18% postmaster libc-2.12.so [.] memcpy
>> 1.12% postmaster postgres [.] mdnblocks
>>
>> Issue starts: at 19:43
>> Mon Mar 16 19:43:04 MSK 2015 Stream: MASTER-rdb04d:70837172337784 SLAVE:70837172314864 Replay:70837172316512 :: REPLAY 21 KBytes (00:00:00.006225 seconds)
>> Mon Mar 16 19:43:09 MSK 2015 Stream: MASTER-rdb04d:70837177455624 SLAVE:70837177390968 Replay:70837176794376 :: REPLAY 646 KBytes (00:00:00.367305 seconds)
>> Mon Mar 16 19:43:14 MSK 2015 Stream: MASTER-rdb04d:70837185005120 SLAVE:70837184961280 Replay:70837183253896 :: REPLAY 1710 KBytes (00:00:00.827881 seconds)
>> Mon Mar 16 19:43:19 MSK 2015 Stream: MASTER-rdb04d:70837190417984 SLAVE:70837190230232 Replay:70837183253896 :: REPLAY 6996 KBytes (00:00:05.873169 seconds)
>> Mon Mar 16 19:43:24 MSK 2015 Stream: MASTER-rdb04d:70837198538232 SLAVE:70837198485000 Replay:70837183253896 :: REPLAY 14926 KBytes (00:00:11.025561 seconds)
>> Mon Mar 16 19:43:29 MSK 2015 Stream: MASTER-rdb04d:70837209961192 SLAVE:70837209869384 Replay:70837183253896 :: REPLAY 26081 KBytes (00:00:16.068014 seconds)
>>
>> We see [k] copy_user_generic_string
>>
>> 12.90% postmaster [kernel.kallsyms] [k] copy_user_generic_string
>> 11.49% postmaster postgres [.] 0x00000000001f40c1
>> 4.74% postmaster postgres [.] hash_search_with_hash_value
>> 1.86% postmaster postgres [.] mdnblocks
>> 1.73% postmaster postgres [.] LWLockAcquire
>> 1.67% postmaster postgres [.] SearchCatCache
>>
>>
>> 25.71% postmaster [kernel.kallsyms] [k] copy_user_generic_string
>> 7.89% postmaster postgres [.] hash_search_with_hash_value
>> 4.66% postmaster postgres [.] 0x00000000002108da
>> 4.51% postmaster postgres [.] mdnblocks
>> 3.36% postmaster [kernel.kallsyms] [k] put_page
>>
>> Issue stops: at 19:51:39
>> Mon Mar 16 19:51:24 MSK 2015 Stream: MASTER-rdb04d:70838904179344 SLAVE:70838903934392 Replay:70837183253896 :: REPLAY 1680591 KBytes (00:08:10.384679 seconds)
>> Mon Mar 16 19:51:29 MSK 2015 Stream: MASTER-rdb04d:70838929994336 SLAVE:70838929873624 Replay:70837183253896 :: REPLAY 1705801 KBytes (00:08:15.428773 seconds)
>> Mon Mar 16 19:51:34 MSK 2015 Stream: MASTER-rdb04d:70838951993624 SLAVE:70838951899768 Replay:70837183253896 :: REPLAY 1727285 KBytes (00:08:20.472567 seconds)
>> Mon Mar 16 19:51:39 MSK 2015 Stream: MASTER-rdb04d:70838975297912 SLAVE:70838975180384 Replay:70837208050872 :: REPLAY 1725827 KBytes (00:08:10.256935 seconds)
>> Mon Mar 16 19:51:44 MSK 2015 Stream: MASTER-rdb04d:70839001502160 SLAVE:70839001412616 Replay:70837260116984 :: REPLAY 1700572 KBytes (00:07:49.849511 seconds)
>> Mon Mar 16 19:51:49 MSK 2015 Stream: MASTER-rdb04d:70839022866760 SLAVE:70839022751184 Replay:70837276732880 :: REPLAY 1705209 KBytes (00:07:42.307364 seconds)
>>
>> And copy_user_generic_string goes down
>> + 13.43% postmaster postgres [.] 0x000000000023dc9a
>> + 3.71% postmaster [kernel.kallsyms] [k] copy_user_generic_string
>> + 2.46% init [kernel.kallsyms] [k] intel_idle
>> + 2.30% postmaster postgres [.] hash_search_with_hash_value
>> + 2.01% postmaster postgres [.] SearchCatCache
>>
>>
>> Could you clarify what types of traces did you mean? GDB?
>>
>> To calculate slave and apply lag I use the following query at the replica site
>>
>> slave_lag=$($psql -U monitor -h$s_host -p$s_port -A -t -c "SELECT pg_xlog_location_diff(pg_last_xlog_receive_location(), '0/0') AS receive" $p_db)
>> replay_lag=$($psql -U monitor -h$s_host -p$s_port -A -t -c "SELECT pg_xlog_location_diff(pg_last_xlog_replay_location(), '0/0') AS replay" $p_db)
>> replay_timediff=$($psql -U monitor -h$s_host -p$s_port -A -t -c "SELECT NOW() - pg_last_xact_replay_timestamp() AS replication_delay" $p_db)
>> master_lag=$($psql -U monitor -h$p_host -p$p_port -A -t -c "SELECT pg_xlog_location_diff(pg_current_xlog_location(), '0/0') AS offset" $p_db)
>> echo "$(date) Stream: MASTER-$p_host:$master_lag SLAVE:$slave_lag Replay:$replay_lag :: REPLAY $(bc <<< $master_lag/1024-$replay_lag/1024) KBytes (${replay_timediff} seconds)"
>>
>> -
>> Best regards,
>> Sergey Shchukin
>>
>
> One more thing
>
> We have upgraded one of our shards to 9.4.1 and expectedly that did not help.
>
> A few things to notice which may be useful.
>
> 1. When replay stops, startup process reads a lot from array with $PGDATA. In iotop and iostat we see the following:
>
> Total DISK READ: 490.42 M/s | Total DISK WRITE: 3.82 M/s
> TID PRIO USER DISK READ> DISK WRITE SWAPIN IO COMMAND
> 3316 be/4 postgres 492.34 M/s 0.00 B/s 0.00 % 39.91 % postgres: startup process
> <...>
>
> Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util
> <...>
> md2 0.00 0.00 6501.00 7.00 339.90 0.03 106.97 0.00 0.00 0.00 0.00
> md3 0.00 0.00 0.00 1739.00 0.00 6.79 8.00 0.00 0.00 0.00 0.00
>
> root(at)rpopdb04g ~ # fgrep 9.4 /proc/mounts
> /dev/md2 /var/lib/pgsql/9.4/data ext4 rw,noatime,nodiratime,barrier=1,stripe=64,data=ordered 0 0
> /dev/md3 /var/lib/pgsql/9.4/data/pg_xlog ext4 rw,noatime,nodiratime,barrier=0,stripe=64,data=ordered 0 0
> root(at)rpopdb04g ~ #
>
> 2. The state of the startup process is changing in such a way:
>
> root(at)rpopdb04g ~ # while true; do ps aux | grep '[s]tartup'; sleep 1; done
> postgres 3316 26.6 3.2 4732052 4299260 ? Rs 18:04 8:11 postgres: startup process
> postgres 3316 26.6 3.2 4732052 4299260 ? Ts 18:04 8:11 postgres: startup process
> postgres 3316 26.6 3.2 4732052 4299260 ? Rs 18:04 8:12 postgres: startup process
> postgres 3316 26.6 3.2 4732052 4299260 ? Ds 18:04 8:12 postgres: startup process
> postgres 3316 26.6 3.2 4732052 4299260 ? Rs 18:04 8:13 postgres: startup process
> postgres 3316 26.6 3.2 4732052 4299260 ? Rs 18:04 8:13 postgres: startup process
> postgres 3316 26.6 3.2 4732052 4299260 ? Rs 18:04 8:14 postgres: startup process
> postgres 3316 26.6 3.2 4732052 4299260 ? Ts 18:04 8:14 postgres: startup process
> postgres 3316 26.6 3.2 4732052 4299260 ? Ds 18:04 8:15 postgres: startup process
> postgres 3316 26.6 3.2 4732052 4299260 ? Rs 18:04 8:15 postgres: startup process
> postgres 3316 26.7 3.2 4732052 4299260 ? Ds 18:04 8:15 postgres: startup process
> postgres 3316 26.7 3.2 4732052 4299260 ? Ds 18:04 8:16 postgres: startup process
> postgres 3316 26.7 3.2 4732052 4299260 ? Rs 18:04 8:16 postgres: startup process
> postgres 3316 26.7 3.2 4732052 4299260 ? Rs 18:04 8:17 postgres: startup process
> postgres 3316 26.7 3.2 4732052 4299260 ? Ts 18:04 8:17 postgres: startup process
> postgres 3316 26.7 3.2 4732052 4299260 ? Rs 18:04 8:18 postgres: startup process
> postgres 3316 26.7 3.2 4732052 4299260 ? Rs 18:04 8:18 postgres: startup process
> postgres 3316 26.7 3.2 4732052 4299260 ? Ds 18:04 8:19 postgres: startup process
> postgres 3316 26.8 3.2 4732052 4299260 ? Rs 18:04 8:19 postgres: startup process
> postgres 3316 26.8 3.2 4732052 4299260 ? Rs 18:04 8:20 postgres: startup process
> postgres 3316 26.8 3.2 4732052 4299260 ? Rs 18:04 8:20 postgres: startup process
> postgres 3316 26.8 3.2 4732052 4299260 ? Ds 18:04 8:21 postgres: startup process
> postgres 3316 26.8 3.2 4732052 4299260 ? Ds 18:04 8:22 postgres: startup process
> postgres 3316 26.8 3.2 4732052 4299260 ? Rs 18:04 8:22 postgres: startup process
> postgres 3316 26.8 3.2 4732052 4299260 ? Rs 18:04 8:23 postgres: startup process
> postgres 3316 26.9 3.2 4732052 4299260 ? Rs 18:04 8:23 postgres: startup process
> postgres 3316 26.9 3.2 4732052 4299260 ? Rs 18:04 8:24 postgres: startup process
> postgres 3316 26.9 3.2 4732052 4299260 ? Ts 18:04 8:24 postgres: startup process
> postgres 3316 26.9 3.2 4732052 4299260 ? Rs 18:04 8:25 postgres: startup process
> ^C
> root(at)rpopdb04g ~ #
>
> 3. confl* fields in pg_stat_database_conflicts are always zero during the pausing of replay.
>
> 4. The stack-traces taken with GDB are not really informative. We will recompile PostgreSQL with —enable-debug option and run it on one of our replicas if needed. Since it is a production system we would like to do it last of all. But we will do it if anybody would not give us any ideas.

We did it. Most of the backtraces (taken while replay_location was not changing) looks like that:

[Thread debugging using libthread_db enabled]
0x00007f54a71444c0 in __read_nocancel () from /lib64/libc.so.6
#0 0x00007f54a71444c0 in __read_nocancel () from /lib64/libc.so.6
#1 0x000000000065d2f5 in FileRead (file=<value optimized out>, buffer=0x7f5470e6ba20 "\004", amount=8192) at fd.c:1286
#2 0x000000000067acad in mdread (reln=<value optimized out>, forknum=<value optimized out>, blocknum=311995, buffer=0x7f5470e6ba20 "\004") at md.c:679
#3 0x0000000000659b4e in ReadBuffer_common (smgr=<value optimized out>, relpersistence=112 'p', forkNum=MAIN_FORKNUM, blockNum=311995, mode=RBM_NORMAL_NO_LOG, strategy=0x0, hit=0x7fff898a912f "") at bufmgr.c:476
#4 0x000000000065a61b in ReadBufferWithoutRelcache (rnode=..., forkNum=MAIN_FORKNUM, blockNum=311995, mode=<value optimized out>, strategy=<value optimized out>) at bufmgr.c:287
#5 0x00000000004cfb78 in XLogReadBufferExtended (rnode=..., forknum=MAIN_FORKNUM, blkno=311995, mode=RBM_NORMAL_NO_LOG) at xlogutils.c:324
#6 0x00000000004a3651 in btree_xlog_vacuum (lsn=71742288638464, record=0x1e48b78) at nbtxlog.c:522
#7 btree_redo (lsn=71742288638464, record=0x1e48b78) at nbtxlog.c:1144
#8 0x00000000004c903a in StartupXLOG () at xlog.c:6827
#9 0x000000000062f8bf in StartupProcessMain () at startup.c:224
#10 0x00000000004d3e9a in AuxiliaryProcessMain (argc=2, argv=0x7fff898a98a0) at bootstrap.c:416
#11 0x000000000062a99c in StartChildProcess (type=StartupProcess) at postmaster.c:5146
#12 0x000000000062e9e2 in PostmasterMain (argc=3, argv=<value optimized out>) at postmaster.c:1237
#13 0x00000000005c7d68 in main (argc=3, argv=0x1e22910) at main.c:228

So the problem seems to be in this part of code - http://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/backend/access/nbtree/nbtxlog.c;h=5f9fc49e78ca1388ab482e24c8b5a873238ae0b6;hb=d0f83327d3739a45102fdd486947248c70e0249d#l507. I suppose, that answers the question why startup process reads a lot from disk while paused replay.

So the questions are:
1. Is there anything we can tune right now? Except for not reading from replicas and partitioning this table.
2. Isn’t there still a function to determine that a buffer is not pinned in shared_buffers without reading it from disk? To optimize current behaviour in the future.

>
> 5. In one of the experiments replay stopped on 4115/56126DC0 xlog position. Here is a bit of pg_xlogdump output:
>
> rpopdb04d/rpopdb M # select pg_xlogfile_name('4115/56126DC0');
> pg_xlogfile_name
> --------------------------
> 000000060000411500000056
> (1 row)
>
> Time: 0.496 ms
> rpopdb04d/rpopdb M #
>
> root(at)pg-backup04h /u0/rpopdb04/wals/0000000600004115 # /usr/pgsql-9.4/bin/pg_xlogdump 000000060000411500000056 000000060000411500000056 | fgrep 4115/56126DC0 -C10
> rmgr: Heap len (rec/tot): 36/ 3948, tx: 3930143874, lsn: 4115/561226C8, prev 4115/56122698, bkp: 1000, desc: hot_update: rel 1663/16420/16737; tid 196267/4 xmax 3930143874 ; new tid 196267/10 xmax 0
> rmgr: Transaction len (rec/tot): 12/ 44, tx: 3930143874, lsn: 4115/56123638, prev 4115/561226C8, bkp: 0000, desc: commit: 2015-03-19 18:26:27.725158 MSK
> rmgr: Heap len (rec/tot): 58/ 90, tx: 3930143875, lsn: 4115/56123668, prev 4115/56123638, bkp: 0000, desc: hot_update: rel 1663/16420/16737; tid 23624/3 xmax 3930143875 ; new tid 23624/21 xmax 0
> rmgr: Transaction len (rec/tot): 12/ 44, tx: 3930143875, lsn: 4115/561236C8, prev 4115/56123668, bkp: 0000, desc: commit: 2015-03-19 18:26:27.726432 MSK
> rmgr: Heap len (rec/tot): 36/ 2196, tx: 3930143876, lsn: 4115/561236F8, prev 4115/561236C8, bkp: 1000, desc: hot_update: rel 1663/16420/16737; tid 123008/4 xmax 3930143876 ; new tid 123008/5 xmax 0
> rmgr: Transaction len (rec/tot): 12/ 44, tx: 3930143876, lsn: 4115/56123F90, prev 4115/561236F8, bkp: 0000, desc: commit: 2015-03-19 18:26:27.727088 MSK
> rmgr: Heap len (rec/tot): 36/ 7108, tx: 3930143877, lsn: 4115/56123FC0, prev 4115/56123F90, bkp: 1000, desc: hot_update: rel 1663/16420/16737; tid 34815/6 xmax 3930143877 ; new tid 34815/16 xmax 0
> rmgr: Transaction len (rec/tot): 12/ 44, tx: 3930143877, lsn: 4115/56125BA0, prev 4115/56123FC0, bkp: 0000, desc: commit: 2015-03-19 18:26:27.728178 MSK
> rmgr: Heap len (rec/tot): 36/ 4520, tx: 3930143878, lsn: 4115/56125BD0, prev 4115/56125BA0, bkp: 1000, desc: hot_update: rel 1663/16420/16737; tid 147863/5 xmax 3930143878 ; new tid 147863/16 xmax 0
> rmgr: Transaction len (rec/tot): 12/ 44, tx: 3930143878, lsn: 4115/56126D90, prev 4115/56125BD0, bkp: 0000, desc: commit: 2015-03-19 18:26:27.728339 MSK
> rmgr: Btree len (rec/tot): 20/ 52, tx: 0, lsn: 4115/56126DC0, prev 4115/56126D90, bkp: 0000, desc: vacuum: rel 1663/16420/16796; blk 31222118, lastBlockVacuumed 0
> rmgr: Heap len (rec/tot): 36/ 6112, tx: 3930143879, lsn: 4115/56126DF8, prev 4115/56126DC0, bkp: 1000, desc: hot_update: rel 1663/16420/16737; tid 23461/26 xmax 3930143879 ; new tid 23461/22 xmax 0
> rmgr: Heap2 len (rec/tot): 24/ 8160, tx: 0, lsn: 4115/561285F0, prev 4115/56126DF8, bkp: 1000, desc: clean: rel 1663/16420/16782; blk 21997709 remxid 0
> rmgr: Transaction len (rec/tot): 12/ 44, tx: 3930143879, lsn: 4115/5612A5E8, prev 4115/561285F0, bkp: 0000, desc: commit: 2015-03-19 18:26:27.728805 MSK
> rmgr: Heap2 len (rec/tot): 20/ 8268, tx: 0, lsn: 4115/5612A618, prev 4115/5612A5E8, bkp: 1000, desc: visible: rel 1663/16420/16782; blk 21997709
> rmgr: Heap len (rec/tot): 36/ 7420, tx: 3930143880, lsn: 4115/5612C680, prev 4115/5612A618, bkp: 1000, desc: hot_update: rel 1663/16420/16737; tid 37456/8 xmax 3930143880 ; new tid 37456/29 xmax 0
> rmgr: Transaction len (rec/tot): 12/ 44, tx: 3930143880, lsn: 4115/5612E398, prev 4115/5612C680, bkp: 0000, desc: commit: 2015-03-19 18:26:27.729141 MSK
> rmgr: Heap len (rec/tot): 36/ 7272, tx: 3930143881, lsn: 4115/5612E3C8, prev 4115/5612E398, bkp: 1000, desc: hot_update: rel 1663/16420/16737; tid 23614/2 xmax 3930143881 ; new tid 23614/22 xmax 0
> rmgr: Heap len (rec/tot): 150/ 182, tx: 0, lsn: 4115/56130048, prev 4115/5612E3C8, bkp: 0000, desc: inplace: rel 1663/16420/12764; tid 11/31
> rmgr: Transaction len (rec/tot): 12/ 44, tx: 3930143881, lsn: 4115/56130100, prev 4115/56130048, bkp: 0000, desc: commit: 2015-03-19 18:26:27.729340 MSK
> rmgr: Heap len (rec/tot): 43/ 75, tx: 3930143882, lsn: 4115/56130130, prev 4115/56130100, bkp: 0000, desc: insert: rel 1663/16420/16773; tid 10159950/26
> rmgr: Btree len (rec/tot): 42/ 74, tx: 3930143882, lsn: 4115/56130180, prev 4115/56130130, bkp: 0000, desc: insert: rel 1663/16420/16800; tid 12758988/260
> root(at)pg-backup04h /u0/rpopdb04/wals/0000000600004115 #
>
> Any help would be really appropriate. Thanks in advance.
>

--
May the force be with you…
https://simply.name