From: | Taras Klioba <info(at)t(dot)lviv(dot)ua> |
---|---|
To: | pgsql-novice(at)lists(dot)postgresql(dot)org |
Subject: | How to solve issues with delayed transactions? |
Date: | 2019-02-03 10:39:31 |
Message-ID: | CAODqpgoOawkKw=nSTRuveYoi1SOm+J_uZxUrq=V33kbN6_8EUQ@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-novice |
Hello, All,
Please give me a piece of advice how could be solved an issue with delayed
transactions as on this video:
https://www.youtube.com/watch?v=PS6t3xO70ec
I have two transactions (two inserts to a table with the column "ins" which
has DEFAULT values as result of executing function NOW(), to receive
timestamp of inserting), and also I have an ETL process to load data from
this table to another place.
A transaction could appear with delay (for example 15-60 seconds).
For ETL I use parameter "last_ins" which equal MAX(ins) from the previous
execution:
SELECT MAX(ins) as last_ins
FROM public.pgsql_novice_example
WHERE ins > :last_ins
In this case, I can miss a delayed transaction.
To solve this problem, I see a few ways:
- Increase the level of isolation (but for my high load system it's not the
best solution);
- Add a trigger on this table to store all changes, and then use this new
table for ETL process (but it increases I/O operations);
- Use an output plugin for logical decoding to receive sequentially changes
from WAL files (it looks like too hard to implement, and why I need to
parse all WAL files if I want to send only one table);
- Load data from my table with the postponement (for example ":last_ins -
interval '30 minutes'") which will be bigger than time of executing a
transaction (but I want to have data near to real-time).
I would be thankful for any suggestions or help. Maybe I could use a
function which will show not start of a transaction, but the end?
--
Best regards,
Solomia K.
Software Developer
From | Date | Subject | |
---|---|---|---|
Next Message | Nico Callewaert | 2019-02-07 17:43:49 | Scaling / Number of simultanous connections |
Previous Message | Stephen Froehlich | 2019-01-31 01:02:18 | RE: Recommended ./configure flags for Ubuntu install? |