Re: Pulling data from Postgres DB table for every 5 seconds.

From: Tony Shelver <tshelver(at)gmail(dot)com>
To: github kran <githubkran(at)gmail(dot)com>, pgsql-general(at)postgresql(dot)org
Subject: Re: Pulling data from Postgres DB table for every 5 seconds.
Date: 2019-01-10 12:41:37
Message-ID: CAG0dhZAzEhuUbQW9tdCJaucqAPdhSEBCD7FhrqmOXvqiV__6OA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

I am not familiar with Aurora, but..

What something like https://github.com/subzerocloud/pg-amqp-bridge?

Set up a message queue in Postgres, which calls into AMPQ (RabbitMQ) to
send a message for consumption by one or more clients.

This provides a function that can be called from a trigger to send the
message.

After that, you have all the goodness of a standards-based open source MQ
platform to distribute your data / notifications.

On Wed, 9 Jan 2019 at 19:02, github kran <githubkran(at)gmail(dot)com> wrote:

>
>> Hi Postgres Team,
>>
>> I have an application using RDS Aurora Postgresql 9.6 version having 4 TB
>> of DB size. In this DB we have a table PRODUCT_INFO with around 1 million
>> rows and table size of 1 GB.
>> We are looking for a implementation where we want to pull the data in
>> real time for every 5 seconds from the DB ( Table mentioned above) and send
>> it to IOT topic whenever an event occurs for a product. ( event is any new
>> product information or change in the existing
>> product information.).
>>
>> This table has few DML operations in real time either INSERT or UPDATE
>> based on the productId. ( Update whenever there is a change in the product
>> information and INSERT when a record doesnt exists for that product).
>>
>> We have REST API's built in the backend pulling data from this backend
>> RDS Aurora POSTGRES DB and used by clients.
>>
>> *UseCase*
>> We dont want clients to pull the data for every 5 seconds from DB but
>> rather provide a service which can fetch the data from DB in real time and
>> push the data to IOT topic by pulling data for every 5 seconds from DB.
>>
>> *Questions*
>> 1) How can I get information by pulling from the DB every 5 seconds
>> without impacting the performance of the DB.
>> 2) What are the options I have pulling the data from this table every 5
>> seconds. Does POSTGRES has any other options apart from TRIGGER ?.
>>
>>
>> Any ideas would be helpful.
>>
>> Thanks !!
>> GithubKran
>>
>

In response to

Browse pgsql-general by date

  From Date Subject
Next Message Dave Cramer 2019-01-10 13:09:22 Re: jdbc PGCopyOutputStream close() v. endCopy()
Previous Message Kim Rose Carlsen 2019-01-10 12:30:49 Postgres wont remove useless joins, when the UNIQUE index is partial