Re: Question about where to deploy the business logics for data processing

On 6/9/23 11:36, Nim Li wrote:
> Hello,
>
> Thank you so so much for all the feedback so far.  :D
>
> About this comment:
>
> > "... an application that requires changing the data model does not
> seem to be well designed...don't allow model change by the business
> logic..."
>
> I work in a science research faculity.  When researchers start a
> project, they don't necessary get the full picture of what they are
> hoping to achive (yet they may get some ideas about the starting point
> that allow them to move forward)  By the time they see 40% percent of
> what they have done, they may start to have a different thought and move
> towards a different direction, or in some cases, they may spin it off to
> something different after a certain period of time  Coming with my Agile
> Development mindset in the research area, it is common for me to see
> users changing their requirement and expectation, with the same buckets
> for the data.  Yes, there is quite a lot of work to keep the researchers
> happy.  ;-)
>
> I suppose when there is a specific end-goal to achive for a project, a
> more specific design can be more feasible based on the goal.  But when
> the end-goal is not necessary clear, and/or change-able, I am not
> exactly clear how we may draw a black-and-white line to determine a
> design is good or not (.. and for how long...)
>

Seems to me you are looking for a two part set up:

1) A experiment play ground where ideas and processes can be tested out
in a more free form manner. Some example software I have used or
experimented with that can fill that role:

Pandas
https://pandas.pydata.org/

Duckdb
https://duckdb.org/

Polars
https://pola-rs.github.io/polars-book/

2) Once something that resembles a solid plan has been developed then
move to Postgres or not.

--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com