The Postgres.ai team is happy to announce the release of version 3.0 of Database Lab Engine (DLE), the most advanced open-source software ever released that empowers development, testing, and troubleshooting environments for fast-growing projects. The use of Database Lab Engine 3.0 provides a competitive advantage to companies via implementing the "Shift-left testing" approach in software development.
Database Lab Engine is an open-source technology that enables thin cloning for PostgreSQL. Thin clones are exceptionally useful when you need to scale the development process. DLE can manage dozens of independent clones of your database on a single machine, so each engineer or automation process works with their very own database provisioned in seconds without extra costs.
Among major changes in DLE 3.0:
Starting with version 3.0.0, DLE collects non-personally identifiable telemetry data. This feature is enabled by default but can be switched off. Read more in the DLE documentation. Keeping telemetry enabled can be considered your contribution to the DLE development because it helps make decisions down the road of the open-source product development.
Further, we discuss the most requested changes that were implemented in DLE 3.0 – all of them were created based on real-life user experience and invaluable feedback from the growing community of users and contributors.
Being open-source software, DLE has always been equipped with API and CLI. As for UI, it was initially available only in the form of SaaS – Database Lab Platform running on Postgres.ai.
In response to numerous requests from DLE users, UI has been integrated into the core distribution of DLE 3.0. This change makes open-source DLE even more attractive to users, bringing ease in use and simplifying adoption in fast-growing companies.
You can watch a short video demonstrating DLE UI.
Some users have told us that with UI in hands, it becomes much easier to explain to colleagues various use cases where DLE can be very helpful. If you like Database Lab, please try to benefit from this change – using UI, demonstrate to others the idea of cloning large databases in a few seconds, and discuss how it can influence your software development, testing processes, as well as incident troubleshooting and SQL optimization.
Another feature added to DLE 3.0 is also something that DLE users have asked a lot about. Before 3.0, any restart of DLE meant the loss of all clones created – so DLE upgrades, VM restarts, and even simple reconfiguration of DLE always needed a maintenance window, interrupting work.
A partial solution to this problem was the ability to reconfigure DLE without restarts introduced in DLE 2.0. However, this wasn't helpful in the cases of DLE upgrades or VM restarts. Now with DLE 3.0, this problem is fully solved:
Of course, in the case of VM restart, DB connections are lost and need to be recreated. However, if you need to restart just DLE, all clone containers will keep running, and users can now continue working with them even during DLE restart, without any work interruptions.
In DLE 2.5, we have implemented the ability to reset to any available snapshot – a convenient way for your clones to travel in time fast. In 2.5, this was supported only for the "physical" provisioning mode (restoring data directory, PGDATA, from physical backups, or obtaining it from the source using pg_basebackup). In other words, it was ready to work only if you manage Postgres yourself and can copy PGDATA or establish physical replication connection to your databases. Something that is not available to the users of RDS and other managed Postgres services.
For DLE running in the "logical" data provisioning mode (based on dump/restore – the only option for most managed Postgres cloud offerings such as Amazon RDS), DLE 2.5 provided the ability to operate with multiple copies of PGDATA was implemented, which allowed having a full refresh without downtime. However, if DLE users were running clones on the "old" PGDATA version, they needed to recreate them to unlock the next full refresh – and this was somewhat inconvenient because of quite unpredictable port allocation.
In DLE 3.0, it is now possible to reset a clone's state to any database version (snapshot), even if that version is provided by another copy of PGDATA running on a different pool/dataset. It means that users can keep their clone running on the same port for a long time, having stable DB credentials (including the port), and when needed, once the full refresh has finished, switch to the freshest database version in seconds. This makes the experience of working with the "logical" almost on par with the "physical" one.
Originally, DLE was designed to run on a dedicated machine, physical or virtual. However, many users found it inconvenient – in some cases, development and testing environments are subject to budget optimization, so it may make a lot of sense to run multiple DLEs on a single machine, especially if the organization has many smaller databases (a typical case for those who deal with microservice architectures).
DLE 3.0 has several improvements that simplify running multiple DLEs on a single machine. Particularly, the new configuration option selectedPool allows having a single ZFS pool and running multiple DLEs in a way where each one of them has its own dataset. This drastically simplifies free disk space management: instead of fragmented space allocation and dealing with many "free disk space" numbers, the DLE administrator now needs to control just a single number and adjust disk size much less often.
We are planning to discuss the aspects of running multiple DLEs on a single machine in a separate article.
Feedback and contributions would be greatly appreciated: