Aestus follows every change in your MongoDB collections and delivers it to your target, in order and exactly where it belongs. It runs continuously, loses nothing, and your data never leaves your network.
A change passes through a few small agents, connected by a durable message broker. Each agent does one job for one kind of system; together they form a flow.
The capture agent follows the MongoDB change stream. It copies existing records first, opening the stream before it starts so no change slips through, and labels every change with its key, group and source version.
RabbitMQ quorum queues hold every change until the next agent confirms it. Related records, such as an item and its stock, share a partition and stay in order; unrelated ones move in parallel.
Optionally reshape the data on the way: map documents to parent and child tables, or plug in your own logic. Flows without a transform store the documents as they are.
The integrate agent writes each change in a way that is safe to repeat, and never lets an older change overwrite a newer one. If a write fails, it asks for the record again and keeps going.
Built and verified end to end on a MongoDB 7 replica set replicating into SQLite.
A new flow copies every existing record, then streams changes. The stream opens before the copy starts, so changes made during the load are never lost, and an interrupted load resumes where it stopped.
Map a collection to a parent table and its embedded arrays to child tables, from nested fields to typed columns. Tables are created and widened automatically when the mapping grows.
Group related collections by a key, such as an item and its stock records. Changes for the same group are applied in order, while different groups are processed in parallel.
Checkpoints only move once the broker has stored a change. Every target row remembers the version of its last change, so late or repeated messages are ignored, and deleted rows cannot come back.
A failed write triggers a recapture: the current record is read again from the source and sent through the flow. Failed transformations retry after growing delays. What still fails waits in a dead-letter queue for one decision: retry, recapture or discard.
Need something no configuration offers, such as masking personal data? Start from the transform template and write one Go function; reading, writing, retries and statistics are handled for you.
Define flows, start, stop and pause agents, watch throughput, lag and events live, and handle dead letters from the terminal UI. Everything it does is also available through a documented REST API.
Run several replicas of a transform or integrate agent: partitions spread across them and move to the others when one stops. A busy collection can be read by several capture readers at once.
Send complete documents, or only the fields that changed. When a change cannot be applied on its own, for example inside an array, Aestus fetches the full record instead of guessing.
Every system gets its own small agent. What is available today, and what comes next.
Where the changes come from.
Replica sets, sharded clusters and MongoDB Atlas, through change streams. Initial load, full documents or changed fields, and split readers for busy collections on MongoDB 7 and later.
Azure Cosmos DB for MongoDB and Amazon DocumentDB.
Google Cloud Firestore and Amazon DynamoDB Streams.
Where the changes land.
Relational tables from the mapping, or documents stored as they are. Version guard per row, tombstones for deletes, and automatic table creation.
Delta tables with documents kept as VARIANT, optional typed columns, and change history.
The same relational row sets SQLite receives today, written to server databases.
The broker, the controller and every agent run as containers inside your own network. Changes go straight from your source to your target.
One small image per agent. A few scripts start the broker, the controller and a development database; the controller starts and supervises the agents. Kubernetes, Docker and systemd are next.
Passwords and connection strings live in Podman secrets and reach agents as environment variables. Configuration files never contain them.
Aestus is an enterprise product, delivered together with consulting: we work with your team to design flows and mappings, run them in your network, and keep them healthy.
Replicate MongoDB collections continuously to SQLite, either as relational parent and child tables or as documents stored as they are, with an initial load, ordered delivery per group, recapture of failed records and dead-letter handling. Databricks is the next target.
You group related collections by a key, such as an item and its stock records by item id. All changes for one group key travel through the same partition queue and are applied in order; other groups move in parallel.
The integrate agent does not stop. It asks the capture agent to read the record again from the source and sends the current version through the flow. If that keeps failing, the record waits in a dead-letter queue until an operator chooses to retry, recapture or discard it.
Yes. Configurable transforms such as the relational mapping need no code. For anything else, the transform template wraps one Go function you write with everything an agent needs.
No. The broker, controller and agents run in your network, and changes travel only between your source, the broker and your target.
Aestus is an enterprise product, delivered with consulting from Helium Consulting Services and Providentia World Wide. Early access is closed at this moment; write to [email protected] to talk about your use case.
Aestus is Latin for the tide. Like the tide, Aestus keeps moving while you sleep, comes back for anything it left behind, and never lets an older wave wash away a newer one.