> DBOS has a special @DBOS.Transaction decorator. This runs the entire step inside a Postgres transaction. This guarantees exactly-once execution for databases transactional steps.
I stopped here. I know, authors really want to chase the exactly-once dragon, but this won't scale. If the step takes a long time, it'll keep the transaction open with it for that time. The master replica has to bookkeeping that. That state has to be replicated. That will also affect MVVC further on. As your scale grows, you'll see disk usage growing and eventually swapping, replica lags, AND vacuum halting for surges. I hope your uncalled engineers have a steady supply of coffee.
Disclaimer: I'm a co-founder of Hatchet (https://github.com/hatchet-dev/hatchet), which is a Postgres-backed task queue that supports durable execution.
> Because a step transition is just a Postgres write (~1ms) versus an async dispatch from an external orchestrator (~100ms), it means DBOS is 25x faster than AWS Step Functions
Durable execution engines deployed as an external orchestrator will always been slower than direct DB writes, but the 1ms delay versus ~100ms doesn't seem inherent to the orchestrator being external. In the case of Hatchet, pushing work takes ~15ms and invoking the work takes ~1ms if deployed in the same VPC, and 90% of that execution time is on the database. In the best-case, the external orchestrator should take 2x as long to write a step transition (round-trip network call to the orchestrator + database write), so an ideal external orchestrator would be ~2ms of latency here.
There are also some tradeoffs to a library-only mode that aren't discussed. How would work that requires global coordination between workers behave in this model? Let's say, for example, a global rate limit -- you'd ideally want to avoid contention on rate limit rows, assuming they're stored in Postgres, but each worker attempting to acquire a rate limit simultaneously would slow down start time significantly (and place additional load on the DB). Whereas with a single external orchestrator (or leader election), you can significantly increase throughput by acquiring rate limits as part of a push-based assignment process.
The same problem of coordination arises if many workers are competing for the same work -- for example if a machine crashes while doing work, as described in the article. I'm assuming there's some kind of polling happening which uses FOR UPDATE SKIP LOCKED, which concerns me as you start to scale up the number of workers.
> # Exactly-once execution
> DBOS has a special @DBOS.Transaction decorator. This runs the entire step inside a Postgres transaction. This guarantees exactly-once execution for databases transactional steps.
Totally awesome, great work, just a small note... IME a lot of (most?) pg deployments have synchronous replication turned off because it is very tricky to get it to perform well[1]. If you have it turned off, pg could journal the step, formally acknowledge it, and then (as I understand DBOS) totally lose that journal when the primary fails, causing you to re-run the step.
When I was on call for pg last, failover with some data loss happened to me twice. So it does happen. I think this is worth noting because if you plan for this to be a hard requirement, (unless I'm mistaken) you need to set up sync replication or you need to plan for this to possibly fail.
Lastly, note that the pg docs[1] have this to say about sync replication:
> Synchronous replication usually requires carefully planned and placed standby servers to ensure applications perform acceptably. Waiting doesn't utilize system resources, but transaction locks continue to be held until the transfer is confirmed. As a result, incautious use of synchronous replication will reduce performance for database applications because of increased response times and higher contention.
I see the DBOS author around here somewhere so if the state of the art for DBOS has changed please do let me know and I'll correct the comment.
[1] https://www.postgresql.org/docs/current/warm-standby.html#SY...
This is very cool. How does this compare to tools like Celery and Dagster?
Like we might use Celery when we need to run specific tasks on specific workers (e.g. accessing some database that sits inside a corporate network, not accessible from the cloud). It seems like we can do something like that using DBOS queues or events, but is there a concept of running multiple workers in different locations?
Compared to Dagster, a DBOS workflow seems like a de facto DAG, running the steps in the right order, and being resumable if a step fails, and the difference here would be that with Dagster the steps are more granular, in the sense that we can re-run a specific step in isolation? In other words, a DBOS workflow will retry failed steps, but can we submit a request to “only re-run step 2”? This comes up often in working with ETL-style tasks, where steps 1 and 2 complete, but we need to resubmit steps 3 and 4 of a pipeline.
Dagster also provides nice visual feedback, where even a non-technical user can see that step 3 failed, and right-click it to materialize it. Maybe I need to play with the OpenTelemetry piece to see how that compares.
Celery and Dagster have their own drawbacks (heavier, complexity of more infrastructure, learning curve), so just trying to see where the edges are, and how far we could take a tool like DBOS. From an initial look, it seems like it could address a lot of these scenarios with less complexity.
I built a small side thing using DBOS ( using python SDK) and the ergonomics were pretty nice.
Then I found out Qian Li and Peter Kraft from that team are sharing breakdowns of interesting database research papers on twitter and I've been following them for that.
Does DBOS offer a way to get messages in or data out of running workflows? (Similar to signals/queries in Temporal) Interested in long-running workflows, and this particular area seemed to be lacking last time I looked into it.
I don’t want to just sleep; I want a workflow to be able to respond to an event.
Could someone clarify on how it can achieve exactly-once processing with idempotency key?
Using the example they provided:
1. Validate payment
2. Check inventory
3. Ship order
4. Notify customer
I'm curious about case when one of the operation times out. The workflow engine needs to then either time out or it may even crash before receving a response
In this scenario, the only option for the workflow is to retry with the same idempotency key it used, but this may re-execute the failed operation which may have been succeeded in the prior run because workflow did not receive the response. The succeeded operations would skip because workflow has run completion record for same idempotency key. Is that correct?
Temporal can use postgres as a backend. What makes this product different?
Maybe I'm not seeing it, but why do none of these "postgres durable packages" ever integrate with existing transactions? That's the biggest flaw of temporal, and seems so obvious that they can hook into transactions to ensure that starting a workflow is transactionally secure with other operations, and you don't have to manage idempotency yourself (or worry about handling "already started" errors and holding up DB connections bc you launched in transaction)
This appears to be an open source MIT library that anyone can just use. The docs say it can be run locally against PG. but then they say to use DBOS cloud in production without mentioning that it can also be ran in production against any PG database. Just checking that I am not missing something. I would like to support this project while using it but adding a vendor is not always possible (for example I don’t see SOC2 mentioned).
I'm having trouble in my head understanding how these workflows are actually executed in the open-source version. Are there workers somewhere? How do we provision those and scale those up and down?
how is it different from Conductor? https://github.com/conductor-oss/conductor
Interesting! Is there anything like this that I could host myself?
Putting your data and workflows close to the database sounds good on paper, but often the performance "gains" are not significant when considering the dangers of vendor or tech stack lock-in. TLDR; try not to do everything on one platform.
Bit disappointed, looked for .net core support but no.
Languages that are supported: Typescript and Python. I know programming languages as a topic is as inflammable as religion, but boy do I feel sad that these two are considered the most important these days. For server applications.
Anyways, can people here recommend alternatives with bindings for .net core?
It is important to note that employing any workflow engine needs careful examination of the benefits vs. drawbacks.
You will need full and complete control over workflow orchestration, restarts, deletes, logging, versioning etc.
Your steps will get stuck, they will error out unexpectedly and you will need complete transparency and operational tools to deal with that.
In many cases you will be better off doing the book-keeping yourself. A process should be defined in terms of your domain data. Typically you have statuses (not the best choice, but I digress) that change over time, so you can report on the whole lifecycle of your shopping cart.
Now, and this is crucial, how you implement state changes - any state change - should exactly be the same for a "workflow" than for a non-workflow! It needs to be. A shipment is either not ready yet or done - this information should not be in a "workflow state".
Let's say you shut down your system and start it back up: do you have a mechanism in place that can "continue where it left off"? If so, you likely don't need a workflow engine.
In our case, on startup, we need to query the system for carts that are waiting for shipments and that are not being dealt with yet. Then fan out tasks for those.
That is robust in the face of changed data. If you employ a workflow engine, changed data always needs to two consider two worlds: your own domain data and any book-keeping that is potentially done in any workflow.