State synchronization is a bit of a hot topic for front end development since most of the “middle end” deals squarely with solving that problem.
I also agree that the way many APIs are built are pushing complexity to the FE: https://bower.sh/dogma-of-restful-api
However, virtually every company I’ve joined has needed to serve multiple clients. In those cases REST is the status quo. Some people have opted for graphql but even that makes some uneasy because tuning gql is non trivial. CRDTs are completely foreign to most organizations unless they deal specifically with multiplayer modes.
So while it sounds nice to ditch REST it’s just not realistic for most orgs. This is why I’ve been developing a side effect and state sync system that can easily interface with REST, websockets, graphql, etc because it’s built off of structured concurrency.
While I superficially agree with the point, the article author states that they haven't tried the alternatives they suggest. Implementing CRDT or OT can be a complex endeavour, especially if you're retrofitting on an existing system that already has a REST or similar API. Use case is ever important, don't spend your all important time implementing a complex state sync if it turns out your customers don't do the kind of collaboration that requires it.
There is a very simple but often missed pair of solutions to this REST+sync-state problem.
1) When fetching data from the source (to appear on your web form that someone will edit), ensure that you always also fetch a "version number" (or version number dictionary for multi-tables) attached with the source data (a la MVCC). (This does imply you control the backend database and can have a version# field in each table or your CRDT event table.)
Only apply updates/deletes on your backend when the version number still matches, otherwise return an error saying the state has changed, refresh the UI and have the user retry their data entry.
OR
2) When sending information back to the server in your REST PUT, include both old+new values. On the backend, only perform the UPDATE or DELETE operations if/WHERE the old values are still present (preferrably wrapped within a transaction). Otherwise return an error saying the underlying data has changed and refresh the UI and have the user retry their data entry.
REST as the author finds out is not for state synchronisation. CRDTs are also not about state synchronisation, only so in an eventual sense, so similar to REST, you can make it work, but you are twisting the approach with assumptions and sticky tape. State synchronisation is a “solved” problem, in the form of distributed data stores, or in some more relaxed sense, database replication, CDC etc. Not an expert, so this is just an opinion even if it sounds assertive.
What part of this implementation is ReST? I see a CRUD API that uses HTTP verbs. State transfer implies multiple steps at least, where I call the initial API which then returns several other API endpoints "transfering the state" typically in a HATEOAS style - perhaps that part isn't required.
https://en.wikipedia.org/wiki/REST
> ... although this term is more commonly associated with the design of HTTP-based APIs and what are widely considered best practices regarding the "verbs" (HTTP methods) a resource responds to, while having little to do with REST as originally formulated ...
Ah, I see, the industry has taken to just calling HTTP ReST for no apparent reason.
As far as not being sure about CRDTs, these protocols were made to overcome the obvious and terrible shortcomings of CRUD APIs (lack of communicability and idempotency). Who ever wants to see "the data on the server has changed, do you want to overwrite the data on the server or overwrite your local copy". If you're not doing some sort of event sourcing for state management (ES, OT, CRDT) you're probably doing it wrong or doing it for a simple project.
Curious to her other people's opinion of this. I arrived at a similar conclusion after making web apps for research control software in my PhD. I went through a Yjs tutorial and looked into integrating it with fastapi websockets. But this seems like a pretty unusual thing; there just isn't enough people doing this.
Nice user-friendly libraries and tutorials don't exist for smoothing the transition from REST to CRDTs, should your app need that.
This is right smack in the territory of the CAP theorem, which a lot of distributed databases using highly reliable networks can now somewhat functionally ignore.
But in the land of highly variable reliability of clients, you got to think about it you are CP or AP. You HAVE to be partition tolerant because clients are so unreliable.
Maybe if the headline said Stop using REST for state synchronization in these scenarios, I would have been on board
Dismissing something outright is usually the wrong answer, and for the same reason I did read the article
Using REST for synchronization isn’t some huge transgression and is fine in a lot of scenarios, silly in others.
Keeping record updates that a user makes in one system in sync with a single record in another system that uses those updates is fine
Trying to sync giant databases in bulk on the other hand might not be
"State transfer is not state synchronization" is such an obscure point to make.
State sync over REST can be rather trivial by keeping resources immutable (hence also cacheable), and delivering a root resource during the initial endpoint contact. I.e. like an index page linking to static assets. Which is all very familiar to REST design.
The article told us we should stop using REST for state sync, yet it didn’t really tell us what to do instead.
Electric SQL and their various packages looking pretty. Can't comment on actually using them though
The pain points described in this post are exquisitely painful. Makes me bullish on sync engines.
I work on the referenced Braid project (https://braid.org), and the author is right on— HTTP needs to evolve from a state transfer protocol to a state synchronization protocol.
What he doesn't seem to realize is (a) how doable this is, and (b) how beneficial it is once you solve it.
First, the Braid extensions to HTTP [1] achieve this already! They transform HTTP from state transfer to state synchronization with four simple, backwards-compatible extensions to HTTP, which work in today's browsers and servers, with a simple polyfill library [2].
You can use this today, with or without any CRDT or OT algorithms. In fact, 99% of apps don't need a fancy CRDT — they just need the subscriptions feature. This is really cool: instead of GET responding just once, a Braid-enhanced GET will keep the response open, and send you new updates each time the resource changes, until you're done with it.
This gives you realtime synchronized HTTP without a Websocket. No CRDT or OT needed.
You only need a CRDT or OT algorithm if multiple writers mutate the same state, and your app needs to guarantee perfect consistency. And if this is needed—no problem! We've got libraries for that that use the other 3 independent Braid extensions — essentially adding a few more headers to carry over the CRDT/OT metadata. They work with any CRDT or OT.
Second, I don't think anyone yet realizes how big of a deal this is, so let me explain. The lack of realtime synchronization in today's HTTP is what drives developers to move their state synchronization traffic to a websocket, with some random pub/sub protocol that meets the needs of their app — but is totally incompatible with other developers, other websites, and other tools. This means that every website uses standard HTTP to serve its pages, but has a non-standard protocol for its internal state. As a result, websites openly link to each other's pages, on the open web — but the internal state of websites has become a walled garden. This basic limitation in the architecture of HTTP has led to the centralization of data silos in websites. It's simply too difficult for a peer-to-peer community of web developers to build apps that re-use each others' internal state, in the same way that they can link to each other's pages, because they lack the infrastructure.
We've been building the tools for this new style of peer-to-peer synchronous web of state. I just presented on it a few weeks back here: https://braid.org/meeting-107
Cheers to the author for correctly identifying this problem. And onward to the glorious future of the synchronous web of state!
[1] https://datatracker.ietf.org/doc/html/draft-toomim-httpbis-b...
[2] https://github.com/braid-org/braid-http?tab=readme-ov-file
The problem is the frontend web application model is not great design. People are designing distributed systems when all they really want is to provide data to the user, or provide a user interface (which provides data from the user back to the system). You can do both those things without being stateful on the frontend.
There are multiple network protocols for synchronizing state, and they are annoying as hell. Necessary in some very specific/limited situations, but not for a broadband mobile or web application. Even when we were on 33.6k modems we didn't need it.
But then again if we were talking about native apps, we'd still be using HTTP APIs to transfer state data, because developers can't be arsed to do anything they're not used to. (Unless you show them something more technically complex, yet easier to use/think about; shit's catnip for coders)
It seems like the author has never worked on an interface that uses realtime UI syncing. The points about cumbersome and mundane issues when working with REST UIs are not wrong, but the challenges of working with realtime synced UIs are substantially more difficult. And I'm not talking about conflict resolution, which can be abstracted by many existing solutions. I'm talking about the intricacies of UI behaviors that are specific to each application, when suddenly every interactive element can potentially be interacted with by multiple parties simultaneously.
For most applications, the status quo whereby a little bit of recent state is maintained locally, and periodically pushed to a centralized store, seems to provide the best overall balance of complexity and functionality. One exception is that most applications do a poor job of actually maintaining the local state (e.g. most forms clear when you refresh the page, which is not hard to fix).