I tried to build this a few years ago using Salesforce’s webhooks and discovered that low-value tenants are on shared infrastructure where things like scheduled jobs can run instantly… or 10 minutes later, and Salesforce make no guarantees about when things will happen — making any attempt to use Salesforce as a source of truth unreliable.
The polling solution is neat but I am imagining it’ll run into issues with API limits and performance, especially for tenants on shared infrastructure: have you encountered that? Are you limited to working with customers that are paying Salesforce enough to have fast/reliable infrastructure?
(I don’t think it’s a problem for your success as your product is most valuable to those paying Salesforce for reliability, I’m just curious how you are thinking about the problems I gave up on! Maybe things with Salesforce have changed in the last 5 years.)
Salesforce reps don't even know what Heroku Connect is. I couldn't get someone to sell it to me! I think the idea is great, glad to see someone picking it up.
As a developer I love the idea. Choosing Postgres over say a GraphQL API is bold, but it makes sense. The customer can then scale up their Postgres instance as they hammer it more and more, and they might already being using this for their app, so developers can add it to their ORM. Or they could wrap this in a microservice.
I also liked the submission recently about a git client that uses SQL. I like the idea that more things can be exposed as SQL (either directly or by syncing with a RDBMS). There is a lot of good tooling around it and despite the S not meaning Standard, the dialects are close enough that it isn't a problem.
This is very cool! I built a similar tool for a project with the need for two-way data syncing between Salesforce and MySQL via Laravel.
Salesforce objects map quite nicely onto Laravel’s Eloquent models (and booleans work fine! ref @ianyanusko in a sibling thread)
On the Salesforce side we use triggers to send a summary of field changes to Laravel to apply to its MySQL database. These are very cheap in terms of Salesforce limits and consumption!
Changes originating from the Laravel side use Salesforce’s REST API. It’s handy taking the round trip through Salesforce when saving changes as it lets flows/processes run and formulas to do their thing before data is persisted in MySQL.
Syncing data from Salesforce (to seed a database for example) is done via REST too. It works OK.
I considered productising it at one point, but ideas are a dime a dozen, it’s a lot harder to execute well and Bracket is doing exactly that, kudos!
This has been a thoroughly interesting post, and I’ll keep my eye on Bracket. You are however out of budget for my client, we had to achieve syncing on a shoestring ;)
Nice i can see the need for this.
> With the polling method, Bracket stores an encrypted copy of your data as an intermediate source of truth. We mostly do this to prevent infinite event loops, but it also helps with merge conflict resolution.
I see you have an on-prem version, but i am still not convinced why you need to store it? Can it not just be stored in an extra table at the client's side?
Neat! How specific is your solution to Postgres? Could it be ported to another db engine?
(And, how are conflicts resolved? In a huge system with millions of records coming from everywhere it can fast become nightmarish?)
This looks great! The UI and docs look very nice. Of course long-term reliability is what really matters in this space. I can definitely see incorporating this into client proposals.
A few things I can't immediately see from the docs: do you support subsetting a data source -- only sync records matching criteria? Do you support to/from different instances of the same connector (e.g. Salesforce to Salesforce)? Can you perform any transformations like map over the data, normalize or denormalize tables, etc?
Many clients I can think of this being most useful for would rather host it themselves, is that an option?
One critique: I can't imagine recommending this to a client without SSL support. I'd highly recommend just baking that in to every tier. It would demonstrate that you're serious about keeping your customers' data secure.
Neat. I once wrote a Salesforce to Oracle sync app. I think the goal was to bulk pull data out of Salesforce for processing in Oracle to avoid Salesforce API costs. This was years ago. It was a fun and aggravating project.
I'm currently on a team tasked with thinning out our Salesforce App Exchange app. Bracket looks very cool.
1. Do you have any plans to release an API rather than utilizing a webapp for defining the sync? 2. Does your Salesforce integration support syncing metadata, including custom metadata? 3. Do you have data on the impact Bracket has on platform events and other Salesforce limits? 4. Can you share any information on pricing?
Thanks!
As the founder of what would become Heroku Connect (aka Cloudconnect), this thread warms my heart. Love to see the innovation - good luck!
Even if this only accomplished one-way directional sync with backup and disaster recovery capabilities, it'd probably find a 10x wider audience.
I really admire the dedication and motivation to be able to work on such incredibly boring things (for me).
Looking forward to the HubSpot + MySQL integration.
Disclosure: I'm one of the HubSpot founders.
I wish you had built this in 2021 but congrats!!
Message me, might have a use-case with a Fortune 100.
Doesn't fivetran/hightouch do this?
Congratulations on not giving up and solving a boring but hard problem.
What's the tech stack?
Get in touch if you want to expand to NetSuite. We can have a chat...
Love the focus. How did you find the people to talk to?
Is bracket using Postgres CDC (via WAL)?
most difficult part building this?
Meta observation: lots of YC companies basically build their startups around some workload (e.g. salesforce <-> db) and sell some managed service around it. This game is quite difficult to scale and sell to enterprise. But it's understandable that many technical founders are most familiar with these types of "painpoints" and thus have their startup thesis around them.
We've built similar thing to sync certain Salesforce objects to our .NET backend but Salesforce has their streaming all built around CometD which uses something I believe an insane person came up with - Bayeux Protocol. It's essentially an HTTP streaming, you shoot a GET call to an API and it starts streaming bytes of data. If it stops or timeouts, you're supposed to shoot another GET to continue streaming bytes, if nothing comes back you timeout and retry.
It's an absolute hell, most community frameworks barely work, there's zero good solutions to error handling, we had our Azure Application Insights constantly red because of timed-out requests that are in fact good, because they just mean there were no events to transmit.
I refuse to believe that multi-billion dollar company that is Salesforce couldn't come up with a RabbitMQ sink or virtually any better solution to that problem, especially that they could gate it behind a subscription most companies would pay for since they spend millions on SF either way...