Don't do a full rewrite ; it won't work, it has to be gradual. We were asked to do this (also PHP, no framework and around 1m LoC) a 2 years ago and we just quietly refactored and containerised parts; the client is super happy but we actually didn't change that much. The major thing was that we didn't change/fix all the fragile stuff that was in there, we just added added microservices and packaged it up in docker. Now they can write new things with Laravel and react (their choice, not mine) and not worry about the old 'core' that runs a lot of stuff and gets called via API's.
Don't build from scratch :-)
It's a rare project that a rebuild succeeds!
My experience is with Laravel, so not too far off your ideal of using Symfony, though I wouldn't let minimalism cause you to avoid Laravel and end up having to do more work. Whether you use Laravel or Symfony alone is probably negligible compared to the application itself.
For such a large application - in keeping with avoiding rebuilds - I'd be looking at ways to subsume parts of the current application into (say) a Laravel application, with glue code at the edges where necessary to paper over differences between the legacy code and the framework-backed code.
Continuing on that path with parts of the system - while (crucially) keeping it up and running for existing users - slowly replacing/enhancing existing code to work in a more coherent fashion with the guide ropes of a framework.
Eventually you ought to end up in a position where you can pick up speed and tap away at the legacy parts.
Ref: your final paragraph, in a decent framework you can quite easily scope queries to restrict the data queried via SQL. Particularly if you have a strong permission system this becomes easier! I like to avoid database-per-customer unless absolutely necessary, it has its own issues, like keeping all databases in sync and on the latest migrations.
Sounds like you've got a fun job ahead of you, holler if your org could use some help :) UK based.
1. Multi-tenant architectures inside of applications is a proven way to keep data separate. Every relevant table/model should have a tenant_id column on it indicating what customer that data is tied to. Indexes on that column and unit test coverage to setup two tenants and try to access each other's data should help convince management. If they want the company and app to grow, they aren't going to want to spend a 6 month project collapsing 100 to 1000 databases and dealing with databases out of sync, etc. Its a maintenance nightmare to have a database per customer at the volume of users you have. I work on systems with 1000's of customers and sharding has not come up at all.
2. Use the Strangler Fig Pattern (aka strangler pattern) to slowly rewrite the app in small iterations with a feedback loop. Write a new component that slowly takes over the old component and brokers between the two until the old one is, well, choked out. You state you're running docker, so figure out what the new tech stack is, call it new_stack (the tech choice here doesn't really matter for a web app and this pattern):
- deploy a docker container running nginx, and new_stack
- deploy a second docker container with the old PHP stack on it (old_stack)
- setup nginx on new_stack to proxy all HTTP requests to both containers
- at first it proxies everything to old_stack container
- rewrite some full service part of app in new_stack and change the proxy rules to route traffic to new_stack for that service
- repeat the last step for each part of the app as its rewritten
- if you run into major issues, update the nginx rewrite rules to route back to old_stack.
DigitalOcean and CloudFlare probably have features to do what I describe above, I don't have details but look into load balancers where you can shard traffic to different servers based on the HTTP route.
> One of the primary complaints we get is that our current system is too slow
Before rewriting the whole thing, are you SURE that you can't make that thing faster?
Speeding things up is not too difficult usually. My experience is that there is always something that takes up 95% of the time. And if you find a way to make your app a lot faster without rewriting it, you'll be a hero.
Replacing the app sounds painful. And btw the architecture you propose sounds way too complicated, the 10k-line controllers are more manageable than your microservice proposal.
A few thoughts:
1. This will last years and never be finished
2. You're not starting from scratch. Not sure why you want advice on that. You're building something that needs to integrate with an existing pile of garbage. Most probably it'll be an absolute ball ache especially around data management.
3. Because of the above I'd just leave. Seriously. It'll be a mammoth task and if all you're getting is a junior salary it's probably not worth it
4. Just use laravel and architect things the laravel way. Using services as well MVC is fine, especially if you'll have an API too, but in general the less you diverge from the framework's usual approach, the more maintainable it'll be.
5. Of course there's a risk with a single DB. However if you architect for a single DB, there's nothing stopping you creating a dedicated one for critical clients.
6. Have you thought about leaving? Expect this to take several years and dominate all dev work until then (or lose steam at some point and be stuck with a mix of both). Ultimately this work adds little to no tangible value to customers and at some point management are likely to want to see new features being added again.
My take on this. We have been in this situation as well, we rebuilt a project formerly built by another team.
1. First, get the database design right, simplify it as much as possible, and consider that you are going to migrate data from the old design to the new design too. Chances are that some of the old structure might not be needed anymore. Work with your client to determine which tables or fields you can already remove. Also, for data that is not expected to be frequently searched or retrieved through complex queries, considering organizing those in JSON format.
2. Do not code right away. Do a very thorough plan of all the controllers and models you need, and review functions that will be reusable, especially in the models; do the same thing for helpers. You mentioned that it was built by a founder without formal training. There’s a good chance a big part of the code does not follow SOLID principles. Your ultimate goal in this planning is to reduce duplicate functions and make it easy to write test cases for your code, both in the BE and FE.
3. For complex or big features, regardless of your team’s experience, I would encourage you to still use flowcharts. Finalize the logic first before coding. You may find logics that can be simplified or is critical to your database design if you do this.
4. Implement TDD, then configure your CI pipeline to automatically trigger test cases and block merging if a single test case fails. This will instill discipline in your team and reduce the number of bugs.
I don't know if it's an option for you to switch to another language. If it's an option, use TypeScript. Strict typing helps early detection of errors caused by incorrect types or accessing missing properties.
From my experience of doing a full migration of a not too large frontend app, its gonna take way longer than anyone expects. And there are going to be higher ups that are going to be upset about it, because whoever sold them on the idea of a full rewrite, said it would take x but its taking 2.5x or how ever long it takes.
I think we need to define the problems.
It sounds like the industry changed. Get business processes or business data requirements first. What do they want to keep, change, get rid of, or add?
Template out your business objects and data structures. On the topic of a single DB... I wouldn't. Unless your clients are sharing data, that's just a huge security risk to keep it all together with website accessible from anywhere.
Then make your changes gradually. I assume you have no tests to look at for requirements, otherwise that would be helpful. It might be possible to migrate some pages or functionality by pulling in the new stuff using some sort of hybrid routing or microfrontend.
Any way you look at this, these sorts of rewrites are painful.
Get Jetbrains PHPStorm: just the ablility to browse classes/functions is worth the price
Use various tools to automate creating documentation/diagrams of different aspects of project. aka uml, php process flow, customer usage, etc.
On dev box, test out/validate assumptions of various modules / interactions noted above.
Should be able to get a feel for code & how being used (internally & externally) for ideas on how to approach end question.
My first question is "How is the business doing after 11 years and where is it headed in terms of current growth, customers, founder(s) vision etc". If the answer is negative, then the technical improvements won't matter much.
But let's say that the business itself is doing good and there truly is a need to improve the tech now to help do even better. In that case, DO NOT DO A FULL RE-WRITE. DO NOT. sorry for caps but it is important.
Like many others have already suggested, start with a smaller piece and improve it. May be create a separate API/service. Then measure the improvement in developer time and customer happiness linked to it. Build that slowly over time.
"I want to keep things simple, and avoid bringing in too many dependencies, so I am leaning towards a minimal set of Symfony components, rather than something like Laravel"
That is good thinking.
"One of the primary complaints we get is that our current system is too slow. In part, this is because most actions trigger a full page reload."
Instead of a full rewrite, focus on improving te performance here. Yes, a full rewrite sounds exciting but how will that impact the business ? Instead, what if you figured out a way to improve the current performance bottleneck first ?
Remember that customer don't care about tech. They care about a solution that solvs their problems fast, efficiently at a reasonable price.
I am also concerned about the fact that a junior dev is being asked to rewrite. I understand it is just 4 of you but depending on the number of customers you serve, do not take this lightly.
Source: I have built software and run my own SAAS. 20+ years of experience breaking shit in production.
I had to remove a lot of details from the original post due to HN's size limit, but I'm happy to answer any questions/provide more information in the comments
1. Write tests as you go!
2. Then write more tests as you find you've missed things.
3. Take time for yourself.Best option quit and join a real company.
In your 4 person company, how many people are developers? Is it just you?
With no offense to you, a junior dev leading a rewrite of a large messy application is almost guaranteed to result in a different messy application. That's really the most likely result regardless of who is doing it.
I would add on to what everyone else is saying and say don't do a full rewrite. Instead, refactor small chunks at a time.
> One of the primary complaints we get is that our current system is too slow. In part, this is because most actions trigger a full page reload.
Full page reloads don't really need to be slow. Measuring and improving web page speed is well established, and IMHO you should work on this and understand it before you build something new; there are a lot of ways to build a page that will be difficult to make fast, having experience making things fast will help you start on the right foot towards fast pages.
You should be measuring at least time to first byte, time to last byte, and data size on the server; times in ms. On the client side, assuming you've got client side analytics, you want to measure the kinds of things that Google PageSpeed Insights measures: https://developers.google.com/speed/docs/insights/v5/about If you can get this from (possibly sampled) real clients, that's best; but if not, at least test occasionally with the tools inside your browsers.
A very fast PHP page can be done in < 10 ms, server measured. But more realistic is time to first byte < 15 ms, full page content in 50-100ms. If you're much beyond that, you need to measure parts until you find the time, and fix it. You can work this problem at the same time as some refactoring by including timing with refactoring; chances are, a lot of the server time is in database queries, and instrumenting timing on database queries and centralizing the database interface into a single place can go well together. Ideally, you can push this timing information into your access logs, and you'd have a list of timings in each log line; time to first byte and total request time are best measured by your webserver rather than in PHP itself; especially if you're using frameworks that do a lot of initialization on each request, you must include that in your total, but it can be difficult to measure from within PHP.
Since your users are all in US/CA (and I assume your server is too), if you get your time to first byte under 15 ms on the server, you should see it under 100 ms on the client. And if your pages aren't too big and the dom isn't too complex, a 100 ms server page time should be at your clients within 0.5 and rendered in < 1 s. You can probably do better with partial page updates, but see what you can do with making full page updates fast first.
It sounds like you're replacing an old monolith with a new faux-monolith that's actually just a bunch of microservices wrapped together under one umbrella? I don't know what your underlying team makeup is like, but it's a truism that your app architecture often reflects your team/company organization. What works well for one dev is often hard to maintain for a team of devs, or many teams. How has your company changed or grown over time, i.e. do all 4 of you work the same things together now, whatever comes up, or are some of you more specialized into frontend/backend etc.?
I second everyone else's opinion of "this is too big to rewrite all at once". Break it up piecemeal and replace one feature/area/service/whatever at a time. Your app sounds way too big to be able to successfully rewrite all at once without breaking edge cases and pissing off customers.
Now to venture into IMHO territory... this is my biased opinion as someone who grew up in PHP (including Laravel/Symfony/Drupal/Wordpress/Twig/etc) and eventually moved to fully Javascript clientside apps.
I think the MVC paradigm works well for environments where your app can inherently be a full-stack monolith, like a desktop app. But it can slow down work in mixed environments like the web where the frontend world moves much much faster than the backend, and is usually made by entirely different vendors (Google, Apple, Meta, Vercel) than the backend (all the FOSS stuff, plus Redis and VMs, etc.). It makes it so that it's really hard to change any one part of the stack without changing or destroying the rest, which means your frontend can only move as fast as your backend.
This might be a situation where decoupling the frontend and backend more might help make it all more modular, i.e. the web UI just calls APIs, as opposed to PHP controlling everything from the server side and rendering changes there. That separation can help you with both a modular rewrite and also future changes. Your backend won't have to be as volatile as the frontend. Basic CRUD stuff hasn't changed all that much and PHP can handle that just fine, but I think it's a bad idea in 2024 to limit yourself upfront to Symfony and HTMX, both because you're tightly coupling your presentation to your data/API layer and because probably by the time you finish the rewrite, the web frontend world will have moved on even more. Both of those are already quite niche compared to the big JS systems, so if your goal is increased velocity, using those systems would make it harder to hire and your codebase harder to maintain. It's also harder to scale with # of users if all your services and rendering for all your customers live in the same VM. And it makes it really hard to shard your services across customers (if you ever did want to move to isolated databases, which is a big if).
On the UI performance side, really it all boils down to using more AJAX, which HTMX and PHP can both do, or you can do it manually. But that works better if your frontend is decoupled from your backend, so the server isn't tied up trying to render new HTML chunks while it's also performing database lookups and mutations. The decoupling allows you to more easily manage complex frontend loading/error/caching/optimistic update/network error/etc states, and more cleanly dictate which UI updates can happen entirely in the browser and which to wait for async server data. It also saves server resources since you're not rendering any HTML for the client. Phones and browsers are plenty fast for that these days, and it makes the end-user UI much snappier, usually.
Even if the network request is a fixed value (i.e. the network roundtrip time is the same regardless of your frontend framework), being able to do dynamic clientside DOM mutations like that will make it feel faster for your visitors, even if your backend stays exactly the same. It also makes clientside component reuse easier, as well as managing global state and inheritance.
I think HTMX is fine for simpler sites but I'm not how far it will do for a more complex app... hard to say without even know what "Star" is (a CMS? a CRM? a spreadsheet? ecommerce? etc.)
This isn't an argument to use the big JS frameworks & latest patterns (like Next.js with React Server Components), because in some ways that would just emulate how it already works in PHP (yeah, we've gone full circle lol). It might be a situation, instead, to use something more like the traditional single-page-apps that takes care of the UI clientside but relies on the server APIs for data lookups and mutations. In that way you can more truly decouple the frontend from the data and give your users fast navigation & interaction updates without having to wait for server renders all the time, like you do in PHP components.
I think HTMX can do some light componentization with HTML Web Components, but those are way less powerful than the JS equivalents, especially when you have complex state and inheritance.
If I were in your shoes -- again, strictly IMHO -- I'd do the rewrite piecemeal, but instead of using a true MVC architecture, just have a bunch of REST endpoints that do simple idempotent operations and then tie them together with something like a backend-for-frontend if needed. That way you don't have to change your backend unless the actual schema changes, you can easily and quickly iterate your frontend to seek the latest optimizations in whatever framework or language is popular this year, and in between, your BFF (or just serverless request proxies) can help smooth over some of the roughness inherent in any such decoupled system. Any one part of the system can be swapped in and out as long as they respect the pureness of their predecessors' inputs and outputs.
I think rebuilding the whole thing on a more traditional MVC PHP setup would just result in you needing to rewrite the whole thing yet again in a few more years when it grows a bit more (both users and maybe your team size).
The TLDR is that a cleaner separation at each layer helps makes your app more maintainable & modular, because each layer can be properly abstracted to only handle its own concerns and not have to worry about anything above or below it. I don't think a MVC architecture can really do that because there is no simple correspondence between MVC and the actual Web boundaries (VMs, backend logic, caching, hosting, CDN, network retries, frontend logic, frontend UI).
But again, just IMHO. Others here would probably disagree entirely.
TL;DR Focus on what's important. What's in it for you? How does this project help you, your resume, future salary, or general well-being?
I would second the previous comment about gradually chipping off pieces of your current app into a new foundation/architecture/framework, rather than a full re-write.
Rewrites don't go how we think they will, and the cut-off and migration between the two certainly doesn't either. It becomes unwieldy very fast, and in ways we couldn't predict.
I've had to do a handful of very large PHP-based rewrites, and I would still recommend a piece-meal approach. It isn't so much that one way is inherently "better" than the other as far as the code goes; git is amazing, and tests do what tests do. It's more that you can reason about the features, changes, and data flow if you take it one piece at a time. And also, lots of tools, libraries, and services that you use will release updates in the time that you're rewriting, which causes its own headache too.
Sure, the tradeoff is that going one piece at a time takes longer, but you have time on your side because you have revenue and a a customer base. If you were a startup that didn't have that, sure, go for the fast approach and fix it later.
But I think when all is said and done, we can go fast and build something that doesn't work, or we can go slow and keep both things running and swap out smaller pieces along the way. As they say, "slow is smooth, and smooth is fast."
Also, holler if your org could use some help. Seattle based.