I think there is something much more fundamental going on...
1. Maintenance is expensive.
2. Maintenance of living systems is even more so.
3. People hate spending money on maintenance.
4. You reap what you sow.
This is not a Linux problem - it's a culture problem (and probably goes well beyond the US).
[0] - https://infrastructurereportcard.org/cat-item/bridges-infras...
> The Case for Oracle Linux
I did not see that twist coming.
Strangely, the author makes a good point. Oracle or not, sticking closer to the upstream kernel is not a bad way to manage an “enterprise” kernel. Maybe we’ve all been too willing to sit back and accept how RHEL does kernels as “the way”.
>Oracle Linux... set aside your automatic (and justified) repulsion
I can't for two reasons:
1) The only way I avoid the repulsive side of Oracle in this is if I don't care about having a reliable guaranteed enterprise support contract. No one is going to have more expertise in how this distro is built & functions than Oracle, and that's the no-go zone.
2) It's not just about the current status of the project but also the future and what happens if this author is successful and lots of people switch over to Oracle Linux. Nothing stops them from switching things up at short notice (unless I pay them for a service guarantee!) and pulling the rug out from underneath the whole thing. And the more that people adopt it as their distro of choice, the more likely Oracle is to look around for some way to monetize it at gun-point (well, crippling legal bills at least)
»Defenders will counter that this is a necessary tradeoff – you can’t have both stability and being fully up-to-date on security fixes. I’m not convinced that’s true as there are other Linux distributions that have shown you can have both. Rolling release distributions like OpenSUSE Tumbleweed follow upstream much more closely while still maintaining stability through thorough automated testing. Additionally, distributions like Fedora Linux, while not technically a rolling release, do tend to hew close to upstream versions at a more regular cadence.«
I'm sorry, but this paragraph proves that the author has not fully understood the purpose of enterprise distributions.
The primary selling point is not the stability of the software, but the stability of the API and ABI. There is closed source software such as SAP that is compiled against the ABI of RHEL and SLES. And lots of expensive enterprise hardware ships Linux kernel drivers in binary form only, so that a stable, i.e. never-changing kernel ABI is required. Examples for such hardware is the SGI UV series whose drivers come in binary form only and therefore the hardware is supported on enterprise distributions only.
Both RedHat and SUSE put a lot of effort into keeping their API and ABI stable, so that enterprise customers are guaranteed that no distribution update is going to break the software that they're running on the hardware they're using.
Also, when you deploy Linux on hundreds or thousands of clients, a rolling release distribution would be a pure nightmare. Given the large amount of clients and applications, there will always be a combination of applications and use-cases that will break after a distribution package was updated to a completely new upstream version.
Companies don't want software that changes all the time since they want to control the time for an update rollout themselves.
What's missing is an example of a case where there is a potentially relevant exploit:
Many times, even Debian has avoided being hit by a vulnerability, just due to a good wack of issues being in fresh code changes.
And many times, even when the vulnerability has been a long existing one: RHEL is pretty focused on defense-in-depth, through compiler flag choices, SELinux setups, etc.
We use Oracle Linux where I work because we used to use Solaris. At one point it looked suspiciously like Redhat had explicitly removed support for some Oracle hardware from their kernel and the UEK saved us. Besides the kernel they also offer a few other extras and optional newer versions like dtrace, support for some extra filesystems and newer KVM. I get the impression that the fact that they're underdogs in the Linux market helps to keep them honest. Much of their key staff are clearly open source advocates even if Larry only cares about money. And while the Oracle support is not up to the standards of the old Sun support it really isn't a bad choice. And it'd be easy to switch away if we ever need to.
> Oracle has also publicly continued to commit to keeping “…the binaries and source code for that distribution publicly and freely available”. At this point their distribution has been around for 17 years, and they’ve never tried to restrict access to their source code in that time.
I’ll be impressed when they open source Oracle DB and let Microsoft fork it and sell support for it. It’s pretty easy to commit to “open source” a copy of RHEL. Especially when linux isn’t core to their business but a bait and switch to get their customers on Oracle’s platform.
As a former hobbyist that is now working a lot with RHEL as a sysadmin, I was really surprised to learn how little advantage there actually is with buying Red Hat Support.
You are always limited to their opinionated decisions on what your deployment should look like* (or risk losing support), but at the same time, the support you actually get is next to none.
If I can't fix it myself, it ain't gonna be fixed.
At this point, I don't know what we are paying for anyways.
*As an example, more recent versions of RHEL only allow the use of NetworkManager for permanent network configuration. In a production hypervisor system, NM is completely unsuitable in my opinion. It's full of footguns and that will bite us at some point
How is Kernel live patching as an approach for reaching the middle ground for enterprise applications? Amazon linux 2023 makes use of it.
Note how “Ksplice was the first project for live patching the Linux kernel; however, ksplice was sold to Oracle and eventually changed to a closed-source tool.”
https://www.redhat.com/en/topics/linux/what-is-linux-kernel-...
> Rolling release distributions like OpenSUSE Tumbleweed follow upstream much more closely while still maintaining stability through thorough automated testing.
That is different kind of stability. No rolling release distro offers API (nevermind ABI) stability the way rhel etc do, which is the big selling point. You can install updates on enterprise distros without needing to worry too much about your application breaking because someone decided to have some fun with the API.
Holy.... that Oracle linux page is so friendly it almost makes me puke. I can't believe anyone would fall for them
I could imagine an updated take on “Enterprise Distro”, in order to take advantage of the constantly shifting sands, being by and large defined & differentiated by some fantastical gauntlet of automated V&V pipelines: big piles of test suites, fuzzers, prop tests, trace-based testing, etc. etc.
More or less setting up a rigorous de facto compliance regime and making the arms race for enterprise distros be about who can establish the best regime and best gauntlet automation.
What's not clear to me is that newer software is safer or just less tested.
Although it's possible that newer versions are more secure, I couldn't find evidence of it yet. (Links are welcome).
>*What's wrong with enterprise Linux*
For me was when I was trying to incorporate Redhat, Intel and Mirantis as a secure platform for on-prem-cloud as a service offering - and we were setting up the environment with RH, Intel and Mirantis (I was the Mirantis tech PM) - and it was a nightmare of IBM-esque beurocracy.
it reminded me of the nightmares of installing equipment in an IBM/Intel DC in the 90s...
Too fn many non-technical stakes in an undefined landscape and RH attempting to "feel enterprise" with Intel - and it was a disaster. (two FN months to get IP allocations????)
Yeah - I signed off on redhat way before this - but this is what made me hate RHEL.
They got too smarmy, just as LinuxCare did...
But to answer the Q -- They attempted to get 'too enterprisey with it' and emulate those who they were previously trying to take down.
(want some linuxcare stories)
If you have to run an "Enterprise Linux", you are probably required by regulations or compliance to have a support agreement in place. Whether it's RedHat or Oracle or something else, security is going take third place to compliance and stability.
The article claims that since security fixes land in latest releases first but must be individually backported to older releases, therefore staying on an older release is less secure.
However I think this is in itself a false premise. The obvious counterargument is that it is newer releases in which new security vulnerabilities are introduced, so after a while, an older release with backported security fixes is more secure.
Which argument is true depends on the relative rates of fixes vs. vulnerability introductions. This will differ per project, and therefore the balance may tip both ways too, depending.
But the article fails to make any assessment of this at all.
I think that one of the biggest problems for my use case is license management for things outside the data center. With windows I can buy a pro license and have it attached to the hardware. I never have to worry about tracking or managing the license. With RHEL I have to make sure that the computer is registered and logged in to the license correctly. There is a lot of extra work for things like developer workstations or instrument controllers etc. If I could just buy a license of a major version of RHEL for $200-$500 and have it locked to the hardware like windows that would relieve a major administrative burden.
>This collection of software will remain locked at its specific version throughout the lifespan of that Enterprise Linux distribution release – which is often 10 years or more.
Is this 10 year assumption true?
Are people running 10 year old versions of operating systems today in non-safety or non-high security aspects?
10 years ago today is when the following were roughly released:
- Linux 3.8.
- RHEL 6.4 with Linux 2.6.32-358
- RHEL 7 was Beta with Linux 3.10.0
Are these still alive out there?
Reminds me of CERN Linux, which was downstream of RHEL when I was there. It took several years to certify all the CERN software on a new release, which surely must've cost the org many more years of cumulative developer productivity.
Enterprise stuff is decided by directors over fancy dinners, charity events, and business class plane flights by bureaucrats. Enterprise Linux needs to learn how to play the lobbying game
This is sometimes called "compliance".
At a previous job we had a bunch of servers that were reaching End of Life and the tech staff already was planning on upgrading, but someone in the bureaucracy with veto and signature powers decided to extend their life to an extended warranty.
We had to keep running those slow and power hungry things for a few more years because we run out of budget.
> By locking packages to specific versions and only backporting select fixes, these distributions lag significantly behind upstream versions when it comes to security bugfixes, albeit unintentionally.
And the "upstream versions" are how better from a security POV when they are mostly an incompatible redesign ? Or the OP means that GTK4 has fewer CVEs than GTK3 because it is newer and better ?
A new version of a library (or a program) does not mean a more secure version. It is mostly "new features" and the bugs are left there to rot.
Erm, enterprise Linux systems backport major new features all the time. The version number of some software means very little. This article's whole premise is flawed.
Doesn't Debian Stable have the very same problem? You get dated software with bugs that are solved in newer versions. The fundamental model of freezing and (endlessly) testing some state of the packages that makes up a distribution simply doesn't make sense in a world with so many and so big changes.
This model is the "waterfall" model for operating systems.
What is Oracle going to do with the userland/RPM compat with RHEL? It's wonderful that they try to ship every fix from the LTS kernel tree and that UEK users already don't have an expectation of a 100% identical kernel. But you still need userland compatibility, which will be much harder to ensure given the sheer number of RPM packages out there.
“The engineers, maintainers, and testers […] put in the enormous amount of thankless work”
No, they actually get paid pretty well to do it. And pretty sure all the paying enterprise customers are “thankful” for the distribution’s stability and security, or they’d take their business elsewhere.
TBH the gist I get from this article is that the way the Linux project deals with security issues is a bit of a joke.
Businesses only care about security fixes as far as compliance and a reasonable level risk.
It’s more important to them type be able to tell auditors that they are following a mitigation process.
In my case that means all I’m doing is pulling Amazon Linux images and running automatic security updates.
Using Amazon Linux is the path of least resistance because it comes with AWS tooling and it’s already optimized for AWS.
Everything else is basically “who cares?”
This whole model is in decline. Most companies have moved to the cloud and are using kubernetes. If you're running servers a datacenter then you might use enterprise linux. You are probably using some kind of virtualization like ESX or Nutanix. You might be using k8s there too. The age of running your own linux servers is ending in a way.
Just move to BSD.
[flagged]
The problem is that there is Enterprise Linux. You don't see a consumer windows, right?
Talk to the CFOs and CTOs. They fund the hardware refresh cycles and have bonuses attached to regular appeasement of shareholders, the market as well as auditors and controllers.
The initiative begets the spend. The spend and roadmap stretches 10-12 years, chassis fans and PSUs have MTBF that conveniently coincide with Moore’s law and the dark arts of finance, where ownership is good but only to that boundary.
There is another problem that wasn't covered in the article. The 10+ years of stability leads to behaviors and outcomes that remind me of the long-lived SSL certificate problem. Updating is done so infrequently that the "how?" is forgotten. As the 10 year support limit approaches, most of the old team members who did it last time are gone, tech debt is through the roof, few people know where everything is or how to build it, and so on. Enterprise Linux "stability" enables all sorts of bad behavior if your company is inclined that way.
LetsEncrypt did us a huge favor by forcing automation vs having the guy who knows how to update the SSL certs every 4.9 years and left 6 months ago. I'd like to see the RHEL stability model go away too and force people to complete their automation and solve the problems of being able to rebuilding on demand - and actually doing it.
(I know, most HN folk are well disciplined but there are a lot of corporate cultures that are not.)