To cap off years of debates on Kubernetes (or K8S) is fitting for everyone or not, I will finally give my deep feeling which took me a bit of time to build after years of use.
In my case I really like K8S but I don't have a particular problem working with "traditional" infrastructures built on to of VM1 (Virtual Machines) especially because it sometimes benefits some of our customers.
Why this debate is still happening in 2024?
First, I'll try to understand why this debate is still hapening in 2024 and it's pretty simple. If I had to give an analogy for developers of using K8S vs. classic IaaS (Infrastructure as a Service) which provides VMs on demand, is comparable as working with the C programing language or a very high-level stack framework like Spring Boot.
So we had to expect this debate which comes exactly like the one of developers who consider that the framework's users are losing their skills and become proletarians vs. the framework's users who observe that they have a better business velocity. It's exactly the same debate and it's running and will continue to run for years exactly the same way.
Understanding the anti-K8S point of view
In Unix/Linux you already have everything you need to automate resilient infrastructures: command/shell interpreters and scripting languages, schedulers (cron, anacron...) and when you take these elements one by one lot of people will say to themselve "there's nothing complicated, it's very easy to use" or for some of them using it for years "why change a winning team?" and it's a an understandable point of view.
For some people who have capitalized their business for years on those technologies that aren't outdated at all, there's no return on investment in telling them to change if we are honest for a few minutes. If we take the time to think and put ourselves in those people's shoes, we should realize that we are asking them to work without any added value because they already got a high satisfaction rate with minimum of time to intervene in case of issue.
Some will even find flaws2 that can be legitimately debated in the current implementation of K8S like every technology including the Linux kernel. Particularly some cloud players which have implemented themselves a resilient control-plane model that has been working for years and hasn't been fully amortized. It's understandable that those are trying to sell their product which isn't necessarily badly done and which can still answer very well to lot of usecases on the market. However everyone should take a few steps back from the argument because no solution is perfect and in the end what matters is the pros and cons and the tradeoffs we choose to make as decision maker in order to keep the better ROI (return of investment).
Understanding the pro-K8S point of view
Now there's a new generation of sysadmin we can sometimes call "SRE" (which stands for System Reliability Engineers) or even Platform Engineers because we ask them to provide their skills and work routine As a Service with a better time to market as competitive as cloud players which are building their IaaS or PaaS (Platform as a Service) for decades. I'll try here to explain why K8S seems the perfect fit for those people.
Let's continue with the example of cron jobs on Linux. cron or anacron are great, very well known and running for years. If we ask those people to schedule some tasks with cronjobs and make them auditable, resilient in case of shutdown and even highly available, using those well known tools will require to add some structured logs with a monitoring system, implement the exponential retry in case of failure, install those crontabs on multiple servers and handle the concurrency with a semaphore/locks design, reconciliation loops...
That's starting to be a lot of things to handle to do something that seemed simple, right? And this is only one of the easiest example of things to handle when you're in charge of the reliability of your systems and it's also something that can matter for everyone including small business nowadays. It'd be a big mistake and bad judgement to think that's a too fancy consideration for small business especially when they are able to handle this for few dollars per months nowadays.
And that's the thing with K8S: it's already implemented by design without any effort, only 5 minutes of work with a single CLI or API invocation. And this only the simplest example I found but it's exactly the same for every deployment automation aspect we use to see as comodity for years.
What I'm trying to explain here is that K8S which is often seen as a distributed orchestrator for large business and which bring values only in case of lot of autoscaling requirements is a wrong perspective. K8S should be seen in fact the new generation of IaaS with a standard API which takes care of every comodity we can have regardless of the size of our business and which is working almost everywhere and avoid the vendor lock-in.
That been said, it's not because it's available almost everywhere that the pricing model or the cost to move what is already running brings the better ROI for everyone. On my side I deeply regret that there's still no serverless implementation of the K8S API in any cloud provider and that this API is still strongly coupled to a single codebase (with some flaws like we said before). The "standard API specification existing everywhere" is only theorical.
My personal point of view
You might know that we are building our DaaS (for Deployment as a Service) and FaaS (for Function as a Service) solution cwcloud because we strongly believe those are the best compromises between IaaS and PaaS. You can check-out this tutorial to get a better understanding.
We insist on continuing to provide, in an agnostic way, those services on both K8S and classical IaaS3 and don't complain about this because we consider that it's our job to adapt and try to make most of people happy because that's what brings us more business.
However we got to admit that the complexity of the DaaS implementation is reduced by far with K8S and it's noticeable just by seeing those two diagrams which present the architecture of the two implementations:
Without K8S
With K8S
You can easily understand here which version took us more efforts ;-)
So, as a developer, I also strongly beleive that the Kubernete's API is lowering the complexity of the deployments exactly the same way a framework and runtime like Spring Boot is lowering as well the comodity we used to develop ourselves to expose our code as a microservice (http exposition of our business logic, abstraction, logs, metrics...).
As a developer, I also love coding with the C programing language as I love understand low level stuffs in Linux/Unix operating systems. I feel more powerfull and more competent with it. However as a manager I'd be a fool to try to avoid frameworks like Spring Boot to increase productivity. And that's also part of the engineering process to analyze the average velocity and ROI.
That been said, I ain't sayin that we should always pick the K8S option. Not at all, I already explained that it brings no added value for lot's of skillful people or already setup infrastructures on one hand, and also that the K8S pricing offers on most of cloud provider is still not great everywhere4 on the other hand. We still miss serverless offers where the billing is based on our pods consumption only to make people beleive that K8S isn't an orchestrator but a standard API definition to deploy everywhere with some kind of agnosticity.
Honestly, we have customers using cwcloud without K8S and it's working as great with sometimes, depending on the choosen cloud provider, a better pricing model. Like always we got to analyze the pros and cons for each of them and help them to accept some tradeoffs in order to bring the better business value possible.
And that's also the thing for companies which are looking for some high-level and uncomplex way to deploy their stack (PaaS, FaaS or DaaS): it's okay if it's running on top of K8S or not. What should only matter to you is if the SLA (Service Level Agreement) and your own velocity are good and if your provider is reliable. Let them sold you their features and not how they achieved them. And it's also okay to change your mind in the future and rebuild everything when you got more funds and more needs...
- I know that K8S is not related to containers anymore and can orchestrate VM (with kubevirt for example) or WASM binaries. Here I'm refering to classical IaaS or hypervisor we used to use almost everywhere before the rise of K8S and wich are still very used.↩
- For example K8S is relying on etcd which is a statefull component and which cannot be "high available" by design. But some distribution like K3S are offering the ability to replace it by something more reliable like NATs (and some cloud players has made their own rewriting of etcd).↩
- We are compliant with the IaaS API of AWS, GCP, Azure, Scaleway and OVH and Openstack for on premises infrastructures. I'm refering to those when I talk about "classical IaaS" which are providing storage and compute as VM.↩
- At the time of writing, Scaleway is offering a free shared control-plane which is pretty promising (it was unstable but getting better in my opinion). This way, you're only paying your nodes with the same pricing of any compute instances (and their pricing is very competitive, you can have a fully functional cluster for less than 40 dollars per month). But this kind of deal is not very common among the biggest cloud players. Anyway Scaleway's becoming a very great deal if you want my honest opinion :)↩