+44 (0) 20 7846 0140
Contact us
Jan 28, 2021| Daniel Jones
There is no single obvious choice for those evaluating Kubernetes-based Cloud Foundry distributions and alternatives. Find out why EngineerBetter chose Google’s Kf for our estate, and why our advice to existing Cloud Foundry users is “stay on BOSH.”
In January VMware shut down their public Cloud Foundry (Pivotal Web Services). All of EngineerBetter’s apps ran on PWS and so we needed to migrate away from the platform and find something equally cheap and simple.
Our choices were:
We’ll get back to the narrative of our own experiences at the end of the post, and instead focus on a comparison of the tools.
cf-for-k8s drops many features, whilst taking the approach of “maintain CLI/API compatibility and replace the internals with cloud-native equivalents.” For example the Cloud Controller stays, routing is implemented with Istio, logging with Fluentd, and staging with VMware’s kpack.
cf-for-k8s is spearheaded by VMware. Despite appearing in the main Cloud Foundry GitHub org, cf-for-k8s is in fact an incubation project.
cf-for-k8s would fail Cloud Foundry Provider Certification. The community maintains a list of unsupported features:
cf sshcf push -b)My personal take is that the project would be more appropriately-named “something-that-is-a-lot-like-cf-for-k8s.”
No. The commercial version (the snappily-named Tanzu Application Service for Kubernetes) has been released only as a technical preview.
No.
VMware are publicly committed to cf-for-k8s.
I would invite readers to consider the amount of activity on the relevant GitHub repositories, and to see how predominantly TAS for Kubernetes features in VMware Tanzu sales pitches and marketing materials. Search results point to reference documentation and a download page, but no shiny marketing material. VMware employees have recently been showing increasing interest in KubeCF on the Cloud Foundry Foundation Slack.
KubeCF uses complex tooling to deploy the exact-same Cloud Foundry components used currently onto Kubernetes. You can think of its approach as “take what we have, deploy it on Kubes.”
KubeCF has had most contributions from SUSE, IBM and HCL.
Yes. IBM are running it in production, SUSE have customers running it in production, and we know folks that have been running KubeCF instances for over a year without problems.
Go and ask SUSE if you can buy their Cloud Application Platform. I suspect you’ll get a sales pitch for Rancher’s RKE.
IBM depend on KubeCF for their production offerings.
I would invite readers to ponder whether the expensive Rancher acquisition made by SUSE incentivises them to continue selling Cloud Foundry. What is the Carrier project recently started by SUSE, involving the folks who have been working on KubeCF?
Again, look at the activity on the GitHub repos, and most importantly, see if you can get SUSE to sell you a CAP.
Kf is a set of components made by Google Cloud and offered as part of their Anthos suite. It offers a core developer experience that is similar to using the CF CLI with a completely new and Kubernetes-native backend. Kf does not intend to implement all the features of Cloud Foundry, rather the core app-pushing experience.
Kf started life as an open source project before being made closed source, presumably for the sake of competitive advantage or due to lack of engagement from other Cloud Foundry Foundation members.
kubectl alonekf CLI is not a like-for-like match for the cf CLIwww. as a host)Yes. We decided to run all our Cloud Foundry apps on GKE using kf. In fact, this very page was served by a kf app!
Given some of the scaling limitations of Kubernetes itself, operators should expect to break one large Cloud Foundry into smaller kf Kubernetes clusters.
Yes, and Kf is offered at no additional cost beyond Anthos Service Mesh and GKE. EngineerBetter as Google Cloud partners can also help you with Kf adoption.
Google continue to invest in kf and Product Manager Micah Baker has publicly mentioned a feature roadmap stretching out at least a year.
We needed to migrate away from PWS before 15th January 2021. Our estate comprised around 15 static web applications, and a few Ruby apps. Some of these needed Redis and Postgres data services.
The team enthusiastically started off using cf-for-k8s. After more than a week of effort, we concluded that it was not ready for real-world use.
Whilst cf-for-k8s makes starting a development environment trivial, it does not offer the feature toggles required for a production deployment. A lot of the conveniences in cf-deployment’s ops files are missing.
The lack of old-school buildpacks was an absolute showstopper for us. There weren’t CNB equivalents available, and even if there were, they probably would not have yielded identical results.
Comments from our engineers included “this is so not ready” and “this feels like an alpha.” They did like the use of Istio and Fluentd, however.
We’ve used KubeCF before, and have delivered training on how to install and operate it. Given the small number of apps that we have, and our lack of need for multi-tenancy, the debugging overhead of all the Quarks-spawned init containers felt like something we should avoid.
Kf was the simplest approach given our low app count (fewer than 20). Whilst we encountered a few gotchas (see the limitations on app and domain names above) it was great that old-school buildpacks are supported (as well as CNBs). The operational overhead is minimal, and the effort to change our CI pipelines was pretty minimal.
As one of our engineers commented: “Kf is how I wish cf-for-k8s had been implemented.” All state is in the kube-api, meaning everything is debuggable with kubectl.
If you’re an existing BOSH-deployed Cloud Foundry operator who does not have a pressing need to make changes, I strongly suggest that you continue using BOSH and cf-deployment.
If you’re using BOSH-deployed Cloud Foundry and are concerned about the dwindling pool of BOSH talent to operate your environment, then you have three options:
kf on any Anthos-connected cluster, and slowly migrate to a Kubernetes-native workflowI don’t see any use cases for which cf-for-k8s is the ideal solution. This is unfortunate, as I already know a bunch of operators who have spent time evaluating it and drawn the same conclusion - it’s not ready yet.
Wasn’t 2021 supposed to be the year of Cloud Foundry’s second wind? Bringing the CF experience to the masses of Kubernetes operators who found building their own platform too costly? It seems like, via a self-fulfilling prophecy of strategic decisions, the future of Cloud Foundry is BOSH.
I think we were all hoping for open source users to be able to ride in the coat-tails of a fully-committed commercial distribution built on top of Kubernetes. The current state of Cloud Foundry is a reminder that open source software comes with risks and responsibilities. If you want stability and assurances, then best speak to a vendor.
For now, open source users are advised to wait until the dust settles. Like most things in 2020, the volatility seems to be going on longer than anyone expected.
We’ve not found anything that matches the Cloud Foundry experience. Yes, the imperative cf CLI is an anachronism in the world of GitOps. But Cloud Foundry is more than that - it’s the sum of all the assurances made to operators too. The simplicity of having a batteries-included platform; being able to rebuild container images long after app teams have disbanded; a curated marketplace of dynamically-provisioned data services; a sensible RBAC model; security-by-default; years of production-hardening in regulated, governmental and military environments.
There are great tools out there like DevSpace. The big vendors are all interested in building something proprietary on top of Knative. We’re yet to find something as vendor-neutral and feature-complete as Cloud Foundry.
If it ain’t broke, don’t fix it. The innards of Cloud Foundry are antiquated. Fewer and fewer people will be able to support BOSH. However, there is not yet a ’no-brainer’ replacement that my peers (in other consultancies - we talk a lot) and I can recommend.
Kubernetes is the substrate on which all cloud platforms will be built. Kubernetes is not a PaaS. I wish vendors had moved Cloud Foundry onto Kubernetes sooner. Goodness knows, Cisco and the team that joined SUSE tried.
Five years ago I made an invitation to deliver technology in a way that most respected the expended heartbeats of those around you.
I look forward to continuing my involvement in the CNCF Business Value Subcommittee, to ensure the passion and learnings of the Cloud Foundry community continue to have a broad and meaningful impact.