Container Orchestration Debate

Regardless of the merits or drawbacks of "de-clouding" for this particular company, it seems to me that their ops team is just really bored or permanently unsatisfied with any solution.They say that they've tried deploying their apps in all of:* Their own Datacenter* ECS* GKE* EKS* and now back to their own DatacenterEven with their new "de-clouded" deployment, it seems like they have created an absolutely immense amount of complexity to deploy what

Companies aren't static. I'd consider Heroku (or similar) to start, but once I had funding/revenue to support having ops, I would transition away.However, once you have developed a workflow, going back in developer experience is hard, and hacks like these become common. (I saw a team do a similar tool to avoid paying for Cloud Foundry.)

I think this is a little unfair.Anecdotally, I've had a good experience with them. I use them as a playground for ideas or learning things like docker/kubernetes.I see no reason why a properly architected deployment would have trouble in their environment.

if you don't have the problem it solves, don't use it. you need clusters services that scale up and down quickly. it's not the best way to deploy one server, it's the best way to deploy 10k servers, turn them off, and deploy them again. that's not even mentioning monitoring etc

Not ideal. But I am using more "cloud native" tooling. Cloud Shell, Cloud Build, Cloud Run on GCP. Spinnaker mapped to the production managed kubernetes cluster. Automation. Speed. Ease. Instant revision control. I think you give up a modicum of control for the convenience. But the cognitive load is gone. You are free to concentrate on app design. Cloud native is becoming standard. And for that reason its worth exploring. There is a nice developer-conscious feature in GCP where you can

Sounds like you moved your complexity from machine virtualization to container orchestration. Doesn't sound like you won anything, least of all agility.

I've seen a deployment of a large number of ec2 instances + other Amazon services handled with Puppet+Ansible, and I've also seen a somewhat smaller but still fairly sizeable deployment using a custom Kube cluster. The custom Kube cluster was cool and definitely seemed to make those working with it happy, but I was much happier working with the Puppet+Ansible bare metal setup. It's just easier to figure out what's wrong. I am now working with hosted Kubernetes, which seems to

Isn’t this stuff supposed to be solved by products like DC/OS?

sure, but i'd wager that most of the pain point is getting orchestration tools to work on other platforms. and also vetting of other platforms, etc.

I'd pick none of them. I'd create a coreos cluster and configure it based on cloudinit files. For more complex projects I'd choose kubernetes or something similar as cloudfoundry.The point being I'd try to get as far as possible from any server configuration. Just limit it to setting up the platform as a service. So engineers can just push containers to that platform.I'd also consider serverless architecture.Long story short, limit server configuration to bare m