(Related to #26036)

I've spent the past year working on the perfect deployment repository for argo-cd. Not continuously, but when I had time (and motivation). I'm slowly getting there, but there are some components ruining the fun.

The goal:
Imagine working in a software company. You're running a small number of k8s clusters (prod, staging, testing, management) and want to provide reusable application templates with predefined values for your developers.

• Some applications are deployed once (like argo on the management cluster)
• Some applications exist once in every cluster (like cert-manager or your favorite ingress controller)
• Some deployments exist multiple times in separate namespaces (like a cnpg-instance for an application)
• and some deployments appear over and over again in even the same namespace (kubevirt vms in my case).

Wouldn't it be nice if you could easily provide predefined template values throughout the whole landscape? Like the schema for your ingress-hosts, <app>.<environment>.<company>.<tld> without needing anyone to think about it? While still providing the option to override this schema of course, for a globally unique deployment.

At the same time I'd love to spare the devs the usual "enabled-features"-list in such deployments. Just providing a file with the deployment-specific values should be enough.

I came up with a directory structure and a matching appSet-Template:

• <repo-root>/apps/<appTemplate>/ for application templates, predefined values and kustomizations
• <repo-root>/clusters/apps/<appInstance> for apps deployed once in every cluster
• <repo-root>/clusters/values.yaml for global values
• <repo-root>/clusters/<environment>/values.yaml for cluster-wide values
• <repo-root>/clusters/<environment>/apps/<appInstance> for apps deployed once in a selected cluster

Each of these appInstance-directories contains a helmfile.yaml or helmfile.yaml.gotmpl that references another helmfile in an appTemplate-directory. These helmfiles would take care of loading the value files for global values, cluster-wide values and appInstance-specific values in correct order and pass all of the values to the application template. My global values would contain primaryDomain: company.tld, the helmfile for the argo-cd template would contain `global.domain: "argo.{{ .Values.primaryDomain }}" and everybody would be happy.

It's not that easy... helmfiles looked like a great idea. Self-contained, can be assembled to bigger deployments, support passing values and secrets, allow patching of helm-charts if necessary, renovateBot can update the used charts, argo can render it with a plugin, a great deal. Sadly the helmfiles project took some design decisions rendering some of these parts unusable. Also, helm and helmfiles using the same templating engine ... those double-templated strings in the app-Set are causing a headache after a while. Also I can only put parameters in a helmfile that are supported by helmfile. .namespace works, .project doesn't. Passing values into the generated argo-applications suddenly becomes much harder.

But what's the alternative? Inventing my own format that includes references to the application template, the helm-values for that template as well as values for the argo-app? That would force me to implement that shiny value-files layering in the appSet-generators that helmfile took care of. The other alternative is using a plugin generator. If I read the documentation properly, this includes building a pod (and service) that responds to http requests and has no access to the file-system/git-repository in question.

A better suited alternative would be adding support for scripts/binaries executed in a sidecar, returning a complete argo app manifest on stdout, very similar to the current implementation of cmp-plugins. They have access to the app source, btw. A solution like that would allow me to chain the clusters-generator, the git-directory generator and a new magic-utility-generator without having to rely on nested helm-values being properly rendered with all the limitations of gotemplate, including the "template must be a string"-limitation. It would also allow things like checking if another app is configured to being deployed on my target cluster, like making sure that my cnpg-instance actually has an operator on that cluster as well.

Yes, it might look more complicated than it needs to be. It might also be easier for devs to use it. We won't find out until we try.
Yes, I know the examples by helm, like storing env-specific values with their charts. Or using a dedicated environments-directory. Or umbrella-charts. I personally think having a hierarchical directory structure might be easier than that.
Honestly, I could as well skip the appSet-part and simply use an app-of-apps approach where every app is fully defined in a single values file. It would surely work, but where's the fun in that?
Oh, using that structure, rendering the final templates locally and storing them in git is a no-go for me, just in case. Git is for source, not results.

If you have another solution that I missed so far, I'd be happy to give it a shot! And if I missed the page in the docs about how to add custom generators for appSets, yell me out, I probably deserved that.