Add ai-telemetry skeleton folders and zookeeper

[RH-csaggin]

• Added ai-telemetry skeleton folders
• Zookeeper:
  • Moved to kustomization structure.
  • Added custom pvc (removed from the integrated template on sts).
  • Added namePrefix: obs- for a better identification.
• Tested on lab cluster with oc kustomize overlays/nerc-ocp-obs | oc create -f - -n zookeeper-obs

[computate]

@RH-csaggin For ArgoCD to properly sync namespaces that need to be created first, we add them to a cluster-scope bundle. Let's add to this PR:

• cluster-scope/base/core/namespaces/zookeeper/ similar to cluster-scope/base/core/namespaces/keycloak/.
• cluster-scope/bundles/zookeeper/ similar to cluster-scope/bundles/keycloak/, but only include the namespace. Don't worry about operatorgroups and subscriptions.
• Add the zookeeper bundle after the other bundles in cluster-scope/overlays/nerc-ocp-obs/kustomization.yaml.

[computate]

@RH-csaggin For ArgoCD to properly sync namespaces that need to be created first, we add them to a cluster-scope bundle. Let's add to this PR:

* `cluster-scope/base/core/namespaces/zookeeper/` similar to `cluster-scope/base/core/namespaces/keycloak/`.

* `cluster-scope/bundles/zookeeper/` similar to `cluster-scope/bundles/keycloak/`, but only include the namespace. Don't worry about operatorgroups and subscriptions.

* Add the zookeeper bundle after the other bundles in `cluster-scope/overlays/nerc-ocp-obs/kustomization.yaml`.

@RH-csaggin I like your idea about using ArgoCD's CreateNamespace option. Let's try that instead of the bundle:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: zookeeper
spec:
  destination:
    namespace: zookeeper
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

[RH-csaggin]

@RH-csaggin For ArgoCD to properly sync namespaces that need to be created first, we add them to a cluster-scope bundle. Let's add to this PR:

* `cluster-scope/base/core/namespaces/zookeeper/` similar to `cluster-scope/base/core/namespaces/keycloak/`.

* `cluster-scope/bundles/zookeeper/` similar to `cluster-scope/bundles/keycloak/`, but only include the namespace. Don't worry about operatorgroups and subscriptions.

* Add the zookeeper bundle after the other bundles in `cluster-scope/overlays/nerc-ocp-obs/kustomization.yaml`.

@RH-csaggin I like your idea about using ArgoCD's CreateNamespace option. Let's try that instead of the bundle:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: zookeeper
spec:
  destination:
    namespace: zookeeper
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

@computate In my testing env using RH GitOps the automated creation of the namespace worked fine with a manifest like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: zookeeper
spec:
  destination:
    namespace: zookeeper
    server: https://kubernetes.default.svc
  source:
    path: ai-telemetry/overlays/nerc-ocp-obs
    repoURL: https://github.com/RH-csaggin/nerc-ocp-config.git
    targetRevision: HEAD
    kustomize:
      namespace: zookeeper
  sources: []
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    managedNamespaceMetadata:
      labels:
        argocd.argoproj.io/managed-by: openshift-gitops

To allow cluster-resources to be created correctly I had to add the label argocd.argoproj.io/managed-by: openshift-gitops please let me know your thoughts.

[schwesig]

checked together this morning
looking good!

[computate]

@larsks In the comment above #547 (comment), @RH-csaggin discovered a clever way for ArgoCD to automatically create the namespace that we need for this PR to add zookeeper to the obs cluster. do you see any issues with adding these syncOptions to our ArgoCD Application instead of creating a cluster-scope/base/core/namespaces/zookeeper and cluster-scope-bundles/zookeeper?

spec:
  destination:
    namespace: zookeeper
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    managedNamespaceMetadata:
      labels:
        argocd.argoproj.io/managed-by: openshift-gitops

[jtriley]

@larsks In the comment above #547 (comment), @RH-csaggin discovered a clever way for ArgoCD to automatically create the namespace that we need for this PR to add zookeeper to the obs cluster. do you see any issues with adding these syncOptions to our ArgoCD Application instead of creating a cluster-scope/base/core/namespaces/zookeeper and cluster-scope-bundles/zookeeper?

This will also need an argocd app defined in the nerc-ocp-apps repo to get this active on the obs cluster. Why not define the namespace in base/zookeeper as you're doing for the other manifests? This would also make it useful outside of argocd (ie manual test deployments) as a standalone set of manifests.

[computate]

Let's rerun the helm chart with replicaCount=3 to fix some StatefulSet configuration for 3 replicas.

[computate]

@RH-csaggin after a discussion with @larsks , the goal for the nerc-ocp-config repo is to have a yaml resource for everything, including namespaces. So we don't want ArgoCD to automatically create our zookeeper namespace resource, because we should be able to use either ArgoCD, or manually add the namespace from yaml.

So let's go back to the original plan. For ArgoCD to properly sync namespaces that need to be created first, we add them to a cluster-scope bundle. Let's add to this PR:

• cluster-scope/base/core/namespaces/zookeeper/ similar to cluster-scope/base/core/namespaces/keycloak/.
• cluster-scope/bundles/zookeeper/ similar to cluster-scope/bundles/keycloak/, but only include the namespace. Don't worry about operatorgroups and subscriptions.
• Add the zookeeper bundle after the other bundles in cluster-scope/overlays/nerc-ocp-obs/kustomization.yaml.