Declarative UI in Kube The user interface for Operator managed custom resources
Declarative UI in Kube - The user interface for Operator managed custom resources<

Background

When building a Kubernetes-native application, CustomResourceDefinitions (CRD) are the primary way to extend the Kubernetes API with custom resources. This post will cover generating a creation form for your custom resource based on the OpenAPI v3 validation schema. Afterward, we are going to talk about the value you can get through Operator Descriptors to fulfill more complex interactions and improve the overall usability of your applications or services.

Generate Creation Form based on OpenAPI Schema

Many of our partners (ISVs) have certain requirements when building a UI form to guide users creating an instance of their application or custom resource managed by their Operators. Starting from Kubernetes 1.8, CustomResourcesDefinition (CRD) gained the ability to define an optional OpenAPI v3 based validation schema. In Kubernetes 1.16 and higher, any new feature use CRD in apiextensions.k8s.io/v1 will be required to have a structural schema (in v1beta1 this is still optional). This is important not only for data consistency and security, but it also enables the potential to design and build a richer user interface to improve the user experience when creating or mutating custom resources.

For example, here is a CRD manifest from one of our partners:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: null
  name: couchbaseclusters.couchbase.com
spec:
  …
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            …
          cluster:
              properties:
                …
                autoFailoverServerGroup:
                  type: boolean
                autoFailoverTimeout:
                  maximum: 3600
                  minimum: 5
                  type: integer
                …
              required:
              …
              - autoFailoverTimeout

With the validation info, we can start associating these fields with corresponding UI input fields. Since the autoFailoverServerGroup field is expecting a boolean data type, we can either assign this field with a checkbox, a radio button or a toggle switch. As for autoFailoverTimeout field, we can simply limit the input type as an integer between 5 to 3600. We can also denote that autoFailoverTimeout is a required field, while autoFailoverServerGroup is optional. So far, everything looks good. However, things start to get complicated for other data types or more complex nested fields.

From our partners who build Operators, one common use case we see is that the custom resource needs a Secret object as a prerequisite for creating an instance. In the CRD manifest, this would be specified similar to the code snippet below:

…
  properties:
    credentials:
      type: string

As we can see, the only viable validation from OpenAPISchema checks only the data type as “string” and is fairly limited in terms of usability. Wouldn’t it be great if the UI could provide a searchable dropdown list of all the existing Secrets in your cluster? It could not only speed up the filling process but also reduce possible human errors in comparison with manually entry. This is where Operator Lifecycle Manager (OLM) descriptors come in.

Operator Descriptors Enhancements

Prerequisites

  • Install Operator Lifecycle Manager
  • The Operator Lifecycle Manager (OLM) can be installed with one command on any Kubernetes cluster and interact with the OKD console. If you’re using Red Hat OpenShift 4, the OLM is pre-installed to manage and update the Operators on your cluster.

  • Generate an Operator manifest, ClusterServiceVersion
  • Generate the ClusterServiceVersion (CSV) that represents the CRDs your Operator manages, the permissions it requires to function, and other installation information with the OLM. See Generating a ClusterServiceVersion (CSV) for more information on generating with Operator SDK, or manually defining a manifest file. You’ll only have to do this once, then carry these changes forward for successive releases of the Operator.

Operator Descriptors

OLM introduces the notion of “descriptors” of both spec and status fields in Kubernetes API responses. Descriptors are intended to indicate various properties of a field in order to make decisions about their content. The schema for a descriptor is the same, regardless of type:

type Descriptor = {
  path: string; // Dot-delimited path of the field on the object
  displayName: string;
  description: string;

  /* Used to determine which "capabilities" this descriptor has, and which
     React component to use */
  'x-descriptors': SpecCapability[] | StatusCapability[];
  value?: any; /* Optional field (type: object).
                  If present, the value of this spec is the same for all instances of the CRD
                  and can be found here instead of on the CR. */
}

The x-descriptors field can be thought of as "capabilities" (and is referenced in the code using this term). Capabilities are defined in types.ts provide a mapping between descriptors and different UI components (implemented as React components) using URN format

specDescriptors - for "spec" field in Kubernetes API

  • "k8sResourcePrefix" specDescriptor:
  • Recall the use case previously mentioned for specifying a Kubernetes resource in the CRD manifest. The “k8sResourcePrefix” is the OLM descriptor for this goal.

    Let’s take CouchbaseCluster as an example to see how this descriptor can be adopted in the Couchbase Operator’s CSV file. First, inside the CRD manifest (couchbasecluster.crd.yaml):

    apiVersion: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
      creationTimestamp: null
      name: couchbaseclusters.couchbase.com
    spec:
      …
      names:
        kind: CouchbaseCluster
      …
      validation:
        openAPIV3Schema:
          properties:
            spec:
              properties:
                …
                authSecret:
                  minLength: 1
                  type: string
                …
                tls:
                  properties:
                    static:
                      properties:
                        member:
                          properties:
                            serverSecret:
                              type: string
                          type: object
                        operatorSecret:
                          type: string
                      type: object
                type: object
              required:
              …
              - authSecret

    The validation block specifies a Secret object (authSecret) that stores the admin credential is required for creating a CouchbaseCluster custom resource. And for TLS (tls), it requires additional two Secret objects, one as Server Secret (tls.static.member.serverSecret), and the other as the Operator Secret (tls.static.operatorSecret).

    To utilize the OLM Descriptors, inside Couchbase Operator’s CSV file, we first specify the k8sResourcePrefix descriptor as a “Secret” object (urn:alm:descriptor:io.kubernetes:Secret) and then point it to the fields on the CouchbaseCluster CRD object in the “path” field.

    apiVersion: operators.coreos.com/v1alpha1
    kind: ClusterServiceVersion
    metadata:
      name: couchbase-operator.v1.1.0
      …
    spec:
      customresourcedefinitions:
        owned:
        - description: Manages Couchbase clusters
          displayName: Couchbase Cluster
          kind: CouchbaseCluster
          name: couchbaseclusters.couchbase.com
          …
          specDescriptors:
          - description: The name of the secret object that stores the admin credentials.
            displayName: Auth Secret
            path: authSecret
            x-descriptors:
            - 'urn:alm:descriptor:io.kubernetes:Secret'
          - description: The name of the secret object that stores the server's TLS certificate.
            displayName: Server TLS Secret
            path: tls.static.member.serverSecret
            x-descriptors:
            - 'urn:alm:descriptor:io.kubernetes:Secret'
          - description: The name of the secret object that stores the Operator's TLS certificate.
            displayName: Operator TLS Secret
            path: tls.static.operatorSecret
            x-descriptors:
            - 'urn:alm:descriptor:io.kubernetes:Secret'
          …
    

    Let’s take a closer look. In the CSV file:

    1. Under the spec.customresourcedefinitions.owned section (i.e. the CRD is that owned by this Operator, which can be multiple CRDs), specify the metadata of your custom resource.
    2. Since this is for “creating or mutating” the custom resource, assign the k8sResourcePrefix descriptor under “specDescriptors" section as input.
    3. description and displayName fields are pretty straightforward being the information that can be displayed on the UI as help text and field title.
    4. path - is used for pointing the field on the object in the dot-delimited path as to where it is inside the CRD (i.e. couchbasecluster.crd.yaml).
    5. x-descriptors - Assign this field with the `k8sResourcePrefix` descriptor and specify the resource type as “Secret” in “urn:alm:descriptor:io.kubernetes:Secret”.

    Now, let’s take a look in OpenShift console. We will have to install the Couchbase Operator from “OperatorHub” first so the Operator is ready to be used on the cluster. We can create a CouchbaseCluster instance via the “Installed Operator” view. Next, switch to the form view and see that “searchable dropdown component” shows up on the UI. This component allows us to look for existing Secrets on the cluster and is pointed to the corresponding fields on the CouchbaseCluster object. It’s that simple.

    Create a CouchbaseCluster instance via the Installed Operator view
    Create a CouchbaseCluster instance via the Installed Operator view.
    A searchable dropdown component for selecting Secrets on the cluster for creating CouchbaseCluster instance.
    A searchable dropdown component for selecting Secrets on the cluster for creating CouchbaseCluster instance.
  • "resourceRequirements" specDescriptor:
  • Specifying how much CPU and memory (RAM) each container needs for a pod is another worth mentioned use case. Again, in the couchbasecluster.crd.yaml manifest, we can see fields for specifying the resource limits and requests for a running pod:

    …
    spec:
      …
      names:
        kind: CouchbaseCluster
      …
      validation:
        openAPIV3Schema:
          properties:
            spec:
              properties:
                …
                servers:
                  items:
                    properties:
                      name:
                        minLength: 1
                        pattern: ^[-_a-zA-Z0-9]+$
                        type: string
                      pod:
                        properties:
                          …
                          resources:
                            properties:
                              limits:
                                properties:
                                  cpu:
                                    type: string
                                  memory:
                                    type: string
                                  storage:
                                    type: string
                                type: object
                              requests:
                                properties:
                                  cpu:
                                    type: string
                                  memory:
                                    type: string
                                  storage:
                                    type: string
                                type: object
                      …
    

    As we can see, these fields are nested and could be tricky to convert and organize them to the form fields. Alternatively, we can take advantage of the resourceRequirements descriptor by including it in Couchbase Operator’s CSV file and pointing to the resources field of the CouchbaseCluster object.

    apiVersion: operators.coreos.com/v1alpha1
    kind: ClusterServiceVersion
    metadata:
      name: couchbase-operator.v1.1.0
      …
    spec:
      customresourcedefinitions:
        owned:
        - description: Manages Couchbase clusters
          displayName: Couchbase Cluster
          kind: CouchbaseCluster
          name: couchbaseclusters.couchbase.com
          …
          specDescriptors:
          …
          - description: Limits describes the minimum/maximum amount of compute Resources required/allowed.
            displayName: Resource Requirements
            path: servers[0].pod.resources
            x-descriptors:
            - 'urn:alm:descriptor:com.tectonic.ui:resourceRequirements'
          …
    

    The Resource Requirement react component will then show up on the UIs for creating or mutating your custom resource in OpenShift console. For example, in Create Couchbase Cluster view, UI shows both Limits and Requests fields.

    Create Couchbase Cluster view shows both Limits and Requests fields for Resource Requirements
    Create Couchbase Cluster view shows both Limits and Requests fields for Resource Requirements.

    On the other hand, in the CouchbaseCluster Details view, you can access the widget to configure the Resource Limits and Requests as shown as the screenshots below, respectively.

    Resource requirements widget is accessible from CouchbaseCluster instance details view.
    Resource requirements widget is accessible from CouchbaseCluster instance details view.
    Resource Limits can be changed on the UI for ongoing managing the CouchbaseCluster instance.
    Resource Limits can be changed on the UI for ongoing managing the CouchbaseCluster instance.
    Resource Requests can be changed on the UI for ongoing managing the CouchbaseCluster instance.
    Resource Requests can be changed on the UI for ongoing managing the CouchbaseCluster instance.
  • "nodeAffinity", "podAffinity", and "podAntiAffinity" specDescriptor:
  • For assigning your running pods to nodes, it can be achieved by the affinity feature, which consists of two types of affinity, Node Affinity and Pod Affinity/Pod Anti-affinity. In the CRD manifest, these affinity related fields could be very nested and fairly complicated (see the nodeAffinity, podAffinity, and podAntiAffinity fields in alertmanagers.monitoring.coreos.com.crd.yaml).

    Similarly, we can leverage nodeAffinity, podAffinity, and podAntiAffinity descriptors and point them to the affinity field of the Alertmanager object.

    apiVersion: operators.coreos.com/v1alpha1
    kind: ClusterServiceVersion
    metadata:
      name: prometheusoperator.0.27.0
      …
    spec:
    …
      displayName: Prometheus Operator
      …
      customresourcedefinitions:
        owned:
        …
        - name: alertmanagers.monitoring.coreos.com
          version: v1
          kind: Alertmanager
          displayName: Alertmanager
          description: Configures an Alertmanager for the namespace
          …
          specDescriptors:
            …
            - description: Node affinity is a group of node affinity scheduling
              displayName: Node Affinity
              path: affinity.nodeAffinity
              x-descriptors:
              - 'urn:alm:descriptor:com.tectonic.ui:nodeAffinity'
            - description: Pod affinity is a group of inter pod affinity scheduling rules
              displayName: Pod Affinity
              path: affinity.podAffinity
              x-descriptors:
              - 'urn:alm:descriptor:com.tectonic.ui:podAffinity'
            - description: Pod anti affinity is a group of inter pod anti affinity scheduling rules
              displayName: Pod Anti-affinity
              path: affinity.podAntiAffinity
              x-descriptors:
              - 'urn:alm:descriptor:com.tectonic.ui:podAntiAffinity'
            …
    

    Later when we go ahead and create an Alertmanager instance in console, we will see those UI widgets with clear visual grouping along with input instruction for guiding how we can specify affinity using key/value pair with the logical operator. The “operator” field is a dropdown that provides viable options and the “value” field is enabled/disabled dynamically based on the operator being specified. And for “preferred” related rules, the weighting will be required.

    The majority of our users treat the UI as the medium to learn and explore the technical or API details

    learned from our customers

    Through talking with customers, we’ve learned the majority of our users treat the UI as the medium to learn and explore the technical or API details. The Affinity descriptor is one good example of the desired UX we strive to provide.

    'nodeAffinity' widget - easily specify which nodes your pod is eligible to be scheduled on based on labels on the node
    "nodeAffinity" widget - easily specify which nodes your pod is eligible to be scheduled on based on labels on the node.
    'podAffinity' widget - easily specify which nodes your pod is eligible to be scheduled based on labels on the pods
															 	that are already running on the node.
    "podAffinity" widget - easily specify which nodes your pod is eligible to be scheduled based on labels on the pods that are already running on the node.
    'antiPodAffinity' widget - easily specify which nodes your pod is eligible to be scheduled based on labels on pods
															  that are already running on the node.
    "antiPodAffinity" widget - easily specify which nodes your pod is eligible to be scheduled based on labels on pods that are already running on the node.

statusDescriptors - for "status" field in Kubernetes API

So far we have covered the OLM descriptors for the spec fields in Kubernetes API responses. In addition, OLM also provides a set of statusDescriptors for referencing fields in the status block of a custom resource. Some of them come with an associated react component too for richer interactions to the API. One example is podStatuses descriptor.

  • "podStatuses" statusDescriptor:
  • "podStatuses" statusDescriptor is usually being paired with "podCount" specDescriptor. User can specify the desired size of the custom resource being deployed with "podCount" specDescriptor. "podStatuses" statusDescriptor provides a dynamic graphical widget for better representing the latest member status of the custom resource being created or mutated.

    Specify the desired size of the custom resource being deployed with 'podCount' specDescriptor.
    Specify the desired size of the custom resource being deployed with "podCount" specDescriptor.
    'podStatuses' statusDescriptor provides a dynamic graphical widget for better representing the latest member status.
    "podStatuses" statusDescriptor provides a dynamic graphical widget for better representing the latest member status.

    Following the same pattern, in the code snippet below, we can see how etcd Operator applies podCount and podStatuses descriptors in the CSV file for users to create, mutating, and displaying etcdCluster custom resource in the console.

    apiVersion: operators.coreos.com/v1alpha1
    kind: ClusterServiceVersion
    metadata:
      name: etcdoperator.v0.9.4
      …
    spec:
    …
     displayName: etcd
     …
     customresourcedefinitions:
        owned:
        …
        - name: etcdclusters.etcd.database.coreos.com
          version: v1beta2
          kind: EtcdCluster
          displayName: etcd Cluster
          description: Represents a cluster of etcd nodes.
          …
          specDescriptors:
          - description: The desired number of member Pods for the etcd cluster.
            displayName: Size
            path: size
            x-descriptors:
            - 'urn:alm:descriptor:com.tectonic.ui:podCount'
            …
          statusDescriptors:
           - description: The status of each of the member Pods for the etcd cluster.
            displayName: Member Status
            path: members
            x-descriptors:
            - 'urn:alm:descriptor:com.tectonic.ui:podStatuses'
            …
    

What’s Next?

We hope the content and examples covered in this post will trigger community-wide discussions on how to improve the overall UX of Operator-managed application for the end-users. If you would like to learn more about OLM Descriptors, check out the github page where you can see the full list of specDescriptors and statusDescriptors that are currently available. Share your experience or feedback to Operator Lifecycle Manager (OLM) with github issues. If you want to explore more and contribute to Operator Descriptors, check out the contributing guide.