Declarative UI in Kube
The user interface for Operator managed custom resources
When building a Kubernetes-native application, CustomResourceDefinitions (CRD) are the primary way to extend the Kubernetes API with custom resources. This post will cover generating a creation form for your custom resource based on the OpenAPI v3 validation schema. Afterward, we are going to talk about the value you can get through Operator Descriptors to fulfill more complex interactions and improve the overall usability of your applications or services.
Many of our partners (ISVs) have certain requirements when building a UI form to guide users creating an instance of their application or custom resource managed by their Operators. Starting from Kubernetes 1.8, CustomResourcesDefinition (CRD) gained the ability to define an optional OpenAPI v3 based validation schema. In Kubernetes 1.16 and higher, any new feature use CRD in apiextensions.k8s.io/v1 will be required to have a structural schema (in v1beta1 this is still optional). This is important not only for data consistency and security, but it also enables the potential to design and build a richer user interface to improve the user experience when creating or mutating custom resources.
For example, here is a CRD manifest from one of our partners:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
creationTimestamp: null
name: couchbaseclusters.couchbase.com
spec:
…
validation:
openAPIV3Schema:
properties:
spec:
properties:
…
cluster:
properties:
…
autoFailoverServerGroup:
type: boolean
autoFailoverTimeout:
maximum: 3600
minimum: 5
type: integer
…
required:
…
- autoFailoverTimeout
…
With the validation info, we can start associating these fields with corresponding UI input fields. Since the autoFailoverServerGroup field is expecting a boolean data type, we can either assign this field with a checkbox, a radio button or a toggle switch. As for autoFailoverTimeout field, we can simply limit the input type as an integer between 5 to 3600. We can also denote that autoFailoverTimeout is a required field, while autoFailoverServerGroup is optional. So far, everything looks good. However, things start to get complicated for other data types or more complex nested fields.
From our partners who build Operators, one common use case we see is that the custom resource needs a Secret object as a prerequisite for creating an instance. In the CRD manifest, this would be specified similar to the code snippet below:
…
properties:
credentials:
type: string
…
As we can see, the only viable validation from OpenAPISchema checks only the data type as “string” and is fairly limited in terms of usability. Wouldn’t it be great if the UI could provide a searchable dropdown list of all the existing Secrets in your cluster? It could not only speed up the filling process but also reduce possible human errors in comparison with manually entry. This is where Operator Lifecycle Manager (OLM) descriptors come in.
The Operator Lifecycle Manager (OLM) can be installed with one command on any Kubernetes cluster and interact with the OKD console. If you’re using Red Hat OpenShift 4, the OLM is pre-installed to manage and update the Operators on your cluster.
Generate the ClusterServiceVersion (CSV) that represents the CRDs your Operator manages, the permissions it requires to function, and other installation information with the OLM. See Generating a ClusterServiceVersion (CSV) for more information on generating with Operator SDK, or manually defining a manifest file. You’ll only have to do this once, then carry these changes forward for successive releases of the Operator.
OLM introduces the notion of “descriptors” of both spec and status fields in Kubernetes API responses. Descriptors are intended to indicate various properties of a field in order to make decisions about their content. The schema for a descriptor is the same, regardless of type:
type Descriptor = {
path: string; // Dot-delimited path of the field on the object
displayName: string;
description: string;
/* Used to determine which "capabilities" this descriptor has, and which
React component to use */
'x-descriptors': SpecCapability[] | StatusCapability[];
value?: any; /* Optional field (type: object).
If present, the value of this spec is the same for all instances of the CRD
and can be found here instead of on the CR. */
}
The x-descriptors field can be thought of as "capabilities" (and is referenced in the code using this term). Capabilities are defined in types.ts provide a mapping between descriptors and different UI components (implemented as React components) using URN format
Recall the use case previously mentioned for specifying a Kubernetes resource in the CRD manifest. The “k8sResourcePrefix” is the OLM descriptor for this goal.
Let’s take CouchbaseCluster as an example to see how this descriptor can be adopted in the Couchbase Operator’s CSV file. First, inside the CRD manifest (couchbasecluster.crd.yaml):
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
creationTimestamp: null
name: couchbaseclusters.couchbase.com
spec:
…
names:
kind: CouchbaseCluster
…
validation:
openAPIV3Schema:
properties:
spec:
properties:
…
authSecret:
minLength: 1
type: string
…
tls:
properties:
static:
properties:
member:
properties:
serverSecret:
type: string
type: object
operatorSecret:
type: string
type: object
type: object
required:
…
- authSecret
…
The validation block specifies a Secret object (authSecret) that stores the admin credential is required for creating a CouchbaseCluster custom resource. And for TLS (tls), it requires additional two Secret objects, one as Server Secret (tls.static.member.serverSecret), and the other as the Operator Secret (tls.static.operatorSecret).
To utilize the OLM Descriptors, inside Couchbase Operator’s CSV file, we first specify the k8sResourcePrefix descriptor as a “Secret” object (urn:alm:descriptor:io.kubernetes:Secret) and then point it to the fields on the CouchbaseCluster CRD object in the “path” field.
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
name: couchbase-operator.v1.1.0
…
spec:
customresourcedefinitions:
owned:
- description: Manages Couchbase clusters
displayName: Couchbase Cluster
kind: CouchbaseCluster
name: couchbaseclusters.couchbase.com
…
specDescriptors:
- description: The name of the secret object that stores the admin credentials.
displayName: Auth Secret
path: authSecret
x-descriptors:
- 'urn:alm:descriptor:io.kubernetes:Secret'
- description: The name of the secret object that stores the server's TLS certificate.
displayName: Server TLS Secret
path: tls.static.member.serverSecret
x-descriptors:
- 'urn:alm:descriptor:io.kubernetes:Secret'
- description: The name of the secret object that stores the Operator's TLS certificate.
displayName: Operator TLS Secret
path: tls.static.operatorSecret
x-descriptors:
- 'urn:alm:descriptor:io.kubernetes:Secret'
…
Let’s take a closer look. In the CSV file:
Now, let’s take a look in OpenShift console. We will have to install the Couchbase Operator from “OperatorHub” first so the Operator is ready to be used on the cluster. We can create a CouchbaseCluster instance via the “Installed Operator” view. Next, switch to the form view and see that “searchable dropdown component” shows up on the UI. This component allows us to look for existing Secrets on the cluster and is pointed to the corresponding fields on the CouchbaseCluster object. It’s that simple.
Specifying how much CPU and memory (RAM) each container needs for a pod is another worth mentioned use case. Again, in the couchbasecluster.crd.yaml manifest, we can see fields for specifying the resource limits and requests for a running pod:
…
spec:
…
names:
kind: CouchbaseCluster
…
validation:
openAPIV3Schema:
properties:
spec:
properties:
…
servers:
items:
properties:
name:
minLength: 1
pattern: ^[-_a-zA-Z0-9]+$
type: string
pod:
properties:
…
resources:
properties:
limits:
properties:
cpu:
type: string
memory:
type: string
storage:
type: string
type: object
requests:
properties:
cpu:
type: string
memory:
type: string
storage:
type: string
type: object
…
As we can see, these fields are nested and could be tricky to convert and organize them to the form fields. Alternatively, we can take advantage of the resourceRequirements descriptor by including it in Couchbase Operator’s CSV file and pointing to the resources field of the CouchbaseCluster object.
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
name: couchbase-operator.v1.1.0
…
spec:
customresourcedefinitions:
owned:
- description: Manages Couchbase clusters
displayName: Couchbase Cluster
kind: CouchbaseCluster
name: couchbaseclusters.couchbase.com
…
specDescriptors:
…
- description: Limits describes the minimum/maximum amount of compute Resources required/allowed.
displayName: Resource Requirements
path: servers[0].pod.resources
x-descriptors:
- 'urn:alm:descriptor:com.tectonic.ui:resourceRequirements'
…
The Resource Requirement react component will then show up on the UIs for creating or mutating your custom resource in OpenShift console. For example, in Create Couchbase Cluster view, UI shows both Limits and Requests fields.
On the other hand, in the CouchbaseCluster Details view, you can access the widget to configure the Resource Limits and Requests as shown as the screenshots below, respectively.
For assigning your running pods to nodes, it can be achieved by the affinity feature, which consists of two types of affinity, Node Affinity and Pod Affinity/Pod Anti-affinity. In the CRD manifest, these affinity related fields could be very nested and fairly complicated (see the nodeAffinity, podAffinity, and podAntiAffinity fields in alertmanagers.monitoring.coreos.com.crd.yaml).
Similarly, we can leverage nodeAffinity, podAffinity, and podAntiAffinity descriptors and point them to the affinity field of the Alertmanager object.
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
name: prometheusoperator.0.27.0
…
spec:
…
displayName: Prometheus Operator
…
customresourcedefinitions:
owned:
…
- name: alertmanagers.monitoring.coreos.com
version: v1
kind: Alertmanager
displayName: Alertmanager
description: Configures an Alertmanager for the namespace
…
specDescriptors:
…
- description: Node affinity is a group of node affinity scheduling
displayName: Node Affinity
path: affinity.nodeAffinity
x-descriptors:
- 'urn:alm:descriptor:com.tectonic.ui:nodeAffinity'
- description: Pod affinity is a group of inter pod affinity scheduling rules
displayName: Pod Affinity
path: affinity.podAffinity
x-descriptors:
- 'urn:alm:descriptor:com.tectonic.ui:podAffinity'
- description: Pod anti affinity is a group of inter pod anti affinity scheduling rules
displayName: Pod Anti-affinity
path: affinity.podAntiAffinity
x-descriptors:
- 'urn:alm:descriptor:com.tectonic.ui:podAntiAffinity'
…
Later when we go ahead and create an Alertmanager instance in console, we will see those UI widgets with clear visual grouping along with input instruction for guiding how we can specify affinity using key/value pair with the logical operator. The “operator” field is a dropdown that provides viable options and the “value” field is enabled/disabled dynamically based on the operator being specified. And for “preferred” related rules, the weighting will be required.
The majority of our users treat the UI as the medium to learn and explore the technical or API details
Through talking with customers, we’ve learned the majority of our users treat the UI as the medium to learn and explore the technical or API details. The Affinity descriptor is one good example of the desired UX we strive to provide.
So far we have covered the OLM descriptors for the spec fields in Kubernetes API responses. In addition, OLM also provides a set of statusDescriptors for referencing fields in the status block of a custom resource. Some of them come with an associated react component too for richer interactions to the API. One example is podStatuses descriptor.
"podStatuses" statusDescriptor is usually being paired with "podCount" specDescriptor. User can specify the desired size of the custom resource being deployed with "podCount" specDescriptor. "podStatuses" statusDescriptor provides a dynamic graphical widget for better representing the latest member status of the custom resource being created or mutated.
Following the same pattern, in the code snippet below, we can see how etcd Operator applies podCount and podStatuses descriptors in the CSV file for users to create, mutating, and displaying etcdCluster custom resource in the console.
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
name: etcdoperator.v0.9.4
…
spec:
…
displayName: etcd
…
customresourcedefinitions:
owned:
…
- name: etcdclusters.etcd.database.coreos.com
version: v1beta2
kind: EtcdCluster
displayName: etcd Cluster
description: Represents a cluster of etcd nodes.
…
specDescriptors:
- description: The desired number of member Pods for the etcd cluster.
displayName: Size
path: size
x-descriptors:
- 'urn:alm:descriptor:com.tectonic.ui:podCount'
…
statusDescriptors:
- description: The status of each of the member Pods for the etcd cluster.
displayName: Member Status
path: members
x-descriptors:
- 'urn:alm:descriptor:com.tectonic.ui:podStatuses'
…
We hope the content and examples covered in this post will trigger community-wide discussions on how to improve the overall UX of Operator-managed application for the end-users. If you would like to learn more about OLM Descriptors, check out the github page where you can see the full list of specDescriptors and statusDescriptors that are currently available. Share your experience or feedback to Operator Lifecycle Manager (OLM) with github issues. If you want to explore more and contribute to Operator Descriptors, check out the contributing guide.
