The feedback loop when deploying to Kubernetes can be quite slow. Not only does the YAML need to be syntactically correct, but we need to ask ourselves:

Is the API version of our resource definition compatible with the version of Kubernetes that it is being deployed to? Kubernetes is constantly evolving, and over time, deprecates older APIs in favor of newer ones. A deployment definition may successfully apply on one version of Kubernetes, but not another.

Are the resources that depend on one another configured properly? For example, when creating a Service, you can specify selectors for which Pods the traffic should ultimately be routed to. In the event that a Service is configured with selectors, but is unable to find any Pods, this mistake would not be known until the Service is deployed to the cluster. What makes this case even more challenging is that this configuration is technically valid but would require additional testing to verify if the traffic is flowing as expected.

Kind aims to help solve a lot of these concerns by allowing developers to spin up a Kubernetes cluster on their local machine, verify their changes, and tear it down with ease. However, it can still take a fair amount of time to bring up a Kind cluster, apply the manifests, and test the outcome.

While these concerns can be caught relatively early, there are additional considerations, especially in the realm of security, that may not be immediately obvious and could go unnoticed until they become a serious problem.

To catch a lot of these problems ahead of time without the need of a Kubernetes cluster, including Kind, it's possible to validate all deployments to Kubernetes against policies.

An example policy

Most of the policies that I write, are written in Rego, a policy language that is interpreted by the Open Policy Agent (OPA). To better understand Rego, and how we can leverage it to write policies for Kubernetes, consider the following scenario:

In our cluster, we want to be able to quickly identify which team owns a given Namespace. This can be useful for being able to notify teams about overutilization, cost reporting, problematic pods, and more. To accomplish this, we require that all namespaces must be created with an owner label.

To show how we can use Rego to validate Kubernetes manifests, let's create a namespace without an owner label:

apiVersion: v1
kind: Namespace
metadata:
  name: missinglabel

We'll also need a policy. A Rego policy to enforce this requirement would look like the following:

package main

violation[msg] {
  input.kind == "Namespace"
  not input.metadata.labels["owner"]
  
  msg := "All namespaces must contain an owner label"
}

NOTE: The input prefix is special when writing policies. It refers to the input document which is one of the base documents provided by the OPA document model.

This policy defines a single rule called violation. While the order of the statements within the rule do not matter, it can be easier to understand how a rule is evaluated if expressed in this way.

Line 04 first evaluates if the input has a kind property and if its value is equal to Namespace. If the kind is not a Namespace, or there does not exist a kind property at all, the input will not be considered a violation. The example Namespace has both a kind property and has a value of "Namespace", so the statement in the rule would return true.

Line 05 then checks to see if there exists a key named owner in the labels map within the manifests metadata. If there is a key named owner, then Namespace must have an owner label. In the example Namespace, there is not an owner label so this statement also returns true.

Line 07 is an assignment operation which will always return true by default.

After all of the statements have been evaluated, the rule itself can be evaluated. In order for a rule to be true, all of the statements inside of the rule must also be true. In this example, all of the statements returned true so the example Namespace would trigger the violation.

Validating Kubernetes manifests with Conftest

It is important to note that the Open Policy Agent always expects JSON in order to evaluate policies. Kubernetes on the other hand, speaks YAML. At its core, the Open Policy Agent is just a policy engine—it's intended to be generic and fit many use cases.

Conftest is a tool that focuses on the user experience when interacting with OPA. Most notably, it handles converting multiple file formats such as .hcl, Dockerfile, and even yaml into JSON so that it can be interpreted by OPA. Because Conftest is just a CLI that can be downloaded onto your machine, or even pipelines, it's possible to verify manifests against your policies before ever creating a pull request.

Here is an example of a policy that enforces resource constraints on all containers:

violation[msg] {
  container := input.spec.template.spec.containers[_]

  not container.resources.requests.cpu
  not container.resources.limits.cpu

  msg := sprintf("(%s): Container resource constraints must be specified.", [input.metadata.name])
}

If we were to run Conftest against the nginx ingress controller bundle, we would see that it fails our policy:

$ conftest test bundle.yaml

FAIL - (ingress-nginx): Container constraints must be specified.

We can then take the necessary steps to add the resource constraints into the Deployment so that our CI will allow the bundle onto our cluster.

Using policy bundles

With Conftest, there's a lot of freedom in the ability to write our own policies, but there are a lot of bundles that the community has written that we also leverage.

A policy bundle can be thought of as a collection of Rego policies that can be pulled from a remote source. A lot of best practices are generic enough that it wouldn't make sense for everyone to have to write and rewrite the same policies. While the concept of bundling and distributing Rego policies for Kubernetes is still quite new, there do exist a couple bundles that have provided immediate value to our pipelines.

Verify API compatibility with Deprek8ion

Deprek8ion is a set of Rego policies that can be used to see if any of our resources are currently, or will be, deprecated in a given Kubernetes release.

$ conftest pull github.com/swade1987/deprek8ion/policies -p policy/deprek8ion
$ conftest test bundle.yaml

WARN - ingress-nginx-admission: API admissionregistration.k8s.io/v1beta1
is deprecated in Kubernetes 1.19, use admissionregistration.k8s.io/v1 instead.

Find security concerns with Kubesec

Kubesec is a set of Rego policies that can be used to see if any of our resources have any insecure configurations.

$ conftest pull github.com/instrumenta/policies/kubernetes -p policy/kubesec
$ conftest test bundle.yaml

FAIL - Deployment ingress-nginx-controller does not drop all capabilities
FAIL - Deployment ingress-nginx-controller is not using a read only root filesystem
FAIL - Deployment ingress-nginx-controller allows privileged escalation
FAIL - Deployment ingress-nginx-controller is running as root

Conftest enables us run policies against multiple resources at once—it is simple, yet powerful. No matter where the Kubernetes manifests originate from, in house or from the open, we can automatically execute our policies to validate that they're compliant with our requirements. This approach allows us to automate our standards and security compliant concerns, freeing up developers to focus on other tasks.

The general-purpose approach that the Open Policy Agent has taken, and the user experience that Conftest provides, enables near unlimited use cases for policy-based validation. While this post focuses a lot on validating Kubernetes manifests before deploying to a cluster, there are other possibilities as well. Notably continuous Kubernetes cluster auditing with Gatekeeper, and infrastructure security compliance with Regula.

Policy-based validation is still relatively new to the Kubernetes ecosystem, but it has already made a large impact on the community and I'm excited to see what's coming next in this space.