When using Kubernetes we often find ourselves troubleshooting workloads by running ephemeral pods or even exec’ing into running ones. This often includes the installation of debugging tools such as, dig or curl, which can eventually lead to bloat or drift in the containers running in those pods. Another, perhaps more important issue with these types of actions is security. When deploying containerized applications we tend to implement security measures on build time only predicting the immutability of the image and we often devalue the possibility of runtime mutation of the image.
We found that the best way to avoid image mutation while allowing engineers to perform live debugging is to delete the pods that have been exec’d into or attached to (yes, you can leave a pod you attached onto without killing the process).
To achieve this we knew we’d have to use something that integrates with Kubernetes’ AdmissionRegistration API. Our first search turned up a project that seemed to do exactly what we needed: box/kube-exec-controller .
kube-exec-controller
kube-exec-controller is a simple admission controller for handling container drift. It works by listening for webhooks with pod information and deleting said pod after a set amount of time. It uses a ValidatingWebhookConfiguration resource to listen to all the exec and attach events and trigger the webhook.
Sadly, it had two issues:
- it wasn’t maintained, the last update was 3 years ago, as of writing
- it introduced either a degree of manual work or the need for other tools, as it didn’t handle it’s own certificate issuing and renewal
This meant we had to find something else.
gatekeeper
The next stop on our search was open-policy-agent/gatekeeper, an open policy agent implementation for Kubernetes. Gatekeeper was very interesting, as it is capable of expressing very complex policies, but as much as it is capable it also is complex and introduces a steep learning curve, due to it’s use of ReGo, OPA’s native policy language.
kyverno
While researching Gatekeeper’s features, we found a similar project: kyverno, which looked more suitable to our approach.
Kyverno is Kubernetes native, so instead of relying on something like ReGo to express the policies Kyverno uses simple CustomResources. It also provided something that was still either being developed or missing in GateKeeper, mutation policies and a cleanup controller.
For a more in-depth comparison of GateKeeper and Kyverno, we recommend the NeonMirrors article that compares both: Kubernetes Policy Comparison: OPA/Gatekeeper vs Kyverno | Neon Mirrors.
Creating Policies
Kyverno policies are divided in 5 types:
- Generate
- Mutate
- Validate
- VerifyImages
- CleanUp
In our case, we were trying to create a policy that would delete pods that were the target of a kubectl exec command, after a set amount of time.
We tried creating a Mutate policy that would label the target pod with a timestamp and then a CleanUp policy that would watch the cluster for that specific label and delete the pod after the timestamp was 5 minutes old.
Mutation Policy
apiVersion: kyverno.io/v2beta1
kind: ClusterPolicy
metadata:
name: label-on-exec
annotations:
policies.kyverno.io/title: Label pods that have been targets of kubectl exec
policies.kyverno.io/category: Other
policies.kyverno.io/severity: high
policies.kyverno.io/subject: Pod, Label, Exec
policies.kyverno.io/description: >-
Label pods that have been targets of kubectl exec
spec:
validationFailureAction: Enforce
background: false
rules:
- name: label
match:
any:
- resources:
kinds:
- Pod/exec
mutate:
targets:
- apiVersion: v1
kind: Pod
name: "{{ request.object.metadata.name }}"
namespace: "{{ request.object.metadata.namespace }}"
patchStrategicMerge:
metadata:
labels:
+(exoawk.com/cleanup-time): "{{ time_now() }}"
CleanUp Policy
apiVersion: kyverno.io/v2beta1
kind: ClusterCleanupPolicy
metadata:
name: delete-pods
annotations:
policies.kyverno.io/title: Cleanup pods that have been targets of kubectl exec
policies.kyverno.io/category: Other
policies.kyverno.io/severity: high
policies.kyverno.io/subject: Pod, Label, Exec
policies.kyverno.io/description: >-
Label pods that have been targets of kubectl exec
spec:
match:
any:
- resources:
kinds:
- Pod
conditions:
all:
- key: "{{ time_since('','{{ target.metadata.labels.\"exoawk.com/cleanup-time\" }}','') || '' }}"
operator: GreaterThan
value: 5m
schedule: "*/1 * * * *"
This approach had 2 issues:
- in the Mutate policy, the actual object of the
execrequest wasn’t the pod, it was the Pod/exec subresource so we couldn’t target the pod - the CleanUp policy would be running very often to achieve our desired outcome, which might lead to performance issues in the cluster (unlikely, but it’s something that we always keep in mind)
To address the first issue, we started looking into how to target the pod based on objects and sub resources created during an exec request to the cluster. We found that on exec requests a PodExecOptions object is created. Might it be the missing piece of our puzzle?
Sadly it wasn’t, as PodExecOptions and other Pod*Options objects do not contain a reference to their meta object, i.e., the pod.
But while digging through logs and Kubernetes API calls, we made a breakthrough!
We discovered that the name of the exec request, when connecting to the pod, is actually the pod’s name with ‘exec-‘ prefixed to it. That meant we could use some of Kyverno’s power to extract the pod name and finally alter the label’s on the pod!
After discovering this, we also realized Kyverno includes a special reserved label (cleanup.kiverno.io/ttl) that can be applied to any resource that needs to be removed, with either the absolute time the resource should be removed on or the remaining time the resource should be kept alive, after the label is applied. This removed the need for an extra CleanUp policy and simplified the whole flow.
Now, we can use a single Mutate policy to achieve our goal!
Mutation Policy:
apiVersion: kyverno.io/v2beta1
kind: ClusterPolicy
metadata:
name: cleanup-on-exec
annotations:
policies.kyverno.io/title: Label pods that have been targets of kubectl exec
policies.kyverno.io/category: Other
policies.kyverno.io/severity: high
policies.kyverno.io/subject: Pod, Label, Exec
policies.kyverno.io/description: >-
Label pods that have been targets of kubectl exec with 'cleanup.kyverno.io/ttl'
spec:
validationFailureAction: Enforce
background: false
rules:
- name: label
match:
any:
- resources:
kinds:
- Pod/exec
preconditions:
all:
- key: "{{ request.operation || 'BACKGROUND' }}"
operator: Equals
value: CONNECT
mutate:
targets:
- apiVersion: v1
kind: Pod
name: "{{ request.name | trim_prefix(@, 'exec-') }}"
namespace: "{{ request.namespace }}"
patchStrategicMerge:
metadata:
labels:
+(cleanup.kyverno.io/ttl): "5m"
Conclusion
This is only one example of the power of Kyverno and its policies. It allows us to define, validate, and enforce policies for resource configurations and operations in a simple and intuitive way.
An example of the many things Kyverno can accomplish, is restricting the image registries that can be used in the cluster:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-image-registries
annotations:
policies.kyverno.io/title: Restrict Image Registries
policies.kyverno.io/category: Best Practices, EKS Best Practices
policies.kyverno.io/severity: medium
policies.kyverno.io/minversion: 1.6.0
kyverno.io/kubernetes-version: "1.26"
policies.kyverno.io/subject: Pod
policies.kyverno.io/description: >-
Images from unknown, public registries can be of dubious quality and may not be
scanned and secured, representing a high degree of risk. Requiring use of known, approved
registries helps reduce threat exposure by ensuring image pulls only come from them. This
policy validates that container images only originate from the registry `eu.foo.io` or
`bar.io`. Use of this policy requires customization to define your allowable registries.
spec:
validationFailureAction: audit
background: true
rules:
- name: validate-registries
match:
any:
- resources:
kinds:
- Pod
validate:
message: "Unknown image registry."
pattern:
spec:
=(ephemeralContainers):
- image: "eu.foo.io/* | bar.io/*"
=(initContainers):
- image: "eu.foo.io/* | bar.io/*"
containers:
- image: "eu.foo.io/* | bar.io/*"
Policies, such as the one above, can be used to validate every resource that is applied to the cluster, assuring all best-practices are followed and the clusters are kept as secure as possible. It can also be coupled with other Kubernetes-native tools, like Falco to provide threat detection or ArgoCD for more powerful automations.
