Writing A Static Site Kubernetes Operator for Self Hosting

#python #selfhosted #operator #kubernetes #helm #docker #jotthatdown

Age: 28

Profession: Engineer

Location: 🇺🇸

Introduction

There was one final piece keeping me from moving my static sites from AWS S3... simplicity. The last thing I wanted to do was deal with configuring a Service, Ingress, Deployment, CI Pipeline, Dockerfile and Certificate more than once. My initial thought was to write a Helm Chart to handle the creation of all of these object but then it dawned on me. For those of us who have home labs, there really is not a great S3 alternative to static site hosting. I can hear your thought, "What about MinIO". I will admit that I am not too familiar with the product, but with the sheer number of services we already run in home labs I wanted to avoid adding another third party one.

Operator

Being that Python is my main language I was poking around and found Kopf. After reading the docs I was thoroughly surprised with how straight forward a simple operator would be.

First, lets define a CustomResourceDefenition or CRD that our operator will watch for creations, modifications or deletions:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: staticsites.example.com
spec:
  scope: Namespaced
  group: porp.me
  names:
    kind: StaticSite
    plural: staticsites
    singular: staticsite
    shortNames:
      - site
  versions:
    - name: v1
      served: true
      storage: true
      subresources: { status: { } }
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true

This definition is fairly open ended. What I mean is we are not limiting the user to what they add in their YAML definition of a StaticSite. We are simply saying "Watch for an object StaticSite and give me the YAML when its created or modified".

At this point we can decide what we want to define. In my case I need at minimum the following:

apiVersion: example.com/v1
kind: StaticSite
metadata:
  name: test
spec:
  replicas: 1
  repositorySecret: test-repo
  ingress:
    host: test.lab.example.com

The StaticSite definition above is saying "I want a StaticSite with 1 replica located internally at test.lab.example.com". Now I am sure your'e wondering to yourself, "What is a respositorySecret?". So, I needed to come up with a way to update these sites without doing a push based CI Pipeline. That is because it is located within my environment and at this time I do not have something like GitLab's Self Hosted Runners. So I figured I will dump the built site into a -public repository and find a way to do a pull based update on my Nginx container. So, this secret contains url: https://oauth2:glpat-<GITLAB_TOKEN>@gitlab.com/<ORGANIZATION>/<REPO>-public.git. I pass this into something I found called git-sync. Let's take a look at my deployment.yaml template to make this make sense:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "{name}-site"
spec:
  replicas: {replicas}
  selector:
    matchLabels:
      app: "{name}-site"
  template:
    metadata:
      labels:
        app: "{name}-site"
    spec:
      volumes:
      - name: nginx-config
        configMap:
          name: "{name}-site-cm"
      - name: git-repo
        emptyDir: {{}}
      containers:
      - name: git-sync
        env:
        - name: REPO_URL
          valueFrom:
            secretKeyRef:
              name: "{repo_secret}"
              key: url
        image: registry.k8s.io/git-sync/git-sync:v4.2.1
        args:
          - --repo=$(REPO_URL)
          - --depth=1
          - --period=60s
          - --link=current
          - --root=/usr/share/nginx/html
        volumeMounts:
          - name: git-repo
            mountPath: /usr/share/nginx/html
      - name: "{name}-site"
        image: nginx:alpine
        volumeMounts:
        - mountPath: /etc/nginx/conf.d/default.conf
          subPath: default.conf
          name: nginx-config
          readOnly: true
        - name: git-repo
          mountPath: /usr/share/nginx/html

As you can see, the git-sync container sits as a sidecar to nginx. This way, every 60 seconds the container will poll my objects to check if they have updated. You may also notice that I am pulling in a ConfigMap for Nginx default.conf. Again, I did not want to deal with any Docker Container builds or pipelines as part of this process. I simply wanted to take an Nginx container and inject what I need. The definition for said ConfigMap is:

apiVersion: v1
kind: ConfigMap
metadata:
  name: "{name}-site-cm"
data:
  default.conf: |
    server {{
      listen 80;
      listen [::]:80;

      root /usr/share/nginx/html/current;
      index index.html index.htm;
      server_name {name};

        location / {{
            # First attempt to serve request as file, then
            # as directory, then fall back to displaying a 404.
            try_files $uri $uri/ /404.html;
        }}
    }}

Unfortunately, since this was originally for personal use I only have Nginx and Cert-Manager capabilities. That aside, let's take a look at some code:

import kopf
import logging
import os
import kubernetes
import yaml

@kopf.on.create('staticsites')
def create_site(spec, name, namespace, logger, **kwargs):

    def create_configmap(core):

        path = os.path.join(os.path.dirname(__file__), 'templates/configmap.yaml')
        tmpl = open(path, 'rt').read()
        text = tmpl.format(name=name)
        data = yaml.safe_load(text)

        kopf.adopt(data)

        obj = core.create_namespaced_config_map(
            namespace=namespace,
            body=data,
        )

        return obj.metadata.name

    def create_deployment(apps):
        
        path = os.path.join(os.path.dirname(__file__), 'templates/deployment.yaml')
        tmpl = open(path, 'rt').read()
        text = tmpl.format(name=name, replicas=spec.get('replicas'), repo_secret=spec.get('repositorySecret'))
        data = yaml.safe_load(text)

        kopf.adopt(data)

        obj = apps.create_namespaced_deployment(
            namespace=namespace,
            body=data,
        )

        return obj.metadata.name

    def create_service(core):
        path = os.path.join(os.path.dirname(__file__), 'templates/service.yaml')
        tmpl = open(path, 'rt').read()
        text = tmpl.format(name=name)
        data = yaml.safe_load(text)

        kopf.adopt(data)

        obj = core.create_namespaced_service(
            namespace=namespace,
            body=data,
        )

    def create_ingress(network):
        ingress_data = spec.get('ingress')

        path = os.path.join(os.path.dirname(__file__), 'templates/ingress.yaml')
        tmpl = open(path, 'rt').read()
        text = tmpl.format(name=name, host=ingress_data['host'])
        data = yaml.safe_load(text)

        kopf.adopt(data)

        obj = network.create_namespaced_ingress(
            namespace=namespace,
            body=data,
        )

    core = kubernetes.client.CoreV1Api()
    apps = kubernetes.client.AppsV1Api()
    network = kubernetes.client.NetworkingV1Api()

    create_configmap(core)
    deployment_name = create_deployment(apps)
    create_service(core)
    create_ingress(network)

    logger.info(f"StaticSite was created: {name}")

    return {'deployment_name': deployment_name}

As you can see the create of each of these object is pretty cut and dry. I am leveraging a template os.path.join(os.path.dirname(__file__), 'templates/ingress.yaml') and simply inserting variables into said template tmpl.format(name=name, host=ingress_data['host']) then Kopf handles the apply via its Kubernetes Service Account and permissions.

Conclusion

Writing a simple operator is not too difficult. I know there are countless improvements I can make. What I will say is, modules like Kopf have made it extremely simple. It already has predefined functions like @kopf.on.create that create the objects when a new custom resource is created! It pretty much handles everything for you!