From Kubernetes to OpenFaaS: A Lesson in Configuration and Container Lifecycles

OpenFaaS has emerged as a powerful platform for deploying functions in the realm of serverless computing. In this blog post, we will cover an intriguing problem we encountered while utilizing OpenFaaS cron jobs for a process that involved JWT authentication. Our exploration took us through the basics of containerization, code configuration, and comprehension of container lifecycles, which ultimately revealed an unforeseen solution.

Setting the stage

Our Data Engineering team had grown accustomed to deploying processes directly within Kubernetes (K8s) environments and managing them through Kubernetes cron jobs. Our familiarity with Kubernetes prompted certain presumptions as we embarked on our exploration of the OpenFaaS domain.

The issue

Our task seemed relatively simple. We needed to create a process that periodically connects to an API endpoint, analyzes the response, and sends messages to a Slack channel when data is present. The process was hosted on OpenFaaS, and JWT was used for authentication. However, an inexplicable issue arose: despite a seemingly correct configuration, the process consistently failed with a 401 Unauthorized status code after the first successful run.

Our authentication configuration was defined using a JWT (JSON Web Token) claims structure and included an expiration time calculated based on the current time. The initial code appeared as follows:

package processor

var conf = configuration{
      SigningKey: []byte(os.Getenv("SHARED_SECRET")),
      SigningMethod: jwt.SigningMethodHS256,
      StaticClaims: jwt.MapClaims{
          "iss": env.GetWithDefault("CREDENTIALS_KEY", "data-engineering"),
          "iat": time.Now().Unix(),
          "exp": time.Now().Add(time.Minute).Unix(),
      },
      ...
}

The logic to execute the process resided within a function that was scheduled using the OpenFaaS cron-connector feature.

func ProcessLatest(ctx context.Context) error {
    p, err := newProcessor(&conf)
    ...
    return p.Process(ctx)
}

Strikingly, the issue presented itself inconsistently: local runs succeeded while executions from OpenFaaS failed.  To further complicate matters, a fresh build of the code resolved the issue temporarily. After meticulous analysis, a pattern emerged: the token expiration was at the heart of the matter because their premature expiration led to unauthorized API access. 

Exploration phase: Understanding container lifecycle

Let’s contrast how Kubernetes cron jobs operate with the unique mechanics of OpenFaaS functions.

Kubernetes cron jobs

Kubernetes operates with a distributed architecture, orchestrated by its control plane and executed on worker nodes. At the core of this architecture are nodes, the worker machines within a Kubernetes cluster. Each node runs containers grouped within pods, forming a unit of deployment.

Containers provide isolated environments tailored for running specific applications. Kubernetes jobs are designed to run pods until a designated task is completed, after which the pods are terminated. A cron job (a specialized type of job) is responsible for executing tasks on a schedule.

Of paramount relevance is the fact that each Kubernetes cron job initiates, executes, and eventually purges the pod upon task completion.

OpenFaaS mechanisms

We encountered a unique set of OpenFaaS mechanics that differentiated it from Kubernetes. OpenFaaS functions are executed within containers thatalign with the ethos of serverless computing. These containers must adhere to certain rules, that include serving HTTP traffic on port 8080, embracing ephemeral storage, and maintaining statelessness.

The concept of the function watchdog lies at the heart of OpenFaaS’s function execution, a lightweight HTTP server. Among its various modes, the classic watchdog is of interest to us. While a detailed examination of how the watchdog operates is warranted, our focus lies on understanding isolation dynamics.

In summary, there are a few things to keep in mind:

1. Container Separation: different functions are encapsulated within separate containers.

2. Container Lifecycle: Containers commence execution upon function deployment or scaling. Depending on scaling options, a function might comprise multiple containers to cater to demand.

3. Handling HTTP Requests: a single container can efficiently handle multiple HTTP requests, each treated as an independent unit with no relation to preceding requests.

Crucially, OpenFaaS operates atop Kubernetes, harnessing its capabilities while offering additional configurations to facilitate the creation of cron jobs for functions.

Untangling the riddle

Armed with this context, the intriguing issue we encountered—JWT token expiration and the inconsistency in OpenFaaS function behavior—began to unravel. The disparity in container lifecycle behavior between Kubernetes and OpenFaaS resided at the core of our confusion. While Kubernetes cron jobs naturally terminated pods after execution, OpenFaaS containers persisted, retaining configuration data. This difference fundamentally impacted the validity of JWT tokens and led to unauthorized API access.

The code segment responsible for computing JWT claim details, including the issued and expiration time, resided as a package variable. This characteristic renders it global in scope, resulting in a single computation during the container’s lifecycle. Consequently, tokens were perpetually outdated, resulting in the recurring authentication failure.

Crafting the solution

With a comprehensive understanding of Kubernetes and OpenFaaS mechanics, the path to resolution was evident. We realized that it was essential to generate configuration values dynamically for every function execution instead of relying on package-level variables. This insight led us to encapsulate the configuration logic within its own function, guaranteeing fresh values with each execution.

func config() *configuration {
    return &configuration{
        ...
        SigningKey: []byte(os.Getenv("SHARED_SECRET")),
        SigningMethod: jwt.SigningMethodHS256,
        StaticClaims: jwt.MapClaims{
            "iss": env.GetWithDefault("CREDENTIALS_KEY", "data-engineering"),
            "iat": time.Now().Unix(),
            "exp": time.Now().Add(time.Minute).Unix(),
        },
        ...
    }
}

func ProcessLatest(ctx context.Context) error {
    conf := config()
    p, err := newProcessor(conf)
    ...
    return p.Process(ctx)
}

Gained insights

Our journey embraced exploration and experimentation, guided by our core values “Try, Fail, Learn and Repeat.” As novices in OpenFaaS, we transformed a stumbling block into a pivotal learning experience. We gained insights into OpenFaaS and containerization, improving our codebase and equipping us for future challenges.

This experience affirmed that simple challenges harbor profound insights. The issue’s resolution reinforced the value of flexibility and platform-specific awareness, highlighting the need for adaptable approaches.

Additional references

1. IXMIUZ - Open FaaS Case Study

2. Kubernetes Documentation