Pods are the smallest and most basic deployable objects in Kubernetes, with each pod representing a single instance of a (running) process within your cluster.
A Pod consists of one or more containers with shared network resources and storage, including a specification for running the containers.
One of the Kubernetes Pod’s uses is to help you execute a graceful shutdown of your application running inside a container.
For instance, you’d want to implement graceful shutdown if you want to wait until all current jobs or requests processing are completed. Other reasons might also include distributed locks or opened connections.
You can manage Kubernetes pods to execute a graceful shutdown using Signals.
For example, when a Pod needs to be terminated, a SIGTERM signal gets sent to the main process or PID 1 in every container. Then, a countdown (a grace period of 30 seconds as a default) starts.
Once the signal arrives, each container should begin a graceful shutdown of your running apps and exit.
If the container doesn’t terminate the running app within the grace period, a SIGKILL signal is sent to force terminate the container.
That’s just one example since you can use other signals to manage Kubernetes pods.
In this guide, we’ll take a closer look into how common signals are used managing Kubernetes pods.
But first…
What are Signals?
Signals are software interrupts sent to a program to indicate that an event has occurred.
Essentially, signals are used to communicate the state of a process to another process, Operating System (OS), or hardware. This makes Signals a core part of any Linux/Unix-based system.
There are two kinds of signals: Maskable and non-maskable.
Users can ignore or change the maskable signals, but it’s not possible to do this with non-maskable signals. Non-maskable signals typically occur if there are non-recoverable hardware failures.
There are multiple types of signals for various functions, such as SIGINT to send interrupts and SIGKILL to kill a process without any cleanup actions.
This way, each signal is associated with a default action, and the targeted process will carry out this action.
Some of these actions are intended to:
- Stop or Terminate a process
- Continue a stopped process
- Dump core (create a dump file containing a memory image of the process)
For comprehensive information about Linux signals, you can visit the online man page for the signals or run the man 7 signal command using the command line to access the man page.
The relationship between containers and signals
When you use Docker, the docker stop command uses SIGTERM signal for the process running as PID 1 in the container.
It instructs it to stop and sends the SIGKILL to completely terminate the process after a specific grace period (default 10 seconds). You can configure this grace period via the CLI using the -t or –time flags.
On the other hand, the docker kill command instantly sends the SIGKILL signal to terminate the container without any chance for graceful termination.
Plus, the docker kill command comes with the flag –signal, which lets users specify various kinds of signals to send to the container.
You can modify this behavior from a Dockerfile itself using the STOPSIGNAL command to assign a different signal. This alters the termination behavior of the container.
How Kubernetes uses signals
Kubernetes relies on the SIGTERM and SIGKILL signals to manage pods.
As such, signals are a core part of the Kubernetes Pods lifecycle. This applies whether you use the kubectl delete pods command to terminate pods or as a part of a deployment to replace applications pods.
Kubernetes behaves the same way, where SIGTERM is sent at first, then waits for a specified time before sending a SIGKILL command.
This waiting period is also known as the graceful termination period of a Pod. You can configure this via the terminationGracePeriodSeconds in a config file or the –grace-period flag in the kubectl command-line interface.
You can refer to this excellent article on Linux graceful termination from a Kubernetes perspective for an extensive guide on how all these things work.
Here’s a quick primer of what happens: If the process does not handle the SIGTERM signal properly, it will get the SIGKILL signal to terminate the process.
Then, it will immediately be removed from the etcd and the K8s API. All this happens without waiting for the actual process to be terminated from the node.
This means that mishandling the termination behavior can lead to unforeseen errors within a Kubernetes cluster.
The SIGTERM signal gets sent to both containers since pods can contain multiple containers. Therefore, it is a developer’s sole responsibility to implement a proper strategy to handle SIGTERM singles, whether it is a single or multi-container pod.
SIGTERM and SIGKILL are not the only signals you can use within the Kubernetes environment. Other Signals, such as SIGQUIT, are also vital to manage the processes.
For example, you can use SIGQUIT to signal a graceful shutdown of a process like a web server.
However, you’ll need to configure these signals since Kubernetes does not have a native way of sending other types of signals.
One way of configuring a different signal is to convert a default signal to a different one within the container.
Suppose you want to implement a graceful shutdown of a process such as Nginx or Apache web servers.
In this case, you can use a lightweight process supervisor and init system such as dumb-init. This allows you to rewrite signals, such as SIGTERM to SIGQUIT or SIGWINCH, within the container by specifying them in a Dockerfile configuration at the time of the image creation.
Conclusion
Signals are common in any Linux/Unix environment, and you can use these to manage and action different processes.
This also translates to most containerized environments for container management throughout their lifecycle.
SIGTERM and SIGKILL are natively implemented in orchestration environments such as Kubernetes. This means that signals should be considered in containerized development.
Moreover, proper strategies should be implemented to handle these signals (or any custom signals) within the containers themselves to define behaviors on handling processors.
