Set resource limits

Introduction

In this exercise, we cover:

  • How to set resource limits
  • What can happen if you don’t

Note: This exercise can be a little finicky in cloud environments. And be careful that you don’t request too much memory if your cluster is running on your machine and doesn’t enforce overall memory limits.

Setup

In this example, we’ll use an app with an exploitable memory exhaustion denial of service. This will be fun to watch…

We’ll use something sort of like a “Billion Laughs”-style attack. Here’s the example from Kubernetes CVE-2019-11253:

apiVersion: v1
data:
  a: &a ["web","web","web","web","web","web","web","web","web"]
  b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a]
  c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b]
  d: &d [*c,*c,*c,*c,*c,*c,*c,*c,*c]
  e: &e [*d,*d,*d,*d,*d,*d,*d,*d,*d]
  f: &f [*e,*e,*e,*e,*e,*e,*e,*e,*e]
  g: &g [*f,*f,*f,*f,*f,*f,*f,*f,*f]
  h: &h [*g,*g,*g,*g,*g,*g,*g,*g,*g]
  i: &i [*h,*h,*h,*h,*h,*h,*h,*h,*h]
kind: ConfigMap
metadata:
  name: yaml-bomb
  namespace: default

The example app just allocates memory based on the value of the parameter you give it.

View the code:

less static/memory-exploder/main.go

package main

import (
	"flag"
	"fmt"
  "io/ioutil"
	"net/http"
  "os"
  "strconv"
  "strings"
)

func main() {
	var addr string

	// The purpose of this flag is to make it easier to change
	// the port; ports don't really matter when containers are
	// exposed using a Kubernetes Service, so it's nice to let
	// your apps take this as a parameter versus requiring code
	// changes to edit the port number.
	flag.StringVar(&addr, "addr", "0.0.0.0:80", "Address to listen on")
	flag.Parse()

	fmt.Fprintf(os.Stderr, "Listening on %q\n", addr)

	err := http.ListenAndServe(addr, http.HandlerFunc(serve))

	if err != nil {
		fmt.Printf("Error serving: %v", err)
	}
}

func logRequest(r *http.Request) {
  fmt.Fprintf(os.Stderr, "Request URL:  %s\n", r.URL.String())
  body, err := ioutil.ReadAll(r.Body)
  if err != nil {
    fmt.Fprintf(os.Stderr, "Unable to print request body: %v", err)
    return
  }
  fmt.Fprintf(os.Stderr, "Request data: %v", body)
}

func alloc(n int) []int {
  return make([]int, 0, n)
}

func serve(w http.ResponseWriter, r *http.Request) {
  logRequest(r)

  if r.Method != "POST" {
    w.WriteHeader(http.StatusMethodNotAllowed)
    return
  }

  // This handler is what does all of the problematic allocation.
  // To use it, POST to /1234 (which will work), and progressively
  // get up to much larger numbers to trip your memory limit.
  // Warning: make sure not to DoS important workloads,
  // or your personal computer!
  n, err := strconv.Atoi(strings.TrimLeft(r.URL.Path, "/"))
  if err != nil {
    w.WriteHeader(http.StatusBadRequest)
    return
  }
  fmt.Fprintf(os.Stderr, "Allocating []int with size %d\n", n)
  _ = alloc(n)

	w.Write([]byte(fmt.Sprintf("Allocated a []int with size %d\n", n)))
}
download main.go

Then deploy:

kubectl apply -f https://securek8s.dev/memory-exploder/buggy.yaml

“Attack”

Call the bad method:

curl -X POST "http://${WORKSHOP_NODE_IP:-localhost}:31304/1234"

This will work. But bump up the number and it will start getting bad!

curl -X POST "http://${WORKSHOP_NODE_IP:-localhost}:31304/123456789012"

You can exit from the request and try to run:

kubectl top pod -n buggy

to see the pod fall over, if your cluster has Heapster installed. (It’s a bit of a race!)

On Docker for Mac you may have more luck with:

docker stats

(Note: for the purposes of the workshop, we won’t try to make nodes completely fall over…)

Countermeasure

Apply a memory and CPU limit.

See the diff:

kubectl diff -f https://securek8s.dev/memory-exploder/buggy-but-limited.yaml

Then deploy:

kubectl apply -f https://securek8s.dev/memory-exploder/buggy-but-limited.yaml

Attack effects after patching

The app gets OOMKilled and restarted instead of causing nodes to become unstable or crash.

You can see a fresh pod being created after your request:

kubectl get pod -n buggy -w

The app will hang up early if you make the same large request from earlier:

curl -X POST "http://${WORKSHOP_NODE_IP:-localhost}:31304/123456789012"

If you check your pods again, you’ll see that the container restarted:

# kubectl get pod -n buggy
NAME                     READY   STATUS    RESTARTS   AGE
buggy-6659cc6895-29j98   1/1     Running   1          2m50s

How to use it yourself

Add requests and limits to each of your pods.

Be careful about what you choose, especially for limits. Too low a CPU limit can interfere with app functionality, especially if you do things like RSA. Too low a memory limit will get your app OOMKilled repeatedly.

Next up

All done for now! 🙂