Network policies

February 22, 2020

Introduction

In this exercise, we cover:

Interesting types of access that pods have
How you can effectively limit network access

Setup

We’ll use an application that simulates a Server Side Request Forgery (SSRF), like the one involved in the Shopify bug bounty report, which ultimately allowed an adversary to steal cloud credentials from the metadata server.

We’ll use the simulated SSRF to see what a real problem like this could expose.

First, let’s take a look at the app. You can use the Cloud Shell editor, or the terminal:

less static/ssrf/main.go

package main

import (
	"crypto/tls"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"time"
)

func init() {
	http.DefaultTransport.(*http.Transport).TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
}

func ssrf(w http.ResponseWriter, r *http.Request) {
	c := http.Client{
		Timeout: 5 * time.Second,
	}
	resp, err := c.Get(r.URL.Query().Get("url"))
	if err != nil {
		w.WriteHeader(http.StatusBadGateway)
		w.Write([]byte(fmt.Sprintf("Couldn't fetch URL: %s\n", err)))
		return
	}
	defer resp.Body.Close()
	b, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		w.WriteHeader(http.StatusInternalServerError)
		w.Write([]byte(fmt.Sprintf("Couldn't read response: %s\n", err)))
		return
	}
	w.Write(b)
}

func main() {
	http.HandleFunc("/fetch", ssrf)
	log.Fatal(http.ListenAndServe("0.0.0.0:8080", nil))
}

main.go

Then let’s check out the Dockerfile, which is quite simple:

less static/ssrf/Dockerfile

FROM scratch

COPY main /main

ENTRYPOINT ["/main"]

Dockerfile

This is an example of using the scratch base image, which has effectively nothing in it. Other similar options include “distroless” containers, or container-focused minimal OSes.

Then, let’s deploy:

kubectl apply -f https://securek8s.dev/ssrf/base.yaml

The service is deployed on a NodePort on port 31302. Open it in your browser by running:

open "http://${WORKSHOP_NODE_IP:-localhost}:31302/fetch?url=http://checkip.dyndns.com"

or create a new browser tab directly.

Attack

We’ll use the fake SSRF exploit to access:

A service that returns your node’s public IP address (shown above)
The cloud provider metadata server
- This works only if you’re in a cloud or similar environment.
- /fetch?url=http://169.254.169.254
- See what you can find in there!
The Kubernetes API
- /fetch?url=https://kubernetes.default
The kubelet read-only API
- This works in most environments, but some (like Docker for Mac) disable the read-only port
- /fetch?url=http://169.254.123.1:10255/pods works in GKE
- This lists out every pod on the node—what interesting things can you find?
The Struts service we deployed earlier
- /fetch?url=http://struts.struts:31301

Countermeasure

We’ll apply an egress NetworkPolicy that blocks access to these services.

See what’s changed:

kubectl diff -f https://securek8s.dev/ssrf/egress-disabled.yaml

Now deploy:

kubectl apply -f https://securek8s.dev/ssrf/egress-disabled.yaml

We’ll see how this egress policy allows us to contact our app from the outside, but doesn’t let adversaries reach back out from inside to download tools. (Kubernetes policies apply to connections–not to packets.)

To do this, just try the examples above again.

Note: Some cluster types (like Docker for Mac) don’t support network policies.

Attack effects after patching

The adversary won’t be able to use your app’s network connection to reach out to the Internet or to underlying infrastructure.

If an adversary can run code or cause network requests in your pods, they will have a harder time finding out more about your infrastructure or spreading through it.

How to use it yourself

Include Network Policy YAMLs in your deployment tooling. Some people have success starting with ingress rules, and applying them to the most sensitive services first; once that rhythm is established, you can move on to the rest.

Check out these posts for more details:

Next up

We’ll move on to Kubernetes service accounts and RBAC in the next exercise:

Tune Kubernetes RBAC and account provisioning