Stripping the layers – how secure is your docker image?

In this post, we’ll walk through an example where a file containing secrets can be found in only one of the intermediate layers that make up the docker image.

Just because a docker image looks secure, it’s possible that one of the layers that makes up the image is insecure.

Take for example, the following Dockerfile which is built from a Jenkins agent.

  1 FROM alpine:3.7
  2
  3 # Create app directory
  4 WORKDIR /usr/target/app
  5
  6 # Bundle app binaries
  7 COPY . .
  8
  9 # execute cmd that keeps container running
 10 CMD tail -f /dev/null

Looks innocent, right?

And, it is.

Let’s say Jenkins goes down and we really need to publish a new Docker image. In the absence of a ‘devops as code’ culture, it would be plausible that we’d ask a developer (Sargon) to locally build and push the Dockerfile using the latest code from github. One small issue… This developer has a .env file included as part of their .gitignore file.

The .env file is used to store environment variables, typically for node apps instead of asking developers to pass in a long string of env vars via the cmd line or resorting to a heavy .bash_rc file. It can contain ports to listen on, secrets, API keys, and DB connection strings.

.env 
AWS_SECRET=ABCDEFG
DB_CONN=mongodb://usr:pswd@mongo.mongo:5414/mydb
API_KEY=123456

“No problem”, says Sargon, as he simply modifies the Dockerfile with the following:

  1 FROM alpine:3.7
  2
  3 # Create app directory
  4 WORKDIR /usr/target/app
  5
  6 # Bundle app binaries
  7 COPY . .
  8
  9 # Remove local secrets file if it exists
 10 RUN rm -f .env
 11
 12 # execute cmd that keeps container running
 13 CMD tail -f /dev/null

He removes the .env file via line 10 : “RUN rm -f .env” and then builds the image:

docker build -t sogwiz/securetest .

When we run this container, we’ll check to see if the .env file exists in the container.

To keep the container running so that we can exec into it, execute the following:

local] docker run -td sogwiz/securetest 
local] docker ps
local] docker exec -it <container_id> sh
/usr/target/app #]  ls -a

You’ll notice that the .env file doesn’t exist, which is expected. We’re good, right?

Not quite…

We’re going to show how you can still view the contents of the .env file from the container itself… in multiple ways.

Way 1: Inspect each layer.tar file and view file contents within each individual layer

Way 2: Use docker tag, rollback to a specific layer and then view the contents of a running container based off of the rollback tag

Way 3: Use the dive tool to explore each layer in a docker image. As wonderful as this cmd line tool is, it doesn’t show the file contents. It lists the files, but not the contents so we won’t be able to see the secrets in the file. For that reason, we’ll skip this method.

Way 1: Inspect each layer.tar file and view file contents within each individual layer

docker save sogwiz/securetest > out.tar
tar -xf out.tar

You’ll notice the following structure

The challenge will be to find the layer that has the .env file that we’re looking for (the .env file in the /usr/target/app dir). I still don’t know how these layer folders relate to the actual layer that can be seen with the docker history command.

Brute forcing our way, we go through each layer dir and extract the contents of the layer.tar file until we find what we’re looking for.

Way 2: Use docker tag, rollback to a specific layer and then view the contents of a running container based off of the rollback tag

An approach is to use the docker ‘tag’ command to rollback to a layer. We can verify it did rollback by looking at the history command again.

docker history sogwiz/securetest

We’ll go through these steps at this point

1. docker tag to rollback
2. docker history to verify rollback
3. View the file contents by running the container in forced wait mode for bash so that we can exec into the container

1. docker tag to rollback

docker tag 48a0d4b65894 sogwiz/securetest
docker history sogwiz/securetest

2. docker history to verify rollback

Note how the history is now changed and the latest layer in the image is the layer just before we did the “rm -f .env” cmd.

View file contents of the running container based off the rolled back tag

local] docker run -td sogwiz/securetest 
local] docker ps
local] docker exec -it <container_id> sh
/usr/target/app #]  ls -a

/usr/target/app # ls -a
..          .env        Dockerfile  layers
/usr/target/app # cat .env
AWS_SECRET=ABCDEFG
DB_CONN=mongodb://usr:pswd@mongo.mongo:5414/mydb
API_KEY=123456

You’ll see the .env file exists, ouch!

To work around this, there are some simple means.

1. .dockerignore : Use a .dockerignore file in a similar fashion as .gitignore to exclude certain files like the .env file
2. devops as code : This would be avoided entirely if we enabled Sargon to kick off the devops pipeline (even from his local environment) which would have built the image using a different sandbox than his local codebase.
3. Other thoughts?

Thanks for reading!