# Creating containers - Part 1

### Michael Crosby

# Creating containers - Part 1

This is part one of a series of blog posts detailing how docker creates containers. We will dig deep into the various pieces that are stitched together to see what it takes to make `docker run ...` awesome.

## First, what is a container?

I think the various pieces of technology that goes into creating a container is fairly commonplace. You should have seen a few cool looking charts in various presentations about Docker where you get a quick "Docker uses namespaces, cgroups, chroot, etc." to create containers. But why does it take all these pieces to create a contaienr?
Why is it not a simple syscall and it's all done for me? The fact is that container's don't exist, they are made up. There is no such thing as a "linux container" in the kernel. A container is a userland concept.

## Namespaces

In part one I'll talk about how to create Linux namespaces in the context of how they are used within docker. In later posts we will look into how namespaces are combined with other features like cgroups and an isolated filesystem to create something useful.

First off we need a high level explanation of what a namespace does and why it's useful. Basically, a namespace is a scoped view of your underlying Linux system. There are a few different types of namespaces implemented inside the kernel. As we dig into each of the different namespaces below you can follow along by running `docker run -it --privileged --net host crosbymichael/make-containers`. This has a few preloaded files and configuration to get your started. Even though we will be creating namespaces inside an container that docker runs for us, don't let that trip you up. I opted for this approach as providing a container preloaded with all the dependencies that you need to run the examples is why we are doing this in the first place. To make things a little easier, I'm using the `--net host` flag so that we are able to see your host's network interfaces within our demo container. This will be useful in the network examples. We also need to provide the `--privilged` flag so that we have the correct permissions to create new namespaces within our container.

If you are interested in what the Dockerfile looks like then here it is:

    FROM debian:jessie

    RUN apt-get update && apt-get install -y \
        gcc \
        vim \
        emacs

    COPY containers/ /containers/
    WORKDIR /containers
    CMD ["bash"]

I'll be doing the examples in C, as it's sometimes easier to explain the lower level details better than the abstractions that Go provides. So lets start...

### NET Namespace

The network namespaces provides your own view of the network stack of your system. This can include your very own `localhost`. Make sure you are in the `crosbymichael/make-containers` and run the command `ip a` to view all the network interfaces of your host machine.

    > ip a
    root@development:/containers# ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host
           valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 08:00:27:19:ca:f2 brd ff:ff:ff:ff:ff:ff
        inet 10.0.2.15/24 brd 10.0.2.255 scope global eth0
           valid_lft forever preferred_lft forever
        inet6 fe80::a00:27ff:fe19:caf2/64 scope link
           valid_lft forever preferred_lft forever
    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
        link/ether 08:00:27:20:84:47 brd ff:ff:ff:ff:ff:ff
        inet 192.168.56.103/24 brd 192.168.56.255 scope global eth1
           valid_lft forever preferred_lft forever
        inet6 fe80::a00:27ff:fe20:8447/64 scope link
           valid_lft forever preferred_lft forever
    4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
        link/ether 56:84:7a:fe:97:99 brd ff:ff:ff:ff:ff:ff
        inet 172.17.42.1/16 scope global docker0
           valid_lft forever preferred_lft forever
        inet6 fe80::5484:7aff:fefe:9799/64 scope link
           valid_lft forever preferred_lft forever

Ok cool, so this is all the network interfaces currently on my host system. Yours may look a little different but you get the idea. Now let's write some code to create a new network namespace. For this we will write a skeleton of a small C binary that uses the `clone` syscall. We will start by using clone to run binaries that are already installed inside our demo container. The file `skeleton.c` should be in the working directory of the demo container. We will use this file as the basis of all our examples. Here is the code incase you don't feel like running the container right now.

    #define _GNU_SOURCE
    #include <stdio.h>
    #include <stdlib.h>
    #include <sched.h>
    #include <sys/wait.h>
    #include <errno.h>

    #define STACKSIZE (1024*1024)
    static char child_stack[STACKSIZE];

    struct clone_args {
            char **argv;
    };

    // child_exec is the func that will be executed as the result of clone
    static int child_exec(void *stuff)
    {
            struct clone_args *args = (struct clone_args *)stuff;
            if (execvp(args->argv[0], args->argv) != 0) {
                    fprintf(stderr, "failed to execvp argments %s\n",
                            strerror(errno));
                    exit(-1);
            }
            // we should never reach here!
            exit(EXIT_FAILURE);
    }

    int main(int argc, char **argv)
    {
            struct clone_args args;
            args.argv = &argv[1];

            int clone_flags = SIGCHLD;

            // the result of this call is that our child_exec will be run in another
            // process returning it's pid
            pid_t pid =
                clone(child_exec, child_stack + STACKSIZE, clone_flags, &args);
            if (pid < 0) {
                    fprintf(stderr, "clone failed WTF!!!! %s\n", strerror(errno));
                    exit(EXIT_FAILURE);
            }
            // lets wait on our child process here before we, the parent, exits
            if (waitpid(pid, NULL, 0) == -1) {
                    fprintf(stderr, "failed to wait pid %d\n", pid);
                    exit(EXIT_FAILURE);
            }
            exit(EXIT_SUCCESS);
    }

This is a small C binary that will allow you to run processes like `./a.out ip a`. It uses the arguments that you pass on the cli as the arguments to whatever process you want to use. Don't worry about the specific implementation too much as it's the changes we will be making that are the interesting aspects. Remember, this will execute the binary and arguments of whatever program you want, this means if you want to run one of these demos below and have it spawn a shell session so that you can poke around in your new namespace then go ahead. It is a great way to explore and inspect these different namespaces at your own pace. So to get started let's make a copy of this file to start working with the network namespace.

    > cp skeleton.c network.c

Ok, within this file there is a very special var called `clone_flags`. This is where most of our changes will happen throughout this post. Namespaces are primarily controlled via the clone flags. The clone flag for the network namespace is `CLONE_NEWNET`. We need to change the line in the file `int clone_flags = SIGCHLD;` to `int clone_flags = CLONE_NEWNET | SIGCHLD;` so that the call to `clone` creates a new network namespace for our process. Make this change in `network.c` then compile and run.

    > gcc -o net network.c
    > ./net ip a
    1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

The result of this run now looks very different from the first time we ran the `ip a` command. We only see a `loopback` interface in the output. This is because our process that was created only has a view of its network namespace and not of the host. And that's it. That is how you create a new network namespace.

Right now this is pretty useless as you don't have any usable interfaces. Docker uses this new network namespace to setup a `veth` interface so that your container has it's own ip address allocated on a bridge, usually `docker0`. We won't go down the path of how to setup interfaces in namespaces at this time. We can save that for another post.

So now that we know how a network namespace is created lets look at the mount namespace.

### MNT Namespace

The mount namespace gives you a scoped view of the mounts on your system. It's often confused with jailing a process inside a `chroot` or similar. This is not true! The mount namespaces does not equal a filesystem jail. So the next time you hear someone say that a container uses the mount namespace to "jail" the process inside it's own root filesystem you can call bullshit because they don't know what they are talking about. Do it, it's fun :)

Let's start by making a copy of `skeleton.c` again for our mount related changes. We can do a quick build and run to see what our current mount points looks like with the `mount` command.

    > cp skeleton.c mount.c
    > gcc -o mount mount.c
    > ./mount mount
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    tmpfs on /dev type tmpfs (rw,nosuid,mode=755)
    shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
    mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
    devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
    /dev/disk/by-uuid/d3aa2880-c290-4586-9da6-2f526e381f41 on /etc/resolv.conf type ext4 (rw,relatime,errors=remount-ro,data=ordered)
    /dev/disk/by-uuid/d3aa2880-c290-4586-9da6-2f526e381f41 on /etc/hostname type ext4 (rw,relatime,errors=remount-ro,data=ordered)
    /dev/disk/by-uuid/d3aa2880-c290-4586-9da6-2f526e381f41 on /etc/hosts type ext4 (rw,relatime,errors=remount-ro,data=ordered)
    devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

This is what the mount points look like from within my demo container, yours may look different. In order to create a new mount namespace we use the flag `CLONE_NEWNS`. You may notice something may look weird here. Why is this flag not `CLONE_NEWMOUNT` or `CLONE_NEWMNT`? This is because the mount namespace was the first Linux namespace introduced and the name was just an undersight. If you write code you will understand that as you start building a feature or an application, you don't often have a full picture of the end result. Anyway, let's add `CLONE_NEWNS` to our `clone_flags` variable. It should look something like `int clone_flags = CLONE_NEWNS | SIGCHLD;`.

Lets go ahead and build `mount.c` again and run the same command.

    > cp skeleton.c mount.c
    > gcc -o mount mount.c
    > ./mount mount

Nothing changed. Whaaat??? This is because the process that we run inside the new mount namespace still has a view on `/proc` of the underlying system. The result is that the new process sort of inherits a view on the underlying mounts. There are a few ways that we can prevent this, like using `pivot_root`, but we will leave that for an additional post on filesystem jails and how `chroot` / `pivot_root` interact with the container's mount namespace.

However, one way that we can try out our new mount namespace is to, well, mount something. Let's create a new `tmpfs` mount in `/mytmp` for this demo. We will do this mount in C and continue to run our same `mount` command as the args to our mount binary. In order to do a mount inside of our mount namespace we need to add the code to the `child_exec` function, before the call to `execvp`. The code within the `child_exec` function is run inside the newly created process, i.e., inside our new namespace. The code in `child_exec` should look like this:

    // child_exec is the func that will be executed as the result of clone
    static int child_exec(void *stuff)
    {
            struct clone_args *args = (struct clone_args *)stuff;
            if (mount("none", "/mytmp", "tmpfs", 0, "") != 0) {
                    fprintf(stderr, "failed to mount tmpfs %s\n",
                            strerror(errno));
                    exit(-1);
            }
            if (execvp(args->argv[0], args->argv) != 0) {
                    fprintf(stderr, "failed to execvp argments %s\n",
                            strerror(errno));
                    exit(-1);
            }
            // we should never reach here!
            exit(EXIT_FAILURE);
    }

We need to first create a directory `/mytmp` before we compile and run with the changes above.

    > mkdir /mytmp
    > gcc -o mount mount.c
    > ./mount mount
    # cutting out the common output...
    none on /mytmp type tmpfs (rw,relatime)

I cut out the common output above from the first time I ran `mount`.
The result is that you should see a new mount point for our `tmpfs` mount. Nice! Go ahead and run `mount` in the current shell just for comparison.
Notice how the `tmpfs` mount is not displayed? That is because we created the mount inside our own mount namespace, not in the parent's namespace.

Remember how I said that the mount namepaces does not equal a filesystem jail? Go ahead and run our `./mount` binary with the `ls` command. Everything is there. Now you have proof!

### UTS Namespace

The next namespace is the UTS namespace that is responsible for system identification. This includes the `hostname` and `domainname`. It allows a container to have it's own hostname independently from the host system along with other containers. Let's start by making a copy of `skeleton.c` and running the `hostname` command with it.

    > cp skeleton.c uts.c
    > gcc -o uts uts.c
    > ./uts hostname
    development

This should display your system's hostname (`development` in my case). Like earlier, let's add the clone flag for the UTS namespace to the `clone_flags` variable. The flag should be `CLONE_NEWUTS`. If you compile and run then you should see the exact same output. This is totally fine. The values in the UTS namespace are inherited from the parent. However, within this new namespace, we can change the hostname without it affecting the parent or other container's that have a separate UTS namespace.

Let's modify the hostname in the `child_exec` function. To do that, you will need to add the `#include <unistd.h>` header to gain access to the `sethostname` function, as well as the `#include <string.h>` header to use `strlen` needed by `sethostname`. The new body of the `child_exec` function should look like the following:

    // child_exec is the func that will be executed as the result of clone
    static int child_exec(void *stuff)
    {
            struct clone_args *args = (struct clone_args *)stuff;
            const char * new_hostname = "myhostname";
            if (sethostname(new_hostname, strlen(new_hostname)) != 0) {
                    fprintf(stderr, "failed to execvp argments %s\n",
                            strerror(errno));
                    exit(-1);
            }
            if (execvp(args->argv[0], args->argv) != 0) {
                    fprintf(stderr, "failed to execvp argments %s\n",
                            strerror(errno));
                    exit(-1);
            }
            // we should never reach here!
            exit(EXIT_FAILURE);
    }

Ensure that `clone_flags` within your main look like `int clone_flags = CLONE_NEWUTS | SIGCHLD;` then compile and run the binary with the same args. You should now see the value that we set returned from the `hostname` command. To verify that this change did not affect our current shell go ahead and run `hostname` and make sure that you have your original value back.

    > gcc -o uts uts.c
    > ./uts hostname
    myhostname
    > hostname
    development

Awesome! We are doing good.

### IPC Namespace

The IPC namespace is used for isolating interprocess communication, things like SysV message queues. Let's make a copy of `skeleton.c` for this namespace.

    > cp skeleton.c ipc.c

The way we are going to test the IPC namespace is by creating a message queue on the host, and ensuring that we cannot see it when we spawn a new process inside it's own IPC namespace. Let's first create a message queue in our current shell then compile and run our copy of the skeleton code to view the queue.

    > ipcmk -Q
    Message queue id: 65536
    > gcc -o ipc ipc.c
    > ./ipc ipcs -q
    ------ Message Queues --------
    key        msqid      owner      perms      used-bytes   message
    0xfe7f09d1 65536      root       644        0            0

Without a new IPC namespace you can see the same message queue that was created. Now let's add the `CLONE_NEWIPC` flag to our `clone_flags` var to create a new IPC namespace for our process. The `clone_flags` var should look like `int clone_flags = CLONE_NEWIPC | SIGCHLD;`. Recompile and run the same command again:

    > gcc -o ipc ipc.c
    > ./ipc ipcs -q
    ------ Message Queues --------
    key        msqid      owner      perms      used-bytes   message

Done! The child process is now in a new IPC namespace and has completely separate view and access to message queues.

### PID Namespace

This one is fun. The PID namespace is a way to carve up the PIDs that one process can view and interact with. When we create a new PID namespace the first process will get to be the loved PID 1. If this process exits the kernel kills everyone else within the namespace. Let's start by making a copy of `skeleton.c` for our changes.

    > cp skeleton.c pid.c

To create a new PID namespace, we will have to set the `clone_flags` with `CLONE_NEWPID`. The variable should look like `int clone_flags = CLONE_NEWPID | SIGCHLD;`. Let's test by running `ps aux` in our shell and then compile and run our `pid.c` binary with the same arguments.

    > ps aux
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root         1  0.0  0.1  20332  3388 ?        Ss   21:50   0:00 bash
    root       147  0.0  0.1  17492  2088 ?        R+   22:49   0:00 ps aux
    > gcc -o pid pid.c
    > ./pid ps aux
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root         1  0.0  0.1  20332  3388 ?        Ss   21:50   0:00 bash
    root       153  0.0  0.0   5092   728 ?        S+   22:50   0:00 ./pid ps aux
    root       154  0.0  0.1  17492  2064 ?        R+   22:50   0:00 ps aux

WTF??? We expected `ps aux` to be PID 1 or atleast not see any other pids from the parent. Why is that? proc. The process that we spawned still has a view of `/proc` from the parent, i.e. `/proc` mounted on the host system. So how do we fix this? How to we ensure that our new process can only view pids within it's namespace? We can start by remounting `/proc`.
Because we will be dealing with mounts, we can take the opportunity to take what we learned from the MNT namespace and combine it with our PID namespace so that we don't mess with the `/proc` of our host system.

We can start by including the clone flag for the mount namespace along side the clone flag for pid. It should look something like `int clone_flags = CLONE_NEWPID | CLONE_NEWNS | SIGCHLD;`. We need to edit the `child_exec` function and remount proc. This will be a simple `unmount` and `mount` syscall for the proc filesystem. Because we are creating a new mount namespace we know that this will not mess up our host system. The result should look like this:

    // child_exec is the func that will be executed as the result of clone
    static int child_exec(void *stuff)
    {
            struct clone_args *args = (struct clone_args *)stuff;
            if (umount("/proc", 0) != 0) {
                    fprintf(stderr, "failed unmount /proc %s\n",
                            strerror(errno));
                    exit(-1);
            }
            if (mount("proc", "/proc", "proc", 0, "") != 0) {
                    fprintf(stderr, "failed mount /proc %s\n",
                            strerror(errno));
                    exit(-1);
            }
            if (execvp(args->argv[0], args->argv) != 0) {
                    fprintf(stderr, "failed to execvp argments %s\n",
                            strerror(errno));
                    exit(-1);
            }
            // we should never reach here!
            exit(EXIT_FAILURE);
    }

Build and run this again to see what happens.

    > gcc -o pid pid.c
    > ./pid ps aux
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root         1  0.0  0.0   9076   784 ?        R+   23:05   0:00 ps aux

Perfect! Our new PID namespace is now fully operational with the help of the mount namespace!

### USER Namespace

The last namespace is the user namespace. This namespace is the new kid on the block and allows you to have users within this namespace that are not equal users outside of the namespace. This is accomplished via GID and UID mappings.

This one has a simple demo application without specifying a mapping, even though it's a totally useless demo. If we add the flag `CLONE_NEWUSER` to our `clone_flags` then run something like `id` or `ls -la` you will notice that we get `nobody` within the user namespace. This is because the current user is undefined right now.

    > cp skeleton.c user.c
    # add the clone flag
    > gcc -o user user.c
    > ./user ls -la
    total 84
    drwxr-xr-x 1 nobody nogroup 4096 Nov 16 23:10 .
    drwxr-xr-x 1 nobody nogroup 4096 Nov 16 22:17 ..
    -rwxr-xr-x 1 nobody nogroup 8336 Nov 16 22:15 mount
    -rw-r--r-- 1 nobody nogroup 1577 Nov 16 22:15 mount.c
    -rwxr-xr-x 1 nobody nogroup 8064 Nov 16 21:52 net
    -rw-r--r-- 1 nobody nogroup 1441 Nov 16 21:52 network.c
    -rwxr-xr-x 1 nobody nogroup 8544 Nov 16 23:05 pid
    -rw-r--r-- 1 nobody nogroup 1772 Nov 16 23:02 pid.c
    -rw-r--r-- 1 nobody nogroup 1426 Nov 16 21:59 skeleton.c
    -rwxr-xr-x 1 nobody nogroup 8056 Nov 16 23:10 user
    -rw-r--r-- 1 nobody nogroup 1442 Nov 16 23:10 user.c
    -rwxr-xr-x 1 nobody nogroup 8408 Nov 16 22:40 uts
    -rw-r--r-- 1 nobody nogroup 1694 Nov 16 22:36 uts.c

This is a very simple example of the user namespace but you can go much deeper with it. We will save this for another post but the idea and hopes for the user namespace is that this will allow us to run as "root" within the container but not as "root" on the host system. Don't forget you can always change `ls -la` to `bash` and have a shell inside the namespace to poke around and learn more.

### In the end...

So to recap we went over the mount, network, user, PID, UTS, and IPC Linux namespaces. The majority of the code that we changed was not much, just adding a flag most of the time.
The "hard work" is mostly managing the interactions between the various kernel subsystems in order to meet our requirements. Like most of the descriptions before this, namespaces are just one of the tools that we use to make a container. I hope the PID example is a glimpse of how we use multiple namespaces together in order isolate and begin the creation of a container.

In future posts we will go into detail on how we jail the container's processes inside a root filesystem, aka a docker image, as well as using cgroups and Linux capabilities. By the end we should be able to pull all these things together to create a container.

Also a thanks to tibor and everyone that helps review my brain dump of a first draft ;)

comments powered by Disqus