Interconnecting ECS Containers with DNS

At Packetly, we run workers' servers as close to customers as possible. These edge locations handle the heavy lifting of Malware and NSFW detection. Only the metadata is sent back to the core console (app.packetly.com). This enables customers to minimise the effects of latency and cloud providers' network costs. No one likes the hyperscaler's high egress costs!

The biggest benefit may be data handling and policy processing.

These edge locations run several servers, including an HTTP server, cache, ClamAV, and an NFWS image service. This sounds like the perfect candidate for Docker, right? You are correct; we run everything as a service. It's just the way we deploy that we think is unique. Instead of running Kubernetes or a Docker swarm over dedicated servers around the world, we decided to use my favourite service, AWS ECS Anywhere.

ECS anywhere allows you to run ECS services with their task deifications, deployments, and service management on your own servers outside of AWS. We love it! We can do the heavy lifting in cheaper providers closer to customers' services while still benefiting from a fully managed API service.

Networking Issues

ECS was not designed to allow containers phyical talk to each other in a typical docker compose setup; this makes sense, really. In ECS, you use awsvpc to connect containers together it does no matter what physical host they are running on. It could even be farmgate. The idea is they are still connected via your VPC.

With ECS anywhere this does not happen. While the containers can communicate with each other, they are not provided with the easy DNS system that you get with Docker compose. You want to be able to tell your worker that the cache service can be looked up locally!

The work around

You are going to love it as its a total hack!

The trick is to make another docker network, this one will support DNS. Unlike the one ECS uses as default.

docker network create --driver bridge "packetly_network"

Then connect your container to it:

Task Family wants to be something like Clamav. You will have more than one and want to enable docker DNS to round-robin requests.

docker network connect --alias "$TASK_FAMILY" "packetly_network" "$CONTAINER_ID"

Do this with all of your containers, and they can now communicate with each other via DNS networking. I use the ECS metadata on the container to find the launch family and then group the DNS together.

TASK_FAMILY=$(docker inspect -f '{{ index .Config.Labels "com.amazonaws.ecs.container-name" }}' "$CONTAINER")

Time for Automation

You won't want to run this by hand every time you start a new task on ECS, so we need to automate it. I created a simple container that runs a looping bash script. This container has a docker, and when you start it, you give it access to the parent docker socket. You need to make sure you know what you are doing here, as this can pose a security risk. If an attacker gets into this container, it's game over!

Dockerfile

FROM alpine:3.14
LABEL authors="Simon Bennett <simon@packetly.com>"

RUN apk add --no-cache bash docker-cli
COPY ./docker.bash /root/docker
RUN chmod +x /root/docker

CMD ["/root/docker"]

The script

https://gist.github.com/mrsimonbennett/e720bde3e31b4ca6f2d92623dcd3dbf7

Task Definition

{
  "family": "ecs",
  "containerDefinitions": [
    {
      "name": "esc",
      "image": "repo:latest",
      "essential": true,
      "mountPoints": [
        {
          "sourceVolume": "docker-socket",
          "containerPath": "/var/run/docker.sock",
          "readOnly": false
        }
      ],
    }
  ],
  "volumes": [
    {
      "name": "docker-socket",
      "host": {
        "sourcePath": "/var/run/docker.sock"
      }
    }
  ]
}

I've skinned this to the bits you need to mount the docker socket into the container in ECS. You will need to add the missing bits, CPU, launch types, etc.

Enjoy abusing ECS anywhere.