Run multi Docker containers with compose file

Docker
2020.11.25

In the previous post, I explained how to communicate with other Docker containers. If you haven’t read the post yet, go to this post first.

Communication with other Docker containers.

You can find the complete source code here

This is one of Docker learning series posts.

  1. Start Docker from scratch
  2. Docker volume
  3. Bind host directory to Docker container for dev-env
  4. Communication with other Docker containers
  5. Run multi Docker containers with compose file
  6. Container’s dependency check and health check
  7. Override Docker compose file to have different environments
  8. Creating a cluster with Docker swarm and handling secrets
  9. Update and rollback without downtime in swarm mode
  10. Container optimization
  11. Visualizing log info with Fluentd, Elasticsearch and Kibana
Sponsored links

Create a docker-compose file

In order to run multiple Docker containers, we can run them one by one with necessary options but it is cumbersome to specify all options in a command line every time we want to starts them. Creating a script file to run them together is one of options but Docker offers another way called docker-compose. Necessary configurations can be written in a file and a user can run and stop them with following commands.

# start containers
docker-compose up
# stop containers
docker-compose down

docker-compose file looks like this below. It is yaml file and you can understand what it configures at the first glance. The Docker images were built in
the previous post.

version: "3.7"

services:

    log-server:
        image:  log-server
        ports:
            - "8001:80"
        networks:
            - app-net

    restify-server:
        image: restify-server
        ports:
            - "8002:80"
        depends_on:
            - log-server
        networks:
            - app-net

networks:
    app-net:
        external:
            name: log-test-nat
  • version: Version of docker-compose format
  • services: top level entry to specify containers
  • log-server, restify-server: Docker containers’ names
  • image: Docker container image
  • ports: ports which the container want to assign. If it is only "80" for example, random port on a host is assigned.
  • networks: Docker network name
  • depends_on: Specify the dependency. The specified Docker containers are running before this container starts.
  • app-net: network name
  • external: Compose expects the network called log-test-nat already exist and it doesn’t try to create it.
Sponsored links

Run multi containers with docker-compose

My result looks like this. As you can see, the log-server started up before restify-server started up because restify-server has dependency to log-server.

$ docker-compose up
Creating docker-compose_log-server_1 ... done
Creating docker-compose_restify-server_1 ... done
Attaching to docker-compose_log-server_1, docker-compose_restify-server_1
log-server_1      | {"message":"restify listening at http://[::]:80","level":"info"}
log-server_1      | {"message":"restify-server: restify listening at http://[::]:80","level":"info"}
restify-server_1  | STATUS: 201
restify-server_1  | HEADERS: {"server":"restify","content-type":"application/json","content-length":"9","date":"Sun, 08 Nov 2020 12:17:41 GMT","connection":"close"}
restify-server_1  | BODY: "Created"
restify-server_1  | No more data in response.

The Docker containers stop when ctrl + c is entered. You can also add an -d option to run it in background. docker-compose makes command simpler but it also offers additional advantage. When we want to run several containers some of them may need to be scaled up to address tons of service requests. docker-compose offers it in very easy way. When it runs with --scale option and the number of the target containers it starts multiple containers up. It is very easy to scale up, isn’t it? I created another docker compose file docker-compose-without-port.yml which doesn’t specify the port bind to a host because it’s not possible to bind the same port to the multiple containers. Arbitrary port is assigned by Docker at start up in this case. The result looks like this.

$ docker-compose -f docker-compose-without-port.yml up -d --scale log-server=3
Starting docker-compose_log-server_1 ... done
Creating docker-compose_log-server_2 ... done
Creating docker-compose_log-server_3 ... done
Starting docker-compose_restify-server_1 ... done

$ docker-compose ps
             Name                            Command               State           Ports
------------------------------------------------------------------------------------------------
docker-compose_log-server_1       docker-entrypoint.sh node  ...   Up      0.0.0.0:32769->80/tcp
docker-compose_log-server_2       docker-entrypoint.sh node  ...   Up      0.0.0.0:32770->80/tcp
docker-compose_log-server_3       docker-entrypoint.sh node  ...   Up      0.0.0.0:32771->80/tcp
docker-compose_restify-server_1   docker-entrypoint.sh node  ...   Up      0.0.0.0:8002->80/tcp

The name became docker-compose_<service-name>_<number>. docker-compose ps lists all running containers that are part of the compose application. It shows 3 log-servers started up and different host ports were bind to them. But does it mean that restify-server calls log API in different server depending on the performance of the containers? We expect that the load balancing functionality works because it offers scale function. But isn’t it too easy? Let’s confirm if it works!

# Send 100 http requests to restify-server
for i in {1..100}; do curl http://localhost:8002/hello/name$i > /dev/null; done

# Check if the requests are load balanced
docker container exec docker-compose_log-server_1 cat server.log
docker container exec docker-compose_log-server_2 cat server.log
docker container exec docker-compose_log-server_3 cat server.log

This is my result.

$ docker container exec docker-compose_log-server_1 cat server.log
{"message":"restify listening at http://[::]:80","level":"info"}
{"message":"restify-server: restify listening at http://[::]:80","level":"info"}
{"message":"restify listening at http://[::]:80","level":"info"}
{"message":"restify-server: GET request with param [name1]","level":"info"}
{"message":"restify-server: GET request with param [name3]","level":"info"}
{"message":"restify-server: GET request with param [name6]","level":"info"}
{"message":"restify-server: GET request with param [name8]","level":"info"}
{"message":"restify-server: GET request with param [name11]","level":"info"}
{"message":"restify-server: GET request with param [name12]","level":"info"}
{"message":"restify-server: GET request with param [name14]","level":"info"}
{"message":"restify-server: GET request with param [name18]","level":"info"}
{"message":"restify-server: GET request with param [name19]","level":"info"}
{"message":"restify-server: GET request with param [name20]","level":"info"}
{"message":"restify-server: GET request with param [name23]","level":"info"}
{"message":"restify-server: GET request with param [name24]","level":"info"}
{"message":"restify-server: GET request with param [name26]","level":"info"}
{"message":"restify-server: GET request with param [name32]","level":"info"}
{"message":"restify-server: GET request with param [name35]","level":"info"}
{"message":"restify-server: GET request with param [name36]","level":"info"}
{"message":"restify-server: GET request with param [name42]","level":"info"}
{"message":"restify-server: GET request with param [name44]","level":"info"}
{"message":"restify-server: GET request with param [name47]","level":"info"}
{"message":"restify-server: GET request with param [name49]","level":"info"}
{"message":"restify-server: GET request with param [name50]","level":"info"}
{"message":"restify-server: GET request with param [name58]","level":"info"}
{"message":"restify-server: GET request with param [name59]","level":"info"}
{"message":"restify-server: GET request with param [name60]","level":"info"}
{"message":"restify-server: GET request with param [name62]","level":"info"}
{"message":"restify-server: GET request with param [name65]","level":"info"}
{"message":"restify-server: GET request with param [name66]","level":"info"}
{"message":"restify-server: GET request with param [name67]","level":"info"}
{"message":"restify-server: GET request with param [name69]","level":"info"}
{"message":"restify-server: GET request with param [name76]","level":"info"}
{"message":"restify-server: GET request with param [name81]","level":"info"}
{"message":"restify-server: GET request with param [name84]","level":"info"}
{"message":"restify-server: GET request with param [name85]","level":"info"}
{"message":"restify-server: GET request with param [name87]","level":"info"}
{"message":"restify-server: GET request with param [name89]","level":"info"}
{"message":"restify-server: GET request with param [name95]","level":"info"}
{"message":"restify-server: GET request with param [name97]","level":"info"}
{"message":"restify-server: GET request with param [name98]","level":"info"}


$ docker container exec docker-compose_log-server_2 cat server.log
{"message":"restify listening at http://[::]:80","level":"info"}
{"message":"restify-server: GET request with param [name5]","level":"info"}
{"message":"restify-server: GET request with param [name7]","level":"info"}
{"message":"restify-server: GET request with param [name9]","level":"info"}
{"message":"restify-server: GET request with param [name13]","level":"info"}
{"message":"restify-server: GET request with param [name16]","level":"info"}
{"message":"restify-server: GET request with param [name17]","level":"info"}
{"message":"restify-server: GET request with param [name22]","level":"info"}
{"message":"restify-server: GET request with param [name37]","level":"info"}
{"message":"restify-server: GET request with param [name38]","level":"info"}
{"message":"restify-server: GET request with param [name39]","level":"info"}
{"message":"restify-server: GET request with param [name40]","level":"info"}
{"message":"restify-server: GET request with param [name43]","level":"info"}
{"message":"restify-server: GET request with param [name51]","level":"info"}
{"message":"restify-server: GET request with param [name54]","level":"info"}
{"message":"restify-server: GET request with param [name57]","level":"info"}
{"message":"restify-server: GET request with param [name61]","level":"info"}
{"message":"restify-server: GET request with param [name63]","level":"info"}
{"message":"restify-server: GET request with param [name64]","level":"info"}
{"message":"restify-server: GET request with param [name68]","level":"info"}
{"message":"restify-server: GET request with param [name70]","level":"info"}
{"message":"restify-server: GET request with param [name71]","level":"info"}
{"message":"restify-server: GET request with param [name74]","level":"info"}
{"message":"restify-server: GET request with param [name78]","level":"info"}
{"message":"restify-server: GET request with param [name79]","level":"info"}
{"message":"restify-server: GET request with param [name80]","level":"info"}
{"message":"restify-server: GET request with param [name82]","level":"info"}
{"message":"restify-server: GET request with param [name83]","level":"info"}
{"message":"restify-server: GET request with param [name90]","level":"info"}
{"message":"restify-server: GET request with param [name91]","level":"info"}
{"message":"restify-server: GET request with param [name96]","level":"info"}
{"message":"restify-server: GET request with param [name100]","level":"info"}

$ docker container exec docker-compose_log-server_3 cat server.log
{"message":"restify listening at http://[::]:80","level":"info"}
{"message":"restify-server: restify listening at http://[::]:80","level":"info"}
{"message":"restify-server: GET request with param [name2]","level":"info"}
{"message":"restify-server: GET request with param [name4]","level":"info"}
{"message":"restify-server: GET request with param [name10]","level":"info"}
{"message":"restify-server: GET request with param [name15]","level":"info"}
{"message":"restify-server: GET request with param [name21]","level":"info"}
{"message":"restify-server: GET request with param [name25]","level":"info"}
{"message":"restify-server: GET request with param [name27]","level":"info"}
{"message":"restify-server: GET request with param [name28]","level":"info"}
{"message":"restify-server: GET request with param [name29]","level":"info"}
{"message":"restify-server: GET request with param [name30]","level":"info"}
{"message":"restify-server: GET request with param [name31]","level":"info"}
{"message":"restify-server: GET request with param [name33]","level":"info"}
{"message":"restify-server: GET request with param [name34]","level":"info"}
{"message":"restify-server: GET request with param [name41]","level":"info"}
{"message":"restify-server: GET request with param [name45]","level":"info"}
{"message":"restify-server: GET request with param [name46]","level":"info"}
{"message":"restify-server: GET request with param [name48]","level":"info"}
{"message":"restify-server: GET request with param [name52]","level":"info"}
{"message":"restify-server: GET request with param [name53]","level":"info"}
{"message":"restify-server: GET request with param [name55]","level":"info"}
{"message":"restify-server: GET request with param [name56]","level":"info"}
{"message":"restify-server: GET request with param [name72]","level":"info"}
{"message":"restify-server: GET request with param [name73]","level":"info"}
{"message":"restify-server: GET request with param [name75]","level":"info"}
{"message":"restify-server: GET request with param [name77]","level":"info"}
{"message":"restify-server: GET request with param [name86]","level":"info"}
{"message":"restify-server: GET request with param [name88]","level":"info"}
{"message":"restify-server: GET request with param [name92]","level":"info"}
{"message":"restify-server: GET request with param [name93]","level":"info"}
{"message":"restify-server: GET request with param [name94]","level":"info"}
{"message":"restify-server: GET request with param [name99]","level":"info"}

It worked as expected. It looks that each container handled similar number of requests. How easy it is to scale up!

Use specific port range

Maybe we want to use the specific port range. It is defined in docker-compose-with-port-range.yml in the following way.

ports:
    - "8003-8010:80"

Just specify the range on left side. In this example, port 8003 – 8010 will be used and max number of containers is 8. Of course, some containers don’t start up when specifying bigger number. In following result, log-server 5 and 10 were failed to start because port number couldn’t be assigned.

$ docker-compose -f docker-compose-with-port-range.yml up -d --scale log-server=10
The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

The "log-server" service specifies a port on the host. If multiple containers for this service are created on a single host, the port will clash.
Creating docker-compose_log-server_1  ... done
Creating docker-compose_log-server_2  ... done
Creating docker-compose_log-server_3  ... done
Creating docker-compose_log-server_4  ... done
Creating docker-compose_log-server_5  ... error
Creating docker-compose_log-server_6  ... done
Creating docker-compose_log-server_7  ... done
Creating docker-compose_log-server_8  ... done
Creating docker-compose_log-server_9  ... error
Creating docker-compose_log-server_10 ... done

ERROR: for docker-compose_log-server_9  Cannot start service log-server: Ports are not available: listen tcp 0.0.0.0:8010: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

ERROR: for docker-compose_log-server_5  Cannot start service log-server: Ports are not available: listen tcp 0.0.0.0:8008: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.

ERROR: for log-server  Cannot start service log-server: Ports are not available: listen tcp 0.0.0.0:8010: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.
Encountered errors while bringing up the project.

Conclusion

We could easily configure all Docker containers that we want them to interact with each other. Download my code sample and try to play with it if you haven’t tried it on your PC yet!

You can find the complete source code here

Sponsored links

Comments

Copied title and URL