Override Docker compose file to have different environments

The dev-environment and production environment is often different. The difference depends on the system but what I can easily imagine is following.

Authentication info
Port number of the service
Environment variables

Docker beginners may create different compose files for dev and production and define all necessary settings in both files. However, it’s getting hard to maintain them because of duplicated settings. When coding we should follow DRY principle and it can be applied to Docker as well. If we specify multiple docker-compose files in the docker-compose command Docker merges them. The command looks like this.

docker-compose -f docker-compose.yml -f docker-compose-dev.yml up

This is everything what this post explains! Let’s check the result one by one.

You can find the complete source code here

This is one of Docker learning series posts.

Simple application code

The sample application is just showing values.

console.log("=== START ===")

console.log(`Running for ${process.env.MODE}`);
console.log(`FOO : ${process.env.FOO}`)
console.log(`HOGE: ${process.env.HOGE} `)

console.log("----secrets----")
const secret = require("./config/secrets.json");
console.log(`user: ${secret.user}`);
console.log(`pass: ${secret.password}`);

console.log("=== END ===")

Dockerfile is also very simple.

FROM yuto/nodejs
WORKDIR /src
CMD [ "node", "./app.js" ]
COPY ./src/app.js /src/

Let’s create the image first.

cd override-docker-compose
docker image build -t show-env .

If you run the container now it throws an error because secrets.json file doesn’t exist there. COPY command for the file is not defined in the Dockerfile because it is sensitive data and shouldn’t be stored with clear text.

$ docker container run --rm show-env
=== START ===
Running for undefined
FOO : undefined
HOGE: undefined
----secrets----
internal/modules/cjs/loader.js:883
  throw err;
  ^

Error: Cannot find module './config/secrets.json'
Require stack:
- /src/app.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
    at Function.Module._load (internal/modules/cjs/loader.js:725:27)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (/src/app.js:8:16)
    at Module._compile (internal/modules/cjs/loader.js:1063:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/src/app.js' ]
}

Basic Docker compose file

We need to have dev and production environments. Both environment have the same configurations and they should be defined in a basic docker-compose file which is based on later phase for both environments.

docker-compose.yml looks like this. There is secrets option in show-env service it needs to be defined in a compose-file. The data defined in secrets is encrypted when managed by orchestration system. Without the system, docker-compose just pass the file to the container and the data isn’t encrypted.

version: "3.7"

x-labels: &app-net
    networks:
        - app-net

services: 
    show-env:
        image: show-env
        secrets:
         - source: test-secrets
           target: /src/config/secrets.json

    log-server:
        image:  log-server
        <<: *app-net

    restify-server: 
        image: restify-server
        depends_on: 
            - log-server
        <<: *app-net

networks:
    app-net:
        external:
            name: log-test-nat

show-env is the image created above. Other two images log-server and restify-server are in Docker folder. Download the source code from my Git repository if you haven’t downloaded it yet and run these commands.

cd log-server
npm run dbuild
cd restify-server
npm run dbuild

Docker compose file for Dev environment

The docker compose file for dev environment is following.

version: "3.7"

services: 
    show-env:
        env_file: 
         - ./config/dev.env

    log-server:
        ports: 
            - "8001:80"

    restify-server: 
        ports: 
            - "8002:80"

secrets:
    test-secrets:
        file: ./config/secrets-dev.json

It specifies a file to set environment variables and pass the secrets json file. This secrets file is copied to /src/config/secrets.json defined in basic docker-compose.yml file. The contents are following.

// dev.env
MODE="dev"
FOO="dev-foo"
HOGE="dev-hoge"

// secrets-dev.json
{
    "user": "dev-user",
    "password": "dev-password"
}

Let’s check the overridden compose file.

$ docker-compose -f docker-compose.yml -f docker-compose-dev.yml config
networks:
  app-net:
    external: true
    name: log-test-nat
secrets:
  test-secrets:
    file: ----\BlogPost\src\Docker\override-docker-compose\config\secrets-dev.json
services:
  log-server:
    image: log-server
    networks:
      app-net: {}
    ports:
    - published: 8001
      target: 80
  restify-server:
    depends_on:
      log-server:
        condition: service_started
    image: restify-server
    networks:
      app-net: {}
    ports:
    - published: 8002
      target: 80
  show-env:
    environment:
      FOO: dev-foo
      HOGE: dev-hoge
      MODE: dev
    image: show-env
    secrets:
    - source: test-secrets
      target: /src/config/secrets.json
version: '3.7'

Let’s run a command and see the result. Environment variables and secrets info are injected correctly from dev.env and secrets-dev.json.

$ docker-compose -f docker-compose.yml -f docker-compose-dev.yml up
Starting override-docker-compose_log-server_1 ... done
Recreating override-docker-compose_show-env_1     ... done
Starting override-docker-compose_restify-server_1 ... done
Attaching to override-docker-compose_log-server_1, override-docker-compose_show-env_1, override-docker-compose_restify-server_1
log-server_1      | {"message":"restify listening at http://[::]:80","level":"info"}
show-env_1        | === START ===
show-env_1        | Running for dev
show-env_1        | FOO : dev-foo
show-env_1        | HOGE: dev-hoge
show-env_1        | ----secrets----
show-env_1        | user: dev-user
show-env_1        | pass: dev-password
show-env_1        | === END ===
override-docker-compose_show-env_1 exited with code 0
log-server_1      | {"message":"restify-server: restify listening at http://[::]:80","level":"info"}
restify-server_1  | STATUS: 201
restify-server_1  | HEADERS: {"server":"restify","content-type":"application/json","content-length":"9","date":"Sun, 15 Nov 2020 13:51:25 GMT","connection":"close"}
restify-server_1  | BODY: "Created"
restify-server_1  | No more data in response.

The used port numbers are 8001 and 8002 which are defined in docker-compose-dev.yml file.

$ docker ps
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                  NAMES
ead3ce9aabdb        restify-server                 "docker-entrypoint.s窶ｦ"   50 minutes ago      Up 10 seconds       0.0.0.0:8002->80/tcp   override-docker-compose_restify-server_1
b47a8d6533a3        log-server                     "docker-entrypoint.s窶ｦ"   50 minutes ago      Up 10 seconds       0.0.0.0:8001->80/tcp   override-docker-compose_log-server_1

Docker compose file for production environment

I changed the port number for production environment.

version: "3.7"

services: 
    show-env:
        env_file: 
         - ./config/pro.env

    log-server:
        ports: 
            - "${LOG_SERVER_PORT}:80"

    restify-server: 
        ports: 
            - "${RESTIFY_SERVER_PORT}:80"

secrets:
    test-secrets:
        file: ./config/secrets-pro.json

Those env variables are defined in .env file. This is different from pro.env which is specified in env_file option. .env file is loaded automatically when executing docker-compose command. The files’ contents are following.

// .env
LOG_SERVER_PORT=8888
RESTIFY_SERVER_PORT=8889
COMPOSE_PATH_SEPARATOR=;
COMPOSE_FILE=docker-compose.yml;docker-compose-pro.yml

// pro.env
MODE="production"
FOO="production-foo"
HOGE="production-hoge"

// secrets-pro.json
{
    "user": "production-user",
    "password": "production-pass"
}

It sets both basic and production compose files to COMPOSE_FILE which means production environment containers start up by default. If we want to start it for production we can start it by docker-compose up. Let’s see the result. The environment variables and secrets info are now production.

$ docker-compose up
# or this command
$ docker-compose -f docker-compose.yml -f docker-compose-pro.yml up
Recreating override-docker-compose_log-server_1 ... done
Recreating override-docker-compose_show-env_1       ... done
Recreating override-docker-compose_restify-server_1 ... done
Attaching to override-docker-compose_log-server_1, override-docker-compose_show-env_1, override-docker-compose_restify-server_1
log-server_1      | {"message":"restify listening at http://[::]:80","level":"info"}
show-env_1        | === START ===
show-env_1        | Running for production
show-env_1        | FOO : production-foo
show-env_1        | HOGE: production-hoge
show-env_1        | ----secrets----
show-env_1        | user: production-user
show-env_1        | pass: production-pass
show-env_1        | === END ===
override-docker-compose_show-env_1 exited with code 0
log-server_1      | {"message":"restify-server: restify listening at http://[::]:80","level":"info"}
restify-server_1  | STATUS: 201
restify-server_1  | HEADERS: {"server":"restify","content-type":"application/json","content-length":"9","date":"Sun, 15 Nov 2020 14:45:36 GMT","connection":"close"}
restify-server_1  | BODY: "Created"
restify-server_1  | No more data in response.

The port numbers are the numbers defined in .env file.

$ docker ps
CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS                  NAMES
9778ebe0b641        restify-server                 "docker-entrypoint.s窶ｦ"   54 seconds ago      Up 52 seconds       0.0.0.0:8889->80/tcp   override-docker-compose_restify-server_1
7b10d7460c50        log-server                     "docker-entrypoint.s窶ｦ"   55 seconds ago      Up 53 seconds       0.0.0.0:8888->80/tcp   override-docker-compose_log-server_1

Conclusion

Create different compose files and define necessary settings there separately in order to have different environments. Then, specify the compose files in a command like following way.

docker-compose -f docker-compose.yml -f docker-compose-pro.yml up

You can also set the default environment by specifying the configurations in .env file and start up the containers by

docker-compose up

Complete source code here

By the way…
When I specified ports in basic docker-compose.yml file and tried to override them by specifying ports in extended compose file it didn’t work as expected.