GitLab Shell Runner. Competitive launch of test services with Docker Compose

GitLab Shell Runner. Competitive launch of test services with Docker Compose



This article will be of interest to both testers and developers, but it is designed mostly for automatists who are faced with the problem of setting up GitLab CI/CD for integration testing under conditions of insufficient infrastructure resources and/or lack of a container orchestration platform. I’ll tell you how to set up deployment of test environments using docker compose on a single GitLab shell runner so that when deploying multiple environments, the services that are started do not interfere with each other.

< br/>

Content




Prerequisites


  1. In my practice, it often happened to "heal" integration testing on projects. And often the first and most significant problem is the CI pipeline, in which integration testing of the developed service (s) is performed in the dev/stage environment. This caused quite a few problems:


    • Due to defects in a particular service during the integration testing process, the test loop may be corrupted by dead data. There were cases when sending a request with a broken JSON format hung the service, which caused the stand to be completely inoperable.
    • Slowing down the work of the test circuit with the growth of test data. I think, to describe an example with cleaning/rollback of a DB does not make sense. In my practice, I have not met a project where this procedure would go smoothly.
    • Risk of disrupting the performance of the test loop when testing general system settings. For example, user/group/password/application policy.
    • Test data from autotests interfere with live manual testers.

    Someone will say that good autotests should clean the data after themselves. I have arguments against:


    • Dynamic stands are very convenient to use.
    • Not every object can be removed from the system through the API. For example, the call to delete an object is not implemented, since it contradicts business logic.
    • When creating an object through the API, a huge amount of metadata can be created, which is difficult to delete.
    • If tests have dependencies among themselves, then the process of clearing data after performing tests turns into a headache.
    • Additional (and, in my opinion, not justified) calls to the API.
    • And the main argument: when the test data starts to be cleaned directly from the database. It turns into a real PK/FK circus! From the developers you can hear: “I just added/deleted/renamed the tablet, why did 100500 integration tests get?”

    In my opinion, the most optimal solution is a dynamic environment.


  2. Many people use docker-compose to run a test environment, but very few people use docker-compose when performing integration testing in CI/CD. And here I do not take into account kubernetes, swarm and other container orchestration platforms. Not every company they are. It would be nice if docker-compose.yml were universal.
  3. Even if we have our own QA runner, how can we ensure that services started via docker-compose do not interfere with each other?
  4. How to collect logs of test services?
  5. How to clean the runner?

I have my own GitLab runner for my projects and I encountered these questions when developing a Java client for TestRail . Or rather, when you run the integration tests. Here we will solve these issues with examples from this project.


To the content



GitLab Shell Runner


For the runner, I recommend a Linux virtual machine with 4 vCPU, 4 GB RAM, 50 GB HDD.
On the Internet, a lot of information on how to configure gitlab-runner, so briefly:


  • Log in via SSH
  • If you have less than 8 GB of RAM, I recommend make swap 10 GB so that the OOM killer does not come in and does not kill us because of the lack of RAM. This can happen when more than 5 tasks are running simultaneously. Tasks will be slower, but stable.


    Example with OOM killer

    If you see bash: line 82: 26474 in the task logs Killed , then simply execute sudo dmesg | grep 26474


      [26474] 1002 26474 1061935 123806 339 0 0 java
     Out of memory: Kill process 26474 (java) score 127 or sacrifice child
     Killed process 26474 (java) total-vm: 4247740kB, anon-rss: 495224kB, file-rss: 0kB, shmem-rss: 0kB  

    And if the picture looks something like this, then either add a swap, or throw out RAM.




  • Install gitlab-runner , docker , docker-compose , make.
  • Add the user gitlab-runner to the docker
    group
      sudo groupadd docker
     sudo usermod -aG docker gitlab-runner  
  • register gitlab-runner.
  • Open for editing /etc/gitlab-runner/config.toml and add


      concurrent = 20
     [[runners]]
      request_concurrency = 10  

    This will allow you to run parallel tasks on one runner. Read more here .
    If your machine is more powerful, for example, 8 vCPU, 16 GB RAM, then these numbers can be made at least 2 times larger. But it all depends on what exactly will run on this runner and in what quantity.



That's enough.


To the content



Preparing docker-compose.yml


The main task is the universal docker-compose.yml, which developers/testers can use both locally and in the CI pipeline.


First of all, we make unique service names for CI. One of the unique variables in GitLab CI is the CI_JOB_ID variable.If you specify container_name with the value "service - $ {CI_JOB_ID: -local}" , then in the case of:


  • if CI_JOB_ID is not defined in the environment variables,
    then the service name will be service-local
  • if CI_JOB_ID is defined in environment variables (for example 123),
    then the service name will be service-123

Secondly, we are making a common network for the services we run. This gives us network-level isolation when running multiple test environments.


  networks:
  default:
  external:
  name: service-network - $ {CI_JOB_ID: -local}  

Actually, this is the first step to success =)


Example of my docker-compose.yml with comments
  version:  "3"

 # To work correctly, web (php) and fmt need,
 # so that containers have shared executable content.
 # In our case, this is the/var/www/testrail directory
 volumes:
  static-content:

 # Isolate the environment at the network level
 networks:
  default:
  external:
  name: testrail-network - $ {CI_JOB_ID: -local}

 services:
  db:
  image: mysql: 5.7.22
  # Each container_name contains $ {CI_JOB_ID: -local}
  container_name: "testrail-mysql - $ {CI_JOB_ID: -local}"
  environment:
  MYSQL_HOST: db
  MYSQL_DATABASE: mydb
  MYSQL_ROOT_PASSWORD: 1234
  SKIP_GRANT_TABLES: 1
  SKIP_NETWORKING: 1
  SERVICE_TAGS: dev
  SERVICE_NAME: mysql
  networks:
  - default

  migration:
  image: registry.gitlab.com/touchbit/image/testrail/migration:latest
  container_name: "testrail-migration - $ {CI_JOB_ID: -local}"
  links:
  - db
  depends_on:
  - db
  networks:
  - default

  fpm:
  image: registry.gitlab.com/touchbit/image/testrail/fpm:latest
  container_name: "testrail-fpm - $ {CI_JOB_ID: -local}"
  volumes:
  - static-content:/var/www/testrail
  links:
  - db
  networks:
  - default

  web:
  image: registry.gitlab.com/touchbit/image/testrail/web:latest
  container_name: "testrail-web - $ {CI_JOB_ID: -local}"
  # If the TR_HTTP_PORT or TR_HTTPS_PORTS variables are not defined,
  # that service rises on port 80 and 443 respectively.
  ports:
  - $ {TR_HTTP_PORT: -80}: 80
  - $ {TR_HTTPS_PORT: -443}: 443
  volumes:
  - static-content:/var/www/testrail
  links:
  - db
  - fpm
  networks:
  - default  

Local Launch Example


  docker-compose -f docker-compose.yml up -d
 Starting testrail-mysql-local ... done
 Starting testrail-migration-local ... done
 Starting testrail-fpm-local ... done
 Recreating testrail-web-local ... done  

But not everything is so simple with the launch of the CI.


To the content



Makefile Preparation


I use the Makefile, as it is very convenient both for local control of the environment and in CI. Further comments inline


  # In my projects, all the auxiliary things are in the `.indirect` directory,
 # including `docker-compose.yml`

 # Use bash with pipefail option
 # pipefail - failit execution of the pipe, if the command was executed with an error
 SHELL =/bin/bash -o pipefail

 # Stop containers and delete network
 docker-kill:
  docker-compose -f $$ {CI_JOB_ID: -. indirect}/docker-compose.yml kill
  docker network rm network - $$ {CI_JOB_ID: -testrail} ||  true

 # Pre-docker-kill
 docker-up: docker-kill
  # Create a network for the environment
  docker network create network - $$ {CI_JOB_ID: -testrail}
  # Pick up the latest images from the docker-registry
  docker-compose -f $$ {CI_JOB_ID: -. indirect}/docker-compose.yml pull
  # Run environment
  # force-recreate - forced re-creation of containers
  # renew-anon-volumes - do not use volumes of previous containers
  docker-compose -f $$ {CI_JOB_ID: -. indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
  # Well, just in case, deduce what we have there in principle running on a typewriter
  docker ps

 # Collective service logs
 docker-logs:
  mkdir ./logs ||  true
  docker logs testrail-web - $$ {CI_JOB_ID: -local} & gt; & amp;  logs/testrail-web.log
  docker logs testrail-fpm - $$ {CI_JOB_ID: -local} & gt; & amp;  logs/testrail-fpm.log
  docker logs testrail-migration - $$ {CI_JOB_ID: -local} & gt; & amp;  logs/testrail-migration.log
  docker logs testrail-mysql - $$ {CI_JOB_ID: -local} & gt; & amp;  logs/testrail-mysql.log

 # Runner cleaning
 docker-clean:
  @echo Stop all testrail containers
  docker kill $$ (docker ps --filter = name = testrail -q) ||  true
  @echo cleaning docker containers
  docker rm -f $$ (docker ps -a -f --filter = name = testrail status = exited -q) ||  true
  @echo cleaning dangling images
  docker rmi -f $$ (docker images -f "dangling = true" -q) ||  true
  @echo cleaning testrail images
  docker rmi -f $$ (docker images - file = reference = 'registry.gitlab.com/touchbit/image/testrail/*' -q) ||  true
  @echo clear all unused volume
  docker volume rm -f $$ (docker volume ls -q) ||  true
  @echo Cleaning all testrail networks
  docker network rm $ (docker network ls --filter = name = testrail -q) ||  true
  docker ps
  

Checking


make docker-up
  $ make docker-up
 docker-compose -f $ {CI_JOB_ID: -. indirect}/docker-compose.yml kill
 Killing testrail-web-local ... done
 Killing testrail-fpm-local ... done
 Killing testrail-mysql-local ... done
 docker network rm network - $ {CI_JOB_ID: -testrail} ||  true
 network-testrail
 docker network create network - $ {CI_JOB_ID: -testrail}
 d2ec063324081c8bbc1b08fd92242c2ea59d70cf4025fab8efcbc5c6360f083f
 docker-compose -f $ {CI_JOB_ID: -. indirect}/docker-compose.yml pull
 Pulling db ... done
 Pulling migration ... done
 Pulling fpm ... done
 Pulling web ... done
 docker-compose -f $ {CI_JOB_ID: -. indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
 Recreating testrail-mysql-local ... done
 Recreating testrail-fpm-local ... done
 Recreating testrail-migration-local ... done
 Recreating testrail-web-local ... done
 docker ps
 CONTAINER ID PORTS NAMES
 a845d3cb0e5a 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp testrail-web-local
 19d8ef001398 9000/tcp testrail-fpm-local
 e28840a2369c 3306/tcp, 33060/tcp testrail-migration-local
 0e7900c23f37 3306/tcp testrail-mysql-local
  

make docker-logs
  $ make docker-logs
 mkdir ./logs ||  true
 mkdir: cannot create directory ‘./logs’: File exists
 docker logs testrail-web - $ {CI_JOB_ID: -local} & gt; & amp;  logs/testrail-web.log
 docker logs testrail-fpm - $ {CI_JOB_ID: -local} & gt; & amp;  logs/testrail-fpm.log
 docker logs testrail-migration - $ {CI_JOB_ID: -local} & gt; & amp;  logs/testrail-migration.log
 docker logs testrail-mysql - $ {CI_JOB_ID: -local} & gt; & amp;  logs/testrail-mysql.log  


To the content



Preparing .gitlab-ci.yml



Running integration tests


  Integration:
  stage: test
  tags:
  - my-shell-runner
  before_script:
  # Authenticate in registry
  - docker login -u gitlab-ci-token -p $ {CI_JOB_TOKEN} $ {CI_REGISTRY}
  # Generate pseudounique TR_HTTP_PORT and TR_HTTPS_PORT
  - export TR_HTTP_PORT = $ (shuf -i10000-60000 -n1)
  - export TR_HTTPS_PORT = $ (shuf -i10000-60000 -n1)
  # create directory with task ID
  - mkdir $ {CI_JOB_ID}
  # copy our docker-compose.yml to the created directory
  # so that the context is different for each task
  - cp .indirect/docker-compose.yml $ {CI_JOB_ID}/docker-compose.yml
  script:
  # raise our environment
  - make docker-up
  # run the tests with the jar being executed (mine)
  - java -jar itest.jar --http-port $ {TR_HTTP_PORT} --https-port $ {TR_HTTPS_PORT}
  # or in a container
  - docker run --network = testrail-network - $ {CI_JOB_ID: -local} --rm itest
  after_script:
  # collect logs
  - make docker-logs
  # stop environment
  - make docker-kill
  artifacts:
  # save logs
  when: always
  paths:
  - logs
  expire_in: 30 days  

As a result of running such a task in artifacts, the logs directory will contain logs of services and tests. Which is very convenient in case of errors. In my case, each test in parallel writes its own log, but I will tell you about it separately.



To the content



Runner cleaning


The task will run only as scheduled.


  stages:
 - clean
 - build
 - test

 Clean runner:
  stage: clean
  only:
  - schedules
  tags:
  - my-shell-runner
  script:
  - make docker-clean  

Next, go to our GitLab project - & gt; CI/CD - & gt; Schedules - & gt; New Schedule and add new schedule



To the content



Result


Run 4 tasks in GitLab CI


In the logs of the last task with integration tests we see containers from different tasks


  CONTAINER ID NAMES
 c6b76f9135ed testrail-web-204645172
 01d303262d8e testrail-fpm-204645172
 2cdab1edbf6a testrail-migration-204645172
 826aaf7c0a29 testrail-mysql-204645172
 6dbb3fae0322 testrail-web-204645084
 3540f8d448ce testrail-fpm-204645084
 70fea72aa10d testrail-mysql-204645084
 d8aa24b2892d testrail-web-204644881
 6d4ccd910fad testrail-fpm-204644881
 685d8023a3ec testrail-mysql-204644881
 1cdfc692003a testrail-web-204644793
 6f26dfb2683e testrail-fpm-204644793
 029e16b26201 testrail-mysql-204644793
 c10443222ac6 testrail-web-204567103
 04339229397e testrail-fpm-204567103
 6ae0accab28d testrail-mysql-204567103
 b66b60d79e43 testrail-web-204553690
 033b1f46afa9 testrail-fpm-204553690
 a8879c5ef941 testrail-mysql-204553690
 069954ba6010 testrail-web-204553539
 ed6b17d911a5 testrail-fpm-204553539
 1a1eed057ea0 testrail-mysql-204553539  

More detailed log
  $ docker login -u gitlab-ci  -token -p $ {CI_JOB_TOKEN} $ {CI_REGISTRY}
 WARNING!  Using --password via the CLI is insecure.  Use --password-stdin.
 WARNING!  Your password will be stored unencrypted in/home/gitlab-runner/.docker/config.json.
 Configure a credential helper to remove this warning.  See
 https://docs.docker.com/engine/reference/commandline/login/#credentials-store
 Login Succeeded
 $ export TR_HTTP_PORT = $ (shuf -i10000-60000 -n1)
 $ export TR_HTTPS_PORT = $ (shuf -i10000-60000 -n1)
 $ mkdir $ {CI_JOB_ID}
 $ cp .indirect/docker-compose.yml $ {CI_JOB_ID}/docker-compose.yml
 $ make docker-up
 docker-compose -f $ {CI_JOB_ID: -. indirect}/docker-compose.yml kill
 docker network rm testrail-network - $ {CI_JOB_ID: -local} ||  true
 Error: No such network: testrail-network-204645172
 docker network create testrail-network - $ {CI_JOB_ID: -local}
 0a59552b4464b8ab484de6ae5054f3d5752902910bacb0a7b5eca698766d0331
 docker-compose -f $ {CI_JOB_ID: -. indirect}/docker-compose.yml pull
 Pulling web ... done
 Pulling fpm ... done
 Pulling migration ... done
 Pulling db ... done
 docker-compose -f $ {CI_JOB_ID: -. indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d
 Creating volume "204645172_static-content" with default driver
 Creating testrail-mysql-204645172 ...
 Creating testrail-mysql-204645172 ... done
 Creating testrail-migration-204645172 ... done
 Creating testrail-fpm-204645172 ... done
 Creating testrail-web-204645172 ... done
 docker ps
 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
 c6b76f9135ed registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of ..." 13 seconds ago Up 1 second 0.0.0.0 entry 1148->80/tcp, 0.0.0.0:25426->  ; 443/tcp testrail-web-204645172
 01d303262d8e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi ..." 16 seconds ago Up 13 seconds 9000/tcp testrail-fpm-204645172
 2cdab1edbf6a registry.gitlab.com/touchbit/image/testrail/migration:latest "docker-entrypoint.s ..." 16 seconds ago Up 13 seconds 3306/tcp, 33060/tcp testrail-migration-204645172
 826aaf7c0a29 mysql: 5.7.22 "docker-entrypoint.s ..." 18 seconds ago Up 16 seconds 3306/tcp testrail-mysql-204645172
 6dbb3fae0322 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of ..." 36 seconds ago Up 22 seconds 0.0.0.0:44202->80/tcp, 0.0.0.0:20151->  ; 443/tcp testrail-web-204645084
 3540f8d448ce registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi ..." 38 seconds ago Up 35 seconds 9000/tcp testrail-fpm-204645084
 70fea72aa10d mysql: 5.7.22 "docker-entrypoint.s ..." 40 seconds ago Up 37 seconds 3306/tcp testrail-mysql-204645084
 d8aa24b2892d registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of ..." About a minute ago Up 53 seconds 0.0.0.0:31103->80/tcp, 0.0.0.0illar 3872-  & gt; 443/tcp testrail-web-204644881
 6d4ccd910fad registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi" ... "About a minute ago Up About a minute 9000/tcp testrail-fpm-204644881
 685d8023a3ec mysql: 5.7.22 "docker-entrypoint.s ..." About a minute ago Up About a minute 3306/tcp testrail-mysql-204644881
 1cdfc692003a registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of ..." About a minute ago Up About a minute 0.0.0.0:44752->80/tcp, 0.0.0.0:23540  - & gt; 443/tcp testrail-web-204644793
 6f26dfb2683e registry ..gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi ..."
 029e16b26201 mysql: 5.7.22 "docker-entrypoint.s ..." About a minute ago Up About a minute 3306/tcp testrail-mysql-204644793
 c10443222ac6 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of ..." 5 hours ago Up 5 hours 0.0.0.0:57123->80/tcp, 0.0.0.0.731657->  ; 443/tcp testrail-web-204567103
 04339229397e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi ..." 5 hours ago Up 5 hours 9000/tcp testrail-fpm-204567103
 6ae0accab28d mysql: 5.7.22 "docker-entrypoint.s ..." 5 hours ago Up 5 hours 3306/tcp testrail-mysql-204567103
 b66b60d79e43 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of ..." 5 hours ago Up 5 hours 0.0.0.0:5632321->80/tcp, 0.0.0.0º8787->  ; 443/tcp testrail-web-204553690
 033b1f46afa9 registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi ..." 5 hours ago Up 5 hours 9000/tcp testrail-fpm-204553690
 a8879c5ef941 mysql: 5.7.22 "docker-entrypoint.s ..." 5 hours ago Up 5 hours 3306/tcp testrail-mysql-204553690
 069954ba6010 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of ..." 5 hours ago Up 5 hours 0.0.0.0:3286869->80/tcp, 0.0.0.0:16066->  ; 443/tcp testrail-web-204553539
 ed6b17d911a5 registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi ..." 5 hours ago Up 5 hours 9000/tcp testrail-fpm-204553539
 1a1eed057ea0 mysql: 5.7.22 "docker-entrypoint.s ..." 5 hours ago Up 5 hours 3306/tcp testrail-mysql-204553539  

All tasks successfully completed

Task artifacts contain logs of services and tests



It seems everything is beautiful, but there is a nuance. Pipeline can be forcibly canceled during the execution of integration tests, in which case running containers will not be stopped. From time to time you need to clean the runner. Unfortunately, the revision task in GitLab CE is still in the status of Open


But we added the launch of the task on a schedule, and no one forbids us to start it manually.
Moving to our project - & gt; CI/CD - & gt; Schedules and run the Clean runner


task


Total:


  • We have one shell runner.
  • There is no conflict between tasks and the environment.
  • We have a parallel launch of tasks with integration tests.
  • You can run integration tests either locally or in a container.
  • Logs of services and tests are collected and attached to the pipeline-task.
  • It is possible to clear the runner from old docker images.

Setup time is ~ 2 hours.
That's all. I will be glad to feedback.


To the content

Source text: GitLab Shell Runner. Competitive launch of test services with Docker Compose