Static testing or save Private Ryan

Static testing or save Private Ryan

The release often sneaks up unnoticed. And any mistake, suddenly discovered before him, threatens us with a shift of time, hotfixes, work until the morning and spent nerves. When such a hindrance began to occur systematically, we realized that it was impossible to live like this anymore. It was decided to develop a comprehensive validation system to save Private Ryan developer Artyom, who went home at 9 pm, or 10, or 11, before the release ... well, you understand. The idea was for the developer to learn about the error, while the changes had not yet reached the repository, and he himself had not lost the context of the task.

Today, changes are carefully checked first locally, and then a series of integration tests on the assembly farm. In this article we will talk about the first stage of testing - static testing, which monitors the correctness of resources and analyzes the code. This is the first subsystem in the chain and accounts for the majority of errors found.

How it all began

Manual process of checking the game before the release began in QA a week and a half before the release. Naturally, bugs that are at this stage need to be fixed as soon as possible.

Because of the lack of time for a good solution, a temporary “crutch” is added, which then takes root for a long time and overgrows with other not very popular solutions.

First of all, we decided to automate the finding of overt errors: falls, impossibility to make a set of actions for the game (open a store, make a purchase, play a level). To do this, the game starts in a special mode of auto-play and, if something went wrong, we will know about it immediately after passing the test on our farm.

But most of the errors that were found by testers and our automated Smoke test is the lack of a resource or incorrect settings of different systems. Therefore, the next step was static testing - checking the availability of resources, their interrelations and settings without launching the application. This system was launched as an additional step on the assembly farm and greatly simplified the finding and repair of errors. But why waste the resources of the assembly farm if you can detect an error before committing and entering the problem code into the repository? You can do this by precomit hooks that are just launched before creating a commit and sending it to the repository.

And yes, we are so cool that static testing before a commit and on an assembly farm is done with one code, which greatly simplifies its support.

Our efforts can be divided into three areas:

  • creating an assembly farm - the very place where everything that commits will be collected and checked;
  • developing static tests - checking the correctness of resources, their interconnections, running code analyzers;
  • runtime test development - launch the application in auto game mode.

A separate task was to organize the launch of tests on the machine from the developer. It was necessary to minimize the execution time locally (the developer does not have to wait 10 minutes to commit one line) and make sure that each system has our system installed.

Many requirements - one system

When developing there is a whole set of assemblies that can be useful: with and without cheats, beta or alpha, iOS or Android. In each case, you may need different resources, settings, or even a different code.Writing scripts for static tests for every possible build results in an intricate system with many parameters. In addition, it is difficult to maintain and modify it, each project also has its own set of crutches, bicycles.

Through trial and error, we arrived at one system, each test in which can take into account the launch context and decide whether or not to run it, what and how to check. At the start of tests, we identified three basic properties:

  • build type: for release and debug check resources will differ in severity, completeness of coverage, as well as identifiers settings and checks for available functionality;
  • platform: what is valid for an android may not be correct for iOS, resources are also collected differently and not all resources in the android version will be in iOS and vice versa;
  • launch location: where exactly we launch - on the build agent, where all available tests are needed or on the user's computer, where the list of the executable must be minimized.

Static Test System

The system kernel and the main set of static tests are implemented in python. The basis is only a few entities:

Test Context is a broad concept. It stores both build and run parameters, which we talked about above, as well as meta-information, which is filled and used by tests.

First you need to understand what tests to run. To do this, the meta-information contains the types of resources that we are interested in specifically in this launch. Resource types are determined by tests registered in the system. A test can be “associated” with a particular type or several, and if at the time of the commit it turns out that the files that this test checks have changed, then the associated resource has changed. It conveniently fits into our ideology - to run locally as few checks as possible: if the files for which the test is responsible have not changed, then it is not necessary to run it.

For example, there is a description of a fish in which a 3D model and texture is indicated. If the description file has changed, then it is checked that the model and texture specified in it exist. In other cases, there is no need to start checking the fish.

On the other hand, changing a resource may require changes and entities dependent on it: if the set of textures that is stored in xml files has changed, then it is necessary to check additionally 3D models, as it may turn out that the necessary model texture has been deleted. The optimizations described above are applied only locally on the user's machine at the time of the commit, and when running on an assembly farm, it is considered that all the files have changed and we run all the tests.

The next problem is the dependence of some tests on others: you cannot check the fish before finding all the textures and models. Therefore, we divided the execution into two stages:

  • context preparation
  • doing checks

In the first stage, the context is filled with information about the resources found (in the case of a fish, with identifiers of models and textures). In the second stage, using the saved information, simply check whether the necessary resource exists. Simplified context is presented below.

  class VerificationContext (object):
  def __init __ (self, app_path, build_type, platform, changed_files = None):
  self .__ app_path = app_path
  self .__ build_type = build_type
  self .__ platform = platform
  # Filled with running tests
  self .__ modified_resources = set ()
  self .__ expected_resources = set ()
  # If the launch comes from the precomit hook, then this list will contain the modified files.
  self.__changed_files = changed_files
  # Meta-data on the resources that found the tests
  self .__ resources = {}

 def expect_resources (self, resources):
  self .__ expected_resources.update (resources)

 def is_resource_expected (self, resource):
  return resource in self .__ expected_resources

 def register_resource (self, resource_type, resource_id, resource_data = None):
  self .__ resources.setdefault (resource_type, {}) [resource_id] = resource_data

 def get_resource (self, resource_type, resource_id):
  if resource_type is not in self .__ resources or resource_id not in self .__ resources [resource_type]:
  return none, none
  return resource_id, self .__ resources [resource_type] [resource_id]  

Having identified all the parameters that affect the test run, all the logic was hidden inside the base class. In a specific test, it remains to write only the test itself and the necessary values ​​for the parameters.

  class TestCase (object):
  def __init __ (self, name, context, build_types = None, platforms = None, predicate = None,
  expected_resources = None, modified_resources = None):
  self .__ name = name
  self .__ context = context
  self .__ build_types = build_types
  self .__ platforms = platforms
  self .__ predicate = predicate
  self .__ expected_resources = expected_resources
  self .__ modified_resources = modified_resources

  # Is the build type and platform suitable for running the test?
  # Have the resources for which the predicate is responsible changed
  self .__ need_run = self .__ check_run ()
  self .__ need_resource_run = False

  def context (self):
  return self .__ context

  def fail (self, message):
  print ('Fail: {}'. format (message))

  def __check_run (self):
  build_success = self .__ build_types is None or self .__ context.build_type in self .__ build_types
  platform_success = self .__ platforms is None or self .__ context.platform in self .__ platforms
  hook_success = build_success
  if build_success and self .__ context.is_build ('hook') and self .__ predicate:
  hook_success = any (self .__ predicate (changed_file) for changed_file in self .__ context.changed_files)
  return build_success and platform_success and hook_success

  def __set_context_resources (self):
  if not self .__ need_run:
  if self .__ modified_resources:
  self .__ context.modify_resources (self .__ modified_resources)
  if self .__ expected_resources:
  self .__ context.expect_resources (self .__ expected_resources)

  def init (self):
  "" "
  Runs after all the tests have been created and the information has been recorded in the context.
  about the changed resources and those resources that are needed by other tests
  "" "
  self .__ need_resource_run = self .__ modified_resources and any (self .__ context.is_resource_expected (resource) for self in self .__ modified_resources)

  def _prepare_impl (self):

  def prepare (self):
  if not self .__ need_run and not self .__ need_resource_run:
  self._prepare_impl ()

  def _run_impl (self):

  def run (self):
  if self .__ need_run:
  self._run_impl ()  

Returning to the example with a fish, you need two tests, one of which finds textures and registers them in context, the other looks for textures for the found models.

  class VerifyTexture (TestCase):
  def __init __ (self, context):
  super (VerifyTexture, self) .__ init __ ('VerifyTexture', context,
  build_types = ['production', 'hook'],
  platforms = ['windows', 'ios'],
  expected_resources = None,
  modified_resources = ['Texture'],
  predicate = lambda file_path: os.path.splitext (file_path) [1] == '.png')

  def _prepare_impl (self):
  texture_dir = os.path.join (self.context.app_path, 'resources', 'textures')
  for root, dirs, files in os.walk (texture_dir):
  for tex_file in files:
  self.context.register_resource ('Texture', tex_file)

 class VerifyModels (TestCase):
  def __init __ (self, context):
  super (VerifyModels, self) .__ init __ ('VerifyModels', context,
  expected_resources = ['Texture'],
  predicate = lambda file_path: os.path.splitext (file_path) [1] == '.obj')

  def _run_impl (self):
  models_descriptions = etree.parse (os.path.join (self.context.app_path, 'resources', 'models.xml'))
  for model_xml in models_descriptions.findall ('.//Model'):
  texture_id = model_xml.get ('texture')
  texture = self.context.get_resource ('Texture', texture_id)
  if texture is None: ('Texture for model {} was not found: {}'. format (model_xml.get ('id'), texture_id))  

Project Spread

Playrix game development takes place on its own engine and, accordingly, all projects have a similar file structure and code using the same rules. Therefore, there are many common tests that are written once and are in common code. It is enough for projects to update the version of the testing system and connect a new test to themselves.

To simplify the integration, we wrote a runner, which is fed into the configuration file and design tests (more about them later). The configuration file contains basic information about which we wrote above: build type, platform, project path.

  class Runner (object):
  def __init __ (self, config_str, setup_function):
  self .__ tests = []

  config_parser = RawConfigParser ()
  config_parser.read_string (config_str)

  app_path = config_parser.get ('main', 'app_path')
  build_type = config_parser.get ('main', 'build_type')
  platform = config_parser.get ('main', 'platform')

  '' '
  get_changed_files returns the list of changed files and depends on the CVS used.
  '' '
  changed_files = None if build_type! = 'hook' else get_changed_files ()
  self .__ context = VerificationContext (app_path, build_type, platform, changed_files)
  setup_function (self)

  def context (self):
  return self .__ context

  def add_test (self, test):
  self .__ tests.append (test)

  def run (self):
  for test in self .__ tests:
  test.init ()

  for test in self .__ tests:
  test.prepare ()

  for test in self .__ tests: ()  

The beauty of the config file is that it can be generated on an assembly farm for different assemblies in automatic mode. But transferring settings for all tests through this file may not be very convenient. To do this, there is a special configuration xml, which is stored in the project repository and lists the ignored files, masks for searching in the code, and so on.
Example configuration file
 app_path = {app_path}
 build_type = production
 platform = ios  

Example of setup xml
  & lt; root & gt;
 & lt; VerifySourceCodepage allow_utf8 = "true" allow_utf8Bom = "false" autofix_path = "ci/autofix" & gt;
  & lt; IgnoreFiles & gt; * android/tmp/* & lt;/IgnoreFiles & gt;
 & lt;/VerifySourceCodepage & gt;
 & lt; VerifyCodeStructures & gt;
  & lt; Checker name = "NsStringConversion"/& gt;
  & lt; Checker name = "LogConstructions"/& gt;
 & lt;/VerifyCodeStructures & gt;
 & lt;/root & gt;

In addition to the common part, the projects have their own features and differences, therefore there are sets of design tests that are connected to the system through runner configuration.For the code in the examples, a couple of lines will be enough to run:

  def setup (runner):
  runner.add_test (VerifyTexture (runner.context))
  runner.add_test (VerifyModels (runner.context))

 def run ():
  raw_config = '' '
  app_path = {app_path}
  build_type = production
  platform = ios
  '' '
  runner = runner (raw_config, setup) ()  

Gathered Rake

Although python itself is cross-platform, we regularly had problems with the fact that users have their own unique environment in which there may not be the version that we expect, several versions, or the interpreter may be absent altogether. As a result, it does not work the way we expect or does not work at all. There were several iterations to solve this problem:

  1. Python and all packages are installed by the user. But there are two “but”: not all users are programmers and installation via pip install for designers, and for programmers too, can be a problem.
  2. There is a script that installs all the necessary packages. This is better, but if the user has the wrong python installed, then collisions may occur in the work.
  3. Deliver the correct version of the interpreter and dependencies from the artifact repository (Nexus) and run tests in a virtual environment.

Another problem is speed. The more tests there are, the longer it takes to check changes on the user's computer. Every few months there is profiling and optimization of bottlenecks. So the context was refined, the cache for text files appeared, the predicate mechanisms (definitions that this file is interesting for the test) were refined.

And then it remains only to solve the problem, how to implement the system on all projects and force all developers to include pre-commit hooks, but this is another story ...


In the development process, we danced on the rake, fought hard, but still got a system that allows us to find errors during a commit, reduced the work of testers, and the tasks before the release of the loss of texture were a thing of the past. For complete happiness, there is not enough simple setting of the environment and optimization of individual tests, but golems from the ci department are working hard on this.

For a complete example of the code used in the article as examples, see our repository .

Source text: Static testing or save Private Ryan