Organizing your Symfony project tests

Published on 2021-12-22 • Modified on 2021-12-22

In this post, we see how to organize your Symfony project tests. We check all the available test types and create consistent and robust test suites. Let's go! 😎

» Published in "A week of Symfony 782" (20-26 December 2021).

Prerequisite

I will assume you have at least a basic knowledge of Symfony and know what automated tests are.

Configuration

• PHP 8.3
• Symfony 6.4.5
• PHPUnit 9.5.26

Introduction

When you start adding tests, sometimes you add files at the root of the tests folder. But, as you add more and more tests, it can quickly be a mess if you continue to do so. Let's see how to do it the right way.

Goal

We will review all the tests types with an example covering each case. I'll show what I put in place in most of my projects and the resulting folder hierarchy

Test cases?

First, let's pass in revue all the differents tests types. But what Symfony says about the subject? We don't even have to read the documentation; for now, let's use the make:test command:

bin/console make:test

We get the following output:

 Which test type would you like?:
  [TestCase       ] basic PHPUnit tests
  [KernelTestCase ] basic tests that have access to Symfony services
  [WebTestCase    ] to run browser-like scenarios, but that don't execute JavaScript code
  [ApiTestCase    ] to run API-oriented scenarios
  [PantherTestCase] to run e2e scenarios, using a real-browser or HTTP client and a real web server

Great! That's a good summary. Let's check each of these "TestCase".

TestCase > Unit tests (PHPUnit tests)

“A unit test is an automated piece of code that invokes a unit of work in the system and then checks a single assumption about the behaviour of that unit of work. ”

I took this definition from this article of Royo Sherove, which sounds straightforward enough. Let's see an example from this project:

<?php

// tests/Unit/Helper/StringHelperTest.php

declare(strict_types=1);

namespace App\Tests\Unit\Helper;

use App\Helper\String\AppUnicodeString;
use App\Helper\String\StringHelper;
use PHPUnit\Framework\TestCase;

/**
 * @see StringHelper
 */
final class StringHelperTest extends TestCase
{
    /**
     * @return iterable<int, array{0:?string, 1:string}>
     */
    public function provideTestFirstCharForUnicode(): iterable
    {
        yield ['ABCD', 'A'];
        yield ['äBCD', 'ä'];
        yield [null, ''];
        yield ['', ''];
        yield ['123456789', '1'];
        yield ['नमस्ते', 'न'];
        yield ['さよなら', 'さ'];
        yield ['🙂👻🐱🌳🙃', '🙂'];
        yield ['🇫🇷🇬🇧🇺🇸🇧🇪', '🇫🇷'];
        yield ['ӚӜҾѾѮ', 'Ӛ'];
    }

    /**
     * @dataProvider provideTestFirstCharForUnicode
     *
     * @see StringHelper::s
     */
    public function testFirstCharFromS(?string $string, string $expected): void
    {
        $appString = new StringHelper();
        self::assertSame($expected, $appString->s($string)->firstChar());
    }

    /**
     * @dataProvider provideTestFirstCharForUnicode
     *
     * @see StringHelper::u
     * @see AppUnicodeString::firstChar
     */
    public function testFirstCharFromU(?string $string, string $expected): void
    {
        $stringHelper = new StringHelper();
        self::assertSame($expected, $stringHelper->u($string)->firstChar());
    }

    /**
     * @return iterable<int, array{0:?string, 1:string}>
     */
    public function provideTestFirstCharForByte(): iterable
    {
        yield ['ABCD', 'A'];
        yield ['abcd', 'a'];
        yield [null, ''];
        yield ['', ''];
        yield ['123456789', '1'];
    }

    /**
     * @dataProvider provideTestFirstCharForByte
     *
     * @see StringHelper::b
     * @see AppByteString::firstChar
     */
    public function testFirstCharFromB(?string $string, string $expected): void
    {
        $stringHelper = new StringHelper();
        self::assertSame($expected, $stringHelper->b($string)->firstChar());
    }
}

This test checks the firstChar() function of the StringHelper class. We instantiate the class then we run assertions with different data thanks to the PHPUnit dataProvider. I put these tests in the tests/Unit folder. Notice that it extends the PHPUnit\Framework\TestCase class. That means we aren't in a Symfony context; we use plain PHP classes.

Let's run it:

./vendor/bin/phpunit --filter=StringHelperTest

PHPUnit 9.5.10 by Sebastian Bergmann and contributors.

Testing
.........................                                         25 / 25 (100%)

Time: 00:00.035, Memory: 20.00 MB

OK (25 tests, 25 assertions)

As we don't even use Symfony, these tests are very fast. You can use the mocking tools provided by PHPUnit (getMockBuilder()) or use your favourite mocking frameworks like Mockery or Prophecy.

KernelTestCase > Integration tests

“Integration testing is the phase in software testing in which individual software modules are combined and tested as a group. ”

This definition comes from Wikipedia. This time we are in a Symfony context. The name of the case is self explicit: we have access to the kernel, which means that we also have access to all services. To avoid extra configuration, all services are marked as "public" in the test environment to get them with the ContainerInterface::get() function. Let's see an example:

<?php

declare(strict_types=1);

namespace App\Tests\Integration\Twig\Extension;

use App\Twig\Extension\EnvExtension;
use Symfony\Bundle\FrameworkBundle\Test\KernelTestCase;

/**
 * @see EnvExtension
 */
final class EnvExtensionTest extends KernelTestCase
{
    private EnvExtension $envExtension;

    protected function setUp(): void
    {
        $this->envExtension = self::getContainer()->get(EnvExtension::class);
    }

    /**
     * @return iterable<int, array{0: string}>
     */
    public function provideGetGlobals(): iterable
    {
        yield ['php_minor_version'];
        yield ['php_version'];
        yield ['sf_major_version'];
        yield ['sf_minor_version'];
        yield ['sf_version'];
    }

    /**
     * @dataProvider provideGetGlobals
     *
     * @see EnvExtension::getGlobals
     */
    public function testGetGlobals(string $global): void
    {
        $globals = $this->envExtension->getGlobals();
        self::assertArrayHasKey($global, $globals);
        self::assertNotEmpty($globals[$global]);
    }
}

The setup() function retrieves the service we want to test and stores it as a property. Then we can use it in our test. This one test a custom Twig extension that dynamically adds some global variables for the current environment (PHP version, Symfony version...).

Let's run it:

./vendor/bin/phpunit --filter=EnvExtensionTest

PHPUnit 9.5.10 by Sebastian Bergmann and contributors.

Testing
.                                                                   1 / 1 (100%)

Time: 00:00.267, Memory: 44.50 MB

OK (1 test, 10 assertions)

We notice the test is about ten times slower than the unit test! Why? Remember, we are now using the kernel, and it has to be booted. It is automatically done when calling self::getContainer(). The great thing is that it has only to be booted once. After, all the following scenarios are much faster.

ApiTestCase > API-oriented scenarios

These tests are a particular type of functional tests. I put them before because they are simpler than the first ones, and we generally only deal with JSON requests and responses. The ApiTestCase file comes from API Platform; that's a good reason to install it even if you don't use it already! Here is an example:

<?php

declare(strict_types=1);

namespace App\Tests\Api\Security;

use ApiPlatform\Symfony\Bundle\Test\ApiTestCase;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Contracts\HttpClient\ResponseInterface;

use function Symfony\Component\String\u;

final class JsonLoginTest extends ApiTestCase
{
    public function testLoginOK(): void
    {
        $response = $this->login('reader', 'test');
        self::assertResponseIsSuccessful();
        $arrayResponse = $response->toArray();
        self::assertArrayHasKey('token', $arrayResponse);
        self::assertNotTrue(u($response->toArray()['token'])->isEmpty());
    }

    public function testLoginNOK(): void
    {
        $this->login('reader', 'wrong password');
        self::assertResponseStatusCodeSame(Response::HTTP_UNAUTHORIZED);
        self::assertResponseHeaderSame('content-type', 'application/json');
        self::assertJsonEquals([
            'code' => Response::HTTP_UNAUTHORIZED,
            'message' => 'Invalid credentials.',
        ]);
        self::assertJsonContains([
            'message' => 'Invalid credentials.',
        ]);
    }

    /**
     * JSON Login try with a given email and password.
     */
    public function login(string $username, string $password): ResponseInterface
    {
        return self::createClient()->request('POST', '/api/login_check', [
            'json' => compact('username', 'password'),
        ]);
    }
}

Let's run it:

./vendor/bin/phpunit --filter=JsonLoginTest

PHPUnit 9.5.10 by Sebastian Bergmann and contributors.

Testing
..                                                                  2 / 2 (100%)

Time: 00:01.184, Memory: 56.50 MB

OK (2 tests, 6 assertions)

This time we are above a second even we only have two tests in this scenario. Why so slow? Because this time, we use actual Symfony requests and responses, and everything needs to be booted at the first test. The following are much faster. In this test, we use the assertJsonEquals assertion, which is very practical to test the content of a response. We also have the assertJsonContains that allows checking partial content. Imagine we have dynamic content in the response like a creation date: we can't test a given value as it would change at each fixtures loading.

WebTestCase > functional or application tests

“Wikipedia: Functional testing is a quality assurance (QA) process and a type of black-box testing that bases its test cases on the specifications of the software component under test. ”

This kind of test allows testing a full feature of your application. This time, we can use the UI, fill and post forms. It also includes smoke tests, which guarantee that a page "work" means that it returns at least a 200 status code and not an error 500. Let's check an example:

<?php

declare(strict_types=1);

namespace App\Tests\Functional\Controller;

use App\Subscriber\NotFoundExceptionSubscriber;
use App\Tests\WebTestCase;
use Symfony\Component\ErrorHandler\ErrorHandler;
use Symfony\Component\HttpFoundation\Response;

final class AppControllerTest extends WebTestCase
{
    /**
     * @see AppController::root
     */
    public function testRootAction(): void
    {
        $client = self::createClient();
        $this->assertUrlIsRedirectedTo($client, '/', '/en');
    }

    /**
     * @see AppController::homepage
     */
    public function testHomepage(): void
    {
        $client = self::createClient();
        $this->assertResponseIsOk($client, '/en');
        $this->assertResponseIsOk($client, '/fr');
    }

    /**
     * @see NotFoundExceptionSubscriber
     */
    public function testNotFoundExceptionSubscriber(): void
    {
        $client = self::createClient();
        $client->request('GET', '/404');
        self::assertResponseStatusCodeSame(Response::HTTP_NOT_FOUND);
        self::assertSelectorTextContains('body', '404 found');
    }

    /**
     * @see ErrorHandler::handleException
     */
    public function test404(): void
    {
        $client = self::createClient();
        $client->request('GET', '/not-found');
        self::assertResponseStatusCodeSame(Response::HTTP_NOT_FOUND);
    }
}

Let's run it:

./vendor/bin/phpunit --filter=AppControllerTest

PHPUnit 9.5.10 by Sebastian Bergmann and contributors.

Testing
...                                                                 3 / 3 (100%)

Time: 00:00.354, Memory: 62.50 MB

OK (3 tests, 5 assertions)

Once again, the kernel needs to be booted. These tests are very straightforward. We check that the root URL is redirected to the correct localized path depending on your prefered language. After, we check status codes. And finally, we test the 404 page. Why this last one? Isn't it something handled by Symfony? Yes, but one can customize the 404 page, and if there is an error in your Twig template, it will return an error 500. So, let's be sure. assertUrlIsRedirectedTo and assertResponseIsOk are not native assertions but shortcuts to simplify the tests.

External tests

I have regrouped some tests into a particular category, "external". These tests are also functional, but they do network access; for example, one test checks that the https://www.strangebuzz.com/index.php URL is not accessible and redirects. The specificity of these tests is that they need to be run online, or they won't work.

PantherTestCase > end-to-end tests

We can take the definition of the maker bundle:

“End-to-end tests use a real-browser or HTTP client and a real webserver where we can test JavaScript. ”

This time, an actual browser like Firefox is used and run. You see it open itself and call the URL to test. I've already made an entire blog post on the subject. You can find it here. These tests are the slower ones as the browser need to be started. This is the only type of test case that allows testing JavaScript thoroughly. Here is an example:

<?php

declare(strict_types=1);

namespace App\Tests\E2E;

use Symfony\Component\Panther\PantherTestCase;

final class BlogPost138Test extends PantherTestCase
{
    private const BUTTON_SELECTOR = '#subscribe_button_panther';
    private const ERROR_MESSAGE_SELECTOR = '#error_msg_panther';
    private const FORM_SELECTOR = '#account_create';

    /**
     * @debug make test filter=BlogPost138Test
     */
    public function testPost138(): void
    {
        $client = self::createPantherClient([
            'browser' => PantherTestCase::FIREFOX,
        ]);
        $crawler = $client->request('GET', '/en/blog/end-to-end-testing-with-symfony-and-panther');
        self::assertSelectorTextContains('h1', 'End-to-end testing with Symfony and Panther');

        // At first load, the error message is shown and the button isn't there
        self::assertSelectorExists(self::ERROR_MESSAGE_SELECTOR);
        self::assertSelectorNotExists(self::BUTTON_SELECTOR);

        // Fill the form so the subscribe button appears
        $crawler->filter(self::FORM_SELECTOR)->form([
            'account_create[login]' => 'Les',
            'account_create[password]' => 'Tilleuls',
        ]);
        $client->waitForVisibility(self::BUTTON_SELECTOR); // wait for the button to appear!

        // Ok, now the button is visble and the error message should be removed from the DOM!
        self::assertSelectorNotExists(self::ERROR_MESSAGE_SELECTOR);
        self::assertSelectorExists(self::BUTTON_SELECTOR);
    }
}

Let's run it:

./vendor/bin/phpunit --filter=BlogPost138Test

[07:32:37] coil@mac-mini.home:/Users/coil/Sites/strangebuzz.com$ ./vendor/bin/phpunit --filter=BlogPost138Test
PHPUnit 9.5.10 by Sebastian Bergmann and contributors.

Testing
.                                                                   1 / 1 (100%)

Time: 00:09.270, Memory: 107.00 MB

OK (1 test, 5 assertions)

As you can see, these tests are indeed very slow: this is almost ten seconds for a single scenario. Of course, the following tests are faster once the browser is open.

That's it; we saw the different tests types. Now, let's see how to run all this. We will use tests suites.

Test suites

Here is what the tests directory looks like now:

ll tests/

0 drwxr-xr-x   5 coil  staff   160B  4 déc 15:44 Api
0 drwxr-xr-x   4 coil  staff   128B 15 déc 08:25 E2E
0 drwxr-xr-x   9 coil  staff   288B  5 déc 07:44 External
0 drwxr-xr-x   4 coil  staff   128B  4 déc 17:44 Functional
0 drwxr-xr-x   7 coil  staff   224B  4 déc 17:43 Integration
0 drwxr-xr-x   6 coil  staff   192B  4 déc 17:35 Unit
8 -rw-r--r--   1 coil  staff   3,6K 10 mai  2021 WebTestCase.php

All the tests are in the folder corresponding to their test case, easy. Everything is already stored correctly. It allows us to execute each test case separately from the other. For example, if you work on unit tests, want only to run all the tests of this type.

Creating test suites

First, we are going to create test suites; in the phpunit.xml file, we can apply the following configuration for the testsuites node:

        <testsuite name="unit">
            <directory>tests/Unit</directory>
        </testsuite>

        <testsuite name="integration">
            <directory>tests/Integration</directory>
        </testsuite>

        <testsuite name="api">
            <directory>tests/Api</directory>
        </testsuite>

        <testsuite name="functional">
            <directory>tests/Functional</directory>
        </testsuite>

        <testsuite name="main">
            <directory>tests/</directory>
            <exclude>tests/External</exclude>
            <exclude>tests/E2E</exclude>
        </testsuite>

        <testsuite name="external">
            <directory>tests/External/</directory>
        </testsuite>

        <testsuite name="e2e">
            <directory>tests/E2E</directory>
        </testsuite>
    </testsuites>
    <listeners>
        <listener class="Symfony\Bridge\PhpUnit\SymfonyTestsListener"/>
    </listeners>
    <extensions>
        <extension class="Symfony\Component\Panther\ServerExtension"/>

We have to specify the folder we want to use. Now, to execute the unit tests, we can run:

./vendor/bin/phpunit --testsuite=unit

This would also work:

./vendor/bin/phpunit tests/Unit

The "all" test suite isn't mandatory because we can run all tests by calling PHPUnit without arguments, but I use it in my makefile, where I have a target that helps me filter the test I want to run. You can find my whole makefile here. There is also a suite "main" that excludes external and e2e tests, the slower ones.

## —— Tests ✅ —————————————————————————————————————————————————————————————————
test: phpunit.xml check ## Run tests with optionnal suite and filter
	@$(eval testsuite ?= 'all')
	@$(eval filter ?= '.')
	@$(PHPUNIT) --testsuite=$(testsuite) --filter=$(filter) --stop-on-failure

I use this target most of the time to run a specific file or scenario.

Summary

We can summarize the features of the tests with the following matrix:

Test is fast: ⚡⚡ = very fast, ⚡ = fast,🚶 = OK, 🐌 = slow

Now that our tests are well organized, we can also run them in parallel. That's not the subject of this blog post but click on the "more on the web" button (at the end of this article) to read the "Improve Symfony Tests Performance" article by Maks Rafalko; you will find many good tips to optimize your test suite. Spoiler alert: Depending on your application, you can speed your test suite up to a 10x ratio!

One important thing to notice is that within a given folder, let's say Functional; the sub-directories takes the exact structure we have in src/. For example, the AppContolerTest file is located here tests/Functional/Controller/AppControllerTest.php, and it tests src/Controller/AppController.php. This way, it's easy to check what is tested or not (of course it doesn't replace a full code coverage report).

We have the following final directory structure:

tests/
├── Api
│   ├── Controller
│   │   └── ApiControllerTest.php
│   ├── Entity
│   │   ├── ArticleTest.php
│   │   └── BookTest.php
│   └── Security
│       └── JsonLoginTest.php
├── E2E
│   ├── BlogPost138Test.php
│   └── BlogPost138ZenstruckTest.php
├── External
│   ├── Controller
│   │   ├── FrontControllerTest.php
│   │   ├── Post
│   │   │   └── Post26TraitTest.php
│   │   ├── Snippet
│   │   │   ├── Snippet99Test.php
│   │   │   └── SnippetsControllerTest.php
│   │   └── ToolsControllerTest.php
│   ├── FeedburnerTest.php
│   └── SslTest.php
├── Functional
│   ├── Controller
│   │   ├── Admin
│   │   │   └── DashboardControllerTest.php
│   │   ├── AppControllerTest.php
│   │   ├── BlogControllerTest.php
│   │   ├── GameControllerTest.php
│   │   ├── LegacyControllerTest.php
│   │   ├── Post
│   │   │   ├── Post13TraitTest.php
│   │   │   └── Post59TraitTest.php
│   │   ├── RedirectControllerTest.php
│   │   ├── SearchControllerTest.php
│   │   ├── SearchPart1ControllerTest.php
│   │   ├── SearchPart2ControllerTest.php
│   │   ├── SitemapControllerTest.php
│   │   ├── SnippetsControllerTest.php
│   │   ├── StaticControllerTest.php
│   │   ├── SuggestControllerTest.php
│   │   ├── TagControllerTest.php
│   │   └── ToolsControllerTest.php
│   └── Entity
│       └── ArticleTypeTest.php
├── Integration
│   ├── Command
│   │   └── ShowVersionCommandTest.php
│   ├── Controller
│   │   └── Snippets
│   │       ├── Snippet100Test.php
│   │       ├── Snippet105Test.php
│   │       ├── Snippet107Test.php
│   │       ├── Snippet108Test.php
│   │       ├── Snippet114Test.php
│   │       ├── Snippet115Test.php
│   │       ├── Snippet116Test.php
│   │       ├── Snippet12Test.php
│   │       ├── Snippet131Test.php
│   │       ├── Snippet132Test.php
│   │       ├── Snippet142Test.php
│   │       ├── Snippet147Test.php
│   │       ├── Snippet14Test.php
│   │       ├── Snippet157Test.php
│   │       ├── Snippet158Test.php
│   │       ├── Snippet160Test.php
│   │       ├── Snippet168Test.php
│   │       ├── Snippet173Test.php
│   │       ├── Snippet176Test.php
│   │       ├── Snippet177Test.php
│   │       ├── Snippet20Test.php
│   │       ├── Snippet2Test.php
│   │       ├── Snippet30Test.php
│   │       ├── Snippet32Test.php
│   │       ├── Snippet33Test.php
│   │       ├── Snippet42Test.php
│   │       ├── Snippet49Test.php
│   │       ├── Snippet50Test.php
│   │       ├── Snippet58Test.php
│   │       ├── Snippet61Test.php
│   │       ├── Snippet6Test.php
│   │       ├── Snippet70Test.php
│   │       ├── Snippet71Test.php
│   │       ├── Snippet74Test.php
│   │       ├── Snippet76Test.php
│   │       ├── Snippet7Test.php
│   │       └── Snippet8Test.php
│   ├── Repository
│   │   └── BaseRepositoryTraitTest.php
│   ├── Twig
│   │   ├── Extension
│   │   │   ├── EnvExtensionTest.php
│   │   │   ├── SeoExtensionTest.php
│   │   │   ├── TypeExtensionTest.php
│   │   │   └── UrlExtensionTest.php
│   │   └── Snippet152Test.php
├── Unit
│   ├── Helper
│   │   └── StringHelperTest.php
│   ├── Log
│   │   └── Processor
│   │       └── EnvProcessorTest.php
│   ├── Tools
│   │   └── FilesystemTest.php
│   └── Utility
│       └── SpamCheckerTest.php
└── WebTestCase.php

28 directories, 81 files

Conclusion

Tests are essentials. You can always write dirty code and take shortcuts if you have a good test suite because it allows you to improve and refactor your code with confidence without fearing breaking everything. So, your test suite is probably the most critical part of your application. Make it shine, optimize it, spoil it , make it stable (and boring). That's the guarantee of an application growing healthily.

That's it! I hope you like it. Check out the links below to have additional information related to the post. As always, feedback, likes and retweets are welcome. (see the box below) See you! COil. 😊

  Read the doc  More on the web

  Work with me!

---