No UI tests for plugin builds when generating travis.yml

[tsteur]

I generated a travis-yml for CustomDimensions and it does not run UI tests: https://github.com/piwik/plugin-CustomDimensions/blob/master/.travis.yml

see https://travis-ci.org/piwik/plugin-CustomDimensions

Am I doing something wrong or is it a bug? I tried to generate it again but it didn't show it. I saw there is a parameter --force-ui-test but I have no clue what it does.

[diosmosis]

The file generation doesn't change the environment variables or matrix (in case users want to define different builds/vars) if a file already exists. You have to add them manually, or you can move the file, generate and then copy over the secure variables. Or you can set environment variables in the repo's settings, delete the file & regenerate.

[tsteur]

So one actually has to set --force-ui-test?

I didn't understand the meaning of that option even when reading the help tests. Can we maybe rename it to something like --add-ui-test-job or so? Or even better apply this one by default? I reckon it is recommended / good practice for us to always create UI tests. If there are no UI tests the build would ideally just succeed anyway.

[diosmosis]

So one actually has to set --force-ui-test?

That's just for the web hook (since an empty plugin will have no UI tests), it won't have any effect in this case.

In this case you have to manually edit the .travis.yml file, because the generation mechanism won't modify the matrix/global envs of existing .travis.yml files.

[tsteur]

If we always applied --force-ui-test the only disadvantage would be to have an additional job? That would be okay for me if it makes this easier.

When generating a UI test, we should check for existence of .travis.yml and if it exists, change .travis.yml matrix automatically (add job for UI tests). And when generating a .travis.yml we could check whether a UI test already exists for this plugin and if so apply it automatically.

[diosmosis]

If we always applied --force-ui-test the only disadvantage would be to have an additional job?

The option was meant for a different use case. If you want to always generate a job for UI tests, you can remove the check for UI tests in the code.

When generating a UI test, we should check for existence of .travis.yml and if it exists, change .travis.yml matrix automatically (add job for UI tests). And when generating a .travis.yml we could check whether a UI test already exists for this plugin and if so apply it automatically.

You're welcome to add the change. I gave up originally when trying to generate human readable .travis.yml files that preserved comments & original formatting.

[tsteur]

If you want to always generate a job for UI tests, you can remove the check for UI tests in the code.

I did not know there is already a check whether UI tests exist in a plugin. So what we basically need to do is to add the following line to the matrix when generating a UI test (if the line does not exist yet):

    - TEST_SUITE=UITests MYSQL_ADAPTER=PDO_MYSQL TEST_AGAINST_PIWIK_BRANCH=$PIWIK_TEST_TARGET

For this I will create an issue in piwik/piwik and link to this one.

[tsteur]

Now I also understand why the option is named force-ui-tests because it tries to detect it automatically :)