Skip to content

Latest commit

 

History

History
208 lines (138 loc) · 6.59 KB

README.md

File metadata and controls

208 lines (138 loc) · 6.59 KB
Raw

Content of the subdirectory vscode-visual-tests

  • Visual (UI) tests for the Analytics extension for the Visual Studio Code

TL;DR;

There are two approaches how to test Analytics plugin in VS Code via UI:

Approach #1

  • use Selenium + Web Driver because VS Code is based on Electron

Pros

  • QA people know this approach a bit

Cons

  • VS Code GUI is autogenerated, there are no IDs for tags etc.
  • it really does not check what users see on their screen

Approach #2

  • use PyAutoGUI library to match regions on screen, click on them etc.

Pros

  • straightforward
  • tests what users really see and can click on

Cons

  • flaky, depends on actual rendering engine settings

Solution

-> Use an interface and add a separated layer between #1/#2 and actual tests

Solution checked so far

  1. Tests are written in Behave (BDD)
  2. Implementation of test steps use PyAutoGUI
  3. Available at https://github.com/fabric8-analytics/fabric8-analytics-common/tree/master/vscode-visual-tests

Examples

Test that installs Analytics plugin using high-level test steps (click on the named widget etc.):

Feature: Visual Studio Code + the Analytics plugin basic functionality

    Scenario: Check that the Analytics plugin can be found in the Marketplace
        Given The PyAutoGUI library is initialized
          And The screen resolution is at least 1024x768 pixels

         When I start the Visual Studio Code
         Then I should find Visual Studio Code instance

         When I click on the extension icon on the activity bar
         Then I should find the Search extension in Marketplace input box

         When I search for Dependency Analytics plugin
          And I select the plugin
         Then I should find the OpenShift logo
          And I should find the Dependency Analytics header
          And I should find the Dependency Analytics title
          And I should find the Plugin install button

         When I start the installation by clicking on the Plugin install button
         Then I should find the Reload and Uninstall buttons
          And I should find the Reload button and Gear icon

         When I close the Visual Studio Code
         Then I should not find any Visual Studio Code instance

Test that installs Analytics plugin using low-level test steps (find region, click on it):

Feature: Visual Studio Code + the Analytics plugin basic functionality

    Scenario: Check that the Analytics plugin can be found in the Marketplace
        Given The PyAutoGUI library is initialized
          And The screen resolution is at least 1024x768 pixels

         # Start the Visual Studio Code
         When I start the Visual Studio Code
         Then I should find Visual Studio Code instance
         When I look at the whole screen

         # Find the Analytics extension
         Then I should find the activity bar
          And I should find the extension icon on the activity bar
         When I click on that region
          And I wait 2 seconds
          And I look at the whole screen
         Then I should find the Search extension in Marketplace input box
         When I click on that region
          And I type in Depencency Analytics
          And I wait 2 seconds
          And I look at the whole screen
         Then I should find the Dependency Analytics info region

         # Install the Analytics extension
         When I click on that region
          And I wait 2 seconds
          And I look at the whole screen
         Then I should find the Plugin install button
         When I click on that region
          And I move mouse cursor to the top left corner
          And I wait 2 seconds
         Then I should find the Reload and Uninstall buttons
          And I should find the Reload button and Gear icon

         # Close the Visual Studio Code
         When I look at the whole screen
          And I click on the File menu
          And I click on the Exit menu entry
          And I wait 2 seconds
         Then I should not find any Visual Studio Code instance

Installation and setup

Litava tool

It is possible to use the Litava project [https://github.com/tisnik/litava] to perform fuzzy pattern matching in raster images. Using this approach, it is possible to find regions in screenshot even when the GUI is distorted a bit (can be caused by various reasons, for example by using the different font rendering engine).

When the Litava project is enabled (see Setup), the pattern on the image is found by using the following algorithm:

  1. the internal algorithm implemented in PyAutoGUI is used to find the exact pattern
  2. if the pattern is not found, then the Litava tool is called to find the region using the fuzzy search

Internal algorithm used by the Litava tool

  1. Inputs: a source image (usually screenshot), a pattern (image to be found in the source image)
  2. Source image pixel values are converted to floats & shifted by computed offset, so the sum of all pixels is zero
  3. Pattern image pixel values are converted to floats & shifted by computed offset, so the sum of all pixels is zero
  4. Autocorrelation is computed for the pattern
  5. Now the pattern is moved over the image in all possible combinations and correlation coefficient is computed for all those combinations
  6. Maximum value of correlation + pattern position are remembered
  7. If maximum correlation == autocorrelation, the exact pattern has been found
  8. If maximum correlation / autocorrelation >= 0.9 (90%) the tool report it as 'fuzzy found'
  9. Otherwise the results is that pattern can't be found

References: https://patents.justia.com/patent/9298598

Examples

Example 1

Source image Pattern Pattern found in the image

Example 2

Source image Pattern Pattern found in the image

Prerequisities

  • GNU C Compiler installed
  • Stadard dev libraries installed (libc or libc6)

Installation

Just use the script:

./install_litava.sh

Setup

Set and export the environment variable USE_LITAVA

Common issues

Error message: Xlib.error.DisplayConnectionError: Can't connect to display ":0.0": b'No protocol specified\n'

You can use the xhost tool to control user access to X.

First run xhost + which should disable ALL checks so the test can be started. This solution is no secure, of course, so use it just for checking.

Better is to use:

xhost -
xhost +SI:localuser:$(you_username)

Error message: ImportError: No module named Xlib.display

sudo dnf install python3-xlib