add support for behave BDD test framework #278

oberhofer · 2022-02-19T19:59:45Z

I want to add support for the behave BDD test framework. I like the clean structure of your project and I hope I got everything covered to support behave (https://github.com/behave/behave).

kondratyev-nv · 2022-02-21T20:40:14Z

Hi! Thanks for the PR! It seems some lint errors are still there. If you want to reproduce this locally, you can run npm run lint. I'll be reviewing your changes this week.

oberhofer · 2022-02-22T11:27:55Z

I will check the lint errors.
In addition I'm not satisfied with my (non existent) behave test suite enumeration. Currently it only works for one test suite in the top level directory. Maybe you could tell me more about how this is supposed to work.
Behave test suites have a dedicated "features" directory containing *.feature files. So in theory I have to scan a project for all directories, that contain a "features" directory and call behave there to get a json file with the test results for this suite.
If you call behave in a directory without a "features" subdir you get a "ConfigError".
Do I have to implement this scheme in load/run or is there another entity which is meant to scan directories ?

- wrap json parsing to catch failures

kondratyev-nv · 2022-02-22T23:13:17Z

I'm not familiar with behave, but I see 2 options there.

You can try to replicate behave CLI behavior. As behave does not search for 'feature' files by itself, let the user specify what folders to include (with 'python.testing.behaveArgs' setting). This way it'd be straightforward on how to configure the extension if you previously called behave with some arguments like 'behave folder1 folder2', IMO it'd be rather obvious to set 'python.testing.behaveArgs' to [" folder1", "folder2"].
Another option is to actually scan folders for 'feature' files. You can scan it and save it in the private property of the adapter code, see something similar here - https://github.com/kondratyev-nv/vscode-python-test-adapter/blob/master/src/pythonTestAdapter.ts#L76. Then pass this list to the testRunner somehow (idk, maybe another property in config object?) and then construct a list of behave arguments based on that.

IMO it's good to start with option 1 as it's simple. If it'll work and you'd feel like changing it afterward, you can upgrade the code following option 2. Let me know if this makes sense and what are your thoughts on this.

kondratyev-nv · 2022-02-23T15:19:06Z

btw, can you please merge the latest master brach to your PR, it should fix most of the CI and test errors.

- wrap json parsing to catch failures

…est-adapter

oberhofer · 2022-02-23T22:12:55Z

IMO it's good to start with option 1 as it's simple. If it'll work and you'd feel like changing it afterward, you can upgrade the code following option 2. Let me know if this makes sense and what are your thoughts on this.

Sounds good. I will go with option 1 as behave supports positional arguments where you can specify either feature directories, single feature files or specific features with file:linenumber.

…aveArgs)

oberhofer · 2022-02-25T19:36:31Z

Ok, seems to work at least on my side, and I can specify multiple locations for my tests. Thanks for your support.

kondratyev-nv

Some nits and comments.

It would be great if you can add some tests. Let me know if you need any help with that.

kondratyev-nv · 2022-03-08T11:49:55Z

src/behave/behaveTestJsonParser.ts

+    try {
+        return JSON.parse(text);
+    } catch (err) {
+        // this.logger.log('warn', 'parse json failed: ${text}');


Should this be uncommented?

I will turn this into a comment as there is no logger accessible here (at the moment).
I also look into adding some test and I may need some help with that.
Which test scenarios do you like to see covered ?
Are there any available tests that may serve as a blueprint ?

Some examples are

test\tests\pytestGeneral.test.ts

test\tests\testplanGeneral.test.ts

test\tests\unittestGeneral.test.ts

The minimal coverage I would say is to replicate the core UI functionality and user flow, for example

Discovery of tests (set testing.behaveEnabled and maybe args) and verify that all expected tests are discovered (see 'should discover tests' tests).

Running tests on multiple levels - run all, run a suite, run a single test (see 'Run pytest tests' tests).

src/behave/behaveTestJsonParser.ts

kondratyev-nv · 2022-03-08T11:56:26Z

src/behave/behaveTestRunner.ts

+// 1: Some tests have failed
+const BEHAVE_NON_ERROR_EXIT_CODES = [0, 1];
+
+const DISCOVERY_OUTPUT_PLUGIN_INFO = {


I think this is not actually used, right? Can you please try removing this?

Not sure what you mean. Behave exit codes are not documented (to my current knowledge). I have seen 0 and 1 as valid exit codes and unfortunately 0 when an error occurs (that then spits out an error string instead of json data).

I meant DISCOVERY_OUTPUT_PLUGIN_INFO constant 🙂

It is used in the loadEnvironmentVariables in the same file (line 154 and 159).

Yes, it's referenced, but it makes sense only for pytest - pytest is run with a custom plugin (in resources/python folder). Since behave does not need any custom plugins this constant and its usages are not actually useful. If you have any doubts, I can suggest writing tests first and then try removing this, I'm pretty sure tests won't break.

src/configuration/vscodeWorkspaceConfiguration.ts

kondratyev-nv · 2022-03-08T12:05:05Z

test/test_samples/behave/behave_runner.sh

@@ -0,0 +1,10 @@
+#!/bin/bash


I don't see any tests that check running behave as a script, so I assume these 2 scripts can be safely removed?

I took the pytest directory as a template, that spawned their existence. Currently they show how to call behave, which may be handy. Maybe I had to add a test to justify their presence :-)

It's used in test\tests\pytestScript.test.ts. If you're not planning to replicate similar tests, I would suggest removing these scripts. Up to you 🙂

I will add a test.

requirements.txt

kondratyev-nv · 2022-03-20T22:09:14Z

src/behave/behaveTestRunner.ts

+        };
+    }
+
+    // @ts-expect-error


Can this be removed?

Yes, should be doable. Currently the test parameter is not used, but it should be.

oberhofer added 2 commits February 19, 2022 20:42

add support for behave BDD test framework

9f5f1a8

fix lint errors

a4061d4

- fix more lint errors

5bab316

- wrap json parsing to catch failures

oberhofer added 4 commits February 23, 2022 22:53

add support for behave BDD test framework

5aa87a0

fix lint errors

07c9a6a

- fix more lint errors

881d3be

- wrap json parsing to catch failures

Merge branch 'master' of https://github.com/oberhofer/vscode-python-t…

20ab035

…est-adapter

oberhofer added 2 commits February 25, 2022 20:21

fix line endings (LF)

0b4c4f9

behaveTestRunner: fix handling of behave specific arguments (e.g. beh…

1a87a0e

…aveArgs)

oberhofer added 3 commits February 25, 2022 21:26

behaveTestRunner: fix handling of behave specific arguments in run()

22344ae

fix json parsing for feature files with no valid scenarios

c5cec0a

fix line endings (LF)

ca8c3ef

kondratyev-nv requested changes Mar 8, 2022

View reviewed changes

oberhofer added 6 commits March 20, 2022 20:09

add behave to requirements.txt

987a087

fix comment

b827ce0

use parseInt() instead of Number() to convert line number

1ead2a6

fix identation

974a3bc

remove unecessary attribute tests

dc3c76e

behave test sample: add .vscode settings

1519b1f

kondratyev-nv reviewed Mar 20, 2022

View reviewed changes

behave runner: fix use of test parameter in getRunArguments()

1b12d4d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add support for behave BDD test framework #278

add support for behave BDD test framework #278

oberhofer commented Feb 19, 2022

kondratyev-nv commented Feb 21, 2022

oberhofer commented Feb 22, 2022

kondratyev-nv commented Feb 22, 2022

kondratyev-nv commented Feb 23, 2022

oberhofer commented Feb 23, 2022

oberhofer commented Feb 25, 2022

kondratyev-nv left a comment

kondratyev-nv Mar 8, 2022

oberhofer Mar 20, 2022

kondratyev-nv Mar 20, 2022

kondratyev-nv Mar 8, 2022

oberhofer Mar 20, 2022

kondratyev-nv Mar 20, 2022

oberhofer Mar 20, 2022

kondratyev-nv Mar 20, 2022

kondratyev-nv Mar 8, 2022

oberhofer Mar 20, 2022

kondratyev-nv Mar 20, 2022

oberhofer Mar 20, 2022

kondratyev-nv Mar 20, 2022

oberhofer Mar 20, 2022

add support for behave BDD test framework #278

Are you sure you want to change the base?

add support for behave BDD test framework #278

Conversation

oberhofer commented Feb 19, 2022

kondratyev-nv commented Feb 21, 2022

oberhofer commented Feb 22, 2022

kondratyev-nv commented Feb 22, 2022

kondratyev-nv commented Feb 23, 2022

oberhofer commented Feb 23, 2022

oberhofer commented Feb 25, 2022

kondratyev-nv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment