Updating bats_test Rule For bats-assert (#11)

* Updating README to use newest tag of bazel-bats.
Moving SHA into its own var, similar to the version number.

* Updating `bazel_bats_dependencies()`'s default version value (and
corresponding SHA) to 1.7.0 (latest bats-core version).
Updating WORKSPACE to be more explicit in versions used.
Adding param checking to ensure that the SHA was passed in.
Updating README to show explicit version passing.

* Extending `bazel_bats_dependencies()` to also (optionally) allow for adding in bats extensions, bats-assert and bats-support (bats-support is required for bats-assert).
The decision to make these optional additions was made to not force extra dependencies on any downstream library that does not make use of them (dep management is good).
The `bats_test()` rule will be updated in a follow-up PR.

* Fixing missing comma.

* Updating `bats_test()` test target rule to support the bats-assert extension.

This was implemented via two separate targets, selected via the `uses_bats_assert` target argument. Using two separate rule definitions is done to allow exclusion of bats-support and/or bats-assert from any downstream library that does not want them. Bazel implicit dependencies are used with bats-assert, thus a separate rule definition must be defined for optional use. Obviously, making use of `uses_bats_assert` without supplying necessary arguments to `bazel_bats_dependencies()` would result in an intended build failure (lacking necessary dependency project).

There are other bats extensions not added here. As discussed already, a large focus with this change is to make these extensions optional. If other bats extensions are added, hopefully that optional mindset can persist. Example: [bats-detik](https://github.com/bats-core/bats-detik) is very niche, and I have no interest in including it into my projects -- but maybe others would (opt-in is fine and dandy).

The two rule definitions were written to try and reuse as much as possible with regard to common functionality.

The extended rule logic copies files from bats-support and bats-assert into the .bats source file path (in bazel runfiles path), for each source file in a given rule. This is necessary for consistency with bats-core layout as defined externally. Bats-core makes use of submodules for extensions, and expects these submodules to be siblings to any/all .bats source test files. This is further backed by the way that bats's `load` system looks in the relative path. Loading of bats-assert is done with `load "test_helper/bats-support/load"`, which means that these extensions must be siblinged to the tests source files (in bazel's runfiles path). These rules allow consistent functionality with any bats tests written externally.

This copy logic works for any number of .bats files, whether they are in the same directory or not (over multiple bazel test targets, or within a single target) -- which is actually more flexible, in terms of project structure, than is afforded with bats directly.

In addition to the rule changes, other changes in this PR:
* Updating README to cover extension support.
* Updating WORKSPACE to add in bats-assert.
* Adding in new test that validates bats-assert statements.

See also: https://github.com/bats-core/bats-assert, https://bats-core.readthedocs.io/en/stable/tutorial.html#quick-installation.

* Slight cleanup on `if`s.

* Asserting using both `assert_failure` overrides.

* More assertions!

* * More better words (improving error messages).

* * Fixing nit.

* * Fixing nit 2.

* * Fixing nit 3.

* * More nit fix.

* * Fixing nit.

README.md

@@ -62,6 +62,8 @@ bats_test(
 )
 ```
 
+If your test would like to make use of the bats-assert extension (`assert_success`, `assert_failure`, `assert_output`, etc), simply add `uses_bats_assert = True` to your `bats_test()` target. This still requires adding the appropriate `load` statements in your test file.
+
 ## Examples
 
 This repository is an example of a repo with BATS tests.  If you have `bazel`

WORKSPACE

@@ -2,11 +2,20 @@
 # https://stackoverflow.com/questions/47192668/idiomatic-retrieval-of-the-bazel-execution-path#
 workspace(name = "bazel_bats")
 
+BATS_ASSERT_VERSION = "2.0.0"
+BATS_ASSERT_SHA256 = "15dbf1abb98db785323b9327c86ee2b3114541fe5aa150c410a1632ec06d9903"
 BATS_CORE_VERSION = "1.7.0"
 BATS_CORE_SHA256 = "ac70c2a153f108b1ac549c2eaa4154dea4a7c1cc421e3352f0ce6ea49435454e"
+BATS_SUPPORT_VERSION = "0.3.0"
+BATS_SUPPORT_SHA256 = "7815237aafeb42ddcc1b8c698fc5808026d33317d8701d5ec2396e9634e2918f"
 
 load("@bazel_bats//:deps.bzl", "bazel_bats_dependencies")
 
 bazel_bats_dependencies(
     version = BATS_CORE_VERSION,
-    sha256 = BATS_CORE_SHA256)
+    sha256 = BATS_CORE_SHA256,
+    bats_assert_version = BATS_ASSERT_VERSION,
+    bats_assert_sha256 = BATS_ASSERT_SHA256,
+    bats_support_version = BATS_SUPPORT_VERSION,
+    bats_support_sha256 = BATS_SUPPORT_SHA256
+)

rules.bzl

@@ -11,6 +11,22 @@ def _test_files(bats, srcs):
         test_paths = " ".join(['"{}"'.format(s.short_path) for s in srcs]),
     )
 
+# Finds shortest path.
+# Used to find commonpath for bats-assert or bats-support.
+# This would not always necessarily return the correct path (in other contexts),
+# but works for the filegroups used.
+# A more robust method of finding the base directory is made more difficult by skylark's
+# divergence from python. Bringing in skylab as a dependency was deemed overkill.
+# So long as a file in the base dir exists (e.g. load.bash), this is fine.
+def _base_dir(files):
+    result = files[0].dirname
+    min_len = len(files[0].dirname)
+    for file in files:
+        if len(file.dirname) < min_len:
+            min_len = len(file.dirname)
+            result = file.dirname
+    return result
+
 def _bats_test_impl(ctx):
     path = ["$PWD/" + _dirname(b.short_path) for b in ctx.files.deps]
     sep = ctx.configuration.host_path_separator
@@ -30,30 +46,84 @@ def _bats_test_impl(ctx):
         output = ctx.outputs.executable,
         content = content,
     )
+
     runfiles = ctx.runfiles(
         files = ctx.files.srcs,
         transitive_files = depset(ctx.files.data + ctx.files.deps),
     ).merge(ctx.attr._bats.default_runfiles)
     return [DefaultInfo(runfiles = runfiles)]
 
-bats_test = rule(
+def _bats_with_bats_assert_test_impl(ctx):
+    base_info = _bats_test_impl(ctx)[0]
+
+    bats_assert_base_dir = _base_dir(ctx.attr._bats_assert.files.to_list())
+    bats_support_base_dir = _base_dir(ctx.attr._bats_support.files.to_list())
+    test_helper_outputs = []
+    for src_file in ctx.files.srcs:
+        test_helper_dir = ctx.actions.declare_directory(
+            "test_helper",
+            sibling = src_file,
+        )
+        test_helper_outputs.append(test_helper_dir)
+        ctx.actions.run_shell(
+            outputs=[test_helper_dir],
+            inputs=depset(ctx.attr._bats_assert.files.to_list() + ctx.attr._bats_support.files.to_list()),
+            arguments=[test_helper_dir.path, bats_assert_base_dir, bats_support_base_dir],
+            command="""
+            mkdir -p $1/bats-support $1/bats-assert \\
+                && cp -r $2/* $1/bats-assert \\
+                && cp -r $3/* $1/bats-support
+            """)
+
+    runfiles = ctx.runfiles(
+        files = test_helper_outputs,
+    ).merge(base_info.default_runfiles)
+    return [DefaultInfo(runfiles = runfiles)]
+
+_bats_test_attrs = {
+    "data": attr.label_list(allow_files = True),
+    "deps": attr.label_list(),
+    "env": attr.string_dict(
+        doc = "A list of key-value pairs of environment variables to define",
+    ),
+    "srcs": attr.label_list(
+        allow_files = [".bats"],
+        doc = "Source files to run a BATS test on",
+    ),
+    "_bats": attr.label(
+        default = Label("@bats_core//:bats"),
+        executable = True,
+        cfg = "exec",
+    ),
+}
+
+_bats_with_bats_assert_test_attrs = {
+    "_bats_support": attr.label(
+        default = Label("@bats_support//:load_files"),
+    ),
+    "_bats_assert": attr.label(
+        default = Label("@bats_assert//:load_files"),
+    ),
+}
+_bats_with_bats_assert_test_attrs.update(_bats_test_attrs)
+
+_bats_test = rule(
     _bats_test_impl,
-    attrs = {
-        "data": attr.label_list(allow_files = True),
-        "deps": attr.label_list(),
-        "env": attr.string_dict(
-            doc = "A list of key-value pairs of environment variables to define",
-        ),
-        "srcs": attr.label_list(
-            allow_files = [".bats"],
-            doc = "Source files to run a BATS test on",
-        ),
-        "_bats": attr.label(
-            default = Label("@bats_core//:bats"),
-            executable = True,
-            cfg = "exec",
-        ),
-    },
+    attrs = _bats_test_attrs,
     test = True,
-    doc = "Runs a BATS test on the supplied source files",
+    doc = "Runs a BATS test on the supplied source files.",
 )
+
+_bats_with_bats_assert_test = rule(
+    _bats_with_bats_assert_test_impl,
+    attrs = _bats_with_bats_assert_test_attrs,
+    test = True,
+    doc = "Runs a BATS test on the supplied source files, allowing for usage of bats-support and bats-assert.",
+)
+
+
+def bats_test(uses_bats_assert = False, **kwargs):
+  if not uses_bats_assert:
+    _bats_test(**kwargs)
+  else:
+    _bats_with_bats_assert_test(**kwargs)

tests/BUILD.bazel

@@ -13,6 +13,15 @@ bats_test(
     ],
 )
 
+bats_test(
+    name = "exit_with_input_bats_assert_test",
+    srcs = ["exit_with_input_bats_assert.bats"],
+    deps = [
+        ":exit_with_input_bin",
+    ],
+    uses_bats_assert = True,
+)
+
 sh_library(
     name = "helper",
     srcs = ["helper.bash"],

tests/exit_with_input

@@ -1,2 +1,7 @@
 #!/usr/bin/env bash
+
+if [ ! -z "$2" ]; then
+    echo "Given for output: $2"
+fi
+
 exit $1

tests/exit_with_input_bats_assert.bats

@@ -0,0 +1,66 @@
+#!/usr/bin/env bats
+
+PATH_TO_EXIT_WITH_INPUT="tests/exit_with_input"
+
+setup() {
+    load "test_helper/bats-support/load"
+    load "test_helper/bats-assert/load"
+}
+
+@test "can run succeeding executable" {
+  run "${PATH_TO_EXIT_WITH_INPUT}" 0
+
+  assert_success
+  refute_output
+}
+
+@test "can run succeeding executable with output" {
+  run "${PATH_TO_EXIT_WITH_INPUT}" 0 'some message'
+
+  assert_success
+  assert_output --partial 'Given for output:'
+  assert_output --partial 'some message'
+  assert_output --regexp '^Given for output: some message$'
+  assert_output 'Given for output: some message'
+  refute_output 'never printed'
+}
+
+@test "can run failing executable" {
+  run "${PATH_TO_EXIT_WITH_INPUT}" 1
+
+  assert_failure
+  assert_failure 1
+  refute_output
+}
+
+@test "can run failing executable with output" {
+  run "${PATH_TO_EXIT_WITH_INPUT}" 1 'some message'
+
+  assert_failure
+  assert_failure 1
+  assert_output --partial 'Given for output:'
+  assert_output --partial 'some message'
+  assert_output --regexp '^Given for output: some message$'
+  assert_output 'Given for output: some message'
+  refute_output 'never printed'
+}
+
+@test "can run failing executable with different return code" {
+  run "${PATH_TO_EXIT_WITH_INPUT}" 2
+
+  assert_failure
+  assert_failure 2
+  refute_output
+}
+
+@test "can run failing executable with different return code with output" {
+  run "${PATH_TO_EXIT_WITH_INPUT}" 2 'some message'
+
+  assert_failure
+  assert_failure 2
+  assert_output --partial 'Given for output:'
+  assert_output --partial 'some message'
+  assert_output --regexp '^Given for output: some message$'
+  assert_output 'Given for output: some message'
+  refute_output 'never printed'
+}