Adding groovy discovery plugin and npm-groovy-lint tool plugins. (ssc…

…pac#370)

Signed-off-by: Alexander Xydes <[email protected]>

.github/workflows/test.yml

@@ -96,6 +96,7 @@ jobs:
           sudo npm install -g n
           sudo n stable
           sudo npm install -g [email protected]
+          sudo npm install -g npm-groovy-lint
           rm nodesource_setup.sh
 
       - name: Test with mypy

CHANGELOG.md

@@ -8,6 +8,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ### Added
 
+- Groovy discovery plugin and tests.
+- [NPM Groovy Lint](https://nvuillam.github.io/npm-groovy-lint/) tool plugin and tests.
+
 ### Fixed
 
 ### Removed

README.md

@@ -352,6 +352,7 @@ File Type | Extensions
 catkin    | `CMakeLists.txt` and `package.xml`
 C         | `.c`, `.cc`, `.cpp`, `.cxx`, `.h`, `.hxx`, `.hpp`
 CMake     | `CMakeLists.txt`
+groovy    | `.groovy`, `.gradle`, `Jenkinsfile*`
 java      | `.class`, `.java`
 Maven     | `pom.xml`
 Perl      | `.pl`
@@ -367,30 +368,31 @@ The `.launch` extension is mapped to XML files due to use with ROS launch files.
 
 Tool | About
 :--- | :----
-[bandit][bandit]             | Bandit is a tool designed to find common security issues in Python code.
-[black][black]               | The uncompromising Python code formatter
-[catkin_lint][catkin_lint]   | Check catkin packages for common errors
-[cccc][cccc]                 | Source code counter and metrics tool for C++, C, and Java
-[clang-format][clang-format] | Format C/C++/Java/JavaScript/Objective-C/Protobuf/C# code.
-[clang-tidy][clang-tidy]     | Provide an extensible framework for diagnosing and fixing typical programming errors.
-[cmakelint][cmakelint]       | The cmake-lint program will check your listfiles for style violations, common mistakes, and anti-patterns.
-[cppcheck][cppcheck]         | static analysis of C/C++ code
-[cpplint][cpplint]           | Static code checker for C++
-[docformatter][docformatter] | Formats docstrings to follow PEP 257
-[flawfinder][flawfinder]     | Examines C/C++ source code and reports possible security weaknesses ("flaws") sorted by risk level.
-[lizard][lizard]             | A simple code complexity analyser without caring about the C/C++ header files or Java imports, supports most of the popular languages.
-[make][make]                 | C++ compiler.
-[mypy][mypy]                 | Optional static typing for Python 3 and 2 (PEP 484)
-[perlcritic][perlcritic]     | Critique Perl source code for best-practices.
-[pycodestyle][pycodestyle]   | Python style guide checker
-[pydocstyle][pydocstyle]     | A static analysis tool for checking compliance with Python docstring conventions.
-[pyflakes][pyflakes]         | A simple program which checks Python source files for errors
-[pylint][pylint]             | It's not just a linter that annoys you!
-[shellcheck][shellcheck]     | A static analysis tool for shell scripts
-[spotbugs][spotbugs]         | A tool for static analysis to look for bugs in Java code.
-[uncrustify][uncrustify]     | Code beautifier
-[xmllint][xmllint]           | Lint XML files.
-[yamllint][yamllint]         | A linter for YAML files.
+[bandit][bandit]                   | Bandit is a tool designed to find common security issues in Python code.
+[black][black]                     | The uncompromising Python code formatter
+[catkin_lint][catkin_lint]         | Check catkin packages for common errors
+[cccc][cccc]                       | Source code counter and metrics tool for C++, C, and Java
+[clang-format][clang-format]       | Format C/C++/Java/JavaScript/Objective-C/Protobuf/C# code.
+[clang-tidy][clang-tidy]           | Provide an extensible framework for diagnosing and fixing typical programming errors.
+[cmakelint][cmakelint]             | The cmake-lint program will check your listfiles for style violations, common mistakes, and anti-patterns.
+[cppcheck][cppcheck]               | static analysis of C/C++ code
+[cpplint][cpplint]                 | Static code checker for C++
+[docformatter][docformatter]       | Formats docstrings to follow PEP 257
+[flawfinder][flawfinder]           | Examines C/C++ source code and reports possible security weaknesses ("flaws") sorted by risk level.
+[npm-groovy-lint][npm-groovy-lint] | This package will track groovy errors and correct a part of them.
+[lizard][lizard]                   | A simple code complexity analyser without caring about the C/C++ header files or Java imports, supports most of the popular languages.
+[make][make]                       | C++ compiler.
+[mypy][mypy]                       | Optional static typing for Python 3 and 2 (PEP 484)
+[perlcritic][perlcritic]           | Critique Perl source code for best-practices.
+[pycodestyle][pycodestyle]         | Python style guide checker
+[pydocstyle][pydocstyle]           | A static analysis tool for checking compliance with Python docstring conventions.
+[pyflakes][pyflakes]               | A simple program which checks Python source files for errors
+[pylint][pylint]                   | It's not just a linter that annoys you!
+[shellcheck][shellcheck]           | A static analysis tool for shell scripts
+[spotbugs][spotbugs]               | A tool for static analysis to look for bugs in Java code.
+[uncrustify][uncrustify]           | Code beautifier
+[xmllint][xmllint]                 | Lint XML files.
+[yamllint][yamllint]               | A linter for YAML files.
 
 ## Customization
 
@@ -730,6 +732,7 @@ His commits were scrubbed from git history upon the initial public release.
 [cpplint]: https://github.com/cpplint/cpplint
 [docformatter]: https://github.com/myint/docformatter
 [flawfinder]: https://dwheeler.com/flawfinder/
+[npm-groovy-lint]: https://nvuillam.github.io/npm-groovy-lint/
 [lizard]: https://github.com/terryyin/lizard
 [logging]: https://docs.python.org/3/howto/logging.html
 [make]: https://gcc.gnu.org/onlinedocs/libstdc++/index.html

clean.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
 
-rm -rf build/ dist/ htmlcov/ output-py* .pytest_cache statick.egg-info/ statick_output/* .tox/
+rm -rf build/ dist/ htmlcov/ output-py* .pytest_cache statick.egg-info/ statick_output/* .tox/ ./*.log
 find . -type d -name .mypy_cache -exec rm -rf {} \;
 find . -type d -name __pycache__ -exec rm -rf {} \;

docs/source/statick_tool.plugins.discovery.rst

@@ -30,6 +30,15 @@ statick_tool.plugins.discovery.cmake_discovery_plugin module
     :show-inheritance:
 
 
+statick_tool.plugins.discovery.groovy_discovery_plugin module
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: statick_tool.plugins.discovery.groovy_discovery_plugin
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
 statick_tool.plugins.discovery.java_discovery_plugin module
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/source/statick_tool.plugins.tool.rst

@@ -100,6 +100,14 @@ statick_tool.plugins.tool.flawfinder_tool_plugin module
     :undoc-members:
     :show-inheritance:
 
+statick_tool.plugins.tool.groovylint_tool_plugin module
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. automodule:: statick_tool.plugins.tool.groovylint_tool_plugin
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
 statick_tool.plugins.tool.lizard_tool_plugin module
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

statick_tool/plugins/discovery/groovy_discovery_plugin.py

@@ -0,0 +1,47 @@
+"""Discover Groovy files to analyze."""
+
+import logging
+from collections import OrderedDict
+from typing import List, Optional
+
+from statick_tool.discovery_plugin import DiscoveryPlugin
+from statick_tool.exceptions import Exceptions
+from statick_tool.package import Package
+
+
+class GroovyDiscoveryPlugin(DiscoveryPlugin):
+    """Discover Groovy files to analyze."""
+
+    def get_name(self) -> str:
+        """Get name of discovery type."""
+        return "groovy"
+
+    def scan(
+        self, package: Package, level: str, exceptions: Optional[Exceptions] = None
+    ) -> None:
+        """Scan package looking for Groovy files."""
+        src_files: List[str] = []
+
+        self.find_files(package)
+
+        for file_dict in package.files.values():
+            if (
+                file_dict["name"].endswith(".groovy")
+                or file_dict["name"].endswith(".gradle")
+                or file_dict["name"].startswith("jenkinsfile")
+            ):
+                src_files.append(file_dict["path"])
+
+        src_files = list(OrderedDict.fromkeys(src_files))
+
+        logging.info("  %d Groovy source files found.", len(src_files))
+        if exceptions:
+            original_file_count = len(src_files)
+            src_files = exceptions.filter_file_exceptions_early(package, src_files)
+            if original_file_count > len(src_files):
+                logging.info(
+                    "  After filtering, %d Groovy files will be scanned.",
+                    len(src_files),
+                )
+
+        package["groovy_src"] = src_files

statick_tool/plugins/discovery/groovy_discovery_plugin.yapsy-plugin

@@ -0,0 +1,3 @@
+[Core]
+Name = Groovy Discovery Plugin
+Module = groovy_discovery_plugin

statick_tool/plugins/tool/groovylint_tool_plugin.py

@@ -0,0 +1,123 @@
+"""Apply GroovyLint tool and gather results."""
+
+import json
+import logging
+import subprocess
+from typing import List, Optional
+
+from statick_tool.issue import Issue
+from statick_tool.package import Package
+from statick_tool.tool_plugin import ToolPlugin
+
+
+class GroovyLintToolPlugin(ToolPlugin):
+    """Apply GroovyLint tool and gather results."""
+
+    def get_name(self) -> str:
+        """Get name of tool."""
+        return "groovylint"
+
+    # pylint: disable=too-many-locals
+    def scan(self, package: Package, level: str) -> Optional[List[Issue]]:
+        """Run tool and gather output."""
+        tool_bin = "npm-groovy-lint"
+
+        tool_config = ".groovylintrc.json"
+        if self.plugin_context:
+            user_config = self.plugin_context.config.get_tool_config(
+                self.get_name(), level, "config"
+            )
+        if user_config is not None:
+            tool_config = user_config
+        if self.plugin_context:
+            format_file_name = self.plugin_context.resources.get_file(tool_config)
+
+        flags: List[str] = []
+        if format_file_name is not None:
+            flags += ["--config", format_file_name]
+        flags += ["--output", "json"]
+        user_flags = self.get_user_flags(level)
+        flags += user_flags
+
+        files: List[str] = []
+        if "groovy_src" in package:
+            files += package["groovy_src"]
+
+        total_output: List[str] = []
+
+        for src in files:
+            try:
+                exe = [tool_bin] + flags + ["-f", src]
+                output = subprocess.check_output(
+                    exe,
+                    stderr=subprocess.STDOUT,
+                    universal_newlines=True,
+                    cwd=package.path,
+                )
+                total_output.append(output)
+
+            except subprocess.CalledProcessError as ex:
+                # npm-groovy-lint returns 1 on some errors but still has valid output
+                if ex.returncode == 1:
+                    total_output.append(ex.output)
+                else:
+                    logging.warning(
+                        "%s failed! Returncode = %d", tool_bin, ex.returncode
+                    )
+                    logging.warning("%s exception: %s", self.get_name(), ex.output)
+                    return None
+
+            except OSError as ex:
+                logging.warning("Couldn't find %s! (%s)", tool_bin, ex)
+                return None
+
+        with open(self.get_name() + ".log", "w") as fid:
+            for output in total_output:
+                fid.write(output)
+                logging.debug("%s", output)
+
+        issues: List[Issue] = self.parse_output(total_output)
+        return issues
+
+    # pylint: enable=too-many-locals
+
+    def parse_output(self, total_output: List[str]) -> List[Issue]:
+        """Parse tool output and report issues."""
+        issues: List[Issue] = []
+
+        # pylint: disable=too-many-nested-blocks
+        for output in total_output:
+            lines = output.split("\n")
+            for line in lines:
+                try:
+                    err_dict = json.loads(line)
+                    if "files" in err_dict:
+                        all_files = err_dict["files"]
+                        for file_name in all_files:
+                            file_errs = all_files[file_name]
+                            if "errors" in file_errs:
+                                for issue in file_errs["errors"]:
+                                    severity_str = issue["severity"]
+                                    severity = "3"
+                                    if severity_str == "info":
+                                        severity = "1"
+                                    elif severity_str == "warning":
+                                        severity = "3"
+                                    elif severity_str == "error":
+                                        severity = "5"
+                                    issues.append(
+                                        Issue(
+                                            file_name,
+                                            str(issue["line"]),
+                                            self.get_name(),
+                                            issue["rule"],
+                                            severity,
+                                            issue["msg"],
+                                            None,
+                                        )
+                                    )
+
+                except ValueError as ex:
+                    logging.warning("ValueError: %s", ex)
+        # pylint: enable=too-many-nested-blocks
+        return issues

statick_tool/plugins/tool/groovylint_tool_plugin.yapsy-plugin

@@ -0,0 +1,3 @@
+[Core]
+Name = NPM Groovy Lint Tool Plugin
+Module = groovylint_tool_plugin

statick_tool/rsc/.groovylintrc.json

@@ -0,0 +1,5 @@
+{
+    "extends": "recommended",
+    "rules": {
+    }
+}

tests/plugins/discovery/groovy_discovery_plugin/exceptions.yaml

@@ -0,0 +1,5 @@
+global:
+  exceptions:
+    file:
+      - tools: all
+        globs: ['*/ignore_this/*']