Merge pull request #50 from intelowlproject/develop

v3.0.0 rewrite (major release)

.github/CHANGELOG.md

@@ -0,0 +1,88 @@
+# Changelog
+
+## 3.0.0 (https://github.com/intelowlproject/pyintelowl/releases/tag/3.0.0)
+
+*Note: Incompatible with previous versions*
+
+This version brings a complete rewrite of the pyintelowl library as well as command line client. We very much recommend you to update to the latest version to enjoy all new features.
+
+- The new CLI is written with [pallets/click](https://github.com/pallets/click) and supports all IntelOwl API endpoints. The CLI is well-documented and will help you navigate different commands; you can use it to request new analysis, view an old analysis, view `analyzer_config.json`, view list of tags, list of jobs, etc.
+- Complete type-hinting and sphinx docs for the `pyintelowl.IntelOwl` class with helper member functions for each IntelOwl API endpoint.
+
+
+## 2.0.0 (https://github.com/intelowlproject/pyintelowl/releases/tag/2.0.0)
+**This version supports only IntelOwl versions >=1.8.0 (about to be released). To interact with previous IntelOwl versions programmatically please refer to pyintelowl version 1.3.5**
+
+* we forced [black](https://github.com/psf/black) style, added linters and precommit configuration. In this way pyintelowl is aligned to IntelOwl.
+* we have updated the authentication method from a JWT Token to a simple Token. In this way, it is easier to use pyintelowl for integrations with other products and there are no more concurrency problems on multiple simultaneous requests.
+
+If you were using pyintelowl and IntelOwl before this version, you have to:
+* update IntelOwl to version>=1.8.0
+* retrieve a new API token from the Django Admin Interface for your user: you have to go in the *Durin* section (click on `Auth tokens`) and generate a key there. This token is valid until manually deleted.
+
+
+## 1.3.5 (https://github.com/intelowlproject/pyintelowl/releases/tag/1.3.5)
+Now optional parameter "runtime_configuration" properly works
+
+Please use this version of pyintelowl with version >= 1.5.x of IntelOwl
+
+
+## 1.3.4 (https://github.com/intelowlproject/pyintelowl/releases/tag/1.3.4)
+see [1.3.3](https://github.com/intelowlproject/pyintelowl/releases/tag/1.3.3) for details
+
+
+## 1.3.3 (https://github.com/intelowlproject/pyintelowl/releases/tag/1.3.3)
+Some fixes:
+
+*   pyintelowl did not work correctly against HTTPS-enabled IntelOwl Servers
+*    fixed parameter name in send_observable_analysis_request
+
+Please use this version of pyintelowl with v1.5.x of IntelOwl
+
+
+## 1.3.2 (https://github.com/intelowlproject/pyintelowl/releases/tag/1.3.2)
+Patch Release after [1.3.0](https://github.com/intelowlproject/pyintelowl/releases/tag/1.3.0).
+
+- renamed `additional_configuration` to `runtime_configuration`.
+- Formatting with psf/black formatter.
+
+**Please use this version of pyintelowl with v1.5.x of IntelOwl.**
+
+
+## 1.3.1 (https://github.com/intelowlproject/pyintelowl/releases/tag/1.3.1)
+Fixes and improvements to "--show-colors" option
+
+
+## 1.3.0 (https://github.com/intelowlproject/pyintelowl/releases/tag/1.3.0)
+reformatted some code + added support for new parameter "additional_configuration"
+
+
+## 1.2.0 (https://github.com/intelowlproject/pyintelowl/releases/tag/1.2.0)
+PR #16 for details.
+
+
+## 1.1.0 (https://github.com/intelowlproject/pyintelowl/releases/tag/1.1.0)
+Added an option when executing pyintelowl as CLI: `-sc` will show the results in a colorful and organized way that helps the user in looking for useful information. By default, the results are still shown in the JSON format. Thanks to tsale to his idea and contribution.
+
+**Example:**
+
+```
+python3 intel_owl_client.py -i <your_intelowl_instance> -sc -a VirusTotal_v2_Get_Observable -a HybridAnalysis_Get_Observable -a OTXQuery observable -v www.google.com
+```
+
+
+## 1.0.0 (https://github.com/intelowlproject/pyintelowl/releases/tag/1.0.0)
+For all the details, check the official blog post:
+
+https://www.honeynet.org/2020/07/05/intel-owl-release-v1-0-0/
+
+This version is compatible only with the related (1.x) IntelOwl release.
+
+## 0.2.1
+
+## 0.2.0
+
+## 0.1.2
+
+## 0.1.1
+

.github/workflows/pull_request_automation.yml

@@ -18,7 +18,7 @@ jobs:
       run: |
         sudo apt-get update
         pip3 install --upgrade pip
-        pip3 install -r test-requirements.txt
+        pip install -e .[test]
 
     - name: Black formatter
       run: |

.gitignore

@@ -7,4 +7,5 @@ test_files/
 *.exe
 __pycache__
 .vscode
-api_token_value.txt
+api_token_value.txt
+docs/_build/

README.md

@@ -1,72 +1,78 @@
-# pyintelowl
+# PyIntelOwl
 
 [![PyPI version](https://badge.fury.io/py/pyintelowl.svg)](https://badge.fury.io/py/pyintelowl)
 [![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/intelowlproject/pyintelowl.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/mlodic/pyintelowl/context:python)
 [![CodeFactor](https://www.codefactor.io/repository/github/intelowlproject/pyintelowl/badge)](https://www.codefactor.io/repository/github/intelowlproject/pyintelowl)
 
-Simple Client for the [Intel Owl Project](https://github.com/intelowlproject/IntelOwl)
+Robust Python **SDK** and **Command Line Client** for interacting with [IntelOwl](https://github.com/intelowlproject/IntelOwl)'s API.
 
-2 ways to use it:
-* as a library
-* as a command line script
+## Features
 
-You can select which analyzers you want to run for every analysis you perform.
+- Easy one-time configuration with self documented help and hints along the way.
+- Request new analysis for observables and files.
+  - Select which analyzers you want to run for every analysis you perform.
+  - Choose whether you want to HTTP poll for the analysis to finish or not.
+- List all jobs or view one job in a prettified tabular form.
+- List all tags or view one tag in a prettified tabular form.
+- Tabular view of the `analyzer_config.json` from IntelOwl with RegEx matching capabilities.
 
-For additional help, we suggest to check the ["How to use pyintelowl" Youtube Video](https://www.youtube.com/watch?v=fpd6Kt9EZdI) by [Kostas](https://github.com/tsale).
+## Installation
 
+```bash
+$ pip3 install pyintelowl
+```
 
-## Generate API key
-You need a valid API key to interact with the IntelOwl server. 
-Keys should be created from the admin interface of [IntelOwl](https://github.com/intelowlproject/intelowl): you have to go in the *Durin* section (click on `Auth tokens`) and generate a key there.
-
-You can use the  with the parameter `-k <api_key>` from CLI
-
-#### (old auth method) JWT Token Authentication
-> this auth was available in IntelOwl versions <1.8.0 and pyintelowl versions <2.0.0
-
-From the admin interface of IntelOwl, you have to go in the *Outstanding tokens* section and generate a token there.
+For development/testing, `pip3 install pyintelowl[dev]`
 
-You can use it by pasting it into the file [api_token.txt](api_token.txt).
+## Quickstart
 
-## Library
-`pip3 install pyintelowl`
+### As Command Line Client
 
-`from pyintelowl.pyintelowl import IntelOwl`
+On successful installation, The `pyintelowl` entryscript should be directly invokable. For example,
 
-#### Endpoints
-`ask_analysis_availability` -> search for already available analysis
+```bash
+$ pyintelowl
+Usage: pyintelowl [OPTIONS] COMMAND [ARGS]...
 
-`send_file_analysis_request` -> send a file to be analyzed
+Options:
+  -d, --debug  Set log level to DEBUG
+  --version    Show the version and exit.
+  -h, --help   Show this message and exit.
 
-`send_observable_analysis_request` -> send an observable to be analyzed
+Commands:
+  analyse              Send new analysis request
+  config               Set or view config variables
+  get-analyzer-config  Get current state of `analyzer_config.json` from the...
+  jobs                 Manage Jobs
+  tags                 Manage tags
+```
 
-`ask_analysis_result` -> request analysis result by job ID
+### As a library / SDK
 
-`get_analyzer_configs` -> get the analyzers configuration
+```python
+from pyintelowl import IntelOwl
+obj = IntelOwl("<your_api_key>", "<your_intelowl_instance_url>", "optional<path_to_pem_file>")
+```
 
+For more comprehensive documentation, please see https://pyintelowl.readthedocs.io/.
 
-## Command line Client
+## Changelog
 
-#### Help
+View [CHANGELOG.md](https://github.com/intelowlproject/pyintelowl/blob/master/.github/CHANGELOG.md).
 
-`python3 intel_owl_client.py -h`
+## FAQ
 
-#### Analyze
-2 Submodules: `file` and `observable`
-
-##### Sample
-Example:
-
-`python3 intel_owl_client.py -k <api_token> -i <url> -a PE_Info -a File_Info file -f <path_to_file>`
+#### Generate API key
+You need a valid API key to interact with the IntelOwl server. 
+Keys should be created from the admin interface of [IntelOwl](https://github.com/intelowlproject/intelowl): you have to go in the *Durin* section (click on `Auth tokens`) and generate a key there.
 
-Run all available analyzers (some of them could fail if you did not implemented the required configuration in the IntelOwl server):
+#### Incompatibility after version 3.0
 
-`python3 intel_owl_client.py -k <api_token> -i <url> -aa file -f <path_to_file>`
+We did a complete rewrite of the PyIntelOwl client and CLI both for the version `3.0.0`. We very much recommend you to update to the latest version to enjoy all new features.
 
-##### Observable
-Example:
+#### (old auth method) JWT Token Authentication
+> this auth was available in IntelOwl versions <1.8.0 and pyintelowl versions <2.0.0
 
-`python3 intel_owl_client.py -k <api_token> -i <url> -a AbuseIPDB -a OTXQuery observable -v google.com`
+From the admin interface of IntelOwl, you have to go in the *Outstanding tokens* section and generate a token there.
 
-#### Get Analyzers Configuration
-`python3 intel_owl_client.py -k <api_token> -i <url> -gc`
+You can use it by pasting it into the file [api_token.txt](api_token.txt).

cli.py

@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+import click
+from pyintelowl import IntelOwl
+from pyintelowl.cli import groups, cmds
+from pyintelowl.cli._utils import (
+    get_logger,
+    ClickContext,
+    get_netrc_obj,
+    get_version_number,
+)
+
+
+@click.group(context_settings=dict(help_option_names=["-h", "--help"]))
+@click.option("-d", "--debug", is_flag=True, help="Set log level to DEBUG")
+@click.version_option(version=get_version_number())
+@click.pass_context
+def cli(ctx: ClickContext, debug: bool):
+    netrc, host = get_netrc_obj()
+    api_key, url, cert = host["password"], host["account"], host["login"]
+    if not api_key or not url:
+        click.echo("Hint: Use `config set` to set config variables!")
+    else:
+        logger = get_logger("DEBUG" if debug else "INFO")
+        ctx.obj = IntelOwl(api_key, url, cert, logger)
+
+
+# Compile all groups and commands
+for c in groups + cmds:
+    cli.add_command(c)
+
+# Entrypoint/executor
+if __name__ == "__main__":
+    cli()

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/conf.py

@@ -0,0 +1,84 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+import pathlib
+
+
+sys.path.append(os.path.abspath("../"))
+
+
+# -- Project information -----------------------------------------------------
+
+project = "PyIntelOwl"
+copyright = "2020, Matteo Lodi"
+author = "Matteo Lodi"
+
+
+VERSION = (pathlib.Path("../version.txt")).read_text()
+
+GITHUB_URL = "https://github.com/intelowlproject/pyintelowl"
+
+# -- General configuration ---------------------------------------------------
+
+html_title = f"{project} Documentation ({VERSION})"
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    "sphinx_rtd_theme",
+    "sphinxcontrib.napoleon",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.viewcode",
+    "sphinx.ext.todo",
+    "sphinxcontrib.asciinema",
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = "en"
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+html_theme = "sphinx_rtd_theme"
+html_theme_path = [
+    "_themes",
+]
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ["_static"]
+
+
+# -- Extension configuration -------------------------------------------------
+
+# -- Options for todo extension ----------------------------------------------
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = True