Skip to content
This repository has been archived by the owner on Dec 17, 2021. It is now read-only.

Commit

Permalink
adjust remote build, document new reqs splitting
Browse files Browse the repository at this point in the history
  • Loading branch information
konklone committed Apr 1, 2018
1 parent ac66061 commit 2a8c711
Show file tree
Hide file tree
Showing 6 changed files with 33 additions and 26 deletions.
41 changes: 22 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,23 +21,26 @@ There is also built-in support for using **headless Chrome** to efficiently meas

### Requirements

`domain-scan` requires **Python 3.5 and up**. To install dependencies:
`domain-scan` requires **Python 3.5 and up**.

To install **core dependencies**:

```bash
pip install -r requirements.txt
```

This will automatically allow the use of two scanners:
You can install scanner- or gatherer-specific dependencies yourself. Or, you can "quick start" by just **installing all dependencies for all scanners and/or all gatherers**:

* `pshtt` - A scanner that uses the [`pshtt`](https://github.com/dhs-ncats/pshtt) Python package from the [Department of Homeland Security's NCATS team](https://github.com/dhs-ncats).
* `sslyze` - A scanner that uses the [`sslyze`](https://github.com/nabla-c0d3/sslyze) Python package maintained by Alban Diquet.
```bash
pip install -r requirements-scanners.txt
pip install -r requirements-gatherers.txt
```

Other individual scanners will require additional externally installed dependencies:

* `trustymail`: The `trustymail` command, available from the [`trustymail`](https://github.com/dhs-ncats/trustymail) Python package from the [Department of Homeland Security's NCATS team](https://github.com/dhs-ncats). (Override path by setting the `TRUSTYMAIL_PATH` environment variable.)
* `a11y`: The `pa11y` command, available from the [`pa11y`](https://www.npmjs.com/package/pa11y) Node package. (Override path by setting the `PA11Y_PATH` environment variable.)
* `third_parties`: The `phantomas` command, available from the [`phantomas`](https://www.npmjs.com/package/phantomas) Node package. (Override path by setting the `PHANTOMAS_PATH` environment variable.)
If you plan on **developing/testing domain-scan itself**, install development requirements:

```bash
pip install -r requirements-dev.txt
```

### Usage

Expand Down Expand Up @@ -65,7 +68,16 @@ Append columns to each row with metadata about the scan itself, such as how long
./scan example.com --scan=pshtt --meta
```

##### Parallelization
### Scanners

* `pshtt` - A scanner that uses the [`pshtt`](https://github.com/dhs-ncats/pshtt) Python package from the [Department of Homeland Security's NCATS team](https://github.com/dhs-ncats).
* `sslyze` - A scanner that uses the [`sslyze`](https://github.com/nabla-c0d3/sslyze) Python package maintained by Alban Diquet.
* `trustymail`: The `trustymail` command, available from the [`trustymail`](https://github.com/dhs-ncats/trustymail) Python package from the [Department of Homeland Security's NCATS team](https://github.com/dhs-ncats). (Override path by setting the `TRUSTYMAIL_PATH` environment variable.)
* `third_parties` - What third party web services are in use, using [headless Chrome](https://developers.google.com/web/updates/2017/04/headless-chrome) to trap outgoing requests. (See documentation for [using](#headless-chrome) or [writing](#developing-chrome-scanners) Chrome-based scanners.)
* `a11y` - Accessibility issues, using [`pa11y`](https://github.com/pa11y/pa11y).
* `noop` - Test scanner (no-op) used for development and debugging. Does nothing.

### Parallelization

It's important to understand that **scans run in parallel by default**, and **data is streamed to disk immediately** after each scan is done.

Expand Down Expand Up @@ -117,15 +129,6 @@ See [`docs/lambda.md`](`docs/lambda.md`) for how to build and deploy Lambda-base

### Options

**Scanners:**

* `pshtt` - HTTP/HTTPS/HSTS configuration, using [`pshtt`](https://github.com/dhs-ncats/pshtt).
* `trustymail` - MX/SPF/STARTTLS/DMARC configuration, using [`trustymail`](https://github.com/dhs-ncats/trustymail).
* `sslyze` - TLS/SSL configuration, using [`sslyze`](https://github.com/nabla-c0d3/sslyze).
* `third_parties` - What third party web services are in use, using [headless Chrome](https://developers.google.com/web/updates/2017/04/headless-chrome) to trap outgoing requests. (See documentation for [using](#headless-chrome) or [writing](#developing-chrome-scanners) Chrome-based scanners.)
* `a11y` - Accessibility issues, using [`pa11y`](https://github.com/pa11y/pa11y).
* `noop` - Test scanner (no-op) used for development and debugging. Does nothing.

**General options:**

* `--scan` - **Required.** Comma-separated names of one or more scanners.
Expand Down
2 changes: 1 addition & 1 deletion lambda/remote_build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ pip install .
cd ..

cd domain-scan
pip install -r requirements.txt
pip install -r lambda/requirements-lambda.txt
cd ..

deactivate
Expand Down
1 change: 0 additions & 1 deletion lambda/requirements-lambda.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,3 @@

strict-rfc3339
publicsuffix

3 changes: 3 additions & 0 deletions requirements-gatherers.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
###
# Requirements used by specific gatherers.

# censys
google-cloud-bigquery
google-auth-oauthlib
10 changes: 6 additions & 4 deletions requirements-scanners.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@

# a11y
pyyaml
requests
###
# Requirements used by specific scanners.

# pshtt
git+https://github.com/dhs-ncats/pshtt.git#egg=pshtt
Expand All @@ -12,3 +10,7 @@ git+https://github.com/dhs-ncats/trustymail.git#egg=trustymail
# sslyze
sslyze>=1.3.4,<1.4.0
cryptography

# a11y / csp
pyyaml
requests
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@
# invocation.
boto3

# Used in Lanbda functions. Also copied to lambda/requirements-lambda.txt.
# Used in Lambda functions. Also copied to lambda/requirements-lambda.txt.
strict-rfc3339
publicsuffix

0 comments on commit 2a8c711

Please sign in to comment.