Further development of CSP header checks and SRI checks

[dhilton]

Context

As outlined in #191 CSP headers are an important (along with other HTTP headers) way of restricting the impact of a XSS or malware infection. The questions around the next stage are as I see the following:

1. What does a good CSP look like? What are we looking for?
2. How do we approach html parsing to check for meta tags and then SRI inclusions?
3. Is there a way we can check a series of headers in one pass? For example what about checking for x-xss-protection: 1; mode=block or Strict-Transport-Security ala https://securityheaders.io/?q=https%3A%2F%2Fwww.gov.uk&hide=on&followRedirects=on ?

Thoughts?

[konklone]

You may want to build on pshtt, which we currently use (through domain-scan as well) to get HTTP/HTTPS behavior analysis of domains.

The pshtt tool will return the observed HTTP headers of the https://, http://, http://www, and https://www endpoints for a given hostname. You can ignore the other HTTPS-specific conclusions, but rely on the work that's gone into making reliable HTTP requests across a variety of configurations to get CSP [and other] headers.

It really depends on what you want to do with the data. You could design a sort of general securityheaders-like scan for a bunch of useful things, or you could dive deep on something like CSP. Our pshtt work is designed around the conclusions that the authors want to visually present to federal agencies, which are based around specific policy/compliance outcomes, so that informs the development.

[konklone]

Note that headless Chrome support is now present in domain-scan: #195

It's documented in the README:

• https://github.com/18F/domain-scan#headless-chrome
• https://github.com/18F/domain-scan/#developing-chrome-scanners

[ghost]

https://github.com/koenbuyens/securityheaders is what you are looking for. I hope it can serve as an inspiration to integrate similar functionality with this awesome project :)