Simplify policies that require constraints on a URL based on its protocol

[GoogleCodeExporter]

Once a: protocol is allowed, policy authors often want to place additional 
constraints: e.g. a data protocol with an image/... mime-type for use with <img 
src>, or a tel: protocol that contains a valid telephone number.

Right now, policy authors are tempted to do

allowUrlProtocols("data", "https", "http", "mailto")

allowAttributes("src").matching(Pattern.compile("^(data:image/(gif|png|jpeg)[,;]
|http|https|mailto|//)", Pattern.CASE_INSENSITIVE)

which requires duplicative effort.

We should provide good alternatives to writing regular expressions to match 
URLs as it is error prone.

Perhaps a URL policy that recognizes structure in URLs.

Original issue reported on code.google.com by [email protected] on 21 Jan 2014 at 4:09

[jmanico]

Bump to revisit.

[cnsgithub]

I would love an abbreviation for this, too.

Could you please help me out getting this to work:

	private static final PolicyFactory htmlSanitizer = new HtmlPolicyBuilder()
	        .allowUrlProtocols("data", "https", "http", "mailto")
	        .allowAttributes("src")
	        .matching(Pattern.compile("^(data:image/(gif|png|jpeg)[,;]|http|https|mailto|//)", Pattern.CASE_INSENSITIVE))
	        .onElements("img")
	        .toFactory()
	        .and(Sanitizers.IMAGES)
	        .and(Sanitizers.BLOCKS);

	public static void main(String[] args) {
		System.out.println(HtmlSanitize("<img src=\"data:image/gif;base64,R0lGODlhEAAQAMQAAORHHOVSKudfOulrSOp3WOyDZu6QdvCchPGolfO0o\" /><p>test</p>"));
	}

I get <p>test</p> here, but the embedded image is missing. Changing src="data to src="http however works. What am I doing wrong?

[cnsgithub]

Got it to work:

	private static final PolicyFactory htmlImageSanitizer = new HtmlPolicyBuilder()
	        .allowUrlProtocols("data", "http", "https")
	        .allowElements("img")
	        .allowAttributes("src")
	        .matching(Pattern.compile("^(data:image/(gif|png|jpeg)[,;]|http|https|mailto|//).+", Pattern.CASE_INSENSITIVE))
	        .onElements("img")
	        .toFactory();
	private static final PolicyFactory htmlSanitizer = htmlImageSanitizer.and(Sanitizers.BLOCKS).and(Sanitizers.FORMATTING).and(Sanitizers.LINKS).and(Sanitizers.STYLES).and(Sanitizers.TABLES);

[mikesamuel]

Your solution seems complicated but the API doesn't obviously allow for a better way, so it's probably the API's fault.

There seem to be some separable concerns here:

• It's hard to restrict data URLs to particular mime-types.
• It's hard to restrict URLs to some attributes and not others.

I think the first problem is a symptom of a larger problem: it's hard to match URLs.
If we had a way to specify a concisely specify a set of URLs, then we could solve the second problem via an API like

    allowAttributes("src")
    .matchingUrls(...)
    .onElements("img")

where the ... encapsulates (http, https, or data with content-type in image/(gif|png|jpeg)).

---

What do you think of https://gist.github.com/mikesamuel/e9720a0acc0601372deba3bf0896f33a as a proposed API for solving the larger problem?

_{^{Note to self: I'm finding excuses to write specifications, so I should probably figure out what work I'm subconsciously avoiding and do it.}}

[cnsgithub]

Your API would be great at this point and definitely its implementation would be worth the effort.

[mikesamuel]

https://github.com/OWASP/url-classifier is an experimental URL classifier API based on that gist.

#126 integrates it into java-html-sanitizer.

Neither is ready for prime-time yet, but you can play around.