Consider gathering anonymous analytics on disable / standard block rate

[pes10k]

Would need to make sure it was anonymous, didn't collect URLs (only aggregate statistics) and in general was respectful to users.

Would need a lot of discussion / planning to make right, but don't want to loose the discussion from issue #24

[jawz101]

My first thought is to only attempt it on sites which provide a robots.txt file. But it would have to be something that was possibly a w3c standard itself since robots.txt is sort of a loose, "handshake deal" protection against web crawlers.

Probably the best references might be in however Internet archival sites choose to crawl the web. If they crawl with some sort of moral standard that doesn't try to compromise a user's or web site's security and infrastructure, maybe that would tell us how to consider what could be logged in a public manner.

[jawz101]

My first thought is to only attempt it on sites which provide a robots.txt file. But it would have to be something that was possibly a w3c standard itself since robots.txt is sort of a loose, "handshake deal" protection against web crawlers.

Probably the best references might be in however Internet archival sites choose to crawl the web. If they crawl with some sort of moral standard that doesn't try to compromise a user's or web site's security and infrastructure, maybe that would tell us how to consider what could be logged in a public manner.

https://archive.org/help/aboutsearch.htm
http://archive.org/wayback/available?url=google.com