Detect bots from HTTP_FROM

[liviuconcioiu]

I think this should be implemented, and I have three reasons for this:

1. User agents change and it's hard to keep up will all bots, worldwide, especially if they don't get reported so we can add them, and also we can detect them right away.
2. Bing uses Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0 and HTTP_FROM is bingbot(at)microsoft.com.
3. Bots are required to send HTTP_FROM https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/From

If HTTP_FROM is detected then it should be treated as generic bot, then further check for specific bot name.

• "<?=print(9347655345-4954366);?>"
Mozilliqa"<?=print(9347655345-4954366);?>"

• TGVnaXRpbWF0ZSBsaW5rIHRyYWNrZXI=
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0

• bingbot(at)microsoft.com
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0

• [email protected]
ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])

• [email protected]
Mozilla/5.0 (compatible; Cotoyogi/4.0; +https://ds.rois.ac.jp/center8/crawler/

• googlebot(at)googlebot.com
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

• gptbot(at)openai.com
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)

• [email protected]
SuggestBot/1.0

• [email protected]
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot

• pigafetta-bot(at)visual-seo.com
Mozilla 5.0 (compatible; Pigafetta/0.5; +http://visual-seo.com/Pigafetta-Bot)

• [email protected]
Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)

• [email protected]
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 [email protected]

• [email protected]
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

• [email protected]
Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)

• [email protected]
The Knowledge AI

• [email protected]
Mozilla/5.0 (compatible; ips-agent)