Don't crawl URLs on x.com

The crawler logic currently skips links to twitter.com; as of cfpb/consumerfinance.gov@846e03a the CFPB website now links to x.com instead. This change adds x.com to the crawl exclusion list.
cfpb · Jul 3, 2024 · 26061f3 · 26061f3
1 parent c21128f
commit 26061f3
Showing 1 changed file with 1 addition and 0 deletions.
diff --git a/crawler/wpull_plugin.py b/crawler/wpull_plugin.py
@@ -27,6 +27,7 @@
         [
             r"^https://www.facebook.com/dialog/share\?.*",
             r"^https://twitter.com/intent/tweet\?.*",
+            r"^https://x.com/intent/tweet\?.*",
             r"^https://www.linkedin.com/shareArticle\?.*",
         ],
     )