Downloading and Installing Crawler on Colab is taking forever

[BukuBukuChagma]

Hey, I'm trying to test out the crawler from haystack.
I'm running the command !pip install farm-haystack[colab,crawler] but it just goes on forever in trying to resolve dependencies issues etc. I have had it running or like 3 hours and its still going. How to resolve this?

Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Requirement already satisfied: farm-haystack[Crawler,colab] in /usr/local/lib/python3.7/dist-packages (1.6.0)
Requirement already satisfied: posthog in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.4.9)
Requirement already satisfied: pydantic in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.9.1)
Requirement already satisfied: elastic-apm in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (6.10.1)
Requirement already satisfied: rapidfuzz<3,>=2.0.15 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (2.3.0)
Requirement already satisfied: jsonschema in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (4.3.3)
Requirement already satisfied: elasticsearch<7.11,>=7.7 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (7.10.1)
Requirement already satisfied: scikit-learn>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.0.2)
Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (3.7)
Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.7.3)
Requirement already satisfied: torch<1.13,>1.9 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.12.0+cu113)
Requirement already satisfied: sentence-transformers>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (2.2.2)
Requirement already satisfied: azure-ai-formrecognizer==3.2.0b2 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (3.2.0b2)
Requirement already satisfied: langdetect in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.0.9)
Requirement already satisfied: mmh3 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (3.0.0)
Requirement already satisfied: seqeval in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.2.2)
Requirement already satisfied: tika in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.24)
Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (4.64.0)
Requirement already satisfied: more-itertools in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (8.13.0)
Requirement already satisfied: azure-core<1.23 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.22.1)
Requirement already satisfied: transformers==4.20.1 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (4.20.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (2.6.3)
Requirement already satisfied: mlflow in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.27.0)
Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.3.5)
Requirement already satisfied: quantulum3 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (0.7.10)
Requirement already satisfied: python-docx in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (0.8.11)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (4.12.0)
Requirement already satisfied: huggingface-hub<0.8.0,>=0.5.0 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (0.7.0)
Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (0.3.5.1)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (2.23.0)
Requirement already satisfied: webdriver-manager in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (3.8.2)
Requirement already satisfied: selenium!=4.1.4 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (4.3.0)
Requirement already satisfied: grpcio==1.43.0 in /usr/local/lib/python3.7/dist-packages (from farm-haystack[Crawler,colab]) (1.43.0)
Requirement already satisfied: six>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from azure-ai-formrecognizer==3.2.0b2->farm-haystack[Crawler,colab]) (1.15.0)
Requirement already satisfied: msrest>=0.6.21 in /usr/local/lib/python3.7/dist-packages (from azure-ai-formrecognizer==3.2.0b2->farm-haystack[Crawler,colab]) (0.6.21)
Requirement already satisfied: azure-common~=1.1 in /usr/local/lib/python3.7/dist-packages (from azure-ai-formrecognizer==3.2.0b2->farm-haystack[Crawler,colab]) (1.1.28)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers==4.20.1->farm-haystack[Crawler,colab]) (1.21.6)
Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in /usr/local/lib/python3.7/dist-packages (from transformers==4.20.1->farm-haystack[Crawler,colab]) (0.12.1)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers==4.20.1->farm-haystack[Crawler,colab]) (21.3)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers==4.20.1->farm-haystack[Crawler,colab]) (2022.6.2)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from transformers==4.20.1->farm-haystack[Crawler,colab]) (6.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers==4.20.1->farm-haystack[Crawler,colab]) (3.7.1)
Requirement already satisfied: urllib3<2,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from elasticsearch<7.11,>=7.7->farm-haystack[Crawler,colab]) (1.24.3)
Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from elasticsearch<7.11,>=7.7->farm-haystack[Crawler,colab]) (2022.6.15)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<0.8.0,>=0.5.0->farm-haystack[Crawler,colab]) (4.1.1)
Requirement already satisfied: isodate>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from msrest>=0.6.21->azure-ai-formrecognizer==3.2.0b2->farm-haystack[Crawler,colab]) (0.6.1)
Requirement already satisfied: requests-oauthlib>=0.5.0 in /usr/local/lib/python3.7/dist-packages (from msrest>=0.6.21->azure-ai-formrecognizer==3.2.0b2->farm-haystack[Crawler,colab]) (1.3.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->transformers==4.20.1->farm-haystack[Crawler,colab]) (3.0.9)
Requirement already satisfied: jarowinkler<2.0.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from rapidfuzz<3,>=2.0.15->farm-haystack[Crawler,colab]) (1.2.0)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->farm-haystack[Crawler,colab]) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->farm-haystack[Crawler,colab]) (2.10)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.5.0->msrest>=0.6.21->azure-ai-formrecognizer==3.2.0b2->farm-haystack[Crawler,colab]) (3.2.0)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=1.0.0->farm-haystack[Crawler,colab]) (1.1.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=1.0.0->farm-haystack[Crawler,colab]) (3.1.0)
Requirement already satisfied: trio-websocket~=0.9 in /usr/local/lib/python3.7/dist-packages (from selenium!=4.1.4->farm-haystack[Crawler,colab]) (0.9.2)
Collecting urllib3[secure,socks]=1.26
Using cached urllib3-1.26.11-py2.py3-none-any.whl (139 kB)
Requirement already satisfied: trio=0.17 in /usr/local/lib/python3.7/dist-packages (from selenium!=4.1.4->farm-haystack[Crawler,colab]) (0.21.0)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.7/dist-packages (from sentence-transformers>=2.2.0->farm-haystack[Crawler,colab]) (0.1.96)
Requirement already satisfied: torchvision in /usr/local/lib/python3.7/dist-packages (from sentence-transformers>=2.2.0->farm-haystack[Crawler,colab]) (0.13.0+cu113)
Requirement already satisfied: outcome in /usr/local/lib/python3.7/dist-packages (from trio~=0.17->selenium!=4.1.4->farm-haystack[Crawler,colab]) (1.2.0)
Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.7/dist-packages (from trio~=0.17->selenium!=4.1.4->farm-haystack[Crawler,colab]) (21.4.0)
Requirement already satisfied: sortedcontainers in /usr/local/lib/python3.7/dist-packages (from trio~=0.17->selenium!=4.1.4->farm-haystack[Crawler,colab]) (2.4.0)
Requirement already satisfied: sniffio in /usr/local/lib/python3.7/dist-packages (from trio~=0.17->selenium!=4.1.4->farm-haystack[Crawler,colab]) (1.2.0)
Requirement already satisfied: async-generator>=1.9 in /usr/local/lib/python3.7/dist-packages (from trio~=0.17->selenium!=4.1.4->farm-haystack[Crawler,colab]) (1.10)
Requirement already satisfied: wsproto>=0.14 in /usr/local/lib/python3.7/dist-packages (from trio-websocket~=0.9->selenium!=4.1.4->farm-haystack[Crawler,colab]) (1.1.0)
Using cached urllib3-1.26.10-py2.py3-none-any.whl (139 kB)
Using cached urllib3-1.26.9-py2.py3-none-any.whl (138 kB)
Using cached urllib3-1.26.8-py2.py3-none-any.whl (138 kB)
Using cached urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Using cached urllib3-1.26.5-py2.py3-none-any.whl (138 kB)
Using cached urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
Using cached urllib3-1.26.3-py2.py3-none-any.whl (137 kB)
Using cached urllib3-1.26.2-py2.py3-none-any.whl (136 kB)
Using cached urllib3-1.26.1-py2.py3-none-any.whl (136 kB)
Using cached urllib3-1.26.0-py2.py3-none-any.whl (136 kB)
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
Collecting urllib3<2,>=1.21.1
Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
INFO: pip is looking at multiple versions of to determine which version is compatible with other requirements. This could take a while.
Using cached urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Using cached urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
Using cached urllib3-1.25.7-py2.py3-none-any.whl (125 kB)
Using cached urllib3-1.25.6-py2.py3-none-any.whl (125 kB)
Using cached urllib3-1.25.5-py2.py3-none-any.whl (125 kB)
INFO: pip is looking at multiple versions of urllib3 to determine which version is compatible with other requirements. This could take a while.
Using cached urllib3-1.25.4-py2.py3-none-any.whl (125 kB)
INFO: pip is looking at multiple versions of to determine which version is compatible with other requirements. This could take a while.
Using cached urllib3-1.25.3-py2.py3-none-any.whl (150 kB)
Using cached urllib3-1.25.2-py2.py3-none-any.whl (150 kB)
Using cached urllib3-1.24.3-py2.py3-none-any.whl (118 kB)
Using cached urllib3-1.24.2-py2.py3-none-any.whl (131 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Using cached urllib3-1.24.1-py2.py3-none-any.whl (118 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Using cached urllib3-1.24-py2.py3-none-any.whl (117 kB)
Using cached urllib3-1.23-py2.py3-none-any.whl (133 kB)
Using cached urllib3-1.22-py2.py3-none-any.whl (132 kB)
Using cached urllib3-1.21.1-py2.py3-none-any.whl (131 kB)
INFO: pip is looking at multiple versions of typing-extensions to determine which version is compatible with other requirements. This could take a while.
Collecting typing-extensions>=3.7.4.3
Using cached typing_extensions-4.3.0-py3-none-any.whl (25 kB)
Using cached typing_extensions-4.2.0-py3-none-any.whl (24 kB)
Using cached typing_extensions-4.1.1-py3-none-any.whl (26 kB)
Using cached typing_extensions-4.1.0-py3-none-any.whl (26 kB)
Using cached typing_extensions-4.0.1-py3-none-any.whl (22 kB)
Using cached typing_extensions-4.0.0-py3-none-any.whl (22 kB)
Using cached typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
INFO: pip is looking at multiple versions of typing-extensions to determine which version is compatible with other requirements. This could take a while.
Using cached typing_extensions-3.10.0.1-py3-none-any.whl (26 kB)
Using cached typing_extensions-3.10.0.0-py3-none-any.whl (26 kB)
Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
INFO: pip is looking at multiple versions of trio-websocket to determine which version is compatible with other requirements. This could take a while.
Collecting trio-websocket~=0.9
Using cached trio_websocket-0.9.2-py3-none-any.whl (16 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Using cached trio_websocket-0.9.1-py3-none-any.whl (16 kB)
Using cached trio_websocket-0.9.0-py3-none-any.whl (16 kB)
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
Collecting attrs>=19.2.0
Using cached attrs-22.1.0-py2.py3-none-any.whl (58 kB)
INFO: pip is looking at multiple versions of trio-websocket to determine which version is compatible with other requirements. This could take a while.
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Downloading attrs-21.3.0-py2.py3-none-any.whl (61 kB)
|████████████████████████████████| 61 kB 557 kB/s
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
|████████████████████████████████| 53 kB 3.2 MB/s
Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
|████████████████████████████████| 49 kB 7.5 MB/s
Downloading attrs-20.2.0-py2.py3-none-any.whl (48 kB)
|████████████████████████████████| 48 kB 5.8 MB/s
Downloading attrs-20.1.0-py2.py3-none-any.whl (49 kB)
|████████████████████████████████| 49 kB 7.1 MB/s
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Downloading attrs-19.2.0-py2.py3-none-any.whl (40 kB)
|████████████████████████████████| 40 kB 6.3 MB/s
INFO: pip is looking at multiple versions of async-generator to determine which version is compatible with other requirements. This could take a while.
Collecting async-generator>=1.9
Using cached async_generator-1.10-py3-none-any.whl (18 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking

[BukuBukuChagma]

I believe there is a dependency conflict with some of the packages that crawler uses.
When tried to use pip's previous version of dependency resolver, here the error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
requests 2.23.0 requires urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1, but you have urllib3 1.26.11 which is incompatible.
msrest 0.7.1 requires azure-core>=1.24.0, but you have azure-core 1.22.1 which is incompatible.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
Successfully installed cryptography-37.0.4 pyOpenSSL-22.0.0 urllib3-1.26.11
WARNING: The following packages were previously imported in this runtime:
[urllib3]
You must restart the runtime in order to use newly installed versions.

[sjrl]

I was able to run this install on colab using the following

!pip install --upgrade pip
!pip install farm-haystack[colab,crawler]

It did take a while ~5min but it did resolve with no errors. Is it possible that you need to run !pip install --upgrade pip first?

[BukuBukuChagma]

Hey that did resolve that problem, but it put up another problem. Can you help with this?

`
from haystack.nodes import Crawler

crawler = Crawler(output_dir="crawled_files") # This tells the Crawler where to store the crawled files
docs = crawler.crawl(
    urls=["https://nutritionfacts.org/blogs"], # This tells the Crawler which URLs to crawl
     filter_urls = ['https://nutritionfacts.org/blogs/'], # Here, you can pass regular expressions that the crawled URLs must comply with
    crawler_depth=10 # This tells the Crawler to follow only the links that it finds on the initial URLs
)
`

When I run this code I get this error:

TypeError Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/haystack/nodes/connector/crawler.py in init(self, output_dir, urls, crawler_depth, filter_urls, overwrite_existing_files, id_hash_keys, extract_hidden_text, loading_wait_time)
87 options.add_argument("--disable-dev-shm-usage")
---> 88 self.driver = webdriver.Chrome(service=Service("chromedriver"), options=options)
89 except:

TypeError: init() got an unexpected keyword argument 'service'

During handling of the above exception, another exception occurred:

Exception Traceback (most recent call last)
2 frames
/usr/local/lib/python3.7/dist-packages/haystack/nodes/connector/crawler.py in init(self, output_dir, urls, crawler_depth, filter_urls, overwrite_existing_files, id_hash_keys, extract_hidden_text, loading_wait_time)
94 !apt install chromium-driver
95 !cp /usr/lib/chromium-browser/chromedriver /usr/bin
---> 96 If it has already been installed, please check if it has been copied to the right directory i.e. to '/usr/bin'"""
97 )
98 else:

Exception:
'chromium-driver' needs to be installed manually when running colab. Follow the below given commands:
!apt-get update
!apt install chromium-driver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
If it has already been installed, please check if it has been copied to the right directory i.e. to '/usr/bin'

I already tried the solution it says in exception. Chromedriver is present at the exact location it says it should be present but the error still presists.
!whereis chromedriver

chromedriver: /usr/bin/chromedriver

[sjrl]

It looks like the version of selenium that is installed is too old which is why you are getting the TypeError: init() got an unexpected keyword argument 'service'. Looking at the selenium github it looks like you need at least version !pip install selenium==4.0.0.

[sjrl]

Hi @Zeni69 this should work now in Google Colab. We just merged the fix in this PR #2921

!pip install --upgrade pip
!pip install git+https://github.com/deepset-ai/haystack.git#egg=farm-haystack[colab,crawler]

[BukuBukuChagma]

Thanks for all the help. Appreciate it ^.^