Replies: 3 comments 13 replies
-
|
Hi @sridhar what happens if you try running this code? It's possible that the code exits with an error which is why you would see no process running on the GPU. Looking at your memory GPU usage it might have to do with your large batch size (the default size is |
Beta Was this translation helpful? Give feedback.
-
|
The code is running (for a day now). I can see all the tqdm status and all the batches being processed. It just runs on a single CPU core :( |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! Could you also show me the step which takes a long time? |
Beta Was this translation helpful? Give feedback.
-
|
Mine negatives seems to take a long time: (the output is a bit jumbled up because tqdm message kind of gets interleaved with my prints sometime). Mine negatives: 1%|█▌ | 8/752 [8:31:37<793:21:43, 3838.8Batches: 100 |
Beta Was this translation helpful? Give feedback.
-
|
Hmm I see. How many documents do you have in your document store (i.e. what is |
Beta Was this translation helpful? Give feedback.
-
|
Hey @sridhar, there are a couple of things that could go wrong here. Let's start with the basics - check your PyTorch installation, is your GPU visible (i.e. torch.cuda.is_available())? How do you set up your document store? GPL relies on properly setting up your document store and embedding retriever to mine negatives. In the mine negatives stage, GPL doesn't use GPU but relies on retrievers to do the work. If the retriever and document store are not GPU bound, then GPL will slow down. Compare your setup with https://haystack.deepset.ai/tutorials/gpl Negative mining stage for the GPL tutorial on 10k documents takes about 6 minutes (Tesla T4), and margin scoring is about that long as well. Retriever adaptation (training) takes about 15 min. Let us know how did it go. |
Beta Was this translation helpful? Give feedback.
-
|
@sjrl Setting up document store: I ran the GPL example you pointed to. I see that margin scoring & retriever adaptation are using 87%+ GPU, but Negative mining is taking at max 1-2% and is the slowest component of all. Here's the output of the whole run: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for such a detailed report @sridhar - the results you get are expected. If you look at the code of negative mining it really shouldn't use much GPU. You could increase the batch size (say 16/32) but I doubt you can load GPU much more than you currently are. I am not sure why your adaptation runs more than a day when the example adaptation above on 10k documents takes roughly 15 min, end-to-end. What happens if you shrink your corpus to 10k documents? |
Beta Was this translation helpful? Give feedback.
-
|
Why not reduce the number of queries to the same ratio? I suspect a good representation of your domain could be sampled from a smaller set of documents, say a 1/4 of your documents. That should work. Which code needs to be cleaned up? |
Beta Was this translation helpful? Give feedback.
-
|
@sridhar If you are using the document_store = FAISSDocumentStore(
faiss_index_factory_str="HNSW",
similarity="cosine",
)which speeds up the |
Beta Was this translation helpful? Give feedback.
-
|
You can find more information about the Just be aware that it will most likely take up more RAM than the |
Beta Was this translation helpful? Give feedback.
-
|
@vblagoje I think the following code needs to be at least parallelized on cpu: for i in tqdm(
range(0, len(question_doc_pairs), batch_size), disable=not self.progress_bar, desc="Mine negatives"
):
# question in batches to minimize network latency
i_end = min(i + batch_size, len(question_doc_pairs))
queries: List[str] = [e["question"] for e in question_doc_pairs[i:i_end]]
pos_docs: List[str] = [e["document"] for e in question_doc_pairs[i:i_end]]
docs: List[List[Document]] = self.retriever.retrieve_batch(
queries=queries, top_k=self.top_k, batch_size=batch_size
)
# iterate through queries and find negatives
for question, pos_doc, top_docs in zip(queries, pos_docs, docs):
random.shuffle(top_docs)
for doc_item in top_docs:
neg_doc = doc_item.content
if neg_doc != pos_doc:
question_pos_doc_neg_doc.append({"question": question, "pos_doc": pos_doc, "neg_doc": neg_doc})
break@sjrl Let me try HNSW and report back |
Beta Was this translation helpful? Give feedback.
-
|
@sridhar I also wanted to let you know that I looked at timings for different parts of the above code and the majority (>99%) of the time is spent on the following step (in our tutorial example): docs: List[List[Document]] = self.retriever.retrieve_batch(
queries=queries, top_k=self.top_k, batch_size=batch_size
)which is why I found using a faster index (HNSW) in the document store helped a lot. And since this is the time-limiting step, I don't think parallelizing the rest of the code will help much at all. |
Beta Was this translation helpful? Give feedback.
-
Hi all,
I'm trying to train an EmbeddingRetriever using GPL on Google Cloud VM with a Nvidia Tesla A100:
This is on a 12 core Xeon box.
I can see that rest of the code is being scheduled on the GPU (tokenizer, qa generation etc).. however, PseudoLabelGenerator is not. Here's the code that uses it. Am I missing something here?
Beta Was this translation helpful? Give feedback.
All reactions