Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: adds some threads #3

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat: adds some threads #3

wants to merge 1 commit into from

Conversation

niquerio
Copy link
Collaborator

@niquerio niquerio commented Sep 30, 2022

Adds a basic threadpool to speed up all of the calls to worldcat

@billdueber Mostly wanting you to look over the thread pool I put in for doing a bazillion calls to the worldcat api. Do I need to do anything to keep it from getting out of control? is 5 threads too many?

* adds a basic threadpool
@billdueber
Copy link

billdueber commented Mar 29, 2023

Your thread code isn't safe -- the gap between barcode_queue.empty? and barcode_queue.shift is asking for a deadlock, because between this thread calling barcode_queue.empty? another thread might have emptied the queue, and it'll wait there forever.

You don't have any shared state, so it seems like it might be a good use for the in_threads gem. I think you can just do something like:

results = @barcodes.in_threads.map do |barcode|
  umich_item = @umich_catalog_items.item_for_barcode(barcode)
  if umich_item.nil?
    nil 
  else
    worldcat_summary = WorldCatSummary.for(umich_item.oclc || [])
    format_line(umich_item: umich_item, worldcat_summary: worldcat_summary)
  end
end
results.compact! # get rid of the nils

If you really just need to print them, skip the map and just call in_threads.each , replacing the if statement with the next if umich_item.nil? you had before.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants