Skip to content

Commit

Permalink
Don't rely on index from future (#71)
Browse files Browse the repository at this point in the history
We're seeing this occasionally
```
TypeError no implicit conversion from nil to integer
[]=(/usr/local/bundle/gems/rb_snowflake_client-1.1.3/lib/ruby_snowflake/result.rb:19)
...
/usr/local/bundle/gems/rb_snowflake_client-1.1.3/lib/ruby_snowflake/client/threaded_in_memory_strategy.rb:35
...
```

I can't really see how we'd ever have a nil index. But maybe having the
index not rely on the result of the future execution would be safer.
  • Loading branch information
nampas authored Nov 5, 2024
1 parent 3f06972 commit 2b96728
Showing 1 changed file with 22 additions and 22 deletions.
44 changes: 22 additions & 22 deletions lib/ruby_snowflake/client/threaded_in_memory_strategy.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,31 +9,31 @@ def self.result(statement_json_body, retreive_proc, num_threads)
result[0] = statement_json_body["data"]

thread_pool = Concurrent::FixedThreadPool.new(num_threads)
futures = []
partitions.each_with_index do |partition, index|
next if index == 0 # already have the first partition
futures << Concurrent::Future.execute(executor: thread_pool) do
[index, retreive_proc.call(index)]
partitions
.each_with_index.map do |partition, index|
next if index == 0 # already have the first partition
[index, Concurrent::Future.execute(executor: thread_pool) { retreive_proc.call(index) }]
end
end
futures.each do |future|
if future.rejected?
if future.reason.is_a? RubySnowflake::Error
raise future.reason
else
raise ConnectionStarvedError.new(
"A partition request timed out. This is usually do to using the client in" \
"multiple threads. The client uses a connection thread pool and if too many" \
"requests are all done in threads at the same time, threads can get starved" \
"of access to connections. The solution for this is to either increase the " \
"max_connections parameter on the client or create a new client instance" \
"with it's own connection pool to snowflake per thread. Rejection reason: #{future.reason.message}"
)
.each do |entry|
next if entry.nil? # 0th index

index, future = entry
if future.rejected?
if future.reason.is_a? RubySnowflake::Error
raise future.reason
else
raise ConnectionStarvedError.new(
"A partition request timed out. This is usually do to using the client in" \
"multiple threads. The client uses a connection thread pool and if too many" \
"requests are all done in threads at the same time, threads can get starved" \
"of access to connections. The solution for this is to either increase the " \
"max_connections parameter on the client or create a new client instance" \
"with it's own connection pool to snowflake per thread. Rejection reason: #{future.reason.message}"
)
end
end
result[index] = future.value
end
index, partition_data = future.value
result[index] = partition_data
end
result
end
end
Expand Down

0 comments on commit 2b96728

Please sign in to comment.