Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Safetensors.write!/2 for streamed write #8

Merged
merged 1 commit into from
Feb 22, 2024
Merged

Conversation

jonatanklosko
Copy link
Member

When persisting to a file, Safetensors.dump/1 calls Nx.to_binary/1 on all tensors upfront (and if we need to convert endianness, we create yet another binary), so this can cause a significant memory bump. This PR adds Safetensors.write!/2 that converts and writes tensors directly to the file, one by one.

Other than lower memory usage, this can also be much faster. For the example below, it's 5 times faster for me:

tensors =
  # 1000 tensors, 8MB each -> 8GB
  for i <- 1..1000, into: %{} do
    {"t#{i}", Nx.iota({1_000_000}, type: :f64)}
  end

# Dump + write
File.write!("/tmp/test.safetensors", Safetensors.dump(tensors))

# Direct write
Safetensors.write!("/tmp/test.safetensors", tensors)

@jonatanklosko jonatanklosko changed the title Add Safetensors.write/2 for streamed write Add Safetensors.write!/2 for streamed write Feb 22, 2024
@jonatanklosko jonatanklosko merged commit ed9267d into main Feb 22, 2024
2 checks passed
@jonatanklosko jonatanklosko deleted the jk-write branch February 22, 2024 09:31
Copy link

@devCaiquedePaula devCaiquedePaula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In lib/safetensors. The Elixir module lacks comprehensive documentation, particularly at the module level, where an explanation of the module's purpose and key concepts would be beneficial. Consider improving code consistency by ensuring uniform indentation and adding vertical spacing for better readability. Address error handling, especially in the File.open!/2 call, by considering alternative approaches such as using File.open/2 to handle potential errors. Refactor code duplication between the write!/2 and dump/1 functions to a common function to enhance maintainability. Introduce named constants or functions for magic numbers and repeated calculations. Validate input parameters, improve error messages for better user guidance, and handle file closing after usage. Verify type specifications for accuracy, and ensure thorough testing to validate the correctness of implemented functions. Overall, these refinements will enhance code organization, maintainability, and user experience.

@josevalim
Copy link
Contributor

Hi @devCaiquedePaula. Please do not use automated/AI code review tools unless requested. If you feel any of these points are relevant though, a PR is welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants