Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read/write files in binary mode #1698

Merged
merged 1 commit into from
Dec 14, 2024
Merged

Read/write files in binary mode #1698

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions mlx/io/load.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,14 @@

#include "mlx/io/threadpool.h"

// Strictly we need to operate on files in binary mode (to avoid \r getting
// automatically inserted), but every modern system except for Windows no
// longer differentiates between binary and text files and for them define
// the flag as no-op.
#ifndef O_BINARY
#define O_BINARY 0
#endif

namespace mlx::core {

namespace io {
Expand Down Expand Up @@ -51,7 +59,8 @@ class Writer {
class ParallelFileReader : public Reader {
public:
explicit ParallelFileReader(std::string file_path)
: fd_(open(file_path.c_str(), O_RDONLY)), label_(std::move(file_path)) {}
: fd_(open(file_path.c_str(), O_RDONLY | O_BINARY)),
label_(std::move(file_path)) {}

~ParallelFileReader() override {
close(fd_);
Expand Down Expand Up @@ -93,7 +102,10 @@ class ParallelFileReader : public Reader {
class FileWriter : public Writer {
public:
explicit FileWriter(std::string file_path)
: fd_(open(file_path.c_str(), O_CREAT | O_WRONLY | O_TRUNC, 0644)),
: fd_(open(
file_path.c_str(),
O_CREAT | O_WRONLY | O_TRUNC | O_BINARY,
0644)),
label_(std::move(file_path)) {}

~FileWriter() override {
Expand Down