Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] File deduplication #6332

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8982dd5
File deduplication
Hocuri Dec 11, 2024
34f4a45
--wip-- [skip ci]
Hocuri Dec 12, 2024
4cf4357
Fix the tests (some of the fixes may need a new test)
Hocuri Dec 13, 2024
6a02bce
Fix some more tests, I'll need to remove some println statements
Hocuri Dec 13, 2024
6b00993
Adapt more tests and fix most of them
Hocuri Dec 18, 2024
9f1c186
test: Assume that the avatar name also changes
Hocuri Dec 18, 2024
d60a309
Adapt summary.rs tests
Hocuri Dec 18, 2024
dba030c
Adapt some more tests, they all pass
Hocuri Dec 18, 2024
4c7b0c2
Adapt src/receive_imf/tests.rs, fails because there is no deduplicati…
Hocuri Dec 18, 2024
f83d5a2
Deduplicate on message reception, fix all tests
Hocuri Dec 19, 2024
27af74b
Small tweaks, clippy
Hocuri Dec 19, 2024
f16ec31
Set deduplicated files as read-only on the file system
Hocuri Dec 23, 2024
47d140a
Set the file modification time so that it's not deleted during housek…
Hocuri Dec 23, 2024
2cbdc01
Deduplicate the code writing a file
Hocuri Dec 26, 2024
ca90133
Use only the first 32 characters of the hash
Hocuri Dec 26, 2024
aa772d7
Keep the code repairing Param::Filename extensions for now
Hocuri Dec 27, 2024
2a25af8
Some renames, leave `set_file_from_bytes()` being pub for now
Hocuri Dec 27, 2024
efabdcd
Create blob dir if it doesn't exist
Hocuri Dec 27, 2024
8b59fc2
Document and expose via the C ffi
Hocuri Dec 27, 2024
d474bfb
Use the actual file's name if `name` is None
Hocuri Dec 27, 2024
d0a3130
Clippy
Hocuri Dec 27, 2024
aa9dc2b
clippy: Make functions that are not async not be async
Hocuri Dec 27, 2024
f492186
Fix mistake I made when rebasing
Hocuri Jan 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ toml = "0.8"
url = "2"
uuid = { version = "1", features = ["serde", "v4"] }
webpki-roots = "0.26.7"
blake3 = "1.5.5"

[dev-dependencies]
anyhow = { workspace = true, features = ["backtrace"] } # Enable `backtrace` feature in tests.
Expand Down
19 changes: 19 additions & 0 deletions deltachat-ffi/deltachat.h
Original file line number Diff line number Diff line change
Expand Up @@ -4726,6 +4726,25 @@ void dc_msg_set_override_sender_name(dc_msg_t* msg, const char* name)
void dc_msg_set_file (dc_msg_t* msg, const char* file, const char* filemime);


/**
* Sets the file associated with a message.
*
* If `name` is non-null, it is used as the file name
* and the actual current name of the file is ignored.
*
* In order to deduplicate files that contain the same data,
* the file will be renamed to a hash of the file data.
* The file must not be modified after this function was called.
*
* @memberof dc_msg_t
* @param msg The message object.
* @param file The file to attach.
* @param name The original filename of the attachment.
* @param filemime The MIME type of the file. NULL if you don't know or don't care.
*/
void dc_msg_set_file_and_deduplicate(dc_msg_t* msg, const char* file, const char* name, const char* filemime);


/**
* Set the dimensions associated with message object.
* Typically this is the width and the height of an image or video associated using dc_msg_set_file().
Expand Down
27 changes: 27 additions & 0 deletions deltachat-ffi/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3813,6 +3813,33 @@ pub unsafe extern "C" fn dc_msg_set_file(
)
}

#[no_mangle]
pub unsafe extern "C" fn dc_msg_set_file_and_deduplicate(
msg: *mut dc_msg_t,
file: *const libc::c_char,
name: *const libc::c_char,
filemime: *const libc::c_char,
) {
if msg.is_null() || file.is_null() {
eprintln!("ignoring careless call to dc_msg_set_file()");
return;
}
let ffi_msg = &mut *msg;
let ctx = &*ffi_msg.context;

ffi_msg
.message
.set_file_and_deduplicate(
ctx,
&as_path(file),
to_opt_string_lossy(name).as_deref(),
to_opt_string_lossy(filemime).as_deref(),
)
.context("failed to set file")
.log_err(&*ffi_msg.context)
.ok();
}

#[no_mangle]
pub unsafe extern "C" fn dc_msg_set_dimension(
msg: *mut dc_msg_t,
Expand Down
Loading
Loading