MacOS cloning as a way to dedupe files: how to detect already cloned files?

[lucbouge]

I have discover the "cloning" facility of recent MacOS version: cp -c. Essentially, it copies a file by just sharing the data block on the disk, that is, without any space consumption. Only the metadata are specific, so that the two copies can live their own lives.

If some block is modified at one of the files, then this very block is duplicated, and only it. It is called "Copy on write". It is of course ideal for files which are mostly read, as for instance archive files.

Is looks like the "fclones dedup" uses this feature, which is great.

However, a major drawback it that the "fclones dedup" does not realize that two copies of file already share the same blocks, so that they do not need to be deduplicated again, which make the repeated runs as long as the first one.

Would it be possible to optimize the "fclones dedup" strategy on MacOS to detect that two identical files are already "cloned", so that nothing is needed?

At least some years ago, it looked almost impossible: https://eclecticlight.co/2021/04/02/how-can-you-tell-whether-a-file-has-been-cloned-in-apfs/

Best regards,

Luc.

[kapitainsky]

And as you noted it is most likely close to impossible. APFS is close source file system with limited documentation. So unless Apple provides clear guidance/tool how to identify two full clones (as clones can be partial and files can be different) I think it is not even worth of time to try given limited gains it would provide.

[lucbouge]

Thanks, kapitainsky. I was expecting that things had moved since 2021. Too bad...

…

Le 21 sept. 2024 à 07:05, kapitainsky ***@***.***> a écrit : And as you noted it is most likely close to impossible. APFS is close source file system with limited documentation. So unless Apple provides clear guidance/tool how to identify two full clones (as clones can be partial and files can be different) I think it is not even worth of time to try given limited gains it would provide.