Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moving from zerocopy to bytemuck #82

Closed
Kerollmops opened this issue Dec 6, 2020 · 4 comments · Fixed by #91
Closed

Moving from zerocopy to bytemuck #82

Kerollmops opened this issue Dec 6, 2020 · 4 comments · Fixed by #91
Labels
breaking A change that is breaking the semver good first issue Good for newcomers

Comments

@Kerollmops
Copy link
Member

This library currently uses the zerocopy crate to provide some useful codecs for common types and types which implements the AsBytes/FromBytes traits (i.e. u8, [u8]). I find it complex/impossible to contribute to this crate as there is no repository for this crate only as it has been developed for the new fuchsia OS; sources of this crate are mixed with the source of the OS (in a sub-repository at least).

The bytemuck library seems to be much easier to contribute to; it is hosted on Github, seems to be much more popular than the former (114k downloads by month compared to 15k). It brings a better API, at least for heed, as it can return information on which kind of problem happens when a cast fails, it would simplify some codecs.

I looked into the crate and found that the change, despite breaking, will be easy. I also found out that we can cast slices of different types by using the try_cast_slice function.

I am just not sure how to expose integers as big-endian, ensuring that the global order is preserved. We will maybe need to create codecs for every integer type like what zerocopy has done using byteorder maybe.

@Kerollmops Kerollmops added the good first issue Good for newcomers label Dec 6, 2020
@Kerollmops Kerollmops pinned this issue Dec 6, 2020
@Kerollmops Kerollmops changed the title Move from zerocopy to bytemuck Moving from zerocopy to bytemuck Dec 6, 2020
@Kerollmops Kerollmops added the breaking A change that is breaking the semver label Dec 29, 2020
@Kerollmops Kerollmops unpinned this issue Jun 28, 2021
@joshlf
Copy link

joshlf commented Oct 26, 2022

Hi @Kerollmops , just wanted to let you know that zerocopy is on GitHub now in case there are still zerocopy features that would be useful to you (like byteorder-aware types). If not, of course disregard this message 🙂

@Kerollmops
Copy link
Member Author

Thank you very much @joshlf,

Is there a page that compares both crates feature-wise? The only issue that I found is this one on bytemuck. I'll evaluate both crates.

Note that I will release a new version of heed with a lot of things rewritten and this choice between heed and bytemuck will be the last choice.

@joshlf
Copy link

joshlf commented Oct 27, 2022

Is there a page that compares both crates feature-wise? The only issue that I found is this one on bytemuck. I'll evaluate both crates.

At this point, we've both adopted features from each other, so it really depends on your specific use cases. If you can describe specific use cases, I can try to speak to what the two crates provide for those use cases (with the obvious bias that I know zerocopy a lot better than I know bytemuck), but I don't maintain a comparison page. For the current state of zerocopy, I'd check out our latest alpha release. For the future state, I'd check out our Q4 2022 roadmap (that roadmap is best effort, so definitely don't make any decisions that rely on that roadmap coming to fruition on schedule).

I'd also love to hear any feedback you have or feature requests! Even just "I want to do <vague, high-level description> and I'm not sure how" is very helpful.

@Kerollmops
Copy link
Member Author

Hey @joshlf,

I'm sorry for not getting back to you sooner. I just released a new version of heed v0.20.0-alpha.0 to let people test the version of the BytesEncode/BytesDecode traits backed by bytemuck. However, as we will also test it at Meilisearch inside milli, we will be able to ensure that it is correct and works great. If we find issues, we can roll back and use zerocopy or ask you for some advice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking A change that is breaking the semver good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants