Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add documentation, improve CI #1

Merged
merged 1 commit into from
Nov 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,41 @@ name: Test
on: [push, pull_request]

jobs:
rustfmt:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@nightly
with:
components: rustfmt
- run: cargo fmt --all -- --check

clippy:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v1
- uses: actions-rs/toolchain@v1
with:
toolchain: nightly
components: clippy
override: true
- uses: actions-rs/clippy-check@v1
with:
token: ${{ secrets.GITHUB_TOKEN }}
args: --all-features

doc-test:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- name: Run doctest
run: cargo test --doc

test:
runs-on: ubuntu-latest

Expand Down
4 changes: 2 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ wasmworker-proc-macro = { version = "0.1", path = "proc-macro" }

[package]
name = "wasmworker"
version = "0.1.0"
version = "0.1.1"
edition = "2021"

description.workspace = true
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ In contrast to many other libraries like [wasm-bindgen-rayon](https://github.com
- [Setting up](#setting-up)
- [Outsourcing tasks](#outsourcing-tasks)
- [WebWorker](#webworker)
- [WorkerPool](#workerpool)
- [WebWorkerPool](#webworkerpool)
- [Iterator extension](#iterator-extension)
- [Feature detection](#feature-detection)
- [FAQ](#faq)
Expand All @@ -32,7 +32,7 @@ Without the `serde` feature, only functions with the type `fn(Box<[u8]>) -> Box<
This is useful for users that do not want a direct serde dependency. Internally, the library always uses serde, though.

You can then start using the library without further setup.
If you plan on using the global `WorkerPool` (using the iterator extensions or `worker_pool()`), you can *optionally* configure this pool:
If you plan on using the global `WebWorkerPool` (using the iterator extensions or `worker_pool()`), you can *optionally* configure this pool:
```rust
// Importing it publicly will also expose the function on the JavaScript side.
// You can instantiate the pool both via Rust and JS.
Expand All @@ -49,7 +49,7 @@ async fn startup() {
### Outsourcing tasks
The library offers three ways of outsourcing function calls onto concurrent workers:
1. `WebWorker`: a single worker, to which tasks can be queued to.
2. `WorkerPool`: a pool of multiple workers, to which tasks are distributed.
2. `WebWorkerPool`: a pool of multiple workers, to which tasks are distributed.
3. `par_map`: an extension to regular iterators, which allows to execute a function on every element of the iterator in parallel using the default worker pool.

All approaches require the functions that should be executed to be annotated with the `#[webworker_fn]` macro.
Expand Down Expand Up @@ -90,7 +90,7 @@ let res = worker.run(webworker!(sort_vec), &VecType(vec![5, 2, 8])).await;
assert_eq!(res.0, vec![2, 5, 8]);
```

#### WorkerPool
#### WebWorkerPool
Most of the time, we probably want to schedule tasks to a pool of workers, though.
The default worker pool is instantiated on first use and can be configured using `init_worker_pool()` as described above.
It uses a round-robin scheduler (with the second option being a load based scheduler), a number of `navigator.hardwareConcurrency` separate workers, and the default inferred path.
Expand Down Expand Up @@ -128,7 +128,7 @@ let res: Vec<VecType> = some_vec.iter().par_map(webworker!(sort_vec)).await;

So far, this library has only been tested with `--target web`.
Other targets seem to generally be problematic in that the wasm glue is inaccessible or paths are not correct.
Both the `Worker` and `WorkerPool` have an option to set a custom path, which should make it possible to support other targets dynamically, though.
Both the `Worker` and `WebWorkerPool` have an option to set a custom path, which should make it possible to support other targets dynamically, though.

3. _Can I use bundlers?_

Expand Down
2 changes: 1 addition & 1 deletion demo/index.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
// Import required functions.
import init, { runWorker, runPool, runParMap } from "./pkg/webworker_demo.js";
import init, { runWorker, runPool, runParMap } from "./pkg/wasmworker_demo.js";

async function run_wasm() {
// Load wasm bindgen.
Expand Down
2 changes: 1 addition & 1 deletion proc-macro/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "wasmworker-proc-macro"
version = "0.1.0"
version = "0.1.1"
edition = "2021"

description.workspace = true
Expand Down
6 changes: 6 additions & 0 deletions src/convert.rs
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
use serde::{Deserialize, Serialize};

/// This wrapper function encapsules our internal serialization format.
/// It is used internally to prepare values before sending them to a worker
/// or back to the main thread via `postMessage`.
pub fn to_bytes<T: Serialize>(value: &T) -> Box<[u8]> {
postcard::to_allocvec(value)
.expect("WebWorker serialization failed")
.into()
}

/// This wrapper function encapsules our internal serialization format.
/// It is used internally to prepare values after receiving them from a worker
/// or the main thread via `postMessage`.
pub fn from_bytes<'de, T: Deserialize<'de>>(bytes: &'de [u8]) -> T {
postcard::from_bytes(bytes).expect("WebWorker deserialization failed")
}
9 changes: 9 additions & 0 deletions src/error.rs
Original file line number Diff line number Diff line change
@@ -1,14 +1,23 @@
use js_sys::wasm_bindgen::JsValue;
use thiserror::Error;

/// This error is returned when a web worker has been configured with a
/// maximum number of tasks to be queued and one of the `try_run` methods
/// is called.
#[derive(Debug, Error)]
#[error("WebWorker capacity reached")]
pub struct Full;

/// This error is returned during the creation of a new web worker.
/// It covers generic errors in the actual creation and import errors
/// during the initialization.
#[derive(Debug, Error)]
pub enum InitError {
/// This error covers errors during the `new Worker()` command.
#[error("WebWorker creation error: {0:?}")]
WebWorkerCreation(JsValue),
/// This error signals that the [`crate::WebWorker`] has been initialized with
/// an invalid path. The path should point to the glue file generated by wasm-bindgen.
#[error("WebWorker module loading error: {0:?}")]
WebWorkerModuleLoading(String),
}
27 changes: 27 additions & 0 deletions src/func.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
use std::marker::PhantomData;

/// This struct describes the function to be called by the worker.
/// It also ensures type safety, when constructed using the [`crate::webworker!`] macro.
pub struct WebWorkerFn<T, R> {
/// The name of the original function.
/// The worker will automatically add the `__webworker_` prefix.
pub(crate) name: &'static str,
_arg: PhantomData<T>,
_res: PhantomData<R>,
Expand All @@ -15,10 +19,20 @@ impl<T, R> Clone for WebWorkerFn<T, R> {
impl<T, R> Copy for WebWorkerFn<T, R> {}

impl<T, R> WebWorkerFn<T, R> {
/// Manually creates a [`WebWorkerFn`] object.
/// This function should be avoided in most cases as it does guarantee that the function
/// has the right type or is exposed to the worker.
///
/// Instead use the [`crate::webworker!`] macro to create an instance of this type.
pub fn new_unchecked(func_name: &'static str, _f: fn(T) -> R) -> Self {
Self::from_name_unchecked(func_name)
}

/// Manually creates a [`WebWorkerFn`] object from only the name of a function.
/// This function should be avoided in most cases as it does guarantee that the function
/// has the right type or is exposed to the worker.
///
/// Instead use the [`crate::webworker!`] macro to create an instance of this type.
pub fn from_name_unchecked(func_name: &'static str) -> Self {
Self {
name: func_name,
Expand All @@ -28,6 +42,19 @@ impl<T, R> WebWorkerFn<T, R> {
}
}

/// This macro safely instantiates a [`WebWorkerFn`] instance to be passed to a [`crate::WebWorker`].
/// It ensures that the function is exposed via the `#[webworker_fn]` procedural macro.
///
/// Example:
/// ```ignore
/// #[webworker_fn]
/// pub fn sort_vec(mut v: VecType) -> VecType {
/// v.0.sort();
/// v
/// }
///
/// let func: WebWorkerFn<VecType, VecType> = webworker!(sort_vec);
/// ```
#[macro_export]
macro_rules! webworker {
($name:ident) => {{
Expand Down
28 changes: 27 additions & 1 deletion src/global.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,28 @@

static WORKER_POOL: OnceCell<SendWrapper<WebWorkerPool>> = OnceCell::const_new();

#[wasm_bindgen]
/// This function can be called before the first use of the global worker pool to configure it.
/// It takes a [`WorkerPoolOptions`] configuration object. Note that this function is async.
///
/// ```ignore
/// # use wasmworker::{init_worker_pool, WorkerPoolOptions};
/// init_worker_pool(WorkerPoolOptions {
/// num_workers: Some(2),
/// ..Default::default()
/// }).await
/// ```
///
/// This function can also be called from JavaScript:
/// ```js
/// // Make sure to use the correct path.
/// import init, { initWorkerPool, WorkerPoolOptions } from "./pkg/wasmworker_demo.js";
///
/// await init();
/// let options = new WorkerPoolOptions();
/// options.num_workers = 3;
/// await initWorkerPool(options);
/// ```
#[wasm_bindgen(js_name = initWorkerPool)]

Check warning on line 30 in src/global.rs

View workflow job for this annotation

GitHub Actions / clippy

unexpected `cfg` condition name: `wasm_bindgen_unstable_test_coverage`

warning: unexpected `cfg` condition name: `wasm_bindgen_unstable_test_coverage` --> src/global.rs:30:1 | 30 | #[wasm_bindgen(js_name = initWorkerPool)] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | = help: expected names are: `clippy`, `debug_assertions`, `doc`, `docsrs`, `doctest`, `feature`, `fmt_debug`, `miri`, `overflow_checks`, `panic`, `proc_macro`, `relocation_model`, `rustfmt`, `sanitize`, `sanitizer_cfi_generalize_pointers`, `sanitizer_cfi_normalize_integers`, `target_abi`, `target_arch`, `target_endian`, `target_env`, `target_family`, `target_feature`, `target_has_atomic`, `target_has_atomic_equal_alignment`, `target_has_atomic_load_store`, `target_os`, `target_pointer_width`, `target_thread_local`, `target_vendor`, `test`, `ub_checks`, `unix`, and `windows` = help: consider using a Cargo feature instead = help: or consider adding in `Cargo.toml` the `check-cfg` lint config for the lint: [lints.rust] unexpected_cfgs = { level = "warn", check-cfg = ['cfg(wasm_bindgen_unstable_test_coverage)'] } = help: or consider adding `println!("cargo::rustc-check-cfg=cfg(wasm_bindgen_unstable_test_coverage)");` to the top of the `build.rs` = note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg/cargo-specifics.html> for more information about checking conditional configuration = note: `#[warn(unexpected_cfgs)]` on by default = note: this warning originates in the attribute macro `wasm_bindgen` (in Nightly builds, run with -Z macro-backtrace for more info)
pub async fn init_worker_pool(options: WorkerPoolOptions) {
WORKER_POOL
.get_or_init(|| async move {
Expand All @@ -19,6 +40,11 @@
.await;
}

/// This function accesses the default worker pool.
/// If [`init_worker_pool`] has not been manually called,
/// this function will initialize the worker pool prior to returning it.
///
/// It will use the options provided by [`WorkerPoolOptions::default()`].
pub async fn worker_pool() -> &'static WebWorkerPool {
WORKER_POOL
.get_or_init(|| async {
Expand Down
16 changes: 16 additions & 0 deletions src/iter_ext/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,27 @@ use serde::{Deserialize, Serialize};

use crate::{func::WebWorkerFn, worker_pool};

/// This extension trait defines the method [`IteratorExt::par_map`],
/// which will use the default [`crate::pool::WebWorkerPool`] as returned by [`worker_pool()`].
pub trait IteratorExt<T>: Sized + Iterator
where
Self::Item: Borrow<T>,
T: Serialize + for<'de> Deserialize<'de>,
{
/// The `par_map` function allows to parallelize a map operation on the default
/// [`crate::pool::WebWorkerPool`] as returned by [`worker_pool()`].
///
/// For each element of the iterator, a new task is scheduled on the worker pool.
/// Only functions that are annotated with the `#[webworker_fn]` macro can be used.
///
/// Example:
/// ```ignore
/// #[webworker_fn]
/// fn my_func(arg: T) -> R { /*...*/ }
///
/// let vec = vec![ /*...*/ ];
/// vec.iter().par_map(webworker!(my_func)).await
/// ```
#[allow(async_fn_in_trait)]
async fn par_map<R>(self, func: WebWorkerFn<T, R>) -> Vec<R>
where
Expand Down
Loading
Loading