Roadmap for pieces

[prrao87]

@sanders41 I thought it made sense to highlight the upcoming pieces that we can each focus on, to revisit over time once we've both worked on them.

I have the following table in mind, and am happy to have any additions/deletions as you see fit (especially for the error-handling, CLI and database categories. I don't think it makes sense to focus on PyO3 until we have at least one of every category before that.

Piece	Category	New Rust concepts
Hello world	Intro	macros
Data structures & constructs	Intro	crates, structs, traits, implementations
Simple CSV parsing	File-handling	serde, vec
Regex JSON	File-handling	match, regex
Mock data generation	File-handling	RNG, sampling
Age grouping	File-handling	enums
Datetime parsing	File-handling	chrono, lifetimes
Capturing pronouns in text	Parallelism	rayon, parallelism
Polars datetimes	DataFrames	datetimes
Polars EDA	DataFrames	TBD
Postgres	Databases	async, sqlx, tokio
Meilisearch	Databases, CLI	async, futures
REST API to Postgres	APIs	axum, async, tokio
REST API to local LLM	APIs	axum, LLMs
PyO3 mock data generation	Unification	TBD
PyO3 query local LLM	Unification	TBD

Focus

I think the main thing to keep in mind is that all pieces, when combined, cover major concepts that one might want to use Rust or Python for.

• Sync/async APIs and databases
  • I see that the Meilisearch Rust client's docs state that it supports simple block on futures as well as tokio - given that we already cover tokio in sqlx, it might make sense to explore another async approach for this one
• Error handling (where Rust is refreshingly different)
• File-handling, parsing and data transformation
• Data analysis and DataFrames (polars)

I feel like we have a very good coverage already, and the net total is a fair amount of work 😅.

Could you maybe take a look at refining the task definitions and concepts covered for CLIs and error-handling? Depending on the ideas you have for error-handling (including anyhow), it could be organized into any of file-handling, databases or CLIs, whatever you think works. A couple of different kinds of CLIs would also be beneficial, if there are significant enough differences between them to illustrate.

Looking forward to hearing your thoughts!

[sanders41]

For exceptions it might be good to do a simple example early on. When I first started with Rust I would frequently try to use the ? operator on functions that didn't return a Result. Coming from Python where you can raise an exception anywhere I didn't think about having to returns a Result to do this. It's a really simple concept, but if you don't know you don't know 😄. Then from there, where it makes since, we could look at moving the concepts into the other pieces. Returning a Result rather than unwrap for example since this is what you would want to do with a real application.

For Meilisearch, the Rust client needs some love, but I think it will do everything we need 😂. That said it was designed to work with different async runtimes. block_on is used in the tests to keep things sync so I don't think you would want to use is for async. An alternative to tokio that I here a lot about but have never used is async_std. I think the client would work with it also so that may be worth exploring.

I'll work on coming up with some CLI ideas.

[sanders41]

Looking through the pieces, another good place to introduce errors could be Simple CSV parsing. That one comes early on and has several places expect is used that could be replaced with returning Result.

[prrao87]

yes! I agree. I realize now that that's where we first bring up the idea of error handling.

[prrao87]

Returning a Result rather than unwrap for example since this is what you would want to do with a real application.

Totally agree, but I think for the earlier pieces like simple_csv, we should make it clear that unwrap and expect are ways to quickly get prototype code up and running, and they serve as a useful marker to actually return Result and Error types in any production application.

And then maybe we could clean up one of the later pieces (like regex_json) to manage errors using external crates and go deeper into the different ways we can handle Result, Option, etc.

[prrao87]

An alternative to tokio that I here a lot about but have never used is async_std.

I'd love to see how async_std works, too. I know even sqlx supports it, but since we used tokio for that, we could look at the async_std API for Meilisearch if it's supported.

[sanders41]

Totally agree, but I think for the earlier pieces like simple_csv, we should make it clear that unwrap and expect are ways to quickly get prototype code up and running, and they serve as a useful marker to actually return Result and Error types in any production application.

And then maybe we could clean up one of the later pieces (like regex_json) to manage errors using external crates and go deeper into the different ways we can handle Result, Option, etc.

My thoughts are going the same way. I have some ideas I’ll work on for this.