Updated formatting for ml-frameworks

[DivyaAmirtharaj]

No description provided.

[profvjreddi]

Great job ML frameowrks team on getting to the first draft stage! 👏 👏 👏

[BaeHenryS]

Leaving Comment to Test Co-Authorship

[sophiacho1]

Testing

[profvjreddi]

Great job on pulling this draft together @DivyaAmirtharaj @sophiacho1 @BaeHenryS and Emeka.

Let the feedback begin! 😄

[oishib]

This paragraph could be reorganized to be clearer and shorter. For example, these two sentences are very similar and could be condensed: "Machine learning frameworks have evolved significantly over time to meet the diverse needs of machine learning practitioners and advancements in AI techniques." and "Machine learning frameworks have evolved considerably over the past decade to meet the expanding needs of practitioners and rapid advances in deep learning techniques"

[oishib]

Are these citations properly linked to corresponding BibTex entries?

[profvjreddi]

I believe @DivyaAmirtharaj @sophiacho1 @BaeHenryS Emeka are still working on those.

[oishib]

It might help to add brief explanations of what a computational graph is, what you would be declaring upfront, whether "on-the-fly" means "during training" or "during inference" or just "during runtime," etc.

[alex-oesterling]

"Dynamic graphs
construct on-the-fly for more iterative development. But around 2016,
frameworks began adopting dynamic graphs like PyTorch and TensorFlow 2.0
which can construct graphs on-the-fly."

These sentences are difficult to read, I would suggest combining them + some tweaks: "But around 2016, frameworks such as Pytorch and TF 2.0 began adopting dynamic graphs which are constructed on-the-fly for more iterative development."

[profvjreddi]

Thanks for providing this suggestion on how to rework things. That makes things so much easier to integrate.

[ushabhalla]

This data loaders section is great, but I think it can be consolidated a bit as some parts are a bit repetitive and it was split up slightly un-intuitively. From what I can tell the main points are that dataloaders (1) read the data from disk or memory, (2) allow for parallel computation on GPU via batching, (3) allow for shuffling (and I think augmentation should be mentioned in this part even though it is the next section), and (4) cacheing for efficiency. I think that points 1 and 4 can be consolidated, as well as 2 and 3 to simply say that dataloaders handle the fetching and cacheing of the inputs to the model, and can be used to batch, shuffle, and augment data before sending it through the model.

[ushabhalla]

Weights and Biases may also be a good reference for this! W&B link

[profvjreddi]

That's a great suggestion, we should definitely add that.

[jared-ni]

Awesome text!

[jared-ni]

It could be great to illustrate this with the image of a common mathematical loss function and explain the different components. It would strengthen this paragraph and make it more informative (since we probably shouldn't assume too much prior knowledge in a potential all-in-one textbook, though the topic may seem simple to people in the field).

[BaeHenryS]

Got it. thank you!

[jared-ni]

It helps to illustrate the loss function and algorithms like gradient descent stochastic methods or learning rates with some equations. Again, I think this part is assuming too much prior knowledge, and those who are not familiar with the buzz words might not understand this paragraph.

[profvjreddi]

Good point.

There will be an AI training chapter that will be coming in before this chapter; we are working on it (just not within the class context), and our hope is that that chapter would have set up the necessary background that would then flow into this chapter. but good feedback though. Thank you.

[jared-ni]

Since the paragraph explains overfitting, wouldn't it also be helpful to touch on underfitting, and in what situations either case may be encountered? I think it would be informative to add examples with potential graphs to illustrate this point.

[jared-ni]

Computational graph might be familiar to people who are exposed to AI/ML, but it doesn't mean much to people who aren't experienced in the field. Since this textbook covers this topic, I think a graphical representation of a computational graph would be good to include here!

[BaeHenryS]

Discussed this earlier. Thanks!

[jared-ni]

Great paragraph on Federated Learning! I think a graph to illustrate how federated learning works might also be cool.

[jared-ni]

Great summary of the tradeoffs when using these frameworks!

[arnaumarin]

Overall the introduction is very clear providing a comprehensive overview of the ML frameworks, utilities and key features + it sets a clear scope for the chapter, which i find very useful in general.

The only thing I feel (perhaps is too personal) it lacks is a specific mention of the challenges of running ML models on resource-constrained devices, which would be crucial for a TinyML-focused book. For example, in the first paragraph you introduce the TF, torch, and "specialized frameworks for
embedded devices". Why don't you just mention Lite/Lite Micro and more here?
You guys might also want to add a sentence about how the constraints of TinyML (power, memory, compute) necessitate different framework capabilities or optimizations. (After this, I know you mention it with TF Lite, Lite Micro, etc.)

[profvjreddi]

This is one of the interesting things that I've been discovering about our notes here. And that is that when you start thinking about ML systems, be it tiny or otherwise, you still need to know the same set of fundamentals before you specialize in the domain of interest, such as tiny.

[AditiR-42]

It would be helpful to provide more explanation on the difference between declarative and imperative APIs. Consider giving an example of the actual function of a declarative API vs an imperative API instead of just citing TensorFlow and Pytorch.

[AditiR-42]

It would be helpful to have a diagram of batch processing here so the reader can visualize how batch processing works. For example: https://media.geeksforgeeks.org/wp-content/uploads/20221020142629/Batchoperatingsystem1-660x330.png

[AditiR-42]

It would be nice to add a more clear comparison between GPUs and TPUs here. What are the tradeoffs of using GPUs vs. TPUs and how does this change depending on use case?

[sjohri20]

Does Pytorch not fall under "low-level frameworks" according to the above definition? It is possible to define custom layers, loss functions, optimizing algorithms in pytorch

[sjohri20]

The following points can be added for challenges faced in federated learning.

Labeled Data is Not Always Naturally Available:

• In many real-world scenarios, data collected on devices might not come with appropriate labels.
• Additionally, users, being a source of data, can be unreliable, meaning that even if data is labeled, there's no guarantee of its accuracy or relevance.

Non-IID Data (covered already, but can be made clearer):

• Each user's data is unique, leading to significant variance in the kind of data generated by different users.
• The unbalanced nature of data, where some users produce more data than others can affect the global model's performance.

Massively Distributed:

• The number of mobile device owners can be significantly larger than the average number of training samples on each device. This leads to a significant communication overhead.

Limited Communication:

• Mobile networks, which are typically used for such communication, can be unstable. This instability can lead to delayed or failed transmission of model updates, affecting the overall training process.

Heterogeneous Device/Silo Resources:

• Devices participating in FL can have varying computational powers and memory capacities, making it challenging to design algorithms that are efficient across all devices.

Privacy, Security, and Trust:

• Malicious clients or servers can attempt to reverse-engineer model updates to infer information about local data.
• With malicious clients or servers in the system, there's a risk of model poisoning or other types of adversarial attacks.

[profvjreddi]

That's excellent feedback. Thanks for providing that.

[BaeHenryS]

Got it. Thanks!

[mpstewart1]

@BaeHenryS @DivyaAmirtharaj @sophiacho1, and Emeka, great job on the frameworks chapter—the content is great! There are still several items that I need your group to resolve before I can merge the pull request:

• Please provide links to all of the frameworks you mention, such as Pytorch, Tensorflow, CNTK, etc., when you first mention them.
• There are some citations missing, notably one for ResNet and GPT-3 towards the start of the chapter. For references, please provide a full bibtex reference and not just a link to the resource (i.e., include authors, year of publication, if it as a website, the date the resource was accessed, and so on).
• Please proofread the chapter, I still see some missing links such as items that say “We will discuss these concepts and details later on in Section XXX.”
• Run ‘quarto render’ in your local repo to make sure that the figures match the column width of the book chapter, and also make sure that these figures (and also tables) have captions and are explicitly referred to in the text at least once.
• If you use an acronym, please define in the chapter at the first opportunity. I see several occasions when TF is used but never defined—for us it is obvious from context but perhaps not for a non-expert.
• Images should not be copied directly into a folder, as some of these images are copyrighted. Instead, please link to the original website image directly within the chapter. Because of this, there is no need to provide an explicit source below the image. Also, please remove lines such as “Source: https://medium.com/mlait/tensors-representation-of-data-in-neural-networks-bbe8a711b93b” and link to these in the figure caption as bibtex references.
• Where code is referenced, please enclose these within double backticks (``) to make it easier for readers to see and interpret.
• Please ensure that all headings are labeled, including subheadings—there are some occasions such as “Data Augmentation” and “Data Loaders” that are not referenced as headings.
• Section 7.9 appears to be completely missing, so I am not sure if this is supposed to be there or not.
• There are some erroneous heading labels for sections 7.5, 7.7, 7,8, and 7.10 (e.g., 7.10.0.1, 7.5.4.0.2) that need to be fixed.
• The table in section 7.4.2.2 appears to be missing content.

Once these items have been resolved, I will rebase and merge the pull request.