diff --git a/publications/publications.rst b/publications/publications.rst index 21e6d3c..ae05748 100644 --- a/publications/publications.rst +++ b/publications/publications.rst @@ -10,6 +10,7 @@ We are working towards making this list complete, meanwhile please see Animesh's Krijn Doekemeijer, Nick Tehrany, Balakrishnan Chandrasekaran, Matias Bjørling, Animesh Trivedi (2023) 2023 IEEE International Conference on Cluster Computing (`CLUSTER `_). :download:`2023-cluster-zns-performance-kdoekemeijer.pdf `. + * test * **Performance Characterization of Modern Storage Stacks: POSIX I/O, libaio, SPDK, and io_uring**, Zebin Ren and Animesh Trivedi, In 3rd Workshop on Challenges and Opportunities of Efficient and Performant Storage Systems (`CHEOPS'23 `_), May 8, 2023, Rome, Italy. diff --git a/teaching/photos/storage-memory-continuum.png b/teaching/photos/storage-memory-continuum.png new file mode 100644 index 0000000..a1bccca Binary files /dev/null and b/teaching/photos/storage-memory-continuum.png differ diff --git a/teaching/slides/stosys/2023-2024-stosys-handbook-4.0.pdf b/teaching/slides/stosys/2023-2024-stosys-handbook-4.0.pdf new file mode 100644 index 0000000..ffb31fd Binary files /dev/null and b/teaching/slides/stosys/2023-2024-stosys-handbook-4.0.pdf differ diff --git a/teaching/slides/stosys/2023-stosys-lecture1-introduction.pdf b/teaching/slides/stosys/2023-stosys-lecture1-introduction.pdf new file mode 100644 index 0000000..212bf50 Binary files /dev/null and b/teaching/slides/stosys/2023-stosys-lecture1-introduction.pdf differ diff --git a/teaching/slides/stosys/2023-stosys-lecture10-ds2.pdf b/teaching/slides/stosys/2023-stosys-lecture10-ds2.pdf new file mode 100644 index 0000000..3ecd721 Binary files /dev/null and b/teaching/slides/stosys/2023-stosys-lecture10-ds2.pdf differ diff --git a/teaching/slides/stosys/2023-stosys-lecture11-emerging-topics.pdf b/teaching/slides/stosys/2023-stosys-lecture11-emerging-topics.pdf new file mode 100644 index 0000000..fc1a3dd Binary files /dev/null and b/teaching/slides/stosys/2023-stosys-lecture11-emerging-topics.pdf differ diff --git a/teaching/slides/stosys/2023-stosys-lecture2-sw-nvme-sw-implications.pdf b/teaching/slides/stosys/2023-stosys-lecture2-sw-nvme-sw-implications.pdf new file mode 100644 index 0000000..6ab6e9a Binary files /dev/null and b/teaching/slides/stosys/2023-stosys-lecture2-sw-nvme-sw-implications.pdf differ diff --git a/teaching/slides/stosys/2023-stosys-lecture3-ftl-gc.pdf b/teaching/slides/stosys/2023-stosys-lecture3-ftl-gc.pdf new file mode 100644 index 0000000..9f70fd4 Binary files /dev/null and b/teaching/slides/stosys/2023-stosys-lecture3-ftl-gc.pdf differ diff --git a/teaching/slides/stosys/2023-stosys-lecture4-fs.pdf b/teaching/slides/stosys/2023-stosys-lecture4-fs.pdf new file mode 100644 index 0000000..bac6650 Binary files /dev/null and b/teaching/slides/stosys/2023-stosys-lecture4-fs.pdf differ diff --git a/teaching/slides/stosys/2023-stosys-lecture5-kv.pdf b/teaching/slides/stosys/2023-stosys-lecture5-kv.pdf new file mode 100644 index 0000000..305632a Binary files /dev/null and b/teaching/slides/stosys/2023-stosys-lecture5-kv.pdf differ diff --git a/teaching/slides/stosys/2023-stosys-lecture6-pmem.pdf b/teaching/slides/stosys/2023-stosys-lecture6-pmem.pdf new file mode 100644 index 0000000..5df8677 Binary files /dev/null and b/teaching/slides/stosys/2023-stosys-lecture6-pmem.pdf differ diff --git a/teaching/slides/stosys/2023-stosys-lecture7-networked-flash.pdf b/teaching/slides/stosys/2023-stosys-lecture7-networked-flash.pdf new file mode 100644 index 0000000..527be2e Binary files /dev/null and b/teaching/slides/stosys/2023-stosys-lecture7-networked-flash.pdf differ diff --git a/teaching/slides/stosys/2023-stosys-lecture8-programmable-storage.pdf b/teaching/slides/stosys/2023-stosys-lecture8-programmable-storage.pdf new file mode 100644 index 0000000..3e03e14 Binary files /dev/null and b/teaching/slides/stosys/2023-stosys-lecture8-programmable-storage.pdf differ diff --git a/teaching/slides/stosys/2023-stosys-lecture9-ds1.pdf b/teaching/slides/stosys/2023-stosys-lecture9-ds1.pdf new file mode 100644 index 0000000..6ff3d9e Binary files /dev/null and b/teaching/slides/stosys/2023-stosys-lecture9-ds1.pdf differ diff --git a/teaching/storage_systems.rst b/teaching/storage_systems.rst index 0d6e209..4e970d2 100644 --- a/teaching/storage_systems.rst +++ b/teaching/storage_systems.rst @@ -1,6 +1,103 @@ -Storage Systems +Storage Systems (XM_0092) Homepage ============== -`Course webpage `_ +Storage System is a unique course due to its sole focus on NVM storage and its impact on research and education. We take inspiration from the 2018 **Data Storage Research Vision 2025** report (https://dl.acm.org/doi/book/10.5555/3316807) which identifies (see section 6.1): -Contents to be added. +*"Many students may only associate storage systems with hard disk drives or a specific file system, which is obviously less attractive compared to, say, self-driving cars. This situation is partly due to the fact that there is no clearly defined course on storage systems in the majority of universities."* + +.. figure:: /teaching/photos/storage-memory-continuum.png + :alt: Group Photo + :width: 600 + :align: center + + The new Storage-Memory Continuum that this course covers. + + +Lecture slides +---------------------------- + +Storage Systems (VU catalogue number XM_0092) is a MSc-level course that is first +established and offered in 2020. The course covers the rise of Non-Volatile Memory (NVM) +storage technologies in commodity computing, their impact on system design (architecture, operating system), +distributed systems, storage services, application designs, and emerging trends (CXL and io_uring). +We cover the following topics in 2023: + +* **1. Introduction:** History, HDDs, NAND flash, internal organization, the new triangle of Storage Hierarchy. + + * :download:`2023-stosys-lecture1-introduction.pdf `. + * `Zoom recording `_ (Passcode: S6qD8tE^) + +* **2. Host Interfacing and Software Implications:** NVMe, storage and block-layer optimizations (multi queue design). + + * :download:`2023-stosys-lecture2-sw-nvme-sw-implications.pdf `. + * `Zoom recording `_ (Passcode: S6qD8tE^) + +* **3. Flash FTL and Garbage Collection:** FTL and GC designs, concerns, and host-managed FTLs. + + * :download:`2023-stosys-lecture3-ftl-gc.pdf `. + * `Zoom recording `_ (Passcode: S6qD8tE^) + +* **4. Flash Filesystems:** Log-structured file systems, F2FS, DFS, and Nameless writes. + + * :download:`2023-stosys-lecture4-fs.pdf `. + * `Zoom recording `_ (Passcode: S6qD8tE^) + +* **5. Flash KV Stores:** B+ Trees, Hash Tables, and LSM trees on flash (LOCS, WiscKey, uTree, SILK). + + * :download:`2023-stosys-lecture5-kv.pdf `. + * `Zoom recording `_ (Passcode: S6qD8tE^) + +* **6. Byte-addressable Persistent Memories:** Optane, NVHeap, and Pmem/PMDK project. + + * :download:`2023-stosys-lecture6-pmem.pdf `. + * `Zoom recording `_ (Passcode: S6qD8tE^) + +* **7. Networked Flash:** Disaggregated storage, NVMoF, Disaggregated Flash, and FlashNet. + + * :download:`2023-stosys-lecture7-networked-flash.pdf `. + * `Zoom recording `_ (Passcode: S6qD8tE^) + +* **8. Programmable Storage:** What is CSD, Willow, Biscuit, INSIDER. + + * :download:`2023-stosys-lecture8-programmable-storage.pdf `. + * `Zoom recording `_ (Passcode: S6qD8tE^) + +* **9. Distributed Storage - I:** Distributed temporary data storage and formats (Crail and Albis). + + * :download:`2023-stosys-lecture9-ds1.pdf `. + * `Zoom recording `_ (Passcode: S6qD8tE^) + +* **10. Distributed Storage - II:** Talks about Corfu and Tango distributed transaction systems. + + * :download:`2023-stosys-lecture10-ds2.pdf `. + * `Zoom recording `_ (Passcode: S6qD8tE^) + +* **11. Emerging Topics:** CXL and io_uring + + * :download:`2023-stosys-lecture11-emerging-topics.pdf `. + * `Zoom recording `_ (Passcode: S6qD8tE^) + + +Practical Work +---------------------------- + +For the practical work, students develop an NVM flash translation layer FTL (the essential part of any modern NVM storage device) for NVMe ZNS devices, and integrate a file system in RocksDB. There are five milestone in the practical work: + + 1. **A new device is in town** - setup the development environment with ZNS devices in QEMU and read the NVMe 1.4 and ZNS specifications, and test the nvme command to interact with nvme devices. + 2. **I can’t read, is there a translator here?** - implement a host-side hybrid log-data FTL. The log segment is page-mapped, while the data-segment is zone-mapped. No GC at this stage. + 3. **It’s 2023, we recycle** - implement a choice of garbage collection algorithm for your FTL. + 4. **We love Rock(sDB) ‘n’ Roll!** - design and implement a file system on top of your FTL and integrate it with the RocksDB FileSystem API. + 5. **Wake up, Neo** - the last milestone requires you to persist and restart your FTL and filesystem states and pass the RocksDB persistency tests. + +The project handbook is publicly available here :download:`2023-2024-stosys-handbook-4.0.pdf `. Drop me an email if you want access to more project related resources. + +License +---------------------------- +This course content are distributed under the Creative Commons Attribution 4.0 International +(CC BY 4.0): https://creativecommons.org/licenses/by/4.0/. + +Feel free to modify and use the slides in your course as you see fit with attribution. + +Acknowledgement +---------------------------- +The project work is generously supported by Western Digital with their donation of ZNS devices and software support. \ No newline at end of file