diff --git a/_posts/2024-1-16-seminar.md b/_posts/2024-1-16-seminar.md index 53b6d74eef71f..c6ed3da188ae3 100644 --- a/_posts/2024-1-16-seminar.md +++ b/_posts/2024-1-16-seminar.md @@ -18,17 +18,17 @@ tags: - Office: Data and Decision Sciences 376 - Phone: (540) 231-4061 - Email: xuanw [at] vt [dot] edu -- Office Hours: TBD +- Office Hours: Monday 4:00 PM - 5:00 PM (by appointment) ## Course Description -Discover the realm of Large Language Models (LLMs) and their burgeoning applications, a domain that has garnered substantial attention, particularly in the wake of ChatGPT's release in November 2022. This course serves as a structured introduction to cutting-edge methodologies and tools aimed at bolstering the **trustworthiness of LLMs**, spanning both their theoretical underpinnings and practical implementations. Organized into three meticulously designed modules, Module I provides a deep dive into the foundational LLMOps stack, coupled with hands-on experience in constructing LLM applications using LlamaIndex, culminating in a rigorous evaluation of a Retrieval-Augmented Generation question-answering application. Module II unveils the pivotal roles LLMs play in critical domains such as healthcare, education, and security, fostering innovative project ideas through collaborative brainstorming sessions. In Module III, students delve into advanced LLM (app) evaluation methodologies, encompassing diverse topics including relevance, groundedness, confidence, calibration, uncertainty, explainability, privacy, fairness, toxicity, and adversarial attacks. Armed with these refined analytical skills, students embark on an immersive quarter-long course project, making this course a comprehensive and academically rigorous exploration of the LLM landscape. +Large Language Models (LLMs) and applications powered by them have recently received tremendous attention, especially since ChatGPT was released in November 2022. This course will provide an introduction to state-of-the-art methods and tools aimed at bolstering the **trustworthiness of LLMs**, spanning both their theoretical underpinnings and practical implementations. This course begins with an introduction to LLMs and a retrieval-augmented generation question-answering application. Various LLM trustworthiness metrics will be discussed in detail, encompassing diverse topics including relevance, groundedness, confidence, calibration, uncertainty, explainability, privacy, fairness, toxicity, and adversarial attacks. Applications in critical domains, such as healthcare, education, and security, will also be covered. Finally, students will work on a course project focusing on open research questions in trustworthy LLMs. ## Prerequisites Students should have experience with machine learning, data analytics, and deep learning. Strong programming skills in a high-level language such as Python, as well as frameworks for rapid deep learning prototyping, e.g., PyTorch, Tensorflow, Keras, etc., are essential for implementing and experimenting with the concepts covered in this course. While not mandatory, familiarity with natural language processing would be advantageous. ## Course Format -The course is a [role-playing paper reading seminar](https://colinraffel.com/blog/role-playing-seminar.html) that is structured around reading, presenting, and discussing weekly papers. Each class will involve the presentation and discussion of two papers. Each student will have a unique, rotating role per week. This role defines the lens through which each student reads the paper and determines what they prepare for the group in-class discussion. All students, irrespective of their role, are expected to have read the paper readings of each corresponding session before class and come to class ready to discuss. There will be no exams or traditional assignments. Instead, throughout the course, students will engage in practical hands-on projects and discussions to identify and work on open research questions on a variety of topics in trustworthy LLMs. +The course is a [role-playing paper reading seminar](https://colinraffel.com/blog/role-playing-seminar.html) that is structured around reading, presenting, and discussing weekly papers. Each class will involve the presentation and discussion of one paper. Each student will have a unique, rotating role per week. This role defines the lens through which each student reads the paper and determines what they prepare for the group in-class discussion. All students, irrespective of their role, are expected to have read the paper before each class and come to class ready to discuss. There will be no exams or traditional assignments. Instead, students will work on a course project focusing on open research questions in trustworthy LLMs. **Presentation Roles**: This seminar is organized around the different “roles” students play each week, that define the lens through which students read the paper. Students will be divided into two groups, one group presenting on Tuesdays and the other on Thursdays. In a given class session, students in the presenting groups will each be given a rotating role (described below): Presenter (two students), Reviewer, Archaeologist, Researcher, Industry Expert, and Blogger OR Hacker (pick one). Presenting groups should create a formal presentation, i.e., have slides prepared for the group in-class discussion. For each student in a presenting group, their assigned role determines what they should include in the slides. The Hacker and Blogger roles are the only exceptions to the rule. Hackers should provide a Jupyter Notebook instead of slides and Bloggers go over their written articles. @@ -57,7 +57,7 @@ Depending on changes in course enrollment, the roles might change, for example, **Final Project**: The main project goal is to engage students in research on trustworthy LLMs. In particular, students should try to extend papers from topics covered in class and present the research outcomes as a research paper, in a standard conference paper format. Students are encouraged to work in groups of no more than four members, taking into consideration that the work produced should be proportional to the number of members in a team. Groups are required to include a “contributions” section in the final project report, listing each member’s contributions in detail. Projects will be hosted on GitHub and should include a written report accompanied by a descriptive Jupyter Notebook, with a format similar to this notebook. In addition, groups will present their final projects during the last two class sessions. A PowerPoint or LaTex final presentation is required. -**Technology**: Piazza will be used for announcements, general questions, and discussions, etc. If you are unable to register to Piazza, please email me. Please familiarize yourself with GitHub, Zoom, LaTeX and paper writing practices. To enhance class participation, and unless restricted by low internet bandwidth, please try to keep your video turned on during class. Please keep your audio muted unless you would like to respond to an ongoing discussion or have a question. You can also use the “raise hand” option, type in the chatbox, or use the Zoom reactions for nonverbal feedback. Please remember that all in-class discussions should adhere to Virginia Tech’s Principles of Community. To keep track of student order during office hours, please type your name in the chat as soon as you enter the Zoom room. For one-on-one interactions with the instructor, please post a private note on Piazza or use Slack. +**Technology**: Piazza will be used for announcements, general questions, and discussions. If you are unable to register to Piazza, please email me. Please familiarize yourself with GitHub, Zoom, LaTeX and paper writing practices. To enhance class participation, and unless restricted by low internet bandwidth, please try to keep your video turned on during class. Please keep your audio muted unless you would like to respond to an ongoing discussion or have a question. You can also use the “raise hand” option, type in the chatbox, or use the Zoom reactions for nonverbal feedback. Please remember that all in-class discussions should adhere to Virginia Tech’s Principles of Community. To keep track of student order during office hours, please type your name in the chat as soon as you enter the Zoom room. For one-on-one interactions with the instructor, please post a private note on Piazza or use Slack. ## Grading Policy **Readings**: 60 points: Each student will be in the presenting role for 12 sessions and the non-presenting role for the remaining 12. You can earn up to 4 points each time you present (all presenting roles are considered equal). You will receive full credit if you do a thorough job of undertaking your role and present it in a clear and compelling way. When you aren’t presenting, you can earn up to 1 point by completing the non-presenting assignment and by participating in the class. At the end of the semester, extra credit of up to 3 points will be assigned to the most well-made presentation, blog, and notebook.