Syllabus

Course Goals

Over the past two decades, psychology, medicine, and other social sciences have recognized a replication crisis in their work and taken measures to improve the quality of their science. Experimental computer science, which includes nearly all work in computing systems and machine learning, uses different forms of evaluation which do not usually incorporate the complexity of human subjects, but still suffers from a similar crisis. The experiments we do to determine if one approach to building a computing system is better than another often produce misleading or incorrect results, and it can take decades (if ever) for these results to be corrected. The goal of this seminar is to: (1) examine how computer scientists perform evaluations and why our methods are often problematic, (2) look at attempts to reproduce computer science results, and (3) conduct our own experiments and develop methods for improving computer science evaluation.

This will be a largely student-driven seminar, with students working together to analyze works in the literature and develop ideas for how to improve the scientific standards of evaluation.

Expected Background: This seminar is open to students at any level (including undergraduates) provided they (1) have sufficient background to be able to read research papers in computer science, (2) have enough experience with computing tools to be able to run and modify typical research-quality code, and (3) have enough interest and initiative to be able to contribute value to the seminar.

Admission to the seminar is by Instructor Permission, so if you are interested but unsure if you are well suited to join, please contact me.

Meetings: Mondays and Wednesdays, 11:00am-12:15pm in Rice Hall 340.

Coordinator: David Evans (Rice 507, evans@virginia.edu)

I will schedule office hours for the fall semester soon, but until then see my office page.

I won’t limit the topics of work we consider in this seminar to ones my research group focuses on, but you can learn more about my research from the Security Research Group web page.

Course Expectations

Students in the seminar are expected to:

• Lead discussions on interesting topics during the class meetings. For each week, there will be a team of students charged with preparing a topic and leading the discussion, and another team charged with writing a blog post about the class. Students responsible for posting the blog summary will be different from the ones charged with leading the topic discussion, but should work closely with the leaders on the posted write-up.

• Participate actively in class meetings. This means being prepared to contribute by doing the assigned preparation (which will typically involve reading a few research papers, but may involve other things also) and thinking about the materials deeply to be able to contribute well to discussions. Part of this preparation will involve responding in writing to questions posed before the class.

• Conduct a project over the course of the semester to explore one or a set of related important computing research papers and how their contributions were evaluated, as well as to perform their own evaluations to test the robustness and validity of previous evaluations. Based on discussions we’ll have in the seminar, this project may be done individually, in a small group, or in a large (full seminar) group.

Seminar Plan. Although the design of this seminar will be different from previous ones I have organized, there will be some aspects in common with previous seminars I have organized so you can get an idea what the seminar will be like from looking at these websites:

• Risks and Benefits of Generative AI and LLMs (Fall 2023)
• Security and Privacy of Machine Learning (Spring 2018)
• Understanding and Securing TLS (Spring 2017)
• Great Works in Computer Science(Spring 2013).

This seminar will be most like the Great Works in Computer Science in its breadth (although we may be looking more for Flawed Works in this seminar rather than Great ones, although there is substantial overlap between the two!), but more like Security and Privacy of Machine Learning and Risks and Benefits of Generative AI and LLMs in its format and expectations.

Readings

We will develop a list of readings as the seminar progresses. For now, here is an initial list of representative readings:

The Elusive Pursuit of Replicating PATE-GAN: Benchmarking, Auditing, Debugging
Georgi Ganev, Meenatchi Sundaram Muthu Selva Annamalai, Emiliano De Cristofaro

A Step Toward Quantifying Independently Reproducible Machine Learning Research
Edward Raff

“Get in Researchers; We’re Measuring Reproducibility”: A Reproducibility Study of Machine Learning Papers in Tier 1 Security Conferences
Daniel Olszewski, Allison Lu, Carson Stillman, Kevin Warren, Cole Kitroser, Alejandro Pascual, Divyajyoti Ukirde, Kevin Butler, and Patrick Traynor

Estimating the reproducibility of psychological science
Open Science Collaboration

Honor and Responsibility

We believe strongly in the value of a community of trust, and expect all of the students in this class to contribute to strenghtening and enhancing that community. The course will be better for everyone if everyone can assume everyone else is trustworthy. The seminar coordinator starts with the assumption that all students at the university deserve to be trusted.

We will be looking critically at other researchers’ work, and questioning strongly held assumptions, but students will be expected to do this in a way that is respectful of both the original researchers and others in the seminar.

Much of the work in the seminar will be done in small teams, and everyone is expected to contribute fairly and constructively. Because this is a graduate-level seminar, the coordinate does not expect it to be necessary to take proactive steps to ensure all students are contributing, but if it becomes apparent that there are any problems, will need to take appropriate measures which will be detrimental to everyone. If you are not able to work well in a team, or expect to free-ride on others’ work, please don’t enroll in the seminar.

Additional Information

This is similar to information that applies to most classes at UVA, and much of it is unlikely to be be relevant for a seminar like this one.

Special Circumstances: The University of Virginia strives to provide accessibility to all students. If you require an accommodation to fully access this course, please contact the Student Disability Access Center (SDAC) at (434) 243-5180 or sdac@virginia.edu. If you are unsure if you require an accommodation, or to learn more about their services, you may contact the SDAC at the number above or by visiting their website https://studenthealth.virginia.edu/sdac

Accommodations: It is the University’s long-standing policy and practice to reasonably accommodate students so that they do not experience an adverse academic consequence when serious personal issues conflict with academic requirements. Although University policy only recognizes religious accomodations, the course instructor believes they are many other valid reasons for accomdations that are at least as justifiable as ones for religious observance and consider family obligations, personal crises, and extraordinary opportunities to all be potentially valid reasons for accomodations. Students who wish to request accommodations should submit their request to Prof. Evans as far in advance as possible.

If you have questions or concerns about the University policy on academic accommodations for religious observance or religious beliefs, visit https://eocr.virginia.edu/accommodations-religious-observance or contact the University’s Office for Equal Opportunity and Civil Rights (EOCR) at UVAEOCR@virginia.edu or 434-924-3200.

Safe Environment: The University of Virginia is dedicated to providing a safe and equitable learning environment for all students. To that end, it is vital that you know two values that we and the University hold as critically important:

1. Power-based personal violence will not be tolerated.
2. Everyone has a responsibility to do their part to maintain a safe community on grounds (including in virtual environments).

If you or someone you know has been affected by power-based personal violence, more information can be found on the UVA Sexual Violence website that describes reporting options and resources available: https://www.virginia.edu/sexualviolence.

As your professor and as a human, know that I each care about you and your well-being and stand ready to provide support and resources as much as I can. As a faculty member, I am classified as a responsible employee, which means that I am required by University policy and federal law to report what you tell us to the University’s Title IX Coordinator. The Title IX Coordinator’s job is to ensure that the reporting student receives the resources and support that they need, while also reviewing the information presented to determine whether further action is necessary to ensure survivor safety and the safety of the University community. If you would rather keep this information confidential, there are Confidential Employees you can talk to on Grounds (see https://eocr.virginia.edu/chart-confidential-resources). The worst possible situation would be for you or your friend to remain silent when there are so many here willing and able to help.

Well-being: If you are feeling overwhelmed, stressed, or isolated, there are many individuals here who are ready and wanting to help. The Student Health Center offers Counseling and Psychological Services (CAPS) for all UVA students. Call 434-243-5150 (or 434-972-7004 for after hours and weekend crisis assistance) to get started and schedule an appointment. If you prefer to speak anonymously and confidentially over the phone, Madison House provides a HELP Line at any hour of any day: 434-295-8255.