Over on the Let-All blog, Georgy Noarov and I wrote post on calibration through the lens of decision making. We think calibration has strong semantics as "trustworthiness", and that lots can be gained by designing uncertainty quantification for particular decision making tasks. You can read the post here:…
Our new paper gives very simple algorithms that promise "multivalid" conformal prediction sets for exchangable data. This means they are valid not just marginally, but also conditionally on (intersecting!) group membership, and in a threshold calibrated manner. I'll explain! Instead of making point predictions, we can quantify uncertainty by producing "prediction sets" --- sets of labels that…
This is a post about a new paper that is joint work with Bastani, Gupta, Jung, Noarov, and Ramalingam. The paper is here: https://arxiv.org/abs/2206.01067 and here is a recording of a recent talk I gave about it at the Simons Foundation: https://www.simonsfoundation.org/event/robust-and-equitable-uncertainty-estimation/ . This is cross-posted to the TOC4Fairness Blog (and this work comes out of…
Reminder to anyone who has forgotten about FORC 2021 --- its a very nice venue --- and also a nice place to highlight recent work that is published or submitted elsewhere, via the non-archival track. Symposium on Foundations of Responsible Computing (FORC) 2021 Call for Papers - Deadline February 15, 2021 AOE (anywhere on Earth) The second annual Symposium on Foundations of Responsible Computing…
This is a post about a new paper Online Multivalid Learning: Means, Moments, and Prediction Intervals , that is joint work with Varun Gupta, Christopher Jung, Georgy Noarov, and Mallesh Pai. It is cross-posted to the new TOC4Fairness blog . For those that prefer watching to reading, here is a recording of a talk I gave on this paper. Suppose you go and train the latest, greatest machine learning…
The existence of no-regret learning algorithms can be used to prove Von-Neumann's min-max theorem . This argument is originally due to Freund and Schapire , and I teach it to my undergraduates in my algorithmic game theory class. The min-max theorem also can be used to prove the existence of no-regret learning algorithms. Here is a constructive version of the argument (Constructive in that in the…
This blog post is about a new paper that I'm excited about, which is joint work with Chris Jung , Changhwa Lee , Mallesh Pai , and Ricky Vohra . If you prefer watching talks, you can watch one I gave to the Wharton statistics department here . Suppose you are diagnosed with hypertension, and your doctor recommends that you take a certain drug to lower your blood pressure. The latest research, she…
Reposting from here: https://thmatters.wordpress.com/2020/06/05/tcs-visioning-workshop-call-for-participation/ The CATCS will be hosting a virtual “Visioning Workshop” the week of July 20 in order to identify broad research themes within theoretical computer science (TCS) that have potential for a major impact in the future. The goals are similar to the workshop of the same name in 2008: to…
The FORC 2020 program is now available here: https://responsiblecomputing.org/program/ (and reproduced below) Note the terrific set of contributed talks at keynotes, and note that registration is free ! There will also be junior/senior breakout sessions into small groups during the lunch breaks to facilitate informal conversation and networking. Please consider attending, and help to spread the…
When we announced the new conference FORC (Foundations of Responsible Computing) we really had no idea what kind of papers folks would send us. Fortunately, we got a really high quality set of submissions, from which we have accepted the papers that will make up the program of the inaugural FORC. Check out the accepted papers here: https://responsiblecomputing.org/accepted-papers/
Can Game Theory Help Us Choose Among Fairness Constraints? This blog post is about a new paper , joint with Christopher Jung, Sampath Kannan, Changhwa Lee, Mallesh M. Pai, and Rakesh Vohra. A lot of the recent boom in interest in fairness in machine learning can be traced back to the 2016 Propublica article Machine Bias . To summarize what you will already know if you have interacted with the…
Here is the CFP: https://responsiblecomputing.org/forc-2020-call-for-paper/ FORC 2020: CALL FOR PAPERS Symposium on Foundations of Responsible Computing The Symposium on Foundations of Responsible Computing (FORC) is a forum for mathematical research in computation and society writ large. The Symposium aims to catalyze the formation of a community supportive of the application of theoretical…
This is a blog post about our new paper, which you can read here: https://arxiv.org/abs/1909.03577 The most basic statistical estimation task is estimating the expected value of some predicate $q$ over a distribution $\mathcal{P}$: $\mathrm{E}_{x \sim \mathcal{P}}[q(x)]$, which I'll just write as $q(\mathcal{P})$. Think about estimating the mean of some feature in your data, or the error rate of a…
Individual Notions of Fairness You Can Use Our group at Penn has been thinking about when individual notions of fairness might be practically achievable for awhile, and we have two new approaches. Background : Statistical Fairness I've written about this before, here . But briefly: there are two families of definitions in the fairness in machine learning literature. The first group of definitions,…
I've had the good fortune to be able work on a number of research topics so far: including privacy, fairness, algorithmic game theory, and adaptive data analysis, and the relationship between all of these things and machine learning. As an academic, we do a lot of writing about the things we work on, but usually our audience is narrow and technical: other researchers in our sub-specialty. But it…
One of the most striking results about fairness in machine learning is the impossibility result that Alexandra Chouldechova , and separately Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan discovered a few years ago. These papers say something very crisp. I'll focus here on the binary classification setting that Alex studies because it is much simpler. There are (at least) three…
Discussion of (un)fairness in machine learning hit mainstream political discourse this week, when Representative Alexandria Ocasio-Cortez discussed the possibility of algorithmic bias, and was clumsily "called out" by Ryan Saavedra on twitter: Socialist Rep. Alexandria Ocasio-Cortez (D-NY) claims that algorithms, which are driven by math, are racist pic.twitter.com/X2veVvAU1H — Ryan Saavedra…
Dear all, Please consider nominating graduating Ph.D. students for the SIGecom Dissertation Award. If you are a graduating student, consider asking your adviser or other senior mentor to nominate you. Nominations are due on February 28, 2019. This award is given to a student who defended a thesis in 2018. It is a prestigious award and is accompanied by a $1500 prize. In the past, the grand prize…
Dear all, Please consider nominating recently graduated Ph.D. students working in algorithmic game theory/mechanism design/market design for the SIGecom Dissertation Award. If you are a graduating student, consider asking your adviser or other senior mentor to nominate you. Nominations are due at the end of this month, March 31, 2018 . This award is given to a student who defended a thesis in…
I've been having fun recently reading about "AI Risk". There is lots of eloquent writing out there about this topic: I especially recommend Scott Alexander's Superintelligence FAQ for those looking for a fun read. The subject has reached the public consciousness, with high profile people like Stephen Hawking and Elon Musk speaking publicly about it. There is also an increasing amount of funding…
Bandit Problems "Bandit problems" are a common abstraction in machine learning. The name is supposed to evoke the image of slot machines, which are also known as "One-armed bandits" (or so I am told... Somehow nobody speaks like this in the circles I run in.) In the classic formulation, you imagine yourself standing in front of a bank of slot machines, each of which is different and might have a…
Adam Smith and I both taught a PhD seminar this semester (at BU and Penn respectively) on adaptive data analysis. We collaborated on the course notes, which can be found here: https://adaptivedataanalysis.com/lecture-schedule-and-notes/ As part of their final project, students will be writing lecture notes for papers that we didn't have time to cover (listed here:…
If you follow the fairness in machine learning literature, you might notice after awhile that there are two distinct families of fairness definitions: Statistical definitions of fairness (sometimes also called group fairness), and Individual notions of fairness. The statistical definitions are far-and-away the most popular, but the individual notions of fairness offer substantially stronger…
The SIGecom Doctoral Dissertation Award recognizes an outstanding dissertation in the field of economics and computer science. The award is conferred annually at the ACM Conference on Economics and Computation and includes a plaque, complimentary conference registration, and an honorarium of $1,500. A plaque may further be given to up to two runners-up. No award may be conferred if the nominations…
Jennifer Wortman Vaughan and I are the track chairs for the the "Web Economics, Monetization, and Online Markets" track of WWW 2018. The track name is a little unwieldy -- we tried to change it to "Economics and Markets" -- but the focus should be of interest to many in the AGT, theory, and machine learning communities. See the call for papers here:…