$\lim_{t \to \infty} V(s_t) \to 0$ so live accordingly
When I was coming up, conferences were a mature technology for disseminating high-quality scientific results at higher speed than journals. The combination of smaller units of work (“one paper, one idea”) and focused peer review were part of the key ingredients. Unfortunately the process is now broken. Arxiv halps fill the gap, but remains caveat emptor; and high quality journals are still slow.…
This slide is from Leon Bottou's amazing ICML 2015 keynote . You should contemplate this image on a regular basis. In the above slide Leon Bottou outlines three distinct, valid, and complementary ways of attempting to understand the world. In AI all of these approaches are used, but with different emphasis as trends change. Arguably, there was too much of the left hand side in the “if it's not…
Here's an example of a research question that started as a practical concern and ended up having science-fiction levels of potential. A Practical Question and Answer Contextual bandits have developed from research prototype to maturing industrial technology. CBs are practical because they incorporate the partial feedback nature of decision making in the real world, while sidestepping difficult…
This blog post is about improved off-policy contextual bandit learning via distributional robustness. I'll provide some theoretical background and also outline the implementation in vowpal wabbit . Some of this material is in a NeurIPS expo talk video , and additional material is in the accepted paper . Motivation In off-policy learning in contextual bandits our goal is to produce the best policy…
If you need to solve a convex optimization problem nowadays, you are in great shape. Any problem of the form $$ \begin{alignat}{2} &\!\inf_z & \qquad & f(z) \\ & \text{subject to} & & h(z) = 0 \\ & & & g(z) \preceq 0 \end{alignat} $$ where $f$ and $g$ are convex and $h$ is affine can be attacked by several excellent freely available software packages: my current favorite is cvxpy , which is a joy…
ICLR 2019 was reminiscent of the early NeurIPS days (sans skiing): a single track of talks, vibrant poster sessions, and a large mid-day break. The Tuesday morning talks were on climate change, modeling proteins, generating music, and modeling the visual cortex. Except for climate change, these were all hot topics at NeurIPS in the late 1990s. History doesn't repeat, but it does rhyme. My favorite…
Operations research (OR) is in the initial stages of a revolution driven by reinforcement learning (RL). When I was at eHarmony years ago, we used classical OR techniques to drive the matchmaking process. Machine learning played a critical role, but was limited to specfiying parameters to the OR solver. In essence, machine learning was to used to estimate the value function, and the OR solver then…
I'm often using positive-unlabeled learning nowadays. In particular for observational dialog modeling, next utterance classification is a standard technique for training and evaluating models. In this setup the observed continuation of the conversation is considered a positive (since a human said it, it is presumed a reasonable thing to say at that point in the conversation) and other randomly…
I only attended NIPS for the Conversation AI workshop , so my thoughts are limited to that. I really liked the subtitle of the workshop: "today's practice and tomorrow's potential." Since I'm on a product team trying to build chatbots that are actually effective, it struck me as exactly the right tone. Several presentations were related to the Alexa prize . When reading these papers, keep in mind…
ICML 2017 has just ended. While Sydney is remote for those in Europe and North America, the conference center is a wonderful venue (with good coffee!), and the city is a lot of fun. Everything went smoothly and the organizers did a great job. You can get a list of papers that I liked from my Twitter feed , so instead I'd like to discuss some broad themes I sensed. Multitask regularization to…
Recently Yoav Goldberg had a famous blog rant . I appreciate his concern, because the situation is game-theoretically dangerous: any individual researcher receives a benefit for aggressively positioning their work (as early as possible), but the field as a whole risks another AI winter as rhetoric and reality become increasingly divergent. Yoav's solution is to incorporate public shaming in order…
I'm on a product team now, and once again I find myself working on a tiered architecture: an “L1” model selects some candidates which are passed to an “L2” model which reranks and filters the candidates which are passed to an “L3”, etc. The motivation for this is typically computational, e.g., you can index a DSSM model pretty easily but indexing a BIDAF model…
I'm working on a task-oriented dialog product and things are going surprisingly well from a business standpoint. It turns out that existing techniques are sufficient to substitute some portion of commercial dialog interactions from human to machine mediated, with tremendous associated cost savings which exceed the cost of developing the automatic systems. Here's the thing that is puzzling: the…
Not all core concepts from software engineering translate into the machine learning universe. Here are some differences I've noticed. Divide and Conquer A key technique in software engineering is to break a problem down into simpler subproblems, solve those subproblems, and then compose them into a solution to the original problem. Arguably, this is the entire job, recursively applied until the…
What is the right way to specify a program that learns from experience? Existing general-purpose programming languages are designed to facilitate the specification of any piece of software. So we can just use these programming languages for reinforcement learning, right? Sort of. Abstractions matter An analogy with high performance serving might be helpful. An early influential page on high…
I've been integrating reinforcement learning into an actual product for the last 6 months, and therefore I'm developing an appreciation for what are likely to be common problems. In particular, I'm now sold on the idea of reinforcement learning as a service, of which the decision service from MSR-NY is an early example (limited to contextual bandits at the moment, but incorporating key system…
There was a really cute paper at the GAN workshop this year, Generating Text via Adversarial Training by Zhang, Gan, and Carin. In particular, they make a couple of unusual choices that appear important. (Warning: if you are not familiar with GANs, this post will not make a lot of sense.) They use a convolutional neural network (CNN) as a discriminator, rather than an RNN. In retrospect this seems…
I'm glad OpenAI exists: the more science, the better! Having said that, there was a strange happenstance at NIPS this year. OpenAI released OpenAI universe , which is their second big release of a platform for measuring and training counterfactual learning algorithms. This is the kind of behaviour you would expect from an organization which is promoting the general advancement of AI without…
Most of the speakers have sent me their slides, which can be found on the schedule page . Overall the workshop was fun and enlightening. Here are some major themes that I picked up upon. Evaluation There is no magic bullet, but check out Helen's slides for a nicely organized discussion of metrics. Many different strategies were on display in the workshop: Milica Gasic utilized crowdsourcing for…
It was a great conference. The organizers had to break with tradition to accommodate the rapid growth in submissions and attendance, but despite my nostalgia, I feel the changes were beneficial. In particular, leveraging parallel tracks and eliminating poster spotlights allowed for more presentations while ending the day before midnight, and the generous space allocation per poster really improved…
The schedule for the workshop has been finalized, and I'm pretty excited. We managed to convince some seasoned researchers in dialog, who don't normally attend NIPS, to give invited talks. We're also devoting some time to “Building Complete Systems”, because it's easy to focus on the trees instead of the forest, especially when the tree is something really interesting like a neural…
I'm co-organizing a workshop on dialogue at NIPS 2016 . NIPS is not a traditional forum for dialogue research, but there are increasing number of people (like myself!) in machine learning who are becoming interested in dialogue, so the time seemed right. From a personal perspective, dialogue is interesting because 1) it smells like AI, 2) recent advances in (deep learning) NLP techniques suggest…
In a recent blog post I discussed two ideas for moving dialogue forward; both ideas are related to the need to democratize access to the data required to evaluate a dialog system. It turns out both ideas have already been advanced to some degree: Having computers “talk” to each other instead of with people : Marco Beroni is on it . Creating an open platform for online assessment :…
ICML is too big for me to ``review'' it per se, but I can provide a myopic perspective. The heavy hitting topics were Deep Learning, Reinforcement Learning, and Optimization; but there was a heavy tail of topics receiving attention. It felt like deep learning was less dominant this year; but the success of deep learning has led to multiple application specific alternative venues (e.g., CVPR,…
In machine learning, assessment isn't everything: it's the only thing. That's the lesson from Imagenet (a labeled data set) and the Arcade Learning Environment (a simulation environment). A simulator is the partial feedback analog of a labeled data set: something that lets any researcher assess the value of any policy. Like data sets, when simulators are publicly available and the associated task…