Ramblings about Machine Learning, Python and scikit-learn.
I’ve been happy to see an increase in funding for open source software across research areas and across funding bodies. However, I observed that a majority of funding from, say, the NSF, goes to projects that do not exist yet, and where the funding is supposed to create a new project, or to extend projects that are developed and used within a single research lab. I think this top-down approach to…
Tldr; You probably shouldn’t be citing the "No Free Lunch" Theorem by Wolpert . If you’ve cited it somewhere, you might have used it to support the wrong conclusion. What it actually (vaguely) says is “You can’t learn from data without making assumptions”. The paper on the “No Free Lunch Theorem” , actually called " The Lack of A Priori Distinctions Between Learning Algorithms " is one of these…
As the title suggests, this is a non-machine-learning, non-vision, non-python post *gasp*. Some people in my network posted about spritz a startup that recently went out of stealth-mode. They do a pretty cool app for speed reading. See this huffington post article for a quick demo and explanation. They say they are still in development, so the app is not available for the public. The app seemed…
Yesterday a week-long scikit-learn coding sprint in Paris ended. And let me just say: a week is pretty long for a sprint. I think most of us were pretty exhausted in the end. But we put together a release candidate for 0.14 that Gael Varoquaux tagged last night. You can install it via: pip install -U https://github.com/scikit-learn/scikit-learn/archive/0.14a1.zip There are also tarballs on github…
The ICML is now already over for two weeks, but I still wanted to write about my reading list, as there have been some quite interesting papers ( the proceedings are here ). Also, I haven't blogged in ages, for which I really have no excuse ;) There are three topics that I am particularly interested in, which got a lot of attention at this years ICML: Neural networks, feature expansion and kernel…
Some time ago I wrote about a structured learning project I have been working on for some time, called pystruct . After not working on it for some time, I think it has come quite a long way the last couple of weeks as I picked up work on structured SVMs again. So here is a quick update on what you can do with it. To the best of my knowledge this is the only tool with ready-to-use functionality to…
As you hopefully have heard, we at scikit-learn are doing a user survey (which is still open by the way). One of the requests there was to provide some sort of flow chart on how to do machine learning. As this is clearly impossible, I went to work straight away. This is the result: [edit2] clarification: With ensemble classifiers and ensemble regressors I mean random forests , extremely randomized…
After a little delay, the team finished work on the 0.13 release of scikit-learn. There is also a user survey that we launched in parallel with the release, to get some feedback from our users. There is a list of changes and new features on the website . You can upgrade using easy-install or pip using: pip install -U scikit-learn or easy_install -u scikit-learn There were more than 60 people…
Recently we added another method for kernel approximation, the Nyström method, to scikit-learn , which will be featured in the upcoming 0.13 release. Kernel-approximations were my first somewhat bigger contribution to scikit-learn and I have been thinking about them for a while. To dive into kernel approximations, first recall the kernel-trick . The Kernel Trick The motivation is to obtain a…
I'm a bit obsessed with MNIST. Mainly because I think it should not be used in any papers any more - it is weird for a lot of reasons. When preparing the workshop we held yesterday I noticed one that I wasn't aware of yet: most of the 1-vs-1 subproblems, are really easy! Basically all pairs of numbers can be separated perfectly using a linear classifier! And even you you just do a PCA to two…
Today there was a workshop at my uni, organized by my Professor Sven Behnke, together with my colleagues Hannes Schulz, Nenard Birešev and me. The target group was a local graduate school with a general scientific background, but not much CS or machine learning. The workshop consisted of us explaining the methods and the students then playing around with them and answering some questions using…
Last week I was at Pycon DE , the German Python conference. After hacking on scikit-learn a lot last week, I decided to to something different on my way back, that I had planned for quite a while: doing a wordl -like word cloud . I know, word clouds are a bit out of style but I kind of like them any way. My motivation to think about word clouds was that I thought these could be combined with…
Recently Jake showed some pretty cool videos in his blog . This inspired me to go back to an idea I had some time ago, about visualizing high-dimensional data via random projections. I love to do exploratory data analysis with scikit-learn , using the manifold , decomposition and clustering module. But in the end, I can only look at two (or three) dimensions. And I really like to see what I am…
Recently I entered my first kaggle competition - for those who don't know it, it is a site running machine learning competitions. A data set and time frame is provided and the best submission gets a money prize, often something between 5000$ and 50000$. I found the approach quite interesting and could definitely use a new laptop, so I entered Detecting Insults in Social Commentary. My weapon of…
Last night I uploaded the new version 0.12 of scikit-learn to pypi . Also the updated website is up and running and development now starts towards 0.13 . The new release has some nifty new features ( see whatsnew ): * Multidimensional scaling * Multi-Output random forests ( like these ) * Multi-task Lasso * More loss functions for ensemble methods and SGD * Better text feature extraction…
Recently some segmentation and superpixel algorithms I implemented were merged into scikits-image . You can see the example here . I reimplemented Felzenszwalb's fast graph based method , quickshift and SLIC . The goal was to have easy access to some successful methods to make comparison easier and encourage experimenting with the algorithms. Here is a a comparison of my implementations against…
This is a short post about an ECCV 2012 paper I just discovered . The paper is Statistics of Patch Off sets for Image Completion by Kaiming He and Jian Sun from Microsoft Asia. There was a recent talk at MSRC about PatchMatch by Dan Goldman . PatchMatch is a simple but very efficient image completion algorithm that is used in Photoshop. The page linked above contains a beautiful illustration of…
At the moment I have the pleasure to be at MSRC , working under the supervision of Carsten Rother and Sebastian Nowozin . We are tackling some low-level vision tasks (as in their recent CVPR paper ) and in this context, filter banks are very useful. They might also be useful for object detection, since one Gabor rules them all , and Google uses collections of Gabor filters for his image retrieval.…
I just pushed an update for my structured SVM in Python. This contains a bugfix in the dual formulation and a subgradient descent version of the structured SVM. The new version now has some options to be really verbose, track the slacks and constraints of all examples and measure the primal objective. Also, quite handy for approximate inference, it complains when the slack of the "most violated…
[EDIT: If you are reading this now, have a look at pystruct.github.io . The project matured quit a bit in the meantime.] Today I pushed some of my code to github that I use for experimenting with CRF learning. This goes along the lines of my recent posts on graphcut and I hope to post a full CRF learning framework for semantic image segmentation soon. This is a pretty standard setup in computer…
I just pushed some of my structured learning code to github and hope that some people might find it useful. Before describing my code here, I wanted to give a basic intro into structured prediction. I hope I can at least convey some intuition for this vast research area. So here goes... What is structured learning and prediction? Structured prediction is a generalization of the standard paradigms…
I have been using the excellent gco library for energy minimization with graph cuts for quite some time. Finally I got around to clean up / rewrite some of my Python wrappers so that maybe someone else can use them, too. So what does this library do? It does discrete energy minimization on loopy graphs. This is an important topic in computer vision, since it can be used for segmentation, stereo…
The ICML2012 accepted papers are officially online . On twitter , Andrej Kaparthy complained that the list is a bit hard to browse through. I agree and even though this is probably not the nice visualization he had in mind, I felt like having topical reading lists would somehow mitigate this problem. Here is my reading list on deep learning and unsupervised feature extraction: A Generative Process…
Yesterday I wanted to try out a "new" superpixel algorithm that seemed quite successful: SLIC superpixels . This is actually a very simple algorithm, basically doing KMeans in the color+(x,y) space. I'm a bit bummed that they named that, since I already tried the same approach a couple of years ago and didn't think it was very useful. Well, apparently it is. The authors have a nice website with…
More than once I came across the problem of rearranging a nested list. I had a nested list of the form X = [['a', 'b', 'c'], ['d', 'e', 'f']] And I want Y = [['a', 'd'], ['b', 'e'], ['c', 'f']] without having to resort to an ugly list comprehension over 3 lines. A friend told me to use Y = zip(*X) So easy! Kind of obvious but I didn't find it on the web. So I thought I'd write it down. Enjoy!