Some of the most popular strategies for handling constraints in gradient-based optimization, namely: reparametrization, projected gradient, and mirror descent, while seemingly very different, are deeply connected. In this post, we will explore these connections. In particular, we show that mirror descent is equivalent to gradient descent on a reparametrized objective with straight-through…
It’s 2018, and your favorite meme pages on facebook constantly come up with quality 3d-post content. How can we mere researchers even begin to compete in terms of social media presence? What chance do we have at going viral? In this post, I show you how to generate 3-d renders of your or your friends’ cool machine learning research.
One of the many reasons why people are beautiful and fascinating is the way we change. I get reminded of how much I have changed everytime I see a photo of myself from high school, and this always triggers memories of how my opinions and attitudes have changed over the years. Some of the beliefs I held most strongly ended up being abandoned completely, while others have not changed much. …
In document classification and other natural language processing applications, having a good measure of the similarity of two texts can be a valuable building block. Ideally, such a measure would capture semantic information. Cosine similarity on bag-of-words vectors is known to do well in practice, but it inherently cannot capture when documents say the same thing in completely different words. …
TL ; DR ¶ I hacked the SocketIO client 0.9.16 to support specifying a special port ( wsport ) to use only for the WebSocket protocol, while keeping all other traffic on the default port. This is required by setups such as OpenShift which require WebSocket traffic to come over a different …
Rank aggregation is a problem with many important applications and naive approaches to it go wrong in subtle ways. Let’s say that your national Quidditch league is dominated by five major wizard sports newspapers. Yes, the ones with moving images and everything. Every week after the games, each of them publishes a ranking of the star players. For now, let’s suppose that the set of players under…
I finally got around to moving my entire website, including the blog, to Pelican . I probably would have gotten away with it too if it weren’t for those meddling kids who hacked my friend’s server and convinced me that it’s worth the effort to go static. It …
Hook ¶ Wouldn’t you like to manage your academic publications list easily within the context of your static website? Without resorting to external services, or to software like bibtex2html , which is very nice but will then require restyling to fit your templates? Look no more, with the help of pelican-bibtex …
A while back I ran into this blog post and it made me wonder. I’m not a native speaker but the idiomatic phrases that they note as common don’t strike me as such. I don’t think I have ever encountered them very often in real dialogue. The …
This summer, I was granted the project called scikit-learn-speed , consisting of developing a benchmarking platform for scikit-learn and using it to find potential speedups, and in the end, make the library go faster wherever I can. On the official closing day of this work, I’d like to take a …
The matrix inverse is a cornerstone of linear algebra, taught, along with its applications, since high school. The inverse of a matrix $latex A$, if it exists, is the matrix $latex A\^{-1}$ such that $latex AA \^{-1} = A\^{-1}A = I_n$. Based on the requirement that the left and …
I am pleased to announce that last night at 2:03 AM , the first fully automated run of the scikit-learn-speed test suite has run on our Jenkins instance! You can admire it at its temporary home for now. As soon as we verify that everything is good, we will move …
This post is about the scikit-learn benchmarking project that I am working on, called scikit-learn-speed . This is a continuous benchmarking suite that runs and generates HTML reports using Wes McKinney’s vbench framework, to which I had to make some (useful, I hope) additions. What it looks like now ¶ You …
EDIT : I made the plots a little more readable, check it out! Last time, I teased you with a screenshot of local output. Now, I will tease you with the benchmarks run on a couple of recent commits, along with some from earlier this year. After some effort and bugfixes …
The scikit-learn-speed project now has memory usage benchmarking! This was accomplished by building on what I described in my recent posts, specifically the extensions to Fabian’s [memory_profiler][] that you can find in my fork , but they will be merged upstream soon. The key element is the %magic_memit function whose …
In my last post I mentioned that I’m not satisfied with the current state of %memit , because some more complicated numerical function calls make it crash. I will start this post with a reminder of a pretty important bug: [On MacOS X (10.7 but maybe more), after forking …
Following up on my task to make it easier to benchmark memory usage in Python, I updated Fabian’s [memory_profiler][] to include a couple of useful IPython magics. While in my last post , I used the new IPython 0.13 syntax for defining magics, this time I used the backwards-compatible …
Everybody loves %timeit , there’s no doubt about it. So why not have something like that, but for measuring how much memory your line takes? Well, now you can; grab a hold of the script in the following gist and run it like in the example. [gist id=3022718] Instead …
It seems that I keep getting handed buggy code to install. These are cases of research software where the developers didn’t make the effort to make sure their tool works on the platforms it should. [ GLARF ][] (Grammatical and Logical Argument Representation Framework) is, in their words, “a typed feature …
MegaM is Hal Daumé III ’s maxent (logistic regression, and much more) modeling software written in OCaml. It is feature-packed and seems to be used a lot, despite being slightly dated. NLTK is able to use it. In order to compile it as of 2012, with the current version of …
To construct a vbench benchmark you need a setup string and a code string. The constructor’s signature is: Benchmark(self, code, setup, ncalls=None, repeat=3, cleanup=None, name=None, description=None, start_date=None, logy=False) . Why generate benchmarks dynamically? ¶ For most scikit-learn purposes, the code string will be …
With a slight delay caused by going to lovely lovely Istanbul for the LREC conference where I presented a poster , I am back to work on the Google Summer of Code project. By the way, this year’s logo and swag looks a lot nicer than last year’s, thank …
Anscombe’s quartet is a set of four toy datasets that look very different, but many of their statistics coincide. They were developed by Francis Anscombe as a striking visual to show that even for small datasets, blindly examining their statistical properties without considering their structure can mislead. Particularly, the …
This summer I hope to be able to put in another full-time amount of effort into scikit-learn. After a successful Google Summer of Code project last year on dictionary learning, I now plan to do some low-level work. The title of my proposal is: “Need for scikit-learn speed” and, in …
So I got my hands of the Google N-gram data for the Romanian language. It’s noisy as hell, has some other subtle issues too, but here’s the first thing I noticed: The Romanian word for coffee is cafea , and the more you crave it, the longer you pronunce …
I have been digging on the internet for the proof of this theorem for the last couple of days without success. The result was published by Sir Crispin Nash-Williams as Valency Sequences which force graphs to have Hamiltonian Circuits . Interim Rep, University of Waterloo Res Rep., Waterloo, Ontario, 1969. However …
Happy new year, friends! I’ve made a New Year’s resolution to build a better web presence and make better use of the domain that I previously only used for mail. This has prompted me to move my blog over to http://localhost:8001 which hopefully is shorter, better …
A while back, Bob L. Sturm blogged about a similar implementation of OMP to the one in scikit-learn. Instead of using the Cholesky decomposition like we did, his Matlab code uses the QR decomposition, to a similar (or maybe even identical) outcome, in theory. So lucky that Alejandro pointed out …
One year ago I had the chance to take a class on Monte Carlo simulation with prof. Ion Văduva, and my assignment for the class was to implement exactly what it says in the title of the blog post. I am going to walk you through the idea behind this …
Last week was marked by the international RANLP (Recent Advances in Natural Language Processing) conference, taking place in a nice spa in Hissar, Bulgaria. The excellent folks from the computational linguistics group at the University of Wolverhampton were behind it, together with the Institute of Information and Communication Technologies from …
Thanks to Olivier, Gaël and Alex, who reviewed the code heavily the last couple of days, and with apologies for my lack of activity during a sequence of conferences, Dictionary learning has officially been merged into scikit-learn master, and just in time for the new scikit-learn 0.9 release. Here …
Anybody reading my blog should have expected me to blog about the end of my GSoC. Sorry to disappoint, but I simply did not experience anything similar to an ending. On the contrary, I feel like things have barely started. Also, I apologize for one of the few posts here …
EDIT : There was a bug in the final version of the code presented here. It is fixed now, for its backstory, check out my blog post on it . When we last saw our hero, he was fighting with the dreaded implementation of least-angle regression, knowing full well that it was …
After intense code optimization work, my implementation of OMP finally beat least-angle regression! This was the primary issue discussed during the pull request, so once performance was taken care of, the code was ready for merge. Orthogonal matching pursuit is now available in scikits.learn as a sparse linear regression …
Since orthogonal matching pursuit ( OMP ) is an important part of signal processing and therefore crucial to the image processing aspect of dictionary learning, I am currently focusing on optimizing the OMP code and making sure it is stable. OMP is a forward method like least-angle regression, so it is natural …
I am happy to announce that the Sparse PCA code has been reviewed and merged into the main scikits.learn repository. You can use it if you install the bleeding edge scikits.learn git version, by first downloading the source code as explained in the user’s guide , and then …
[![Dictionary learned with K-Means on the LFW dataset with whitening PCA ][]][][![Dictionary learned with K-Means on the LFW dataset without whitening PCA ][]][] One of the simplest, and yet most heavily constrained form of matrix factorization, is vector quantization ( VQ ). Heavily used in image/video compression, the VQ problem is a …
I am presenting an image denoising example that fully runs under my local scikits-learn fork. Coming soon near you! The 400 square pixels area covering Lena’s face was distorted by additive gaussian noise with a standard deviation of 50 (pixel values are ranged 0-256.) [ ][] The dictionary contains 100 atoms …
Closing in on the goal of integrating J. Mairal’s dictionary learning in the scikit, I stitched together a couple of examples. The code is not yet integrated according to our standards, but here is the kind of results you can expect. Here is how a dictionary obtained from 8x8 …
After a little busy while, I have graduated and entered the summer vacation, which means time for serious GSoC work. [ ][] So we had a little conference in order to discuss what will be done and when. We gathered quite a few code snippets since the official start of the project …
I am working on implementing the Orthogonal Matching Pursuit ( OMP ) algorithm for the scikit. It is an elegant algorithm (that almost writes itself in Numpy!) to compute a greedy approximation to the solution of a sparse coding problem: $$ \text{argmin} \big|\big|\gamma\big|\big|_0 \text{ subject to }\big …
I have been working on the integration into the scikits.learn codebase of a sparse principal components analysis (SparsePCA) algorithm coded by Gaël and Alexandre and based on [[1]][]. Because the name “sparse PCA ” has some inherent ambiguity, I will describe in greater depth what problem we are actually solving …
Scikits.learn is a great general library, but machine learning has so many different application, that it is often very helpful to be able to extend its API to better integrate with your code. With scikits.learn, this is extremely easy to do using inheritance and using the pipeline module …
My GSoC proposal is titled “Dictionary learning in scikits.learn” and in the project, I plan to implement methods used in state of the art research and industry applications in signal and image processing. In this post, I want to clarify the terminology used. Usually the terms dictionary learning and …
I am sharing some Python code code that I wrote as a school assignment. This computes the Newton form of the interpolation polynomial of a given set of points, and allows for the evaluation of both the polynomial and its derivative, at a given point. This is an accurate way …
One of the problems we tackled here at my university is one as old as the modern Romanian language. It is a problem for linguists, as well as for foreigners trying to learn the language. We call it the root alternations problem. Similar to French and other languages, Romanian verbs …
When plotting multiple subplots using matplotlib, the axes rarely look pretty with the default configuration. Since matplotlib figures are abstract objects, designed for consistency in print as well as on screen, tweaking their layout can get tricky. An example ¶ The following code is taken from the face recognition example in …
Today, I spent more than two hours trying to figure out why, despite things working out fine in my development scikits.learn folder, python setup.py install would completely ignore the module I refactored into a subpackage. I imagined that simply adding it to the parent __init__.py __all__ attribute …
The fifth scikits.learn coding sprint took place Friday, April 1st 2011. For anyone who is not familiar with it, scikits.learn is a fast and easy to use machine learning toolkit for the pylab environment (Python, NumPy, SciPy, Matplotlib.) This was a good opportunity for me to get code …
My name is Vlad, I am a master’s student at the University of Bucharest, I work there at the Centre for Computational Linguistics, and I am a contributor to the Python machine learning library scikit-learn .