Intro #

So, I’m ready to announce that I have finished working on a new Ruby parser. It’s called lib-ruby-parser.

Key features:

1. It’s fast. It’s written in Rust and it’s slightly faster than Ripper. The difference is about 1-2% on my machine.
2. It has a beautiful interface. Every single node has its own type that is documented. For example, take a look at CSend node that represents “conditional send” like foo&.bar. Here’s a list of all defined nodes . Both Ripper and RubyVM::AST have no documentation of their AST format. whitequark/parser has a great documentation , but its AST is not “static”.
3. What’s “static AST”? By saying that I mean that if documentation says that “N is not-nullable” then it’s true no matter what. whitequark/parser does a great job, but the nature of dynamic language does not allow it to provide such guarantees. I’ll show a few examples later.
4. It’s precise. Unlike whitequark/parser, its lexer (or tokenizer if it sounds better for you) is based on MRI’s parse.y. What does it mean? It means that I was not able to find any difference in tokenizing on 3 million lines of code that I have got by pulling sources of top 300 gems (by total downloads). I’ll mention how I track it soon.
5. It does not depend on Ruby. In fact, it has absolutely no “required” dependencies (only a few optional ones). So, it’s possible to write bindings for any other language, and I have made them for C/C++/Node.js. Of course, it’s possible to have bindings for Ruby (because there are bindings for C and it’s easy to reuse them)

Implementation #

Current performance (in release mode, with jemalloc) is ~200000 LOC/s. I think it can even be used for syntax highlighting (and in the browser, too, haha).