RocksDB is built on an LSM-tree, which never modifies data in-place - every write creates a new version of the key. That's half of what you need to implement Multi-Version Concurrency Control ( MVCC ). Real-world databases serve many clients reading and writing the same data at the same time. The simplest way to make concurrent access safe is locking (think of a hash map protected with a mutex).…
Note This blog post was co-authored with May Lee and is cross-posted on the Datadog blog . Datadog collects billions of events from millions of hosts every minute and that number keeps growing and fast. Our data volumes grew 30x between 2017 and 2022. On top of that, the kind of queries we receive from our users has changed significantly. Why? Because our customers have grown in sophistication:…
Introduction # Over the past years, the adoption of RocksDB increased dramatically. It became a standard for embeddable key-value stores. Today RocksDB runs in production at Meta, Microsoft , Netflix , Uber . At Meta RocksDB serves as a storage engine for the MySQL deployment powering the distributed graph database. Big tech companies are not the only RocksDB users. Several startups were built…
Full-Text Search is one of those tools people use every day without realizing it. If you ever googled "golang coverage report" or tried to find "indoor wireless camera" on an e-commerce website, you used some kind of full-text search. Full-Text Search (FTS) is a technique for searching text in a collection of documents. A document can refer to a web page, a newspaper article, an email message, or…
String interning is a technique of storing only one copy of each unique string in memory. It can significantly reduce memory usage for applications that store many duplicated strings. The built-in string is represented internally as a structure containing two fields. Data is a pointer to the string data and Len is a length of the string: type StringHeader struct { Data uintptr Len int } In Go…
Note This post is outdated, please read the new design document on GitHub . A few months ago I released the first version of an embedded on-disk key-value store written in Go. The store is about 10 times faster than LevelDB for random lookups. I'll explain why it's faster, but first let's talk about the reason why I decided to create my own key-value store. Why another key-value store? # I needed…
Running Go on AWS Lambda is not something totally new - developers figured out how to launch Go binaries from Python a while ago, but it wasn't convenient and had some performance implications. A few days ago Amazon announced an official Go support for AWS Lambda. The API Gateway integration is straightforward, all you need to do is to import the github.com/aws/aws-lambda-go package, implement a…
Cgo is a mechanism that allows Go packages call C code. The Go compiler enables cgo for every .go source file that imports a special pseudo package "C" . The text in the comment before the import "C" line is treated as a C code. You can include headers, define functions, types and variables - everything a normal C code can do: package main /* #include <stdio.h> void foo(int x) { printf("x: %d\n",…
Note This post was updated on 2021-04-25. Go has a powerful built-in profiler that supports CPU, memory, goroutine and block (contention) profiling. Enabling the profiler # Go provides a low-level profiling API runtime/pprof , but if you are developing a long-running service, it's more convenient to work with a high-level net/http/pprof package. All you need to enable the profiler is to import…
Here are the slides from my talk "Scraping the Web with AWS Lambda and PhantomJS" given at Greater Philadelphia AWS User Group meetup on May 25, 2016. You can find the source code of PhantomJS/Node.js web scraper for AWS Lambda at https://github.com/akrylysov/lambda-phantom-scraper .
Note This post was updated on 2016-08-13: added python-rapidjson ; updated simplejson and ujson . A couple of weeks ago after spending some time with Python profiler, I discovered that Python’s json module is not as fast as I expected. I decided to benchmark alternative JSON libraries. Libraries # json simplejson 3.8.2 ujson 1.35 python-rapidjson 0.0.6 python-cjson , yajl-py and jsonlib are not…
Столкнулся недавно с простой задачей - нужно было найти позицию открывающегося тега <body> в HTML странице. Не долго думая я решил использовать регулярные выражения, через минуту у меня родился регексп <body[^>]*> . Все работало хорошо, пока дело не дошло до тестирования на больших объемах данных. Я решил создать тестовое приложение, дабы замерить скорость работы regex_search : #include <regex>…