Recent content on Openinx Blog
I have been working at a US company for almost a month recently. An interesting fact is that I have counted the SaaS services I have used for specific scenarios since joining, and surprisingly, there are as many as 23 of them. All of these SaaS services are provided by third-party companies. In other words, Databricks is a customer of all these companies. This aroused my curiosity and made me want…
I recently had the pleasure of joining Databricks, but I often reflect on my profoundly rewarding experience at ByteDance. Before joining ByteDance, I was a software engineer who had written code for a decade. My core responsibility was to tackle system design and code development for complex software while ensuring robust operations in production environments. At ByteDance, however, I led a…
This is the second article of my thoughts for Iceberg Summit 2025 (Here is Part-1), which is not limited to Iceberg but focuses on Data Lake. I am trying to share what these creative teams are doing in this field, why it is a problem, what the solution is, and how it will develop in the future. These insights come from various sharing, discussions, and debates, but are limited by my personal…
I attended the Apache Iceberg Summit 2025 in the US in the past two days. The conference was held at the Hyatt Regency Hotel next to Union Square in downtown San Francisco. This time, two floors of the hotel were rented as the venue. Mainstream companies in the Data field are basically sponsors of the conference, including AWS, Databricks, Snowflake, Cloudera, Dremio, Microsoft, Airbyte,…
Hi, I’m Zheng Hu — a Staff Software Engineer with deep roots in big data infrastructure and open-source communities. I’m a PMC member of Apache HBase and Apache Iceberg, and the author of the book ‘HBase Principles and Practice’. Over the past decade, I’ve worked at Databricks, ByteDance, Alibaba Cloud, Xiaomi, and more, building large-scale systems in cloud data…
I recently read Microsoft’s 2015 VLDB paper, Real-Time Analytical Processing with SQL Server [1]. SQL Server was among the earlier products to ship and productionize an HTAP row/column update design. Row-store (row-wise index) tradeoffs for OLTP are well understood; efficient millisecond-scale column store updates have several designs in the wild—Kudu [2], Positional Delta Tree [3], and…
I joined TiDB’s Hackathon 2019 and finally have time to write this recap. A 48-hour hackathon around TiDB: build a demo, present in six minutes. Scoring weights practicality, ease of use, and performance for TiDB (40%), completeness (30%), innovation (20%), and presentation (10%). Our team of three: captain Yi Wu from PingCAP’s US office (ex-Facebook RocksDB, now RocksDB at PingCAP),…
How good is HBase 2.x write performance? Let’s benchmark it. Test environment: five-node cluster; each node has twelve 800GB SSDs, 24 CPU cores, 128GB RAM. HBase and HDFS are co-located—RegionServer and DataNode on the same host for better write locality (at least one replica local). Software: HBase 2.1.2, HDFS 2.6.0, OpenJDK 1.8.0_202. Per RegionServer we use 50GB heap and 50GB off-heap…
HBase 2.0.0 shipped on April 30, 2018—about 15 months ago as of this writing. HBase 2.0.x is EOL with no further releases. HBase 2.1 is at 2.1.6 and likely will not be maintained much longer. Stable 2.x lines will be 2.2.x and 2.3.x, especially 2.2.x as the version battle-tested at large shops. Work I think deserves focus on HBase 2.x: Procedure V2 and Assignment V2 give distributed workflows…
After working in the Apache HBase community for a while, I noticed a few relentlessly productive leaders: Michael Stack, contributing day in and day out for a decade, and Duo Zhang, recently promoted to HBase project chair. Stack—a senior engineer I could call “uncle” by age—works across my entire day: morning mail shows a JIRA reply, afternoon another patch, late night comments on…
The third Apache HBaseConAsia summit takes place in Beijing on July 20. As the Apache HBase community’s premier user conference in Asia, HBaseCon has run since 2012. More than 20 experts and community leaders from leading internet and big-data companies will share the latest on HBase and the surrounding ecosystem. Apache HBase is a highly available, high-performance, multi-version…
Motivation Databases need flexible query patterns. A KV store with only Get, Put, and Scan would frustrate users—real workloads are richer. For orders: “this user’s orders in the last three months” needs at least (1) filter by user and (2) filter by time range, combined with AND. Scanning the whole table on the client and filtering locally would crush the cluster. Server-side…
Nebula held its second Beijing meetup in Jingdong Beichen. CEO Xiaomeng Ye led Geobase, Ant Financial’s graph database. Two technical directors—Heng Chen and Fenglin Hou (dutor)—drive storage (similar to TiKV) and the query engine (similar to TiDB), respectively. The architecture separates storage and compute. The compute layer is SQL-like, with Go syntax for edge hops and pipelines to…
In HBASE-21879, we redesigned the offheap read path: read the HFileBlock from HDFS to pooled offheap ByteBuffers directly, while before HBASE-21879 we just read the HFileBlock to heap which would still lead to high GC pressure. After few months of development and testing, all subtasks have been resovled now except the HBASE-21946 (It depends on HDFS-14483 and our HDFS teams are working on this, we…
HBase Off-Heap Today HBase is a widely used distributed NoSQL database. Many workloads—feeds, ads, and similar—demand high throughput and low latency. HBase 2.0 off-heaped the core read and write paths: allocations go to JVM off-heap memory, which is not GC-managed and must be freed explicitly. On the write path, request buffers are allocated off-heap until data is written to the WAL and memstore.…
HBaseConWest2018 was held on June 18 in San Jose, California, hosted by Hortonworks. Attending HBaseCon West in Silicon Valley each year has become routine for the Xiaomi HBase team—our community presence is well known (seven HBase Committers, two PMC members), and the company is willing to share a year-in-review of internal practice and community contributions. In 2018 we submitted the talk…
On October 20, I accepted an invitation from the Apache HBase community and became an HBase Committer. At Xiaomi I maintain our internal HBase branch and production clusters, and the company already had six HBase Committers—including one PMC member—so becoming a Committer was a natural next step rather than something extraordinary. Compared with someone doing HBase at a company with no Committers,…
HBase is a distributed key-value database that supports automatic load balancing. With the balance switch (balance_switch) enabled, the HMaster process automatically selects regions according to a specified policy and assigns them to RegionServers with lower load. The official distribution currently supports two region-selection policies: DefaultLoadBalancer and StochasticLoadBalancer, both…
Notes on HBaseCon West 2017 presentations: 1. HBase at Xiaomi Presented jointly by Zhe Yang and Guanghao Zhang—both became HBase Committers in 2016 (Xiaomi has produced eight HBase Committers in total, including two PMC members, and has resolved hundreds of issues). Highlights included: Lessons from upgrading clusters from 0.94 to 0.98. Experience using G1GC for internal HBase deployments.…
In an HBase master-slave replication cluster, as shown on the left in the figure below, Region-Server-X and Region-Server-Y are two RegionServers in the master cluster. Under normal conditions, writes to Region-A append logs to Hlog-X on Region-Server-X, and Region-Server-X asynchronously applies those HLog entries in batches to the slave cluster. If Region-Server-X then crashes, Region-A is taken…
This article covers transaction isolation in TokuDB. The source implementation is complex; for clarity, we focus on the most essential parts and omit minor details. Background In traditional relational databases (Oracle, MySQL, SQL Server, and others), transactions are central to both engineering and discussion. The core properties of a transaction are ACID. A (atomicity) means a…
This article analyzes TokuDB’s index structure—the fractal tree—from an engineering implementation perspective. It describes the on-disk storage layout of ft-index in detail, how ft-index implements point queries, range queries, and insert/delete/update operations, and throughout the discussion attempts a detailed comparison with InnoDB’s B+ tree from multiple angles. Introduction to…
Online version: PPT This is a technical talk on Vitess I gave internally at my company. The main topics are: Vitess feature modules, capabilities, and system architecture Vitess sharding approach SQL syntax supported by Vitess How Vitess resharding works Vitess data backup internals Pros and cons of Vitess compared with traditional relational databases and NoSQL A side note To try building an…
I took Google’s online campus hiring written test yesterday. Skip algorithms for a day and your skills slide right back down. Problem A. Password Attacker Description How many passwords of length M can be formed from N distinct characters, such that every password uses all N characters at least once? Solution 1: Brute Force For each solution to the equation below, add the number of distinct…
Basic LevelDB constraints With the default options, LevelDB follows these basic constraints: LevelDB has 7 levels: 0, 1, 2, 3, 4, 5, and 6. SSTables on level 0 are about 4 MB each. On level i (i > 0), each SSTable is at most 2 MB. Level 0 ideally has 4 SSTables, should stay within 8, and must not exceed 12. The total storage used by all SSTables on level i (i > 0) should stay around 10^i MB.
After grinding LeetCode for several days, I finally finished. Code is here. Reverse Words in a String Simulation on strings Evaluate Reverse Polish Notation Simulation — evaluate postfix expressions Max Points on a Line Given N points in the plane, find a line that passes through the most points. Enumerate each point as the origin, compute relative coordinates, then compute y/x and use a hash…
Zookeeper hardly needs an introduction — it is a distributed coordination service. A few questions I was curious about: How does ZooKeeper implement asynchronous watcher callbacks? (code-level details) How does ZooKeeper implement distributed locks? How do Queue, barrier, and similar recipes work? I read the Python client kazoo and got the general picture. A simple client example #!/usr/bin/python…
Source code and related articles are available here. This article discusses the K-th largest or K-th smallest element in a sequence. Since the K-th largest can be reduced to finding the (N−K+1)-th smallest (where N is the sequence length), we focus on the K-th smallest element. Problems covered in this article: Given an integer sequence, find the K-th smallest element. Given an integer sequence…
Hash Table A hash table is an array of size buckets, table[0...size-1]. Hashing an object yields an index; we store the object in table[value]. When a bucket holds multiple objects, they are chained into a linked list. This collision strategy is called separate chaining. Load Factor If a hash table has size buckets and used stored elements, used / size is the load factor. When loadFactor <= 1,…
Abstract: Plane sweep is widely used in computational geometry, computer graphics, grid computing, and related areas. Many classic algorithms use plane sweep to greatly reduce time complexity — e.g. segment intersection, union of axis-aligned rectangle perimeters, rectangle intersection, spatial collision detection, Voronoi diagram construction, closest pair of points, and more. This article…
2021.01.08 Flink Forward Asia 2021 Beijing: The Best Practice of Integrating Apache Flink with Apache Iceberg 2021.04.25 Flink+Iceberg Shanghai Meetup: How Flink and Iceberg Solve the Challenges of Data Lake Ingestion 2020.12.15 Flink Forward Asia 2020 Beijing: How to Analyze CDC Data in Iceberg Data Lake Using Flink, Blog 2019.07.20 HBaseConAsia 2019 Beijing: Further GC optimization for HBase…