Warning Acknowledgments Let’s take a moment to thank my PhD sponsors: InfluxData , Bauplan , SpiralDB , and the taxpayers of the State of Wisconsin and the federal government. LLMs have become so important that I (probably you as well) want to understand them better, and the best way to learn is to build one. In this blog post, I’ll share what I’ve learned about LLM inference, from the perspective…
I’m currently having an end-of-PhD crisis, where I keep thinking about what to do next. It’s a good time to slow down and reflect, to avoid repeating the same mistakes. I have made many mistakes during my PhD. Here, I list four major ones I’m currently aware of. Mistake 1: Giving up too late Since childhood, I was told to never give up, stick to it, and I would eventually win. Maybe that’s true,…
Warning Acknowledgments My work is supported by funding from InfluxData , Bauplan , SpiralDB , and the taxpayers of the State of Wisconsin and the federal government. Much appreciation! Try it out here: https://github.com/XiangpengHao/parquet-linter > cargo install parquet-linter-cli > parquet-linter data.parquet warning low-compression-ratio -- > column [ 2 ] ( "pid" ) aggregated compression…
Warning Acknowledgments My work is supported by funding from InfluxData , Bauplan , SpiralDB , and the taxpayers of the State of Wisconsin and the federal government. Much appreciation! To search vector embeddings, the standard advice is almost always “spin up a vector database.” Tools like Pinecone , Milvus , or specialized formats like Lance are fantastic pieces of engineering. But they come…
Warning Acknowledgments My work is supported by funding from InfluxData , Bauplan , SpiralDB , and the taxpayers of the State of Wisconsin and the federal government. Much appreciation! The Wrong Direction Everyone knows today’s systems are sub-optimal for agents. They waste context, confuse models with unstructured output, and prioritize vision-based human interaction over text-based LLMs. So we…
Warning Acknowledgments This work is supported by funding from: 1. InfluxData , Bauplan , and SpiralDB . 2. The taxpayers of the State of Wisconsin and the federal government. Your support for science is greatly appreciated! LiquidCache is a caching layer that unifies the design goals of compute and storage 1 . 1 Check out our research paper (VLDB 2025) for more technical details. It accelerates…
Warning Acknowledgments This work was supported by funding from: InfluxData , Bauplan , and SpiralDB . The taxpayers of the State of Wisconsin and the federal government. Your support for science is greatly appreciated! Strings are the most common data type we encounter in analytics. When we built LiquidCache – a distributed pushdown cache for DataFusion – we discovered that the way strings are…
Warning Acknowledgement This blog post was made possible by funding support from: InfluxData Taxpayers of the state of Wisconsin and the federal government. Your support for science is greatly appreciated! TL;DR S3-Select can filter S3 data before sending it back to you, significantly saving network bandwidth and time. Unfortunately, AWS killed this feature in 2024. Good news: you can build your…
Warning Acknowledgement This work (the PR, this blog post, parquet-viewer , and LiquidCache ) was made possible by funding support from: InfluxData Taxpayers of the state of Wisconsin and the federal government. Your support for science is greatly appreciated! In the previous post , we discussed how DataFusion prunes Parquet files to skip irrelevant files/row_groups (sometimes also pages ). This…
We, as system researchers, face an existential crisis. We find ourselves questioning our unique strengths and fundamental purpose where industry increasingly outpaces academic innovation . Research is a privilege Research is paid for by taxpayers, but we often forget this is a gift. We think we deserve this money because we’re smart or have fancy degrees, without asking if we really earned it. I…
Quick look Online here: https://parquet-viewer.xiangpeng.systems Source code: parquet-viewer Run SQL queries Visualize query plans Examine metadata How it works It compiles Parquet/Arrow/DataFusion to webassembly which runs in the browser. Specifically, it uses trunk to pack wasm files, leptos to build reactive UI components, and Tailwind CSS for styling. And most importantly, LLM wrote most of…
📢📢📢 Checkout LiquidCache , an open-source caching solution that reduces latency by 10x for cloud-native DataFusion. Note: my research is funded by InfluxData , which made this blog post possible. In the last post , we discussed how DataFusion prunes Parquet files to read only the necessary data. This post explores DataFusion’s caching mechanisms, which help avoid repeating reads to object…
Note: special thanks to InfluxData for funding this blog post. Apache Parquet has become the industry standard for storing columnar data, and reading Parquet efficiently – especially from remote storage – is crucial for query performance. Apache DataFusion implements advanced Parquet pruning techniques to effectively read only the data that matters for a given query. Achieving high performance…
Editor note: This blog post is adapted from the InfluxData blog post series, make sure to also check out the original posts 1 2 . Note: Thanks to InfluxData for sponsoring this work as a summer intern project This blog describes our experience implementing StringView in the Rust implementation of Apache Arrow , and integrating it into Apache DataFusion , significantly accelerating string-intensive…
North star Research should connect to real people. Impact definition Research impact is defined by the number of connections between the paper and the users. Note that the research paper itself does not produce an impact, as nobody’s life is changed because of reading a paper—we are not philosophers. This means that the research impact is always indirect; we have to apply the research to a system…
I have interned at Google, Microsoft (Research), and InfluxData. The experience at InfluxData has been the most enjoyable, and this blog post explains why. Smaller companies are more connected to the real business, more connected within the company, and more willing to connect you to the rest of the world. Problem definition Non-problems I have worked on great projects: (1) auto-tuned BigTable,…
Good advisors Innovators, educators, and entrepreneurs. Innovator : good at finding the right thing to do. They have a clear vision of what important problems are. They are often smart people who deeply understand the problems they work on. Students learn a lot about their methodologies, visions, and ways of thinking/reasoning. Example: Mike Stonebraker . Educator : good at teaching and empowering…
Preface A database can be complex; it involves almost all aspects (research communities) of computer science: PL (programming language), SE (software engineering), OS (operating system), networking, storage, theory; more recently, NLP (natural language processing), and ML (machine learning). The database community is centered around the people interested in making the database (the product) better…