静水可铭，丰山须镂。 - Silent Water and Curved Mountain
“Use-faced” computing: any computing that needs to happen in the tight loop of users interacting with each other or themselves. By this definition, most business insight generations, log processing, server health stats, data reprocessing for cold storage, automated trading, and automatic surveillance analysis are not user-faced computing. Conversely, when you send a message to another user, every…
With broader access to Llama, Alpaca, ChatGLM, and Dolly available on ordinary consumer-level hardware, we have entered a new phase where every technology-literate person can access GPT-3 level perplexity language models, which also inherit all the shortcomings of GPT-3 level models. These models hallucinate a lot, lack the knowledge of their more parameter-heavy counterparts, and are not nearly…
Every year, we have a new iPhone that claims to be faster and better in every way. And yes, these new computer vision models and new image sensors can exercise the phone as hard as they can. However, you could already take good pictures on an iPhone 10 years ago. These are incremental improvements. These incremental asks only deserve incremental improvements. Once in a few years, there are…
Reinforcement learning algorithms are ridiculously fragile. Older deep Q-learning algorithms may or may not train just with a different seed. Newer algorithms such as PPO and SAC are much more stable. Well-configured ones can train on a wide range of tasks with same / similar hyper-parameters. Because of that, these algorithms are now used more and more in real-world settings, rather than just a…
Back in late 2012, it was not nearly as obvious as now how long the deep learning AI renaissance was going to last. Alex Krizhevsky has been working on cuda-convnet from code.google.com for over a year, and only then, people started to download the code and to reproduce “CIFAR-10 under 30-minute” with their own GPUs. History can take many different paths at that point. If Alex & Hinton’s startup…
I did the Underactuated Robotics course at the beginning of this year (for anyone hasn’t, highly recommended! https://underactuated.mit.edu/ ). One thing interesting to watch is to see how Russ opened the notebook, did the computation instanously, and then spent a few seconds to a minute to generate the animated illustrations. If you previously Googled about how to show animations in a Jupyter…
Once for a while, we asked questions like: what can we do with more computations? When we asked that question in 2010, CUDA came along and Jesen Huang gifted everyone under the sun a GTX 980 or GTX Titan in search of problems beyond graphics that these wonderful computation workhorses can help. Then suddenly, we found out that we can not only solve simulation problems (3D graphics, physics…
There are three things in the past decade that I am not only wrong once about, but actively being wrong while the situation is evolving. Introspecting how that happened would serve as a useful guide for the future. Cryptocurrencies I’ve been exposed to cryptocurrencies, particularly Bitcoin pretty early on, somewhere around 2009 on Hacker News. I’ve run the software at that time (weirdly, on a…
After Elon’s announcement of the Tesla Bot [1], many people mocked the silly on-stage presentation and joked about the forever Level-5 Tesla Autopilot. The last time when humanoid robots caught my eye was the Honda Asimo, when I was a kid. I saved money to buy Robosapien the toy because that was the only bipedal humanoid robot toy available. A decade passed, and the world has given us videos such…
My first time going skiing was in my 20s. The software engineering job was paying me well at that time already. Why not just hire a private trainer for my ski lesson, I was thinking. Afterall, more money spent and a more focused time from the trainer ought to produce better outcomes, right? I definitely enjoyed the lesson for the first two days, but I cannot ski. The trainer was fun, and we did…
This series of essays explores how cryptocurrencies, despite the frauds and scams in the system, can possibly go beyond the speculative asset category. Rather than investigating the underlying utility value for a cryptocurrency system, this series will focus on power interplays between people, and how that can draw a narrow scenario in which cryptocurrencies become mainstream. When reading…
One thing in the past decade the academics failed to appreciate is the power of believing. In democracies like the United States, if the power of believing can be maintained long enough, a supporting structure from both the political and economic side will emerge. In every aspect, cryptocurrencies are moving along the right direction to establish themselves with such supporting structure despite…
I am a mediocre marathon runner that hasn’t done any long-distances since Covid-19. But out of serendipity today, I was thinking about marathon and the interesting choices you would make during it. Flat road marathon with support stations every half a mile is … bland. I probably have done that only twice, one in Maui, one in Denver. The only interesting bits for these two are hot temperature or…
前段时间人人影视组因为侵犯版权问题被调查了。国内引起了对知识产权保护的大讨论。在一定的时期和特定的领域，知识产权有着积极的意义。但是，知识产权制度本身并不是真理，它只是人为的一种创造。 知识产权的本质是将知识，这一属于人类共有资源（Public common）的事物通过国家机器，在有限的时间内赋予其完整的私有产权，将其私有化。定义私有产权的四种性质：即可使用性（right to use），可转移性（right to transfer），排他性（right to exclude）和可销毁性（right to destory）对于知识产权都是适用的。 需要注意的一点是，将知识产权化私有这一行为本身会将知识从人类公有资源中排除。长此以往，人类共享的公有资源将会减少消失。因此，知识产权均有时间限制。比如专利有20年的有效期，而著作权在作者死后50年后失效（美国是70年）。…
For many machine learning practitioners, training loop is a universally agreed upon concept as demonstrated by numerous documentations, conference papers to use the word without any reference. It would be a helpful concept for many beginners to get familiar with before diving into the rabbit holes of many deep learning tools. The field of machine learning becomes vast, diverse and ever-more open…
I started to work on a new data science related project last year. Our initial setup was fairly traditional: it is Python-centric, with Anaconda for dependency management and environment setup. Anaconda is interesting. It is probably useful if you have a ton of dependencies and the system tries really hard to figure out the package compatibilities with each other based on their claims (version…
每两年，我都会做 一次 四年 为期的预测。对 2020年的预测 ，怎么也不会预料到这样的突发事件。这样的突变，反而会使得两年前和四年前过于悲观的预测显得乐观不少。 回顾2016年的预测，辅助驾驶技术已经广泛应用于汽车之中。Cruise和Waymo也分别在旧金山和凤凰城开始了无驾驶员的自动驾驶汽车的运营。存活的移动消息服务，基本也都达到了三亿日活左右的标准。 许多长线航班的取消，虽然确实实现了，却是因为不同的原因。而石油价格虽然大幅变化，且维持在40美元左右，变化的方向却完全错了。 VR设备并没有在19年底售出三千万，仅仅售出了三百万台左右。 写了这么几篇的四年预测之后，我的目的也从单纯的追求准确变成了在追求准确的同时，尽量写一些以前没写过的新趋势。这也就是为什么在絮絮叨叨了将近十年的自动驾驶之后，这一篇并没有提了。因为他终于开始走上了产业化的正轨，接下去就是按部就班了。…
From the onset of implementing libnnc , it meant to be a common ground for higher-level language bindings beyond Python. The underlying architecture has been stable for a year or so, and I have been using it for some personal projects for a while. But the raw C interface is not the easiest to use, it is the time to implement some high-level language bindings for that library. The default…
For the past a few months, I’ve worked on Dflat , among many other things . The theme is to explore a workflow using Bazel as the build and deployment system, and Swift as the main language for data exploration, modeling, and production system. Dflat appears to be an outlier for this theme. I simply have the urge to implement what I considered to be the one-true-way of doing structured data…
I plan to write a series of articles to discuss some simple but not embarrassingly parallel algorithms. These will have practical usages and would most likely be on many-core CPUs or CUDA GPUs. Today’s is the first one to discuss a parallel algorithm implementation for CSV file parser. In the old days, when our spin disk speed maxed out at 100MiB/s, we only have two choices: either we don’t care…
In the past a few days, I’ve been experimenting with Bazel as a library distribution mechanism for ccv . I am pretty familiar with hermetic build systems at this point. My main knowledge comes from Buck dating 8 years back. At that time, it never occurred to me such a build system could eventually be a library distribution mechanism. During the same 8 years, NPM has taken over the world. New…
During the past few years, I’ve been involved in migrations from Xcode workspace based build system to Buck twice (Facebook and Snapchat). Both of these experiences took talented engineers multi-months work with many fixes upstreamed to get it work. Recently, I’ve been helping another company to migrate their iOS project from Xcode workspace based build system to Bazel. This experience may be…
At times, a year can be a day and a day can be a year. After the past several years since January 2020, it feels far-reaching nowadays to preach how awesome the human world is. But the progress cannot be stalled, the problem cannot solve itself with inaction. So, good people continue to publish great works. The recent GPT-3 model shows some interesting abstract reasoning capability. While I still…
I had some of the fondest memories for Visual Basic in 1998. In January, I enlisted myself to a project to revive the fun part of programming . There are certain regrets in today’s software engineering culture where we put heavy facades to enforce disciplines. Fun was lost as the result. With Visual Basic, you can create a GUI and start to hack a program in no time. You write directives and the…
There are some articles referring to the past 20 years as the modern Gilded Age. While Gilded Age is an American specific reference, it nevertheless carries a certain western culture mark in the early 20th century. However, the characterization of the Gilded Age, particularly railroads v.s. the internet, would come across to me as lazy. The fanatic obsession with literal bits over atoms comparison…
Yesterday, I was listening to an interview by Oxide Computer people with Jonathan Blow on my way back to San Francisco. There were usual complaints about how today’s programmers buried themselves into the pile of abstractions. As a game programmer Jonathan Blow himself, they also discussed some fond memories about programming basic games in his childhood. All that kept me thinking, how…
The past 3 months ought to be the best 3 months for television shows. As a prolific TV viewer, the shows from the past 3 months are all impressive and well-executed. Some of these due to better technologies. Some of these we can probably thank to the investment coming from the streaming war. The pouring of money and talents certainly worked. On top of all these, one of the most interesting turns…
Grand Central Dispatch is the de-facto task-based parallelism / scheduling system on macOS / iOS. It has been open-sourced as libdispatch and ported to many platforms including Linux and FreeBSD. libdispatch has been designed to work closely with the Clang extension: Blocks . Blocks is a simple, yet powerful function closure implementation that can implicitly capture variables to facilitate the…
I’ve discussed a stackful coroutine implementation to coordinate CUDA stream last year . That was an implementation based on swapcontext / makecontext APIs. Increasingly, when I thought about porting nnc over to WASM, it becomes problematic because these APIs are more or less deprecated. Popular libc implementations such as musl don’t have implementation of these methods. After the article , it…
To train large deep neural network, you need a lot of GPU and a lot of memory. That is why a Titan RTX card cost more than 3 times of a RTX 2080 Ti with just a bit more tensor cores. It has 24GiB memory and that makes a lot of models much easier to train. More memory also means bigger batch size. Many GPU kernels run faster with larger batch size. If somehow we can reduce memory footprint at…
War starts when people fail to communicate with each other. The current U.S. and China dispute is so complex and overreaching, any rational discussions online can devolve into flame wars. There are so many topics, making the multi-variable optimizations difficult. Overlaying all this with a gloomy long-term implications of technology, it is far easier to just pick a side and rooting for the red /…
When programming with CUDA, there are several ways to exploit concurrency for CUDA kernel launches. As explained in some of these slides , you can either: Create thread corresponding each execution flow, execute serially on stream per thread, coordinate with either cudaEventSynchronize or cudaStreamSynchronize ; Carefully setup CUDA events and streams such that the correct execution flow will…
NNC is a tiny deep learning framework I was working on for the past three years. Before you close the page on yet another deep learning framework . let me quickly summarize why: starting from scratch enables me to toy with some new ideas on the implementation, and some of these ideas, after implemented, has some interesting properties. After three years, and given the fresh new takes on both APIs…
十年前，我开始每两三年写一篇 BLOG ，内容是关于四年之后的一些预测。预测的原理也很简单，就是根据和大家聊的内容，还有每天看的新闻，做一些合理的臆测。后来也开始正儿八经的先描述一下大致的人文市场政治环境，再做猜测。这么一个设置，让很多地方也就看起来扯得更有道理了。然而追根究底之所以如此，不过是其实接下去几年大家要做什么也都了解得八九不离十。说四年后苹果会生产第十四代 iPhone ，这又有什么好猜的呢。 那么，站在2018年的的刚开始看2022年，又是什么样的呢？ 首先，是中国的消除贫困。在2021年，也就是建党一百周年的时候，中国领导人会宣布全面建成小康社会。而在中国的语境下，全面建成小康社会是明确可量的，那就是消除贫困。在2022年，中国人均 GDP…
Decades have passed before we had a yet high quality consumer software. It is now taken that software supposes to be crashy, laggy and barely functional. Why and how we get here? When the question is asked, many people felt the nostalgia, where the software is simpler, and people crafts their cathedral. They often overlooked the fact that the software we built today, was many orders of magnitudes…
四岁的小孩子，睁开眼睛。星辰，银河，月亮，太阳。我站着河边，望着你大笑。都市里的灯光，桌上一碗稀米汤。
离上次 预测未来2016 已经过去了四年。本打算每两年一次预测，每次预测四年之后的情况。因盖茨曾说过，我们总是高估两年之后的科技发展，而低估十年的科技发展。而十年之前，将奔腾四芯片的运算能力放到手掌大小的运算设备上几乎是不可能的事情。哪怕是八年之前，当我们要订制有当时高端台式机处理能力的一体化电视也是困难重重。 回顾过去四年的预测 虽然四年前的预测现在看来相当精准。但是经济政治稳定的大前提在各国政府的量化宽松政策主导下得以保持，否则这样的预测难言准确。回顾四年前的预测，互联网接入、3D电视悲观市场前景、电视剧点播、计算机性能、辅助驾驶、自拍技术均和现实没有太大出入。然而，对于无线电源的普及、Pad和笔记本融合的趋势、商用超音速客机投产、失业率以及人工智能的发展预测都出现了较大失误，没有预测到无人机的发展。总结起来，一些是过于乐观，而另一些是知识的储备不够。 未来四年经济社会总体概况的预测…
八月的夏天总是特别漫长。暑假的最后几天，家里百无聊赖，休整得已经不能再好了就等着开学，和同学们扯蛋聊暑假发生的大小事情。暑假的尾巴，八六版的西游记也快结尾了，都开始放西游记续了。 嗯，其实从来也没有过过这样的暑假。大概除了去年的夏天，也总是忙忙忙。忙工作啊，忙实习啊，忙一些上学时候不应该忙的事情。太阳公公晒啊晒，把白天晒得格外长。漫长的白天，又跟着漫长的黑夜。重复的八月，赶着同一个Deadline。结果也都有好有坏。有让大家不满意的地方，也有让自己开心的时候。 九局下半，两人出局。 最早的漫长的八月在上海，出奇地重复一致。每天早上在学校门口吃碗馄饨，然后把主机搬到实验室，开始下载东西，准备资料，写代码。傍晚的时候，把主机再搬回寝室，继续做没做完的事情。…
我是表现得有多不热爱LA啊！过去的八周，只有三个周末呆在LA，忙着各种Routines就过了。人家一问这个城市怎么样啊，还是噼里啪啦地答不上来。所以，趁着长周末，就，嗯，跑到大岛来了。 不过，我是有多热爱在这样的海边小镇瞎逛。一不小心就在Kona的小镇上溜达了一下午。啊，刚刚才意识到这里土特产应该是咖啡！在Venice工作的时候也算是喜欢在Oceanfront Walk上瞎逛。但最近都是匆匆忙忙走过，连抓杯咖啡的时间也没有。这大概或许是人生走上正轨的写照？那当年在Facebook没事抓个咖啡，找人扯淡聊天喝奶茶的人生或许大概就是在瞎胡闹？或者现在这样假装很忙才是在瞎胡闹，在办公室里面看党报喝茶可能才是正途，谁知道呢。 之前有段时间，对自己想住在哪这件事一直很纠结。嗯，纽约，那个，是吧。旧金山湾区，现在回去也有了一层陌生感。LA总是在慌乱地工作，夏天好不容易到了，Hollywood…
Birthdays are often joyless for me. I’ve yet to find this as an excuse to celebrate for. But the clock is ticking, and every time, it kicks me hard on the back this day of the year. It is always a thing for me to accomplish something, to set a goal, and work towards it. But looking back, I’ve done nothing tangible, not even to mention worthy causes. When I die, I die.…
上周四晚上，和同事们去特Sketchy的一Neighborhood吃饭。洗手间在楼上，酒足饭饱之后去，发现小便池上面一排「THIS PLACE IS HAUNTED」的警告。匆忙跑下来，接到了姥姥过世的噩耗。不到30个小时之后，就回到了家乡，老房子的楼下。 当周遭都回到了十几年前的模样，看世界的眼睛才会变得光怪陆离。桌上的小泥人在夜里站起来自己活动脚肢。七月十五深夜的马路边，会有穿着白裙子的长头发坐着哭泣。从古墓里跑出来的僵尸还在被解放军追捕。半夜河边的桥上，三五个少年在无声地嬉戏打闹。而这一切，在脚边的公鸡打鸣破晓之后都会不见啦。…
05年2月份的时候，写下了第一篇BLOG。十年，当时的伤心事和不得意，也终究完全不记得了，还是一如既往地憎恨生活。 十七八岁的时候就离开家到上海上学，人生成长的关键时期，以至于连审美都被培养得只对上海姑娘充满好感，无法自已。 翻看十年前的BLOG，却总是显得过分高尚不物质。当时批判着大家买Porsche和Ferrari，现在还是觉得橙色的488 GTB好不错，应该努力赚钱给有些人买呢。十年前是表面不物质，但雄心勃勃地想赚好多好多钱，买好大好大的房子。十年后却表面各种物质，实际想的，不过是能自由地到处游学，做一些让所有人都受益的好事情。十年后，当年认识的同伴们有的已经实现了我十年前的梦想，可是现在的我看着他们却只能摇摇头，觉得那也不是自己想要的。…
上周和友人聊天的时候，他莫名就提到了少年情怀。快到三十，还一事无成的时候，谈起少年情怀总是有点Pretentious啊。最近的日子过得都太物质，而在十七八岁的时候，这个物质的世界反而不是这么具象。十七八岁的少年，不止有一个物质的世界。虽然对于大部分的少年吧，那个非物质的世界往往就在网吧的屏幕里。但至少在那里，可以打怪升级，耀武扬威。…
一开始抒情，陈升这个男人就在我脑子里开始「依然是，不懂爱也不懂情」。本来说年前的时候写一篇像话的小散文，讲讲现在的乐观情绪。结果一玩起来就没了神，一周什么都没干，就像做梦一样。一年用这样的方式结束总是觉得有点邋遢，跑马拉松的时候，最后一英里就算再累，也要冲过去啊。 去年刚开始的时候，想做的事情才做了一半，安排的事情也都八字没有一撇。却没有任何征兆地，就这样到了年底。 2014年呢： 跑完了一趟50Mi又跑完了一趟250K。 认识了更多有趣的朋友。 读完了四五本书。 从旧金山搬到了洛杉矶。 这样一条一条数下来，却发现和自己期待的差了好多。但是到了这个年纪，也知道好多事情并不是能在一年的时间段完成的。人性的伟大之处，不就在于能够牺牲短期的欢愉，而争取长期的进步吗。
只有一个月就到年底了，一方面蠢蠢地想着年底出去玩，另一方面什么都没做也很慌张！本来说感恩节的时候好好在家工作，结果眨眼功夫，就到了San Diego呆两天。一年时间什么都没干成，还坐在Grand Hyatt楼顶干嘛，早点回去干活才是正事。 故事到最后，都是我想要离开，你却停不下来。 喂，少年，新生活刚开始的时候要勇敢面对啊！
有的时候，两年什么事情也没发生。转眼间，跑第一次马拉松就已经过去了一年，在Mountain View的夏天训练已经是那个夏天而不是这个夏天。但转眼间，很多事情又发生在两周之间，就这样为三十岁收尾了吗？ 有时候也会吓一跳午夜冷汗醒来，怕泡沫破灭之后自己一无所有。有时候也会固执地觉得，最后会是Kentucky的某个老太太的养老金来给我们买单。上周末在GSB，就是看看更不靠谱的人怎么为自己找台阶的，也要一边摇头，一边学习学习呢。 无论环境怎么变，还是得慢慢积累，和以前一样好好工作，才会继续有好运呢。能在新的小公寓里望着Bay Bridge，想着还有点小激动。
去年的时候提起Burning Man，有人告诉我：「It is a very spiritual experience.」可是，在沙漠里面一堆人唱歌跳舞又有什么可稀奇的呢。 可是当自己真的到一个没什么人的地方，每天只是跑步，吃饭，睡觉之后，才觉得，真的很Spiritual。 也会担心回家之后成堆的邮件和事情，也会担心赶不上进度。可是在那几天，只有南半球的星星在头顶，太阳和月亮照着前进的道路。自己和自己说话，想着过去的往事，想着未来，想着五年之后，十年之后。想到回到加州之后，夏天就结束了。 自己总是不够努力，懒惰。有事情做的时候不努力向上，没事情做的时候又懈怠。这样的人格硬伤，和不会法语放在一起，真是残疾。 人生的问题，总结下来，无非就是，从约翰内斯堡回来的航班要十六个小时，但是，但是为什么南非航空的航班会有爱情公寓看啊！
而我站在原地。下楼的时候到底是穿短裤还是穿牛仔裤呢，到底是穿T恤还是套个hoodie呢，还没想清楚，晃动就停了。 现在想起来才后怕。四川地震的时候，还能一溜烟地跑到楼底求生。反而6年之后，求生的欲望却没有那么强烈，想的却是只过了三年却多了这么多身外之物，想的却是就这样死去却也不是最坏的结果。最坏的结果不是壮志未酬，而是壮志难酬啊。而是拥有这么多值得期待的品质，却胡吃海花，下半生落魄街头，谁也不认识那个创立了实用主义的人。 就算自己离开家十一年了，也是没办法很好处理各种生活细节。今天整理马上去马达加斯加的背包的时候，把各种食物，分门别类，按照每天的卡路里所需打包起来，贴上小贴士，装得很专业，其实里面的东西，除了方便面，对怎么弄成一顿饭一点头绪也没有。就这么去七天真的没问题吗喂！不会就这么横尸非洲丛林吗！很羡慕那些把生活和工作打理得井井有条的人，而我总是手忙脚乱的狗刨才好歹不被淹死。…
来耶路撒冷之前，大家都告诉我，这是一个多么特别的地方。总归是半信半疑。来了也就打算去Old City照照破石头，特游客地发发Facebook。谁想到雇了一个这么Hippie的导游，见面就拉着我聊，来来来，我们从Ottoman Empire时候的Zionism谈起！喂！见面就聊这么沉重的话题，我们还能不能好好拍照看石头了！ 穿过巴勒斯坦区，路过以色列的旗帜和梵蒂冈的旗帜，被边警拦着非穆斯林勿入。只能远远地望着Haram al-Sharif，竟然没有缘分见到这么一块石头。 周四，在Western Wall边，一群十三岁少年的Synagogues。小摊小贩叫卖的画里，Dome of the Rock被幻想中的the Third Temple给取代了。这时候，望着墙后面的穹顶再拍照是多么的不合适啊！一个巴勒斯坦的老头走过来，向我们诉说自己被关押了两年半的遭遇。…
看舌尖才觉得，回家的时候，火锅完全没吃到饱啊。不止是火锅，连早餐的红烧牛肉面都没吃到！有的，是妈妈存的甜柚，冬天风干的香肠和桦树木熏的腊肉。所以就算再节制，不跑步不锻炼的呆一周半，也狠长了几磅的。 十几年前第一次离开故乡的时候，还是十四五岁的小孩，有着青春期的叛逆和心比天高。当时也不会想到，十几年后自己也没什么进步，还在挣扎和学着走路说话。即使自己没进展，也有着奇怪的断层。当朋友们都天各一方，也会有时候分不清家乡的方向。陌生地陪着父母散步几个小时，模糊地记得这里，又不记得那里。找个餐馆都要打开手机地图。…