2016 年是我第一次参加 CVPR. 我以 FAIR 实习生的身份注册, 在会场和旷视的伙伴们 Tim, 舒畅汇合, 一起搭起了旷视的展台. 那一年,…
In Python, when a set of objects constructs a reference cycle, none of them would reach a zero refcount. In this case, even if these objects all go out-of-scope and are no longer accessible, they will not be immediately released. The Python ecosystem typically accepts reference cycles as an inevitable issue, and relies on garbage collection (GC) to avoid leaks. A GC is triggered by the Python…
在过年的这几天, 为了从焦虑的工作中换一个心情, 我给我的 wechat-dump 项目添加了几个当年没做出来的功能,…
Meta 与 Google 内部的代码管理工具都支持一种被称作 "stacked diffs / stacked PRs" 的 workflow. 然而, 基于 git 的主流平台 (github, gitlab) 都不支持这种 workflow. 许多离开 Meta…
People have many different opinions about config systems. Having worked with various styles of configs, I also want to write about what a great config subsystem in a large-scale (in terms of system complexity, number of users, etc.) system should look like. The design space is complex, so in this article I'll start with a smaller topic: registration in config systems. I'll show why this…
Since I joined Google Brain, I brought PyTorch to Google's internal infra and owned its maintenance. Being a "tech island", it's well known that almost everything in Google works differently from the outside world, and that creates many challenges when building a massive library like PyTorch. Among those challenges, there are a few tricky bugs related to static initialization order…
最近学习到了一些 Terminal Escape Sequences, 其中尤其对 OSC52 相见恨晚. 这里稍微记录一下各种 Sequences. Terminal Escape Sequences 是终端应用向 stdout…
A typical PyTorch training program on 8 GPUs with 4 dataloader workers per GPU would create at least processes. A naive use of PyTorch dataset and dataloader can easily replicate your dataset's RAM usage by 40 times . This issue has probably affected everyone who has done anything nontrivial with PyTorch. In this post, we will explain why it happens, and how to avoid the 40x RAM usage .
"Loss function" is one of the most basic concepts today in deep learning. Despite that, it is actually not necessarily a good programming abstraction when designing general-purpose systems. A system should not assume that a model always comes together with a separate "loss function".
Building a library for research and experiments is quite different from building other types of software. A key challenge is that, in research, abstractions and APIs are rarely set in stone: users may want to propose a slight variant or modification to literally ANYWHERE in the whole program, just because they have a new idea.
This post is about a small functionality that is found useful in TensorFlow / JAX / PyTorch. Low-level components of these systems often use a plain list of values/tensors as inputs & outputs. However, end-users that develop models often want to work with more complicated data structures: Dict[str, Any] , List[Any] , custom classes, and their nested combinations. Therefore, we need bidirectional…
PyTorch provides two methods to turn an nn.Module into a graph represented in TorchScript format: tracing and scripting. This article will: Compare their pros and cons, with a focus on useful tips for tracing. Try to convince you that torch.jit.trace should be preferred over torch.jit.script for deployment of non-trivial models.
这篇文章说说用户怎么提出好的 feature request / pull request, 以及维护者如何对待它们.
我听过不少人凭借爱好开源了自己的项目后, 却对 issue 太乱感到困扰, 甚至想干脆直接禁用 issue. 其实,…
报告错误 / 报 bug 是用户与开发者间最常见的一类交流, 也是常见的 github issue. 但是很多用户并不会科学的报 bug, maintainer…
相比传统的邮件列表 / bugzilla/sourceforge 等开源平台, github 把开源社区交流的成本 / 门槛降的很低, 因此交流的质量也常常随之下降.…
In large systems, logs can be terrifying: they are huge in volume, and hard to understand. This note lists some suggestions and common misuse of Python's logging module, with the aim of: Reduce redundant logs & spams from libraries. Allow more control of logging behaviors. Make logs more informative to users.
延续 上一篇文章 , 再说一说怎么科学的在 paper 里做 ablations.
Technically, an image is a function that maps a continuous domain, e.g. a box , to intensities such as (R, G, B). To store it on computer memory, an image is discretized to an array array[H][W] , where each element array[i][j] is a pixel . How does discretization work? How does a discrete pixel relate to the abstract notion of the underlying continuous image? These basic questions play an…
这几年来, 从 FAIR 的几位大佬身边学习到的最多的是对待 research 的态度. 因此说说写 paper 和做实验的体会. 实验与 claims…