Meta 与 Google 内部的代码管理工具都支持一种被称作 "stacked diffs / stacked PRs" 的 workflow. 然而, 基于 git 的主流平台 (github, gitlab) 都不支持这种 workflow. 许多离开 Meta 后不得不使用 github 的朋友表示, stacked diffs 对于工程师是一个 "ultimateproductivity ...继续阅读 (97)

People have many different opinions about config systems. Having worked withvarious styles of configs, I also want to write about what a greatconfig subsystem in a large-scale (in terms of system complexity, number ofusers, etc.) system should look like.The design space is complex, so in this article I'll start with a smaller topic:registrationin config systems. I'll show why this common pattern, though works fine forsmall-scale projects, does not scale well in the long term. I'll also discussan alternative.Registration for configs¶Configs are often constrained to include o ...继续阅读 (36)

Since I joined Google Brain, I brought PyTorch to Google's internal infra andowned its maintenance. Being a "tech island", it's well known that almosteverything in Google works differently from the outside world, and thatcreates many challenges when building a massive library like PyTorch.Among those challenges, there are a few tricky bugs related tostaticinitialization order fiasco(SIOF) and their destructions. This time I was forced to learn a lot more detailsthan I'd like to know about these topics, so it's good to write them down before I forget.Terminology"Static initi ...继续阅读 (48)

最近学习到了一些 Terminal Escape Sequences, 其中尤其对 OSC52 相见恨晚. 稍微记录一下.Terminal Escape Sequences 是终端应用向 stdout 打出的一些具有特殊含义的字符串. 终端看到这些串之后不会显示它们, 而是执行这些 ...继续阅读 (38)

A typical PyTorch training program on 8 GPUs with 4 dataloaderworkers per GPU would create at leastprocesses.A naive use of torch dataset and dataloader can easilyreplicate your dataset's RAM usage by 40 times. This issue has probably affected everyone who has done anything nontrivial with PyTorch.In this post, we will explain why it happens, and how toavoid the 40x RAM usage.All code examples and experiment results are available on github atppwwyyxx/RAM-multiprocess-dataloader.The content is not specific to PyTorch: it applies to any user of Python's multiprocessing library on Linux ...继续阅读 (44)

"Loss function" is one of the most basic concepts today in deep learning.Despite that,it is actually not necessarily a good programming abstraction whendesigning general-purpose systems. A system should not assume thata model always comes together with a "loss function"."Loss Function": Separation of Logic"Loss function" may mean different things in different systems.The version I'm going to criticize is the most common one that looks like below:BadWorsedeftrainer_bad(data, model, loss_func):whileTrue:inputs =next(data)predictions = model(inputs)loss = loss_func(predictions, inputs)# comp ...继续阅读 (40)

Building a library for research and experiments is quite different from building other types of software.A key challenge is that, in research, abstractions and APIs are rarely set in stone:users may want to propose a slight variant or modification to literallyANYWHEREin the whole program,just because they have a new idea.In deep learning libraries, these variants can be a different implementation of a layer,a change in optimization algorithm, or a small modification to the training logic, etc.When users want to make such changes, they often implement variants by simply adding featuresto the ta ...继续阅读 (38)

This post is about a small functionality that is found useful in TensorFlow / JAX / PyTorch.Low-level components of these systems often use a plain list of values/tensorsas inputs & outputs.However, end-users that develop models often want to work with morecomplicated data structures:Dict[str, Any],List[Any], custom classes, and their nested combinations.Therefore, we need bidirectional conversion between nested structures and a plain list of tensors.I found that different libraries invent similar approaches to solve this problem, and it's interesting to list them here.Nested Containers A ...继续阅读 (39)

PyTorch provides two methods to turn annn.Moduleinto agraph represented in TorchScript format: tracing and scripting.This article will:Compare their pros and cons, with a focus on useful tips for tracing.Try to convince you thattorch.jit.traceshould be preferred overtorch.jit.scriptfor deployment of non-trivial models.The second point might be an uncommon opinion:If I Google "tracing vs scripting", thefirst articlerecommends scripting as default.But tracing has many advantages.In fact, by the time I left,"tracing as default, scripting only when necessary"is thestrategy all detection & segmenta ...继续阅读 (39)

本系列文章谈谈 Github 上如何交流 (1)如何科学的报 bug如何管理 issue{Feature,Pull} Request这篇文章说说用户怎么提出好的 feature request / pull request, 以及维护者如何对待它们.这里, 我们忽略那种特别简单的 (例如 10 行代&# ...继续阅读 (40)

本系列文章谈谈 Github 上如何交流 (1)如何科学的报 bug如何管理 issue{Feature,Pull} Request我听过不少人凭借爱好开源了自己的项目后, 却对 issue 太乱感到困扰, 甚至想干脆直接禁用 issue. 事实上, 任何项目达到 ...继续阅读 (44)

本系列文章谈谈 Github 上如何交流 (1)如何科学的报 bug如何管理 issue{Feature,Pull} Request报告错误 / 报 bug 是用户与开发者间最常见的一类交流, 也是常见的 github issue. 但是很多用户并不会科学的报 bug, maintainer 对此也&# ...继续阅读 (41)

本系列文章谈谈 Github 上如何交流 (1)如何科学的报 bug如何管理 issue{Feature,Pull} Request相比传统的邮件列表 / bugzilla/sourceforge 等开源平台, github 把开源社区交流的成本 / 门槛降的很低, 因此交流的质量也常常随之Ӡ ...继续阅读 (42)

In large systems, logs can be terrifying: they are huge in volume, and hard to understand.This note lists some suggestions and common misuse of Python'sloggingmodule,with the aim of:Reduce redundant logs & spams from libraries.Allow more control of logging behaviors.Make logs more informative to users.Libraries must not configure the root loggerLoggers are globally identified by a dot-separated name given tologging.getLogger(name), such aslibrary.module.submodule.The logger named by an empty string is the "root logger".Libraries must not calllogging.basicConfigor configure the root logger ...继续阅读 (38)

本系列文章Deep Learning Experiments and ClaimsHow to do Ablation Experiments延续上一篇文章, 再说一说怎么科学的在 paper 里做 ablations.Ablation / 控制变量一组理想的 ablation 实验, 应当所有实验尽量使用一份代码实现, 和相同的实验 recipe, 这样才 ...继续阅读 (37)

Technically, an image is a function that maps acontinuousdomain, e.g.a box, to intensities such as (R, G, B).To store it on computer memory, an image isdiscretizedto an arrayarray[H][W], where each elementarray[i][j]is apixel.How does discretization work? How does a discrete pixel relate to the abstract notion of the underlying continuous image?These basic questions play an important role in computer graphics & computer vision algorithms.This article discusses these low-level details, and how they affect our CNN models and deep learning libraries.If you ever wonder which resize function to use ...继续阅读 (42)

这几年来, 从 FAIR 的几位大佬身边学习到的最多的是对待 research 的态度. 因此说说写 paper 和做实验的体会.实验与 claims实验是为了证明或强化文章里给出的 claim/hypothesis 的.Ross ICCV 2019 tutorial最后谈了谈怎么写 pa ...继续阅读 (39)

开源工具链里有很多陈年小 "feature", 最初由于各种原因 (例如作为 workaround) 实现了之后, 即使语义模糊或设计不合理, 也因为兼容性被留到了今天.STB_GNU_UNIQUE就是 ELF 中一个不太好的设计, 带来了不&# ...继续阅读 (39)

吐个小槽. 很久以前有次我在知乎上的一个回答里夸了 TensorFlow 1.x, 然后被人抱怨说 graph mode 写不了 IfElse 不能忍.然而, PyTorch 就可以写 IfElse 了?最近 detectron2 遇到的产品 / 部署的需求越来越多, 看看 PyTorch 五花八&#x ...继续阅读 (43)

TL;DR: How to find out if your favorite deep learning library is occasionally giving youwrong results? Such bugs happen from time to time, and are extremely difficult to notice, report, and debug.Three years ago, I wrote an articleUnawareness of Deep Learning Mistakes:buggy code can still train and appear to work, so it's difficultfor users to realize that their code is wrong.What's apparently more difficult to find out, is when the bug comes from the deep learning library we use.Imagine, what if the library unfortunately computes wrong results for certain parts of our model during t ...继续阅读 (40)

Python's package management is a mess.I'm involved in a few open source projects and Ioften help users address their environment & installation issues.A large number of these environment issues essentially come down toincorrectly / accidentally mixing multiple different python environment together.This post lists a few common pitfalls and misconceptions of such.The ProblemsPeople often unfortunately have multiple python binaries and multiple installations of python pckages, e.g.:The OS's package manager can install python and some python packages.Example:/usr/bin/python,/usr/lib ...继续阅读 (35)

安全比赛 Geekpwn 今年开始强调 "人工智能安全", 办了一个 CAAD 对抗样本攻防赛. 这让我在多年没碰 CTF 之后又有了一个 CTF 的机会, 只是这次攻击的是 ImageNet 分类器..整个 CAAD 比赛分为 online 和 live CTF 两部分.o ...继续阅读 (36)

House3D是一个用于 research 的交互式 3D 环境. 用户可以载入一个来自 SUNCG 数据集的房子的模型, 然后在里面走来走去, 并获得 first-person view 的图片输入.我写了 House3D 的渲染代码, 过程中踩到了不少神奇的坑 ...继续阅读 (41)

TL;DR: People are hardly aware of any deep learning mistakes they made,because things alwaysappearto work, and there are no expectations on how well theyshouldwork.The solution is to try toaccurately reproduce settings & performanceof high-quality papers & code.The Dropout MistakeI'll start with a true story.Years ago I was playing with different ConvNets with a couple of students at Berkeley.There weren't so many great softwares at that time, and we just wrote everything on top ofTheano, which was also very premature back then.We just started to learn about neural networks, and I wa ...继续阅读 (39)

因为工作原因, 第一次开始尝试使用OSX. 这里列一下一些主要的问题, 方便从GNU/Linux切换过来的的时候参考. 有了解决方案或新的问题我会继续更新. 这里的问题有一些只是习惯不一致, 我可以通过在Linux上配置新的使用习惯来调整自己, 另外一些则是功能缺失.现在还没决定要不要长期用下去, 因为以下不少问题还解决不好. 当然OSX的优点也很明显:能够使用outlook客户端处理公司邮件/calendar; UI好看(主要是字体渲染); 不容易出硬件问题(连不上wifi/投影之类的).Chrome由于在可见的未来仍然会主要使用Linux, 因此在两边维持尽量一致的key binding是很重要的. 这主要是针对核心程序(浏览器+终端)和窗口管理.Chrome的key modifier有时候是cmd, 有时候是ctrl. 凡是menu bar点开能看到的快捷键, 都可以在System Preference - Keyboard - Shortcuts - App Shortcuts中进行override, 所以ctrl+T, ctrl+W等等都能用. 目前剩下的主要问题是, 打开新标签页的cmd+click, 不知道如何改为ctrl+click.Linux可以设置gtkrc来统一所有gtk程序的key binding, 因此我可以在Chrome的所有输入框里使用emacs ...继续阅读 (39)

因为工作原因, 第一次开始尝试使用 macOS. 这里列一下一些主要的问题, 方便从 GNU/Linux 切换过来的的时候参考. 有了解决方案或新的问题我会继续更新. 这里的问题有一些只是习惯不一 ...继续阅读 (39)

THU本科期间很幸运接触了CS的很多个topic,每一个topic都有一些让我觉得十分吸引人或者印象深刻的小事.Machine Learning: 第一次做ML是用LogisticRegression打了个比赛,中间一天晚上亲眼看到自己想的feature work了.Security: 和室友在寝室一起打CTF.以及去FIT楼听蓝莲花大黑阔们说我听不懂的黑话. 15年Las Vegas看到奇装异服的DEFCON参会者.Database:SIGMOD14, 每天程序能快一秒就觉得今天没白过.中间有 一天下午和tim在C楼顶试图写一个类似于OS kernel的内存管理器,被各种如何加锁的问题搞的头晕.OS: 考前和宋教授一起研读ucore代码到两三点,亲眼看见page swap,interrupt这些东西如何在一个简易OS里实现.HPC: ASC比赛一结束中大就跑来问"你们是不是用了乘法分配律!".ISC我的两个task都很刺激, 第一个task临时发现需要有大RAM和X的机器做显示,多亏国内同学帮忙出图;第二个提心吊胆45分钟怕机器烧掉.Vision: 在Google的manager给MTV demo了我的项目,当场几个人喊launch it.Graphics: 在学堂调了一下午的kdtree corner case,以及一晚上把一个非真实感模型改成了刚跟tim学的MCPT,画出 ...继续阅读 (39)

1一眨眼就已经毕业了, 在 CMU 呆了一年半, 大概只做了三件事情.第一个学期又看了一点 geometry, 然后趁机把全景拼接整体上做的科学了. 尤其是写了 BA 之后, 能把理论上该拼好的图都拼好& ...继续阅读 (37)

这个领域里,什么都特别快.三个月前看到Bengio组的BinaryConnect. 脸草的同事都很喜欢模型加速/压缩的主题,因此立刻就重现了结果开始改进. 当时就说要做成Binary Activation,并且搞一个GPU runtime. 正当同事们回家过年,我在yy这学期parallel大作业要不就写这个runtime的时候,昨天看到Bengio新的paper挂出来,已经都做完了. 更夸张的是,在前天arxiv挂了另一篇文章,方法基本一样.三个月,能专心做的话并不难,然而我要应付作业,要去oculus写代码,还有其他好玩的东西在分心. 想着有空慢慢做的时候,别人已经不等你了.一个月前Yuandong学长说可以来FAIR跟他做围棋. 目前的CNN方法很promising,问题本身也很酷,觉得是个好项目. 然而过几天DeepMind出来说,我们早都做差不多了,三月份韩国比赛,指不定就problem solved了. 我都不知道该期待李世乭赢还是输.看了一篇博文和Yuandong学长的专栏, 感慨万千. DeepMind的paper看下来,很多人说没看到新鲜的东西. 用CNN做估值不算难以想到的idea, 用SL和RL像是他们试了试不同模型挑了个最好的. 然而DeepMind有一帮有时间的人,有钱有系统,能以极致的规模把问题给solve掉, 上了nature封面. 而我们只能在家里看 ...继续阅读 (87)

This is a summary of the algorithms I used to write the open source panorama stitcher. You can find the source code ongithub.SIFT FeatureLowe's SIFT [1] algorithm is implemented infeature/. The procedure of the algorithm and some results are briefly described in this section.Scale Space & DOG Space (feature/dog.cc)A Scale Space consisting of $ S \times O $ grey-scale images is built at the beginning. The original image is resized in $ O$ different sizes (AKA. octaves), and each is then Gaussian-blurred by $ S$ different $ \sigma$. Since features are then detected on different resized versi ...继续阅读 (57)