A blog by sunfishcode
Uninitialized buffers in Rust are a long-standing question, for example: https://rust-lang.github.io/rfcs/2930-read-buf.html https://doc.rust-lang.org/nightly/unstable-book/library-features/core-io-borrowed-buf.html https://blog.yoshuawuyts.com/uninit-read-write/…
Consider the following Unix shell command: $ a | b | c > d This runs program a , pipes the output to program b , pipes that output to program c , and finally redirects that output to file d . If we think of the process tree, we could draw a diagram for it like this: a b c ↑ ↑ ↑ └─shell─┘ The shell spawns processes for a , b , and c , wires up their file descriptors to connect their I/O in the…
Streams are an essential feature of any large modular system. Whether they're in the form of channels, pipes, sockets, iterators, async coroutines, or other things, streams represent sequences of data over time, and can connect two pieces of software with minimal coordination. WASI is being designed to support large modular systems, so WASI needs a solid stream design. This is a blog post…
A while ago I wrote a blog post about "Bugs in Hello World" , about how a lot of programming languages' default way of printing to stdout silently swallow errors. This led to a repo for maintaining lists of languages that do and don't have this bug, and some examples of how to fix the bug. For example, in C, the fixed Hello World looks like this: #include < stdio.h > #include < stdlib.h…
When we first designed the filesystem API in WASI, we had a lot of questions. When a Wasm module has an import , what does that mean? How are import names resolved? What do we know about the state of the world outside of what the spec calls the "store"? Capabilities are pretty cool; what's the best way to incorporate capabilities into filesystem APIs? At the time, we mainly wanted to keep our…
One of the core design goals for Wasm is to support code compiled from many different programming languages. To this end, core Wasm's type system is very low-level. It's designed in view of the fact that programming languages all have their own ways of doing things. Even seemingly simple things like strings or dictionaries can have different semantics or performance tradeoffs between…
The WASI Subgroup has just voted to launch WASI 0.2, also known as WASI Preview 2 or WASIp2! This blog post is a brief look at the present, past, and future of WASI. The present The Subgroup voted to launch WASI 0.2! This is a major milestone! We made it! At the same time, the journey is only just beginning. But let's talk this moment to step back and look at what this means. Most…
WASI Preview 2 is also known as WASI 0.2. and I've gotten feedback that that name makes more sense to people, so I've renamed my post accordingly. Here's the new URL for this page
Eyra is challenging ideas about what it means to be a libc. In doing so, it's doing a few things often considered to be... impossible 😉. Fixing Rust's set_var unsoundness rust-lang/rust#27970 is a soundness bug in Rust. It is a way that Rust programs can segfault without using any unsafe code. The bug was opened 8 years ago, and it's widely believed to be impossible to fully…
So yeah, what... What is Wasm? Well, I guess, one can find various one-sentence descriptions out there. The webassembly.org website leads with: WebAssembly (abbreviated Wasm ) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable compilation target for programming languages, enabling deployment on the web for client and server applications.…
This post is in a series about "Everything Is A File": Is Everything A File? Measuring System Interface Complexity What does Everything Is A File do? The Filesystem Namespace (this post) Blog post content coming soon!
This post is in a series about "Everything Is A File": Is Everything A File? Measuring System Interface Complexity What does Everything Is A File do? (this post) The Filesystem Namespace In particular, this post is looking at the "Everything is a file descriptor " meaning. To see how the Everything Is A File approach works in practice, let's look at three kinds of classic resources that have…
Mustang gained lots of new features recently. It's now complete enough to run lots of real-world Rust programs, with no libc: Tide's hello web server example (epoll and async!) Tokio's tinyhttp web server example (more epoll and async!) Most of coreutils (lots of stuff!) Ripgrep (threads!) See here for instructions on how to port programs to mustang. And if you give it a try, post…
This post is in a series about "Everything Is A File": Is Everything A File? Measuring System Interface Complexity (this post) What does Everything Is A File do? The Filesystem Namespace One of the most easily quantifiable ways to measure the "simplicity" or "elegance" or an operating system this is to count the number of functions in its system interface, or "syscalls". Linux has over 300.…
This blog post aims to provide a simple answer to the question: What is a Capability? I answer this question from my own perspective, as someone who didn't previously know anything about component models. This post is forward-looking; not all of the pieces described here are usable yet. It's a look at what's coming. Preliminaries When I say "capabilities", I'm talking about…
This post is the first in a series about "Everything Is A File": Is Everything A File? (this post) Measuring System Interface Complexity What does Everything Is A File do? The Filesystem Namespace "Everything Is A File" is one of the most fundamental and widely influential pillars of Unix design philosophy. It's one of the major sources of simplicity that propelled Unix beyond its predecessor…
The research paper A Fork in The Road contains a good summary of the problems with the fork call in Unix. As the paper points out, posix_spawn provides an alternative to fork which solves some of the performance problems. However, if we set aside the performance side for the moment, posix_spawn 's API kind of gives an impression that, if that's the alternative, perhaps we should…
Worlds have emerged as an important tool for WASI. Luke Wagner's The Path to Components talk gives a great introduction to Worlds, with the story to how we got here, and the path ahead. WebAssembly/wasi-kv-store#2 is a PR to the wasi-kv-store showing a concrete example of Worlds, and WebAssembly/component-model#83 is a PR to the component-model spec with a lot of detailed…
When I first heard people in the Wasm world talking about these things called "components" for Wasm, I had no idea what they were talking about. Other people have now written about what a Wasm component is, such as this blog post from Fermyon which makes an analogy relating Wasm concepts to native code and OS concepts, and this podcast with Lin Clark which makes an analogy between Wasm components…
This post proposes and explores a design principle for components in complex software systems: No Ghosts! The ideas in this post aren't new; they come from papers and blog posts such as Robust Composition , Capabilities: Effects for Free , Parse, don't validate , the nanoprocess model , and the design choices in the Wasm component model , which itself incorporates ideas from Erlang,…
Hello World might be the most frequently written computer program. For decades, it's been the first program many people write, when getting started in a new programming language. Surely, this humble starting-point program should be bug free, right? After all, hello world programs only do one thing. How could there be a bug? Hello world in C There are a lot of different ways to write hello…
Rustix is a system-call wrapper library with multiple backends. It has a raw Linux syscalls backend, as well as a libc backend, and other backends are in development. Rustix is designed for memory safety, I/O safety , and performance. And this is a branch of Rust's std partially ported to use rustix in place of direct libc calls. Read on for why this is cool, and stay for the benchmarks!…
This is a blog post brainstorming about contexts . I'll us the term contexts here, as tmandry is leaning to , since it seems to make sense to keep capabilities distinct concepts. Idiomatic capability-based code and the Principle of Least Authority prefer fine-grained access to resources , which contexts don't seem like a good fit for. So let's keep these concepts distinct for now.…
In order to make large applications modular, we need to think about resources and sharing. One axis for thinking about this is the spectrum of granularity. It's a spectrum, but we can identify several notable levels: Level 0: File::open Level 1: with root: &Dir Level 2: dir.open Level 3: thing: &File Level 4: thing: &StreamReader This post describes each of these levels and considers the…
Mustang , a system for running Rust programs entirely written in Rust, has make a lot of progress since the last blog post: New targets: riscv64 and arm, joining x86_64, aarch64, and x86; thanks to @Urgau for arm support in rustix! Threading support, including TLS, TLS destructors, and detaching Panic and unwind support, thanks to the unwinding crate! Math library support, thanks to the libm…
mustang is a system for writing Rust programs entirely in Rust, meaning they don't use libc , crt1.o , or any C code. It's experimental, but it's complete enough to run a std -using Hello World and other simple programs on Linux on x86-64, x86, and aarch64. See mustang 's README for information on how to use it. This post walks through the major steps leading to this point.…
What kinds of bugs should safety in Rust protect against? Rust clearly wants to say that safety is about protecting programs against some kinds of bugs, but not all possible bugs. Where should the boundary be? Safety should at the very least mean protection against memory corruption through dangling and out-of-bounds pointers. To do that, it's necessary to protect against all Undefined…
Posish has been renamed to Rustix. Here's the new URL for this page !
Rsix has been renamed to Rustix. Here's the new URL for this page !
Rustix: I/O Safety in practice The I/O Safety RFC is now merged! But it's all fairly abstract, so what will this look like in practice with real APIs? One of the ways is rustix, the fastest POSIX-ish, Unix-ish, Linux-ish, and libc-ish API for Rust! And it's also I/O-safe! And memory-safe! These are actually the bigger motivators for this crate, and it helped guide the…
Introducing cap-std cap-std is a project to create capability-based versions of Rust standard library and related APIs. Capability-based here means that the APIs don't access files, directories, network addresses, clocks, or other external resources implicitly, but instead operate on handles that are explicitly passed in. This helps programs that work with potentially malicious content avoid…
@withoutBoats makes this observation about Rust in "Notes on a smaller Rust" : Pure functional programming is an ingenious trick to show you can code without mutation, but Rust is an even cleverer trick to show you can just have mutation. A particular aspect I'd like to explore here is: Can we apply this observation to I/O? Haskell also has an ingenious trick to do first-class I/O…
Canonicalization and canonical forms are one dimension of organizing the work of an optimizing compiler. Intro A lot of code constructs can be written in multiple ways. For example: x + 4 4 + x (x + 2) + 2 Canonicalization means picking one of these forms to be the canonical form , and then going through the program and rewriting all constructs which are equivalent to the canonical form into the…