Steve Klabnik — Rust

Arguing about arguments

Mon, 21 Sep 2026 00:00:00 GMT

Here’s some Rust code:

// define a function
fn foo(x: i32, y: i32) -> i32 {
    // body elided
}

// call it
let z = foo(5, 6);

In programming language jargon, we call x and y parameters and 5 and 6 arguments.

Rust does not have many fancy features related to parameters and arguments. But other languages do. For example, here’s a function in Ruby, another language I’ve written a ton of:

# its definition in Rails
def redirect_to(options = {}, response_options = {})
  # body elided
end

# you can call it in all of these ways:

# pass a string with a URL
redirect_to "http://www.rubyonrails.org"

# pass an instance of a model to go to it, if post.id = 5 then this might go to
# "/posts/5"
redirect_to @post

# invoke an action directly, might be equivalent to the above in a Posts controller
redirect_to action: "show", id: 5

# redirect to a model's URL, passing a flash message
redirect_to post_url(@post), alert: "Watch it, mister!"

# do the same but also set a specific HTTP code
redirect_to post_url(@post), status: :found

# do both
redirect_to post_url(@post), status: 301, flash: { updated_post_id: @post.id }

I used to love Ruby, and now I love Rust. And to be honest, redirect_to and friends are a huge reason why I am so against Rust gaining these sorts of fancy features: they can be really concise, and beautiful to a certain kind of eye, but it also makes it really, really hard to know what all you can do with a function. You pretty much have to hope that someone has written good documentation, and while Rails’s documentation is pretty good, not every library is going to have it.

To be clear about it, there are several things bundled together here: flexible argument types, options hashes, defaults, and keyword-looking syntax. It’s that whole style of API that shaped my skepticism, not argument labels by themselves.

So, what exactly are all of these features?

Named parameters

The simplest one is named parameters. As you might guess, this means that we can explicitly use parameter names at the call site:

// without named parameters
let z = foo(5, 6);

// if Rust had them
let z = foo(x: 5, y: 6);

Named parameters are nice because they can give you more information at the call site, but also they can be more flexible. Some languages also let you supply named arguments in a different order:

// since we have names, we can write this:
let z = foo(x: 5, y: 6);

// but we can also write this:
let z = foo(y: 6, x: 5);

The downside of named parameters is that they can get quite verbose:

# FastAPI in Python, UserOut class not shown
@app.get(
    "/me", 
    response_model=UserOut, 
    status_code=200, 
    tags=["Users"],
    summary="Get current user profile"
)
def get_current_user():

I tried to not strawman this by showing an extreme example, but get can take 23 different keyword arguments, and so this could get quite, quite long.

Named arguments can be good when passing expressions, so we can understand something more at the call site. In the above, status_code=200 is much nicer than a bare 200, if you know HTTP well it’s kind of obvious what it is, but the additional clarity is nice. But it can be common to break truly complex examples down into variables that we end up forwarding to the function, and then it becomes verbose and redundant. As an example from inside FastAPI where get is invoked directly with all of those options:

self.router.get(
    path,
    response_model=response_model,
    status_code=status_code,
    tags=tags,
    dependencies=dependencies,
    summary=summary,
    description=description,
    response_description=response_description,
    responses=responses,
    deprecated=deprecated,
    operation_id=operation_id,
    response_model_include=response_model_include,
    response_model_exclude=response_model_exclude,
    response_model_by_alias=response_model_by_alias,
    response_model_exclude_unset=response_model_exclude_unset,
    response_model_exclude_defaults=response_model_exclude_defaults,
    response_model_exclude_none=response_model_exclude_none,
    include_in_schema=include_in_schema,
    response_class=response_class,
    name=name,
    callbacks=callbacks,
    openapi_extra=openapi_extra,
    generate_unique_id_function=generate_unique_id_function,
)

We actually snuck in another two features into this example: how did we only pass five arguments into a function that has 23 parameters?

Optional and/or Default arguments

As the name implies, optional arguments is a feature where you can choose to not pass in an argument, and default arguments lets you set a default when an argument is not passed.

To go back to that example:

@app.get(
    "/me", 
    response_model=UserOut, 
    status_code=200, 
    tags=["Users"],
    summary="Get current user profile"
)
def get_current_user():

We only passed in five of the 22 possible arguments. If we had to pass all 22 every time, that would be extremely verbose. So by declaring defaults for some parameters, we can leave them off of the call site, and not pass arguments for them, and get the defaults instead. We saw this in the Ruby:

# its definition in Rails
def redirect_to(options = {}, response_options = {})

The = {} are defaults for options and response_options: they’re an empty hash map.

You’ll often find optional and default arguments come together: after all, if you don’t pass an argument in, what should the value of that parameter be? If your language has some sort of null value, that can be one solution, and it can feel like optional arguments without default arguments.

But you can truly get one without the other with our next feature, function overloading:

Function Overloading

Languages that support function overloading allow you to define multiple functions with the same name, but different signatures. Which function gets invoked depends on which arguments you pass. This lets us truly have optional arguments without default arguments. In Java:

// Version 1: Requires both arguments
void connect(String url, int timeout) { ... }

// Version 2: Timeout is optional to the caller, and handled internally
void connect(String url) {  ... }

Here, if you pass in a timeout, you get the first function, and if you don’t, you get the second. Instead of null, the parameter just doesn’t exist at all.

This is a specific form of something called “function dispatch,” which basically means “hey when a function is invoked, what exactly gets called?” There are a lot of different ways to do this, and I don’t want this post to get into all of that right now, but there is a multitude of possibilities here: static vs dynamic, single vs multiple, predicate dispatch, pattern matching dispatch, prototype based dispatch… maybe I’ll write a post about that someday too.

So, what are the pros and cons here?

These features make me uneasy

As I said above, Rust doesn’t support any of these three features right now. And they have been a persistent request from the community for years. Here is a 12 year old GitHub issue about this support, and it even includes something we didn’t talk about, “variable arity argument lists.”

That gets me to the first thing that I don’t like about these features: there are a lot of them. And they are all intertwined. As we mentioned before, if you have optional arguments, you probably want default arguments. If you have named parameters, do you want to support only named parameters, or do you want to support both? It’s so tempting to just support all of them, and then you’ve added a huge mountain of complexity.

In Rust right now, you have to write multiple functions with slightly different names:

// a new empty vector
let v = Vec::new();

// one with a set capacity
let v = Vec::with_capacity(5);

For the simple cost of “write two different names for your functions,” you get to keep the rules very simple: there is one function definition, and you invoke it by name. Done. Would it be nice to be able to say all of these:

let v = Vec::new();
let v = Vec::new(5);
let v = Vec::new(capacity: 5);

Sure, maybe. But the cost feels very, very high to me. If you’re used to these features, maybe it doesn’t seem that high, but I have a Ruby tattoo on my body. I’ve seen some shit. And so I’ve grown to enjoy Rust’s simplicity in this area. Yes, if you have a function with tons of parameters, you may need to write a builder instead, and that comes with its own form of verbosity:

// if Vec had a builder
let v = Vec::new()
    .with_capacity(5)
    .build();

// all of the code you have to write to make the builder elided

But the language itself stays simple, and the rules are easy. For a language that’s already perceived as complex, this has always felt right to me. And that’s why I’ve pushed back against the various proposals to extend Rust in this way all of these years.

But recently, I changed my opinion on one of these features, and one alone, and I would consider it okay if Rust gained them. Maybe. Also, it’s important to note that I haven’t worked on Rust in years, and so my opinions are kind of irrelevant, but whatever: it’s my blog, this is what opinions are for.

I’m okay with named parameters now

I think Rust could be okay with named parameters, but not optional or default ones. And what changed my opinion is coding agents, actually. I’m sorry, maybe you’re sick of me talking about AI, but hear me out.

One of my beefs with named parameters, as mentioned above, is verbosity. But what you gain for that verbosity is clarity. I don’t think that verbosity and clarity are always the same thing; I like {} rather than do/end, and find it more readable, personally. But let’s take this function from the image crate:

// definition
pub fn crop_imm<I: GenericImageView>(
    image: &I,
    x: u32,
    y: u32,
    width: u32,
    height: u32,
) -> SubImage<&I> { ... }

// calling it
let cropped = image::imageops::crop_imm(&img, 10, 20, 200, 100);

// if we had named arguments
let cropped = image::imageops::crop_imm(
    image: &img,
    x: 10,
    y: 20,
    width: 200,
    height: 100,
);

This does really increase readability, no question about it. But for me, typing all of that out just isn’t really super worth the squeeze. But when Claude is gonna write it? I care a lot less. And the readability advantage for humans is even stronger for agents: the extra clarity of the text inline seems to (I haven’t run real evals on this yet…) be more helpful and use less tokens and calls when looking at the actual call site. If an agent were reading that named example above, it could tell that 10 is the x value without having to go look up the function signature itself. “What’s good for humans is true for agents” strikes again.

This is also why I’m still against optional arguments: they deliberately hide what is being passed in to the function, and obscure clarity at the call site. What’s nice about them is that you have to type less. But I’m not typing myself anymore. So the benefit just isn’t there, but the drawbacks are. This also doesn’t solve the verbosity side of passing foo=foo in our FastAPI example, it is still redundant. But one part of the downsides have been ameliorated.

Notably, all of this reasoning also applies to the builder pattern, and it’s why I try to use builders sparingly in Rust. It can emulate named arguments, which is good, but it can also emulate optional/default/variable arguments, which is bad. The juice is worth the squeeze sometimes, but not all the time, and probably not even the majority of the time.

So we’ve ameliorated the largest downside, but get to keep the largest upside. That seems good. But even if we like the feature, there are still lots of practical concerns with Rust specifically that make it not a slam-dunk to add them to the language.

There’s tons of practical issues though

Now, “is this feature good in the abstract” is very different from “should this feature” be implemented? There are a lot of open questions and drawbacks to implementing named parameters. A simple one is that you don’t get rid of all of those previous functions, with_capacity will exist forever. That’s a relatively minor downside to some of the harder problems.

First, strictly speaking Rust parameters aren’t names, they’re patterns. A name is a simple pattern, but you can get more complex:

fn foo((x, y): (i32, i32)) {

This parameter doesn’t have a name, it’s a pattern that introduces two bindings. You could come up with some syntax that gives things external names vs internal names, but there’s that complexity revealing its head again.

What happens when functions become values? I can write this today:

fn resize(width: u32, height: u32) {
    // body elided
}

fn offset(dx: u32, dy: u32) {
    // body elided
}

let f: fn(u32, u32) = if resizing { resize } else { offset };

What name do we use for f’s parameters? They can be named different things. Does this become an error? What names can you use when invoking f? Making this more complicated, Rust actually will accept this code right now:

type Callback = fn(width: u32, height: u32);

fn f(g: Callback) {}

fn bar(x: u32, y: u32) {}

fn main() {
    f(bar) ;
}

The names in Callback are just documentation, as you can see they’re not required. Does this become an error? Do we require bar to change its names? The same goes for trait definitions, actually:

trait Writer {
    fn write(&mut self, data: &[u8]);
}

struct Sink;

impl Writer for Sink {
    fn write(&mut self, bytes: &[u8]) {
        // body elided
    }
}

Is this an error? Is it okay?

Here’s another problem: evaluation order.

fn consume(data: Vec<u8>, length: usize) {
    // body elided
}

Right now, Rust always evaluates parameters from left to right at the call site. So if we use the named version:

fn consume(data: Vec<u8>, length: usize) {
    // body elided
}

fn consume2(length: usize, data: Vec<u8>) {
    // body elided
}

fn main() {
    let data = vec![1, 2, 3];

    // this is an error, borrow after move
    consume(data, data.len());
    // this is okay
    consume2(data.len(), data);

    // does this compile or not?
    consume(length: data.len(), data: data);
}

If we keep left to right evaluation order according to the definition, but let you use named values in any order, suddenly some orders at the call site compile and some orders do not. Do we change evaluation order to do whatever compiles? That feels very dangerous and confusing.

Rust does let you do this with struct fields:

struct Args {
    data: Vec<u8>,
    length: usize,
}

fn main() {
    let data = vec![1, 2, 3];
    
    // works
    let args = Args {
        length: data.len(),
        data,
    };

    // doesn't
    let args = Args {
        data,
        length: data.len(),
    };
}

This is one reason why a lot of proposals for named arguments try to unify them with structs. As I said before, not insurmountable, just things that need to be sorted.

Another small thing is a compatibility hazard: if we start naming parameters, renaming them breaks callers. Is that okay? Could you introduce aliases to help with this? That’s more complexity…

These are just some of the problems that come up in the design space, and this post is getting long, so I’ll leave it at this for now. But proposals for doing this have to engage with all of this and more, which is why language design is difficult.

In conclusion

I like to think that I change my opinions over time as the world or my understanding of it changes. So I thought it might be interesting to share a way in which an opinion that I’ve held for over a decade has changed a bit lately. I’m still not sure if named parameters are right for Rust, but I’m at least somewhat open to the idea these days. We’ll see what the language team decides if anyone does decide to keep trying to push this forward. And people do, for example, I am writing this in part because I saw this post go by today. This post isn’t intended to be support or a rebuttal of it, I actually didn’t even read it yet, it’s just that I’ve been thinking about this a lot lately (while working on Rue) and seeing that go by pushed me to write some of the thoughts down. You should probably go check it out, as it’s an actual proposal in this space and this post is more of an exploration of the general idea and my own thoughts on it.

Here’s my post about this post on BlueSky:

Steve Klabnik @steveklabnik.com

Arguing about arguments steveklabnik.com/writing/argu...

Arguing about arguments

Here’s some Rust code: // define a function fn foo (x : i32 , y : i32 ) -> i32 { // body elided } // call it let z = foo ( 5 , 6 ); In programming language jargon, we call x and y parameters and 5 and 6 arguments . Rust does not have many fancy features related to parameters and…

steveklabnik.com

September 21, 2026 at 5:30 PM

92 11

Read 21 replies on Bluesky

Thirteen years of Rust and the birth of Rue

Sun, 21 Dec 2025 00:00:00 GMT

I started using Rust on December 21, 2012. It’s now December 22, 2025. That’s a long time.

I don’t have a ton of thoughts about Rust to share today, like I thought I might. I program in the language regularly, it’s nice, I prefer it to all others. I have my beefs and my regrets, like any real project, but for the most part, I am pretty happy these days. To be honest, after a pretty rough last few years, that’s been the theme of 2025: I’m pretty happy these days.

I have long been wondering if I should try and make my own language. I really like them! That’s part of why I got involved in Ruby and then Rust in the first place! But I’ll be turning 40 in a month, and I barely have time for things anymore. So I had kind of put that on the back burner.

But one thing that 2025 changed: I went from an AI skeptic to writing most of my code with Claude. I’m still a bit unsure about LLMs outside of programming, but as a software development tool, I’ve found them finally to be good enough to be legitimately useful. Your mileage may vary, I’m not really going to try and tell you that they should be for you, only that they are for me.

And so, earlier this year, I embarked on starting my own little language project. I decided to call it “Rue”, becuase:

It has to start with Ru, duh. I did Ruby and then Rust, so my hands are basically tied here…
It has a negative connotation, “to rue the day”, like Rust has a possible negative interpretation.
It has a nature connotation, a Rue, like Rust has a possible fungi connotation.
It’s short.

I have also been wondering for a few years if you can build a language without financial backing and a team. I think LLMs might change that equation. Yes, Claude costs money, but I mean like, you don’t need to do what Mozilla did and drop seven or eight figures worth of money on a development team to make meaningful progress. And since I don’t have a ton of personal time these days, what better way to test that than to actually give it a go.

Anyway, I got into it, and kind of made a mess of the first go. And then I had a tremendously busy fall, and had to put things by the wayside.

But this week, I had some time after work, and decided to start over. And I’m much more happy with how it’s going this time. So I’ve decided it’s time to at least let the world know I’m working on it, just because why not.

I don’t expect it to grow to anything more than my hobby project, but then again, that’s also true of Rasmus and Graydon and a ton of other language authors, so, never say never. But that’s not an explicit goal here. I reserve the right to just stop working on this when I think it’s not fun. I’m not trying to build a community or even add committers. It’s just something for me for fun.

You can find Rue over at https://rue-lang.dev/ if you want to check it out. There’s a blog with an RSS feed. I may blog there, I may not. But I’ll probably keep it over there more than over here, in general.

I’m not saying that Claude is amazing, or that this code is fantastic, or anything. The language is still very janky. I’m sure I have codegen bugs. It’s missing basic features. It is very, very much a work in progress.

Thank you to everyone for making 2025 a nice year. There’s a lot of terrible things in this world right now, but I’m more hopeful than ever, personally. I hope to still be writing Rust in another ten years, and maybe even writing some Rue. We’ll see :)

Here’s my post about this post on BlueSky:

Steve Klabnik @steveklabnik.com

Thirteen years of #rustlang and the birth of #ruelang steveklabnik.com/writing/thir...

Thirteen years of Rust and the birth of Rue

Blog post: Thirteen years of Rust and the birth of Rue by Steve Klabnik

steveklabnik.com

December 21, 2025 at 8:05 PM

111 10

Read 19 replies on Bluesky

Is Rust faster than C?

Mon, 09 Jun 2025 00:00:00 GMT

Someone on Reddit recently asked:

What would make a Rust implementation of something faster than a C implementation, all things being the same?

I think this is a great and interesting question! It’s really tough because it ultimately relies on what exactly you mean by “all things being the same.” And I think this is something that makes it hard to compare languages.

Here’s some ways in which you can argue that things are “the same” but also, that they’re not, and what they imply for runtime performance.

Inline Assembly

Rust has inline assembly built into the language. C has inline assembly as a very common compiler extension, to the degree where saying it’s not part of the language is an arguable nitpick.

Here’s an example in Rust:

use std::arch::asm;

#[unsafe(no_mangle)]
pub fn rdtsc() -> u64 {
    let lo:  u32;
    let hi:  u32;
    unsafe {
        asm!(
            "rdtsc",
            out("eax") lo,
            out("edx") hi,
            options(nomem, nostack, preserves_flags),
        );
    }
    ((hi as u64) << 32) | (lo as u64)
}

This reads the time stamp counter with rdtsc, and returns its value.

Here’s the example in C:

#include <stdint.h>

uint64_t rdtsc(void)
{
    uint32_t lo, hi;
    __asm__ __volatile__ (
        "rdtsc"
        : "=a"(lo), "=d"(hi)
    );
    return ((uint64_t)hi << 32) | lo;
}

These (under -0 in both rustc 1.87.0 and clang 20.1.0) produce the same assembly:

rdtsc:
        rdtsc
        shl     rdx, 32
        mov     eax, eax
        or      rax, rdx
        ret

Here’s a link on Godbolt: https://godbolt.org/z/f7K8cfnx7

Does this count? I don’t know. I don’t think it really speaks to the question asked, but it is one way to answer the question.

Similar code, different results

Rust and C can have different semantics for similar code. Here’s a struct in Rust:

struct Rust {
    x: u32,
    y: u64,
    z: u32,
}

Here’s “the same” struct in C:

struct C {
    uint32_t x;
    uint64_t y;
    uint32_t z;
};

In Rust, this struct is 16 bytes (on x86_64, again) and in C, it is 24. This is because Rust is free to reorder the fields to optimize for size, while C is not.

Is this the same, or different?

In C, you can re-order the fields to get the same size. In Rust, you can write #[repr(C)] to get the same layout as C. Does this mean we should have written different Rust or different C to get “the same” thing?

Some people have reported that, thanks to Rust’s checks, they are more willing to write code that’s a bit more dangerous than in the equivalent C (or C++), where they’d do a bit more copying to play it a bit safer. This would be “the same” in the sense of the same devs on the same project, but the code would be different, due to judgement calls. You can make an argument that that’s not the same, but is different too.

An example of this from a long time ago is the Stylo project. Mozilla tried to parallelize Firefox’s style layout twice in C++, and both times the project failed. The multithreading was too tricky to get right. The third time, they used Rust, and managed to ship. This is the same project, (though not the same programmers, I believe) by the same organization, but one was possible and one was not. Is that “the same”? In some senses, but not in others.

This also goes for a similar question: assuming we have a junior developer writing Rust, and also writing C, for the same task. Are we going to get faster code in one or the other? This controls for ability, but not for the same code. Is that “the same”? I don’t know. What about an expert in each language, someone who knows Rust super well but doesn’t know C, and vice versa, being given the same task? Is that different than a junior, or an “average” developer?

Compile time vs runtime?

Another redditor asks:

I’m not Rust expert, but aren’t most (all?) of the safety checks compile time checks? They shouldn’t have any runtime impact.

A great question! Part of this is another example of defaults being different.

array[0] is valid in both languages.

In Rust, there’s a bounds check at runtime. In C, there is not. Does this mean that they’re the same? In Rust, I could write array.get_unchecked(0), and get C’s semantics. In C, I could write a bounds check to get Rust’s semantics. Are any of those “the same”?

In Rust, that check may be optimized away, if the compiler can prove it’s safe. In C, if we wrote the bounds check by hand, the check may be optimized away, if the compiler can prove it’s safe. Are any of those “the same”?

They aren’t wrong that a lot of Rust’s safety checks are at compile time. But some are at runtime. But this raises another interesting question: That compile-time check may cause you to write different code for the same task as in C. A common example is using indices rather than pointers. That may mean that the generated code performs differently. Is that check truly “at compile time”? Technically, at the micro level, yes. At the engineering level? Possibly not!

My Conclusions

I think the most important part of this question is related to possibility, that is:

If we assume C is the ‘fastest language,’ whatever that means.
Is there any inherent reason why Rust could not do the same things?

I think the answer to that is “no,” even ignoring the inline assembly case. So on that most important, fundamental level, the answer is “there’s no difference between the two.”

But we’re not usually talking about that. We’re usually talking about something in the context of engineering, a specific project, with specific developers, with specific time constraints, and so on. I think that there are so many variables that it is difficult to draw generalized conclusions.

Here’s my post about this post on BlueSky:

Steve Klabnik @steveklabnik.com

Is #rustlang faster than C? steveklabnik.com/writing/is-r...

Is Rust faster than C?

Blog post: Is Rust faster than C? by Steve Klabnik

steveklabnik.com

June 9, 2025 at 6:04 PM

91 19

Read 10 replies on Bluesky

Rust 1.0, ten years later

Thu, 15 May 2025 00:00:00 GMT

It’s kind of hard to believe it’s been ten years since Rust 1.0 was released.

I stopped working on Rust three years ago, and frankly, the last three years have been pretty rough. But 2025 is shaping up to be a really good year for me, with lots of positive changes.

With that in mind, this post is going to be a bit of a different one than you might expect. For a fantastic post that’s a bit more traditional, check out what Graydon wrote for the Rust Foundation: 10 years of stable Rust: an infrastructure story. It’s fantastic, of course.

I’d instead like to reflect a bit on a comment I saw on the internet the other day:

[Rust is] not a great hobby language but it is a fantastic professional language, precisely because of the ease of refactors and speed of development that comes with the type system and borrow checker.

Patrick Walton, one of the earliest contributors to Rust, had this to say:

I never thought I’d live to see the day when someone would say this. The first 5 years of Rust were all “this is interesting for hobby projects but nobody will ever adopt this in industry”.

The release of Rust 1.0 was great, but it was also stressful. The few years before Rust 1.0 were marked by a deep, deep anxiety among the team. I’m sure someone else who was around back then will disagree with me, but that’s my recollection. It may not have always been overt, but it was there. A large question was looming over us: would anyone actually use this thing? Would it become a hobbyist language, or would it be something that people actually used in production?

At this point, we now know the answer: yes, Rust is used a lot. It’s used for real, critical projects to do actual work by some of the largest companies in our industry. We did good.

The release of 1.0 was the start of that journey in earnest. It was time to stop changing things all the time, and to start focusing on stability. It was time to gain more users, and to fix the issues that they found. We had to start shipping stable releases, and we had to start doing it on a regular basis. We had to start doing all of the things that a real programming language does.

Over the last ten years, a lot changed. It wasn’t easy, and it took a lot of blood, sweat, and many, many tears. But we did it. We built a real programming language, and we built a real community around it.

Rust is very much not perfect. There’s more work to be done, and arguably, someone out there should probably be starting a language that will someday eat Rust’s lunch.

But ten years ago, we were pushing a button and praying that it would all work out. And now, ten years later, I’m doing this:

❯ rustup update stable
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2025-05-15, rust version 1.87.0 (17067e9ac 2025-05-09)
info: downloading component 'rust-src'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: downloading component 'rustfmt'
info: removing previous version of component 'rust-src'
info: removing previous version of component 'cargo'
info: removing previous version of component 'clippy'
info: removing previous version of component 'rust-docs'
info: removing previous version of component 'rust-std'
info: removing previous version of component 'rustc'
info: removing previous version of component 'rustfmt'
info: installing component 'rust-src'
info: installing component 'cargo'
info: installing component 'clippy'
info: installing component 'rust-docs'
 19.9 MiB /  19.9 MiB (100 %)  16.3 MiB/s in  1s ETA:  0s
info: installing component 'rust-std'
 29.4 MiB /  29.4 MiB (100 %)  22.2 MiB/s in  1s ETA:  0s
info: installing component 'rustc'
 76.3 MiB /  76.3 MiB (100 %)  23.9 MiB/s in  3s ETA:  0s
info: installing component 'rustfmt'

  stable-x86_64-unknown-linux-gnu updated - rustc 1.87.0 (17067e9ac 2025-05-09) (from rustc 1.86.0 (05f9846f8 2025-03-31))

and then going about my day. It is a rare privilege to be able to spend ten years building a tool you want to use, and then be able to spend your time at work using it. I am grateful to have had that opportunity, and I hope that everyone else who has worked on Rust over the years feels the same way.

Here’s my post about this post on BlueSky:

Steve Klabnik @steveklabnik.com

#rustlang 1.0, ten years later steveklabnik.com/writing/rust...

Rust 1.0, ten years later

steveklabnik.com

May 15, 2025 at 5:49 PM

198 35

Read 13 replies on Bluesky

Thinking like a compiler: places and values in Rust

Mon, 07 Apr 2025 00:00:00 GMT

A while back, someone on the internet asked about this syntax in Rust:

*pointer_of_some_kind = blah;

They wanted to know how the compiler understands this code, especially if the pointer wasn’t a reference, but a smart pointer. I wrote them a lengthy reply, but wanted to expand and adapt it into a blog post in case a broader audience may be interested.

Now, I don’t work on the Rust compiler, and haven’t really ever, but what I do know is language semantics. If you’re a language nerd, this post may not be super interesting to you, other than to learn about Rust’s value categories, but if you haven’t spent a lot of time with the finer details of programming languages, I’m hoping that this may be a neat peek into that world.

Programming languages are themselves languages, in the same sense that human languages are. Well, mostly, anyway. The point is, to understand some Rust code like this:

*pointer_of_some_kind = blah;

We can apply similar tools to how we might understand some “English code” like this:

You can't judge a book by its cover.

Oh, you also might ask yourself as you’re reading the next sections, “why so many steps?” The short answer is a classic one: by breaking down the big problem of “what does this mean?” into smaller steps, each step is easier. Doing everything at once is way more difficult than a larger number of smaller steps. I’m going to cover the classical ways compilers work, there’s a ton of variety when you start getting into more modern ways of doing things, and often these steps are blended together, or done out of order, all kinds of other things. Handling errors is a huge topic in and of itself! Consider this a starting point, not an ending one.

Let’s get into it.

Lexical Analysis (aka ‘scanning’ or ‘tokenizing’)

The first thing we want to do is to try and figure out if these words are even valid words at all. With computer languages, this process is called “lexical analysis,” though you’ll also hear the term “tokenizing” to describe it. In other words, we’re not interested in any sort of meaning at all at this stage, we just want to figure out what we’re even talking about.

So if we look at this English sentence:

You can't judge a book by its cover.

We follow a two step process in order to tokenize it: we first “scan” it to produce a sequence of “lexemes.” We do this by following some rules. I’m not going to give you a sample of the rules for English here, as this post is already long enough. But you might end up with something like this:

You
can't
judge
a
book
by
its
cover
.

Note how we do have ' in can't, but . is separate from cover. These are the kinds of rules we’d be following: the ' is because this is a contraction, but the . is not really part of cover, but its own thing.

We then run a second step and evaluate each individual string of characters, turning them into “tokens.” A token is some sort of data type in your compiler, probably, so for example we could do this in Rust:

enum Token {
    Word(String),
    Punctuation(String),
}

And so the output of our tokenizer might be an array of something like

[
    Word("You"),
    Word("can't"),
    Word("judge"),
    Word("a"),
    Word("book"),
    Word("by"),
    Word("its"),
    Word("cover"),
    Punctuation("."),
]

At this point, we know we have something semi-coherent, but we’re still not sure it’s valid yet. On to the next step!

Syntactic Analysis (aka ‘parsing’)

Funny enough, this is an area where human-language linguistics and compilers mean things that are slightly different. With human language, parsing is often combined with our next step, which is semantic analysis. But we (most of the time) try to separate syntax and semantic analysis in compilers.

Again, I’m going to massively simplify with the English. We’re going to use these rules for what a sentence is:

A sentence is a sequence of words.
A sentence has a “subject,” which is the first word, and must be capitalized.
A sentence ends with a period.

Obviously this is a tiny subset of English, but you get the idea. The goal of syntactic analysis is to turn our sequence of tokens into a richer data structure that’s easier to work with. In other words, we’ve figured out that our sentence is made up of valid sequences of characters, but do they fit the grammatical rules of our language? Note that we also don’t need to store everything; for example, maybe our English data structure looks like this:

struct Sentence {
    subject: String,
    words: Vec<String>,
}

Where’s the period? How do we know subject must be capitalized? That’s the job of syntactic analysis. Since every sentence ends with a period, we don’t need to track it in our data structure: the analysis makes sure that it’s true, and then we aren’t doing it again. Likewise, we don’t need to store our subject as a capitalized string if we don’t want to: we determined that the input was, but we can transform it as needed. So our value after syntax analysis might look like this:

Sentence {
    subject: "you",
    words: ["can't", "judge", "a", "book", "by", "its", "cover"],
}

Often, for computer languages, a tree-like structure works well, and so you’d see an “abstract syntax tree,” or “AST” at the end of this stage. But it’s not strictly speaking required, whatever data structure makes sense for you can work.

Now that we have a richer data structure, we’re almost done. Now we have to get into meaning.

Semantic Analysis (aka ‘wtf does this mean’)

Imagine our sentence wasn’t “You can’t judge a book by its cover.” but instead this:

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua.

This is a famous bit of text that’s incoherent. The words are all Latin words, and it feels like it might be a sentence, but it’s nonsense. We could parse this into a Sentence:

Sentence {
    subject: "Lorem",
    words: ["ipsum", "dolor", "sit", "amet", /* and continued */ ],
}

But it’s not valid. How do we determine that?

Well, in the context of English, “Lorem” isn’t a valid English word. So if we were to check that the subject is a valid word, we’d successfully reject this sentence. In computer languages, we’d do similar things like type checking: 5 + "hello" might lex and parse just fine, but when we try and figure out what it means, we learn it’s nonsense. Except if your language lets you add numbers and strings!

After semantic analysis, we’ve determined that our program is good, aka “well formed.” In a compiler, we’d then go on to generate machine code or byte code to represent our program. But that stuff, while incredibly interesting, isn’t what we’re talking about here: remember our original objective? It was to understand this:

*pointer_of_some_kind = blah;

That’s semantics. So now that we have some background, let’s talk about how to understand this code.

So… how do I understand this code?

Well, to understand the code, we first need to understand how it lexes and parses. In other words, what the grammar of our language is. How our language would lex, tokenize, and then parse our code. In this case, it’s Rust. Rust’s grammar is large and complex, so we’ll only be talking about part of it today. We’re going to focus on statements vs expressions.

You may have heard that Rust is an “expression based language” before. Well, this is what people mean. You see, when it comes to most of the things you say in a program, they’re often one of these two things. Expressions are things that produce some sort of value, and statements are used to sequence the evaluation of expressions. That’s a bit abstract, so let’s get concrete.

Statements

Rust has a few kinds of statements: first, there’s “declaration statements” and “expression statements,” and each have their own sub-kinds as well.

Declaration statements have two kinds: item declarations, and let statements. Item declarations are things like mod or struct or fn: they declare certain things exist. let statements are probably the most famous form of statement in Rust, they look like this:

 OuterAttribute* let PatternNoTopAlt ( : Type )? (= Expression † ( else BlockExpression) ? ) ? ;

That’s… a mouthful. We haven’t talked about * or ? yet, and we don’t really want to cover some of the more exotic parts of Rust right now. So we’re going to talk about this via a simpler grammar first:

let Variable = Expression;

This is how we create new variables in Rust: we say let, and then a name, an =, and then finally some expression. The result of evaluating that expression becomes the value of the variable.

This is leaving out a lot: the name isn’t just a name, it’s a pattern, which is very cool. let else exists in Rust now, and that’s cool. We’re ignoring types here. But you can get the basics with just this simple version.

Expression statements are much simpler:

ExpressionWithoutBlock ; | ExpressionWithBlock ;?

The | there is an or, so we can either have a single expression followed by a ;, or a block (denoted by {}s, which can optionally (the ? means it can exist or not) be followed by a ;.)

So to think like a compiler, you can start to figure out how to combine these rules. For example:

let x = {
    5 + 6
};

Here, we have a let statement, but the expression on the right hand side of the = is an ExpressionWithBlock. Here’s a pop quiz for you: is the ; part of the let expression, or part of the expression on the right hand side?

The answer is, it’s part of the let. The let expression has a mandatory ;, but the block does not, and so:

let x = ExpressionWithBlock;

If we had the semicolon with the block, we’d still need the one for the let, and so we’d have };;. Which the compiler accepts, but it warns about it.

Going back to our original code:

*pointer_of_some_kind = blah;

We don’t have a let, and this isn’t an item declaration: this is an expression statement. We have a ExpressionWithoutBlock, followed by a ;. So now we have to talk about expressions.

Expressions

There are a lot of expression types in Rust. Section 8.2 of the Reference has 19 sub-sections. Whew! In this case, this code is an Operation Expression, and more specifically, an Assignment Expression:

Expression = Expression

Easy enough! So the left hand side of the = is an expression with *pointer_of_some_kind, and the right hand side is blah. Easy enough!

But these two expressions are in some ways, the entire reason that I wrote this post. We just finally got here! You see, the reference has this to say about assignment expressions:

An assignment expression moves a value into a specified place.

What are those?

Places and Values

C, and older versions of C++, called these two things “lvalue” and “rvalue,” for “left” and “right”, like the sides of the =. More recent C++ standards have more categories. Rust splits the difference: it only has two categories, like C, but they map more cleanly to two of C++‘s categories. Rust calls lvalues, the left hand side, a “place,” and rvalues, the right hand side, a “value.” Here’s two more precise definitions, from the Unsafe Code Guidelines:

A place is basically a pointer, but might contain more information such as size or alignment. A place has a type, indicating the type of values that it stores.
A value A value is what gets stored in a place. A value has a type.

Both of these have expression forms, so a place expression produces a place when it’s evaluated, and a value expression produces a value. And that’s how = works: we have a place expression on the left, a value expression on the right, and we put that value in that place. Easy enough!

Once again, the code we were trying to figure out:

*pointer_of_some_kind = blah;

So *, the dereference operator, takes the pointer, and evaluates to the place where it points: its address. And blah gives us the value to put there.

Case closed? Not quite!

Deref

Rust has a trait, Deref, that lets us override the * operator. So let’s talk about this example, to make things easier:

use std::ops::{Deref, DerefMut};

struct DerefMutExample<T> {
    value: T
}

impl<T> Deref for DerefMutExample<T> {
    type Target = T;

    fn deref(&self) -> &Self::Target {
        &self.value
    }
}

impl<T> DerefMut for DerefMutExample<T> {
    fn deref_mut(&mut self) -> &mut Self::Target {
        &mut self.value
    }
}

fn main () {
    let mut x = DerefMutExample { value: 'a' };
    *x = 'b';
    assert_eq!('b', x.value);
}

You can play with it here.

We didn’t talk about the kind of expression that’s relevant here yet: a path expression.

Path expressions that resolve to local or static variables are place expressions, other paths are value expressions.

We talked about let earlier: in let mut x = DerefMutExample { value: 'a' }; above, the x is a path expression, and since it’s resolving to our new variable, that means it’s a place expression. The DerefMutExample { value: 'a' } is a value expression, because it doesn’t resolve to a variable.

Let’s talk about *x = 'b';, remember our assignment expression:

expression = expression;

And what it does: it moves a value into a place.

To understand how * works, we only need to add one more thing: the Dereference expression. It’s produced by the deference operator, *. It looks like this:

*expression

Its semantics are pretty straightforward:

If the expression has the type &T, &mut T, *const T, or *mut T, then this evaluates to the place of the value being pointed to, and gives it the same mutability.
If the expression is not one of those types, then it is equivalent to either *std::ops::Deref::deref(&x) if it’s immutable or *std::ops::DerefMut::deref_mut(&mut x)

That’s it. Now we have enough to fully understand *x = 'b':

'b' is a value expression
*x is not a pointer type, so we expand it to *std::ops::DerefMut::deref_mut(&mut x) and try again
std::ops::DerefMut::deref_mut(&mut x) returns the type &mut char in this case, and it is pointing at the place of self.value (which I’m gonna call <that place> for short), which currently has the value 'a' stored in it. Now we have *&mut <that place>.
We now use the other rule of the dereference operator, we’re operating on a &mut T, so *&mut <that place> refers to <that place>.
We now have <that place> = 'b', and so we move that 'b' into that place.

Whew! That’s that.

Conclusion

Thinking like a compiler can be fun! Once you’ve mastered the idea of grammars and gotten used to substituting things, you can figure out all sorts of interesting stuff. In this specific case, sometimes people wonder why Deref returns a reference if the whole goal is to show where something should point… and if you didn’t know that dereference expressions expanded to something with * in it, it would be confusing! But now you know. And hopefully learned some interesting things about values, places, and how compilers think about code along the way.

Here’s my post about this post on BlueSky:

Steve Klabnik @steveklabnik.com

Thinking like a compiler: places and values in #rustlang steveklabnik.com/writing/thin...

Thinking like a compiler: places and values in Rust

steveklabnik.com

April 7, 2025 at 7:09 PM

85 11

Read 4 replies on Bluesky

Does unsafe undermine Rust's guarantees?

Mon, 17 Mar 2025 00:00:00 GMT

When people first hear about unsafe in Rust, they often have questions. A very normal thing to ask is, “wait a minute, doesn’t this defeat the purpose?” And while it’s a perfectly reasonable question, the answer is both straightforward and has a lot of nuance. So let’s talk about it.

(The straightforward answer is “no”, by the way.)

Memory safe languages and their implementations

The first way to think about this sort of thing is to remember that programming languages and their implementations are two different things. This is sometimes difficult to remember, since many programming languages only have one implementation, but this general principle applies even in those cases. That is, there are the desired semantics of the language you are implementing, and then the codebase that makes programs in that language do those things.

Let’s examine a “real” programming language to talk about this: Brainfuck. Brainfuck programs are made up of eight different characters, each one performing one operation. Brainfuck programs can only do those eight things, and nothing more.

However, this simplicity leads to two interesting observations about the differences between languages in the abstract and their implementations.

Unsafety in implementation

The first is that properties of the language itself are distinct from properties of an implementation of the language. The semantics of the Brainfuck language don’t provide an opportunity to call some sort of re-usable function abstraction, for example. Yet, when we write a Brainfuck interpreter or compiler, we can use a programming language that supports functions. That doesn’t change that Brainfuck programs themselves can’t define or call functions.

And in fact, we often want to implement a language in another language that has more abilities than the language we’re implementing. For example, while it’s possible to implement Brainfuck in Brainfuck, it’s much easier to do so in a language that lets you define and call functions. Another classic example is interacting with hardware. Brainfuck has the . instruction that allows you to produce output. But in order to actually produce output to say, a terminal, you interact with your platform’s API to do so. But inside of that API, eventually you’ll hit a layer where there are no abstractions: you need to talk to hardware directly. Doing so is not statically able to be guaranteed to be memory safe, because the hardware/software interface often boils down to “put some information at this arbitrary memory location and the hardware will take it from there.” Our Brainfuck program is operating in an environment just like any other; at the end of the day it has to coordinate with the underlying operating system or hardware at some point. But just because our implementation must do so, doesn’t mean that our Brainfuck programs do so directly. The semantics of programs written in Brainfuck aren’t changed by the fact that the implementation can (and must) do things that are outside of those semantics.

Let’s take Ruby as a more realistic example than Brainfuck. Ruby does not let us modify arbitrary memory addresses from within Ruby itself. This means that pure Ruby programs should not be able to produce a segmentation fault. But in the real world, Ruby programs can segfault. This is possible if there’s a bug in the Ruby interpreter’s code. Sure, it’s our Ruby code that’s making the call to a function that results in a segfault, but the real fault lies with the code outside of Ruby’s purview.

That this can happen doesn’t mean that Ruby’s guarantees around memory manipulation is somehow useless, or suffers segmentation faults at the same rate as programs in a language that allow you to manipulate arbitrary memory as a matter of course. But it does also mean that we don’t need to look at our entire program to figure out where the problem comes from: it instead comes from our non-Ruby code. Ruby’s guarantees have helped us eliminate a lot of suspects when figuring out whodunit.

Extending languages and unsafety

The second property is that certain implementations may extend the language in arbitrary ways. For example, I could write a Brainfuck interpreter and say that I support a ninth instruction, @, that terminates the program when invoked. This would be a non-portable extension, and Brainfuck programs written in my variant wouldn’t work in other interpreters, but sometimes, this technique is useful.

Many languages find that these sorts of extensions are useful, and so offer a feature called a “foreign function interface” that allows for your program to invoke code in a different language. This provides the ability to do things that are outside of the domain of the language itself, which can be very useful. Brainfuck does not offer an FFI, but if it did, you could imagine that @ could be implemented in terms of it, making programs that use @ portable again, as long as the extended functionality was included somehow, often as a library of some kind.

Just like our implementation has the ability to do things outside of our languages’ semantics, FFI and similar extension mechanisms also give us the ability to do arbitrary things. I can write an extension for Ruby that writes to arbitrary memory. And I can cause a segfault. But we’re in the same place that we were with our implementation issues; we know that if we get a segfault, the blame lies not with our Ruby code, but instead, with either the implementation or our FFI.

So how is this memory safe?

It may seem contradictory that we can call a language “memory safe” if real-world programs have the ability to cause memory problems. But the thing is, it isn’t really programs in that language that caused the issue: it was FFI, which is in a different language entirely, or it was implementation issues, and implementations must do memory unsafe things, thanks to a need to interact with the OS or hardware. And so the definition of “memory safe language” is commonly understood to refer to languages and their implementations in the absence of either implementation bugs or FFI bugs. In practice, these bugs occur at such a low rate compared to languages that are clearly not memory safe that this practical definition serves a good purpose, even if you may feel a bit uncomfortable with these “exceptions.”

But there’s actually a deeper reason why these exceptions are acceptable, and that’s due to how we understand properties of programs and programming languages in the first place. That is, this isn’t just a practical “well these exceptions seem fine” sort of thing, they’re actually okay on a deeper level.

Programs, properties, and proofs

So how do we know anything about how programs and programming languages work at all?

Computer science is no more about computers than astronomy is about telescopes.

Edsger Wybe Dijkstra

A fun thing about computer science is that it’s closely related to other disciplines, including math! And so there’s a long history of using math techniques to understand computers and their programs.

One technique of building knowledge is the idea of proofs. If you get a degree in computer science, you’ll engage with proofs quite a lot. I even took a class on logic in the philosophy department as part of my degree.

I don’t intend to give you a full introduction to the idea of proofs in this blog post, but there’s some high-level concepts that are useful to make sure we’re in the same page about before we go forward.

Aristotelian syllogisms

Here is a very classic example of a form of reasoning called a “syllogism”, given by Aristotle in 350 BCE:

All men are mortal.
Socrates is a man.
Therefore, Socrates is mortal.

These first two lines are called “propositions,” and the third is a conclusion. We can base our conclusion on a logical relationship between the information given to us by the propositions.

But how do we know propositions are true? Are all men mortal? What’s important here is that we, for the purposes of our proof, assume that propositions are true. And we do this because, on some level, we cannot know everything. And so to begin, we have to start somewhere, and make some assumptions about what we know. It’s true that later, we may discover a fact that disproves our proposition, and now our proof no longer works. But that’s just the way that the world works. It doesn’t prevent these sorts of proofs from being useful to help us gain knowledge about the world and how it works, as best as we can tell at the current time.

So on some level, this is also why Ruby is a memory safe language even though a C extension can segfault: there’s always some kind of things that we have to assume are true. Memory safe languages are ones where the amount of code we have to assume is memory safe, rather than is guaranteed to be by the semantics, is small, and preferably indicated in the code itself. Put another way, the amount of code we need to trust is memory safe is large in a memory unsafe language, and small in a memory safe language, rather than zero.

Logic and programs

As time went on, so did our understanding of logic. And, of course, we even have competing logics! And then this gets fun, because terms can mean something slightly different. For example, in more recent logics, we’d call something like “All men are mortal” to be an axiom, rather than a proposition. Same idea: it’s something that we accept without proof.

As computers appeared, people sought to apply the rules for mathematical logic onto them. We even call circuits that carry out classic logical operations “logic gates.” Number theory and logic were foundational to making computers work. And so, once high level languages appeared on the scene, there was interest in applying these mathematical tools to understanding programs as well. This discipline is called “formal verification,” and the general idea is to describe various properties we wish a system to have, and then use formal methods from mathematics to demonstrate that this is true.

This area of study is very deep, and I don’t plan to cover the vast majority of it here. However, I do want to pursue one particular thread in this area.

Hoare Logic

“Hoare Logic” is, well:

In this paper an attempt is made to explore the logical foundations of computer programming by use of techniques which were first applied in the study of geometry and have later been extended to other branches of mathematics. This involves the elucidation of sets of axioms and rules of inference which can be used in proofs of the properties of computer programs. Examples are given of such axioms and rules, and a formal proof of a simple theorem is displayed. Finally, it is argued that important advantage, both theoretical and practical, may follow from a pursuance of these topics.

Incidentally, C. A. R. Hoare, the author of this paper, and Graydon Hoare, the creator of Rust, are unrelated.

How does Hoare logic work? Yet again, not going to cover all of it, but the general idea is this: In order to figure out if a program does what it is supposed to do, we need to be able to reason about the state of the program after execution happens. And so we need to be able to describe the state before, and its relationship to the state after. And so you get this notation:

P { Q } R

P is a precondition, Q is a program, and R is the result of the program executing with those preconditions.

But most programs have more than one statement. So how could we model this? Hoare gives us the Rule of Composition:

If P { Q1 } R1 and R1 { Q2 } R then P { Q1; Q2 } R.

This allows us to build up a program by proving each statement in turn.

Hoare logic is very neat, and I’ve only scratched the surface here. People did a lot of work to extend Hoare logic to include more and more aspects of programs. But then, something else happened.

Separation Logic

In 2002, Separation Logic: A Logic for Shared Mutable Data Structures was published.

In joint work with Peter O’Hearn and others, based on early ideas of Burstall, we have developed an extension of Hoare logic that permits reasoning about low-level imperative programs that use shared mutable data structure.

Hmm, shared mutable data structures? Where have I heard that before…

Let’s see what they have to say:

The problem faced by these approaches is that the correctness of a program that mutates data structures usually depends upon complex restrictions on the sharing in these structures.

For sure. Well, what are we to do about this?

The key to avoiding this difficulty is to introduce a novel logical operation P * Q, called separating conjunction (or sometimes independent or spatial conjunction), that asserts that P and Q hold for disjoint portions of the addressable storage.

What disjoint portions of addressable storage might we care about?

Our intent is to capture the low-level character of machine language. One can think of the store as describing the contents of registers, and the heap as describing the contents of an addressable memory.

Pretty useful!

Before we talk a bit about how separation logic works, consider this paragraph on why it’s named as such:

Since these logics are based on the idea that the structure of an assertion can describe the separation of storage into disjoint components,we have come to use the term separation logics, both for the extension of predicate calculus with the separation operators and the resulting extension of Hoare logic. A more precise name might be storage separation logics, since it is becoming apparent that the underlying idea can be generalized to describe the separation of other kinds of resources.

The plot thickens.

Anyway, in Separation Logic, we use slightly different notation than Hoare Logic:

{ P } C { Q }

This says that we start with the precondition P, and if the program C executes, it will not have undefined behavior, and if it terminates, Q will hold.

Furthermore, there is the “frame rule”, which I am going to butcher the notation for because I haven’t bothered to install something to render math correctly just for this post:

If { p } c { q }

then { p * r } c { q * r }

where no free variable in r is modified by c.

Why is it interesting to add something to both sides of an equation, in a sense? Well what this gives us the ability to do is add any predicates about parts of the program that c doesn’t modify or mutate. You might think of &mut T or even just ownership in general: we can reason about just these individual parts of a program, separately from the rest of it. In other words, we have some foundational ideas for ownership and even bits of borrowing, and while this original paper doesn’t involve concurrency, eventually Concurrent Separation Logic would become a thing as well.

I think the paper explains why the frame rule matters better than I can:

Every valid specification {p} c {q} is “tight” in the sense that every cell in its footprint must either be allocated by c or asserted to be active by p; “locality” is the opposite property that everything asserted to be active belongs to the footprint. The role of the frame rule is to infer from a local specification of a command the more global specification appropriate to the larger footprint of an enclosing command.

What this gives us is something called “local reasoning,” and local reasoning is awesome. Before I talk about that, I want to leave you with one other very interesting paragraph:

Since our logic permits programs to use unrestricted address arithmetic,there is little hope of constructing any general-purpose garbage collector. On the other hand, the situation for the older logic, in which addresses are disjoint from integers, is more hopeful. However, it is clear that this logic permits one to make assertions, such as “The heap contains two elements” that might be falsified by the execution of a garbage collector, even though, in any realistic sense, such an execution is unobservable.

Anyway. Let’s talk about local vs global analysis.

Global vs local analysis

Proving things about programs isn’t easy. But one thing that can make it even harder is that, for many properties of many programs you’d want to analyze, you need to do a global analysis. As an example, let’s use Ruby. Ruby is an incredibly dynamic programming language, which makes it fairly resistant to static analysis.

Global analysis

Here is a Ruby program. Do you know if this program executes successfully?

class Foo
  def self.bar
    "baz"
  end
end

p Foo.bar

Yeah, it prints baz. But what about this Ruby program?

class Foo
  def self.bar
    "baz"
  end
end

require "foo_ext.rb"

p Foo.bar

We can’t know. foo_ext.rb may contain no relevant code, but it also might include something like this:

Foo.class_eval { undef :bar }

In which case, when we try and call Foo.bar, it no longer exists:

<anonymous>': eval:13:in `block in <main>': undefined method `bar' for class `Foo' (NameError)
eval:13:in `class_eval'
eval:13:in `<main>'
-e:in `eval'

Ouch. So in this case, we’re going to need the whole code of our program in order to be able to figure out what’s going on here.

Incidentally, Sorbet is a very cool project to add type checking to Ruby. They do require access to the entire Ruby source code in order to do their analysis. But they also made some decisions to help make it more tractable; if you try Sorbet out on the web, the type checker is fast. What happens when you try the above code with Sorbet?

editor.rb:5: Unsupported method: undef https://srb.help/3008
     5 |Foo.class_eval { undef :bar }
                         ^^^^^^^^^^

This is a very fair tradeoff! It’s very common with various forms of analysis to choose certain restrictions in order to make what they want to do tractable. Ruby has so much evil in it, just not supporting some of the more obscure things is completely fair, in my opinion.

Local analysis

The opposite of global analysis is, well, local analysis. Let’s consider a Rust equivalent of our Ruby:

struct Foo;

impl Foo {
    fn bar() -> String {
        String::from("baz")
    }
}

fn main() {
    dbg!(Foo::bar());
}

Can we know if this program works? Sure, everything is here. Now, if we tried the same trick as the Ruby code, we know that it would work, because Rust doesn’t have the ability to remove the definition of bar like Ruby does. So let’s try something else:

struct Foo;

impl Foo {
    fn bar() -> String {
        // body hidden
    }
}

fn main() {
    dbg!(Foo::bar());
}

Can we know if main is properly typed here? We can, even though we know nothing about the body. That’s because dbg! takes any value that implements the Debug trait, and we know that Foo::bar() returns String, which implements Debug. Typechecking main is local to main’s implementation, we don’t need to look into the bodies of any function it calls to determine if it’s well-typed. We only need to know their signatures. If we didn’t require bar to have a type signature, we’d have to peek into its body to figure out what it returns, and so on for any function that bar calls in its body. Instead of this arbitrarily deep process, we can just look at the signature and be done with it.

Composability of proofs

So why is local analysis so helpful? The first reason is either speed or scalability, depending on how you want to look at it. If you are running global analysis checks, they will get more expensive the larger your codebase, since they will require checking the entire thing to work. Whereas local analysis only requires a local context, it doesn’t get any more expensive when you add more code to your project, only when you change that local context. So when you’re trying to scale checks up to larger projects, local analysis is crucial.

But I also like to personally think about it as a sort of abstraction. That is, global analysis is a leaky abstraction: changes in one part of the codebase can cascade into other parts. Remember this line about the frame rule?

The role of the frame rule is to infer from a local specification of a command the more global specification appropriate to the larger footprint of an enclosing command.

If we have local reasoning, we can be sure that changes locally don’t break out of the boundaries of those changes. As long as the types of our function don’t change, we can mess with the body as much as we want, and we know that the rest of the analysis of the program is still intact. This is really useful, in the same way that abstractions are useful when building programs more generally.

Rustbelt

Okay, we have taken a real deep dive here. What’s this all have to do with unsafe Rust?

Well, seven years ago, RustBelt: Securing the Foundations of the Rust Programming Language by Ralf Jung, Jaques-Henri Jourdan, Robbert Krebbers, and Derek Dreyer, was published:

In this paper, we give the first formal (and machine-checked) safety proof for a language representing a realistic subset of Rust.

But it’s more exciting than that:

Our proof is extensible in the sense that, for each new Rust library that uses unsafe features, we can say what verification condition it must satisfy in order for it to be deemed a safe extension to the language.

Very good!

The paper begins by talking about why verifying Rust is such a challenge:

Consequently, to overcome this restriction, the implementations of Rust’s standard libraries make widespread use of unsafe operations, such as “raw pointer” manipulations for which aliasing is not tracked. The developers of these libraries claim that their uses of unsafe code have been properly “encapsulated”, meaning that if programmers make use of the APIs exported by these libraries but otherwise avoid the use of unsafe operations themselves, then their programs should never exhibit any unsafe/undefined behaviors. In effect, these libraries extend the expressive power of Rust’s type system by loosening its ownership discipline on aliased mutable state in a modular, controlled fashion

This is basically the question asked at the start of this post: can you honestly say that a function is safe if it contains unsafe code as part of its implementation?

They go on to describe the first challenge for the project: choosing the correct logic to model Rust in (some information cut for clarity):

Iris is a language-generic framework for higher-order concurrent separation logic which in the past year has been equipped with tactical support for conducting machine-checked proofs of programs in Coq. By virtue of being a separation logic, Iris comes with built-in support for reasoning modularly about ownership. Moreover, the main selling point of Iris is its support for deriving custom program logics for different domains using only a small set of primitive mechanisms. In the case of RustBelt, we used Iris to derive a novel lifetime logic, whose primary feature is a notion of borrow propositions that mirrors the “borrowing” mechanism for tracking aliasing in Rust.

Separation logic!

So how does this work? There are three parts to the proof. I’m going to summarize them because I’m already quoting extensively, please go read the paper if you want the exact details, it is well written.

Verify that the typing rules are sound when interpreted semantically.
Verify that if a program is well typed semantically, it does not exhibit unsafe behavior.
For any library that uses unsafe, check that it satisfies the semantics that its interface requires.

If all three of these things are true, then the program is safe to execute. This is also exciting because, thanks to 3, when someone writes new unsafe code, they can get RustBelt to let them know which properties they need to satisfy to make sure that their unsafe code doesn’t cause problems.

Weaknesses

Of course, there is more to do following up on RustBelt:

We do not model (1) more relaxed forms of atomic accesses, which Rust uses for efficiency in libraries like Arc; (2) Rust’s trait objects (comparable to interfaces in Java), which can pose safety issues due to their interactions with lifetimes; or (3) stack unwinding when a panic occurs, which causes issues similar to exception safety in C++.

However, the results were good enough to find some soundness bugs in the standard library, and if I remember correctly, widen one of the APIs in a sound manner as well. Another interesting thing about RustBelt is that it was published before Rust gained Non-lexical lifetimes, however, it modeled them, since folks knew they were coming down the pipeline.

The theoretical vs the empirical

So, we have some amount of actual proof that unsafe code in Rust doesn’t undermine Rust’s guarantees: what it does is allow us to extend Rust’s semantics, just like FFI would allow us to extend a program written in a language that supports it. As long as our unsafe code is semantically correct, then we’re all good.

… but what if it’s not semantically correct?

When things go wrong

At some point maybe I’ll write a post about the specifics here, but this post is already incredibly long. I’ll cut to the chase: you can get undefined behavior, which means anything can happen. You don’t have a real Rust program. It’s bad.

But at least it’s scoped to some degree. However, many people get this scope a bit wrong. They’ll say something like:

You only need to verify the unsafe blocks.

This is true, but also a bit misleading. Before we get into the unsafe stuff, I want you to consider an example. Is this Rust code okay?

struct Foo {
    bar: usize,
}

impl Foo {
    fn set_bar(&mut self, bar: usize) {
        self.bar = bar;
    }
}

No unsafe shenanigans here. This code is perfectly safe, if a bit useless.

Let’s talk about unsafe. The canonical example of unsafe code being affected outside of unsafe itself is the implementation of Vec<T>. Vecs look something like this (the real code is different for reasons that don’t really matter in this context):

struct Vec<T> {
    ptr: *mut T,
    len: usize,
    cap: usize,
}

The pointer is to a bunch of Ts in a row, the length is the current number of Ts that are valid, and the capacity is the total number of Ts. The length and the capacity are different so that memory allocation is amortized; the capacity is always greater than or equal to the length.

That property is very important! If the length is greater than the capacity, when we try and index into the Vec, we’d be accessing random memory.

So now, this function, which is the same as Foo::set_bar, is no longer okay:

impl<T> Vec<T> {
    fn set_len(&mut self, len: usize) {
        self.len = len;
    }
}

This is because the unsafe code inside of other methods of Vec<T> need to be able to rely on the fact that len <= capacity. And so you’ll find that Vec<T>::set_len in Rust is marked as unsafe, even though it doesn’t contain unsafe code.

And this is why the module being the privacy boundary matters: the only way to set len directly in safe Rust code is code within the same privacy boundary as the Vec<T> itself. And so, that’s the same module, or its children.

This is still ultimately better than any line of code in the whole codebase, but it’s not quite as small as you may think at first.

How can we check that things haven’t gone wrong?

Okay, so while RustBelt could give you some idea if your code is correct, I doubt you’re about to jump into Coq and write some proofs. What can you do? Well, Rust provides a tool, miri, that can interpret your Rust code and let you know if you’ve violated some unsafe rules. But it’s not complete, that is, miri can tell you if your code is wrong, but it cannot tell you if your code is right. It’s still quite useful.

To get miri, you can install it with Rust nightly:

$ rustup +nightly component add miri

And then run your test suite under miri with

$ cargo +nightly miri test

For more details, consult miri’s documentation.

Do things go wrong?

Do people write good unsafe code, or bad?

Well, the obvious answer to the above question is “yes.” Anyone who says all unsafe code in Rust is sound is obviously lying. But the absolute isn’t what’s interesting here, it’s the shade of grey. Is unsafe pervasive? Does Rust have memory related bugs at the same rate as other MSLs, or closer to memory-unsafe languages?

It’s still sort of early days here, but we do have some preliminary results that are helpful. In 2022, Google reported that:

To date, there have been zero memory safety vulnerabilities discovered in Android’s Rust code.

I am very interested to see a more up to date report about how that’s changed, but as they say:

We don’t expect that number to stay zero forever, but given the volume of new Rust code across two Android releases, and the security-sensitive components where it’s being used, it’s a significant result. It demonstrates that Rust is fulfilling its intended purpose of preventing Android’s most common source of vulnerabilities. Historical vulnerability density is greater than 1/kLOC (1 vulnerability per thousand lines of code) in many of Android’s C/C++ components (e.g. media, Bluetooth, NFC, etc). Based on this historical vulnerability density, it’s likely that using Rust has already prevented hundreds of vulnerabilities from reaching production.

We’ll see if these results replicate themselves, in the future, and also outside of Google. But it at least looks like in practice, most unsafe code has not led to problems. At least so far!

Here’s my post about this post on BlueSky:

Steve Klabnik @steveklabnik.com

Does unsafe undermine #rustlang 's guarantees? steveklabnik.com/writing/does...

Does unsafe undermine Rust's guarantees?

steveklabnik.com

March 19, 2025 at 3:43 PM

138 41

Read 17 replies on Bluesky

A Happy Day for Rust

Wed, 05 Mar 2025 00:00:00 GMT

A while back, I wrote A sad day for Rust about the Actix-web unsafe controversy. The short of it was, Actix-web was using some unsound unsafe code. When this was discovered, people responded pretty harshly, and the maintainer quit. Others picked up the work, but it still wasn’t great:

I’m not sure where we go from here, and I’m not sure what we could have done to prevent this from happening. But I feel like this is a failure, and it’s set Rust back a bit, and I’m just plain sad.

While the Rust community surely isn’t perfect, today I’d like to share with you a different story: one where there was some controversy, but it was handled in the complete opposite way.

Rustup is the official installer and toolchain manager for Rust. It has a much smaller team than Rust itself does, and releases on a much slower schedule.

Back in August of last year, it was decided to change some behavior. The details aren’t super important, what matters is that it’s a breaking change to a critical component of the ecosystem. I’m not here to say if this change is good or bad; they had good motivations, there’s also good reasons to keep the previous behavior, it’s one of those classic “you can’t make everyone happy” sort of deals.

The team knew that they’d need to let people know that this had happened:

Yes, you’ve understood it correctly, and this breakage is intentional: #3635 (comment)

Before the actual release, we’ll definitely clearly communicate this change with the community, and we’ll use the beta phase to collect user feedback.

And they did make a post on internals about the change, seeking feedback. This also landed in a This Week in Rust, in my understanding.

The problem is, not everyone reads internals, and not everyone reads This Week in Rust. And so yesterday, some folks woke up to a broken CI, leading to this bug report among other upset posts elsewhere on the internet.

I have been on the other side of this before. It’s frustrating when you try and communicate something, and people don’t hear about it, and then they find out later and get mad at you about it. I also remember the Actix debacle, and many other controversies in the broader open source ecosystem.

Steve Klabnik @steveklabnik.com

at one point i questioned my desire to help people get into open source image unrelated

March 3, 2025 at 8:04 PM

1,158 86

Read 13 replies on Bluesky

However, this time, things went differently. As you can see, the issue is very respectful, and the comments are also generally respectful. The team responded graciously, and decided to put in the work to release a new version to restore the previous behavior.

I was kind of worried yesterday that this would end up being another Actix situation. The Rust community has changed a lot over time, and while it’s still pretty good overall, sometimes things can go poorly, just like with any random group of thousands of people on the internet. But today, I’m grateful that feedback was given constructively, it was taken with the spirit that it was intended, and everyone should end up happy.

If only every controversy went this way, maybe I wouldn’t be wondering about coyotes.

Here’s my post about this post on BlueSky:

Steve Klabnik @steveklabnik.com

A happy day for #rustlang steveklabnik.com/writing/a-ha...

A Happy Day for Rust

I’m not sure where we go from here, and I’m not sure what we could have done to prevent this from happening. But I feel like this is a failure, and it’s set Rust back a bit, and I’m just plain sad.

steveklabnik.com

March 5, 2025 at 3:39 PM

185 23

Read 4 replies on Bluesky

When should I use String vs &str?

Wed, 16 Oct 2024 00:00:00 GMT

Rust has two main string types: String and &str. Sometimes, people argue that these two types make Rust code difficult to write, because you have to think about which one you should be using in a given situation. My experience of writing Rust is that I don’t really think about this very much, and this post is about some rules of thumb that you can use to be like me.

Level 1: Don’t think about it at all

The very first thing you can do is follow the simplest rule:

Always use String, never use &str.

That looks like this:

struct Person {
    name: String,
}

fn first_word(words: String) -> String {
    words
        .split_whitespace()
        .next()
        .expect("words should not be empty")
        .to_string()
}

This style means you may need to add .to_string() or .clone() for things to work sometimes:

fn main() {
    let sentence = "Hello, world!";

    println!("{}", first_word(sentence.to_string()));

    let owned = String::from("A string");

    // if we don't clone here, we can't use owned the second time
    println!("{}", first_word(owned.clone()));
    println!("{}", first_word(owned));
}

But that’s okay, the compiler will let you know when you need to:

error[E0382]: use of moved value: `owned`
  --> src/main.rs:21:31
   |
18 |     let owned = String::from("A string");
   |         ----- move occurs because `owned` has type `String`, which does not implement the `Copy` trait
19 |
20 |     println!("{}", first_word(owned));
   |                               ----- value moved here
21 |     println!("{}", first_word(owned));
   |                               ^^^^^ value used here after move
   |
note: consider changing this parameter type in function `first_word` to borrow instead if owning the value isn't necessary
  --> src/main.rs:5:22
   |
5  | fn first_word(words: String) -> String {
   |    ----------        ^^^^^^ this parameter takes ownership of the value
   |    |
   |    in this function
help: consider cloning the value if the performance cost is acceptable
   |
20 |     println!("{}", first_word(owned.clone()));
   |                                    ++++++++

Hey, compiler, that’s a great idea. Let’s move on to level 2.

Level 2: prefer `&str` for function parameters

A rule that’s a little better is this one:

Always use String in structs, and for functions, use &str for parameters and String types for return values.

This is what the compiler error for level 1 was suggesting we do instead of .clone.

That results in code that looks like this:

struct Person {
    name: String,
}

fn first_word(words: &str) -> String {
    words
        .split_whitespace()
        .next()
        .expect("words should not be empty")
        .to_string()
}

fn main() {
    let sentence = "Hello, world!";

    println!("{}", first_word(sentence));

    let owned = String::from("A string");

    println!("{}", first_word(&owned));
    println!("{}", first_word(&owned));
}

We’re now doing much less copying. We do need to add a & on to String values that we wish to pass into first_word, but that’s not too bad, the compiler will help us when we forget:

error[E0308]: mismatched types
  --> src/main.rs:20:31
   |
20 |     println!("{}", first_word(owned));
   |                    ---------- ^^^^^ expected `&str`, found `String`
   |                    |
   |                    arguments to this function are incorrect
   |
note: function defined here
  --> src/main.rs:5:4
   |
5  | fn first_word(words: &str) -> String {
   |    ^^^^^^^^^^ -----------
help: consider borrowing here
   |
20 |     println!("{}", first_word(&owned));
   |                               +

Following this rule will get you through 95% of situations successfully. Yes, that number was found via the very scientific process of “I made it up, but it feels correct after writing Rust for the last twelve years.”

For 4% of that last 5%, we can go to level 3:

Level 3: return `&str` sometimes

Here’s a slightly more advanced rule for certain circumstances:

Always use String in structs, and for functions, use &str for parameters. If the return type of your function is derived from an argument and isn’t mutated by the body, return &str. If you run into any trouble here, return String instead.

That would look like this:

struct Person {
    name: String,
}

// we're returning a substring of words, so &str is appropriate
fn first_word(words: &str) -> &str {
    words
        .split_whitespace()
        .next()
        .expect("words should not be empty")
}

fn main() {
    let sentence = "Hello, world!";

    println!("{}", first_word(sentence));

    let owned = String::from("A string");

    println!("{}", first_word(&owned));
    println!("{}", first_word(&owned));
}

This lets us remove a copy, we no longer have a .to_string in the body of first_word.

Sometimes, we can’t do that though:

// because we are going to uppercase the first word, our return type can't
// be &str anymore, because we aren't actually returning a substring: we are
// creating our own new string.
fn first_word_uppercase(words: &str) -> String {
    words
        .split_whitespace()
        .next()
        .expect("words should not be empty")
        .to_uppercase()
}

fn main() {
    let sentence = "Hello, world!";

    println!("{}", first_word_uppercase(sentence));

    let owned = String::from("A string");

    println!("{}", first_word_uppercase(&owned));
    println!("{}", first_word_uppercase(&owned));
}

How do you know that this is the case? Well, in this specific case, to_uppercase already returns a String. So that’s a great hint. If we tried to return a &str, we’d get an error:

// this can't work
fn first_word_uppercase(words: &str) -> &str {
    &words
        .split_whitespace()
        .next()
        .expect("words should not be empty")
        .to_uppercase()
}

would give us

error[E0515]: cannot return reference to temporary value
  --> src/main.rs:7:5
   |
7  |        &words
   |  ______^-
   | | ______|
8  | ||         .split_whitespace()
9  | ||         .next()
10 | ||         .expect("words should not be empty")
11 | ||         .to_uppercase()
   | ||                       ^
   | ||_______________________|
   |  |_______________________returns a reference to data owned by the current function
   |                          temporary value created here

And that’s really it. Following this rule will get you through virtually every scenario where you need to wonder about String and &str. With some practice, you’ll internalize these rules, and when you feel comfortable with a level, you can go up to the next one.

What about that last 1% though? Well, there is a next level…

Level 4: When to use `&str` in a struct

Here’s the rule for level 4:

Should you use a &str in a struct? If you’re asking that question, use String. When you need to use &str in a struct, you’ll know.

Storing references in structs is useful, for sure, and it’s good that Rust supports it. But you’re only going to need it in fairly specific scenarios, and if you feel like you’re worrying about String vs &str, you’re just not in the position to be worrying about the complexity of storing a &str in a struct yet.

In fact, some people believe in this rule so strongly that they’re working on a language where storing references in structs isn’t even possible, and it’s a language I’ve found very interesting lately: Hylo. They go a bit farther than that, even: in Hylo, you think of everything as being values, rather than having references at all. They think that you can write meaningful programs with this model. But this isn’t a post about Hylo. I’ll write one of those eventually. My point is just that you can really, truly, get away with not storing &strs in structs for a lot of useful Rust programs. So it’s not really worth spending mental energy on, until you determine that you must do so. That is the case when you’ve profiled your program and determined that copying strings to and from your struct is a big enough issue to bother with lifetimes.

I hope you’ve found this helpful, and if you have any other rules of thumb like this, I’d love to hear about them!

Here’s my post about this post on BlueSky:

Steve Klabnik @steveklabnik.com

When should you use String vs &str? Some #rustlang rules of thumb steveklabnik.com/writing/when...

When should I use String vs &str?

steveklabnik.com

October 16, 2024 at 6:27 PM

287 43

Read 30 replies on Bluesky

Memory Safety is a Red Herring

Thu, 21 Dec 2023 00:00:00 GMT

TL;DR:

I think that a focus on memory safe languages (MSLs) versus non memory-safe languages is a bit of a red herring. The actual distinction is slightly bigger than that: languages which have defined behavior by default, with a superset where undefined behavior is possible, vs languages which allow for undefined behavior anywhere in your program. Memory safety is an important aspect of this, but it is necessary, not sufficient. Rust’s marketing has historically focused on memory safety, and I don’t think that was a bad choice, but I do wonder sometimes. Finally, I wonder about the future of the C++ successor languages in the face of coming legislation around MSLs for government procurement.

Additionally, I would like to thank ixi for discussing some of this stuff with me. I do not claim that they agree with me, these opinions are solely my own! But talking about it helped me form those opinions.

ThreadId(1)

Today I have been programming in Rust for 11 years. I usually write some sort of blog post reflecting on the year. This year, I am reflecting on some old questions:

Should Rust have focused so much on memory safety in its advocacy?
Does Rust conflate a more general concept of “safety” with “memory safety?”
What do people even mean by “memory safe languages?”

But before we get there, we have to talk about some other things.

ThreadId(2)

The other day, someone asked an interesting question on Hacker News:

Can someone with a security background enlighten me, on why Python is on the list of “memory safe” languages? Most of the python code I have worked with is a thin wrapper on C. Wouldnt that make python vulnerable as well?

This is a very reasonable question! If you click the link, you’d see my answer:

You are correct that if you call C from Python, you can run into problems. But the fault there lies with the C, not the Python. Pure Python itself is memory safe.

Because memory safety is built on top of memory unsafety, it is generally understood that when discussing things at this sort of level, that we are speaking about the Python-only subset. (Same as any other memory safe language.)

But this has been gnawing at me for a while. Like it’s one of those things that “everybody knows what I mean.” But isn’t that kind of… bullshit? Like something is wrong with your definitions. But on the other hand, if we accept FFI as being not safe, and that since you can FFI from anywhere, that means it’s also unsafe, then every single program is unsafe. Your operating system does not offer memory-safe APIs. To do anything useful, you must call into the operating system. This means every program would be infected via this conception of safety, and therefore, as a definition, it would be useless.

If we instead make an exception for a language’s runtime, which is allowed to make unsafe calls, but users’ code is not, that would draw an appropriate boundary: only write code in the guest language, and you don’t have to worry about safety anymore. And I think that this sort of definition has been the default for many years. You have “managed” languages which are safe, and you have “unmanaged” languages which are unsafe. Unmanaged languages are used to implement managed languages. What’s that? A Java runtime written in Java? A Ruby runtime written in Ruby? Yeah those are weird exceptions. Ignore those. And those “native extensions” that segfault? Well they aren’t written in the host language, so of course that will happen, and yeah it’s kinda annoying when it happens, but it’s fine. Sometimes you have to risk it for performance.

I just don’t find this satisfying. It’s too vague. You just kinda know it when you see it. Ignore FFI, and we’re good. And sometimes, you just can’t have safety. What if you want to write a runtime? You’re gonna need to call into the system directly, and now we’re back to unsafe, so just use an unsafe language. It is what it is.

ThreadId(3)

That comment is on an article titled The Case for Memory Safe Roadmaps: Why Both C-Suite Executives and Technical Experts Need to Take Memory Safe Coding Seriously . This document, published by the Five Eyes, suggests… that executives and technical experts need to take memory safety seriously. Call it boring if you want, but it is what it says on the tin.

More specifically, it says:

Memory safe programming languages (MSLs) can eliminate memory safety vulnerabilities.

As well as an appendix, “Memory Safe Languages,” which describes C#, Go, Java, Python, Rust, and Swift as memory safe languages.

Finally, it says:

Programming languages such as C and C++ are examples of memory unsafe programming languages that can lead to memory unsafe code and are still among the most widely used languages today.

But it also acknowledges the reality that sometimes you can get around memory safety:

MSLs, whether they use a garbage collection model or not, will almost certainly need to rely on libraries written in languages such as C and C++. Although there are efforts to re-write widely used libraries in MSLs, no such effort will be able to re-write them all anytime soon.

For the foreseeable future, most developers will need to work in a hybrid model of safe and unsafe programming languages. Developers who start writing in an MSL will need to call C and C++ libraries that are not memory safe. Likewise, there are going to be situations where a memory unsafe application needs to call into a memory safe library. When calling a memory unsafe component or application, the calling application needs to be explicitly aware of—and limit any input passed to—the defined memory bounds.

The memory safety guarantees offered by MSLs are going to be qualified when data flows across these boundaries. Other potential challenges include differences in data marshalling, error handling, concurrency, debugging, and versioning.

Okay, that’s enough quoting from the report. And I swear I’m not about to say nice things about it because I am cited as #42, though to be honest it’s real fucking weird that searching “klabnik” on a page on defense.gov returns 1/1. But I think this is an excellent formulation of the critical distinction between the languages on one side of the line versus the other, though they make the same conceptual mistake that I believe Rust may have: it’s not just about memory unsafety. “safety” itself is basically the idea that most of the time, you’re in a universe where everything is peachy keen, and occasionally, you’ll need to reach across that line into the scary other place, but you gotta do what you gotta do. But there is a line there. But that line is more abstract than just memory safety.

ThreadId(4)

Rust often describes itself as “data race free.” But what the heck is a data race anyway? Here’s John Regehr:

A data race happens when there are two memory accesses in a program where both:

target the same location

are performed concurrently by two threads

are not reads

are not synchronization operations

There are other definitions but this one is fine; it’s from Sebastian Burckhardt at Microsoft Research.

Oh. Well, I guess “Here’s Sebastian Burckhardt:” instead.

Point is, data races are undefined behavior in Rust, and they’re also undefined behavior in C++. When looking for a succinct citation for that fact, I came across this 2006 post by Hans Boehm. Its first line:

Our proposed C++ memory model gives completely undefined semantics to programs with data races.

I believe this was the model that was accepted, so that’s the best citation you’re gonna get from me. Okay, as soon as I read that I said “come on Steve, that’s stupid, make sure it’s not just trivial to actually find in the standard” and lo and behold n4849 says:

The execution of a program contains a data race if it contains two potentially concurrent conflicting actions, at least one of which is not atomic, and neither happens before the other, except for the special case for signal handlers described below. Any such data race results in undefined behavior.

Okay, the real citation was easier. The article by Bohm is still fascinating.

But data races aren’t undefined behavior in every programming language. For example, let’s consider both Java and Go. Now I hate that I even have to preemptively say this, but I am not saying Java and Go are bad. I think these are good decisions for both of the two languages. And I think their decisions here are very informative. Language wars are stupid. Stop it.

Java was the first time I remember hearing, “yeah you can have data races but like, it’s fine” about. I kinda lodged that fact in the back of my brain, and didn’t think about it much. But lately, I was wondering. So I went and read the Java specification, and more specifically, Chapter 17. It contains this text:

The semantics of operations other than inter-thread actions, such as reads of array lengths (§10.7), executions of checked casts (§5.5, §15.16), and invocations of virtual methods (§15.12), are not directly affected by data races.

Therefore, a data race cannot cause incorrect behavior such as returning the wrong length for an array.

This is morally equivalent to “not undefined behavior” even though the specification doesn’t define “undefined behavior” in the way that the C, C++, and Rust specifications do. Notably, the specification also defines “data race” in a slightly different way:

When a program contains two conflicting accesses (§17.4.1) that are not ordered by a happens-before relationship, it is said to contain a data race.

Not just any ordering or synchronization primitive, a happens-before relation specifically. Fun! We have three technically-slightly different but basically morally equivalent ways of defining a data race. Let’s do a fourth. Go also allows data races. Sort of. From the Go memory model:

A data race is defined as a write to a memory location happening concurrently with another read or write to that same location, unless all the accesses involved are atomic data accesses as provided by the sync/atomic package.

Given that Rust provides atomic ordering operations in a submodule of its standard library, Rust and Go agree here, though Go’s more specific about naming it.

But they also have this to say:

While programmers should write Go programs without data races, there are limitations to what a Go implementation can do in response to a data race. An implementation may always react to a data race by reporting the race and terminating the program. Otherwise, each read of a single-word-sized or sub-word-sized memory location must observe a value actually written to that location (perhaps by a concurrent executing goroutine) and not yet overwritten.

So it’s not right, but it’s also not exactly UB. It goes on:

These implementation constraints make Go more like Java or JavaScript, in that most races have a limited number of outcomes, and less like C and C++, where the meaning of any program with a race is entirely undefined, and the compiler may do anything at all. Go’s approach aims to make errant programs more reliable and easier to debug, while still insisting that races are errors and that tools can diagnose and report them.

While a Go program may exhibit what a Rust or C++ program would consider undefined behavior, and it does also consider it an error, the consequences are very different. You don’t get time travel. You get 998 instead of 1,000.

(After publication, Edoardo Spadolini reached out to me and we had a productive conversation about the above paragraph: while my overall point here is still valid, there are slightly more serious consequences to UB in a Go program than slightly incorrect sums. For example, torn writes to a slice can lead to serious consequences, and there are other ways to write code that exploits UB to be actively malicious, such as a function with the signature func Cast[T any](addr uintptr) *T that works. However, it’s also worth mentioning that Go ships with a race detector, available by simply passing -race. For some interesting usage of this at scale, check out this post by Uber. At the end of the day, this is still very different than the “your whole program is invalid” style of issues you can end up with in C or C++, I am slightly underselling the danger above.)

Go also provides an unsafe package:

The built-in package unsafe, known to the compiler and accessible through the import path “unsafe”, provides facilities for low-level programming including operations that violate the type system. A package using unsafe must be vetted manually for type safety and may not be portable.

Oh yeah, I forgot, Java also has sun.misc.unsafe, essentially the same thing. They have actually wanted to remove it for a long time, and some progress has been made, but it’s not quite gone yet. The Unsafe Class: Unsafe at Any Speed is a good explanation of the topic.

ThreadId(5)

One weird thing about suggesting that the borrow checker is only about memory safety, and that memory safety means the absence of data races, is that memory safety is more than just the absence of data races. Consider a problem that requires no threads to become an issue: iterator invalidation. And different languages have different answers to this issue:

In C++, it is undefined behavior.
In Java, some data structures detect this and throw a ConcurrentModificationException.
In JavaScript, Ruby, and other languages, you often get “weird” behavior:

numbers = [1, 2, 3, 4]
numbers.each do |number|
  p number
  numbers.shift(1)
end
p numbers

prints

1
3
[3, 4]

But you don’t get full-blown time-travel undefined behavior. Sort of like this code in Rust:

fn main() {
    let i = i32::MAX + 1;
    println!("{i}");
}

Okay in this case because it’s all at compile time Rust can detect it:

error: this arithmetic operation will overflow
 --> src/main.rs:2:13
  |
2 |     let i = i32::MAX + 1;
  |             ^^^^^^^^^^^^ attempt to compute `i32::MAX + 1_i32`, which would overflow
  |
  = note: `#[deny(arithmetic_overflow)]` on by default

But in the general case, you won’t get a compile error. You may get a panic, you may get -2147483648. It’s weird. But it’s not full-blown time-travel undefined behavior.

Anyway, back to iterator invalidation:

In Rust, it is prevented at compile time.

This is neat! This is cool! This is good! And it’s pretty unique to Rust.

It is nice when our programs do what we expect, instead of what we don’t expect. It is even nicer when, if our programs don’t do what we expect, we have some sort of waypost, some sort of sign where the problem might be. What “nice” means may be a bit different in different languages, but the boundary is the important part: we’re gonna have to potentially violate the rules somewhere, so at least give us some help when that inevitably goes wrong. Over here, logic bugs may happen, some weird behavior may result, but over there? There be dragons. Nobody can save you out there. Tread carefully.

ThreadId(6)

If we think about all of these designs, they all are very similar conceptually: safe code is the default, but you can call into some sort of unsafe facility. And everyone is very clear on the relationship between the two: while the unsafe facility exists to be used, it must uphold the rules that the safe world relies on. And that means that safety and unsafety have a super/sub-set relationship. In the core is unsafety. But at some point, we draw a line, and on top of that line, safety exists.

In Rust, we often talk about how unsafe is great, because it clearly draws a line around code with which serious bugs may happen. More specifically, while most people describe it as “where memory safety can originate from” or similar, it’s actually slightly broader than that: it’s where undefined behavior can originate from.

The nomicon:

Rust code is incorrect if it exhibits any of the behaviors in the following list. This includes code within unsafe blocks and unsafe functions. unsafe only means that avoiding undefined behavior is on the programmer; it does not change anything about the fact that Rust programs must never cause undefined behavior.

It is the programmer’s responsibility when writing unsafe code to ensure that any safe code interacting with the unsafe code cannot trigger these behaviors. unsafe code that satisfies this property for any safe client is called sound; if unsafe code can be misused by safe code to exhibit undefined behavior, it is unsound.

This is not a new insight, but basically, unsafe is like FFI in a managed language. It’s not something you do often. But it is something you sometimes have to do. And when you do, you can at least contain the danger: the problem has to lie somewhere behind that veil. You have clear points to begin your search into where the issues lie, should the abyss claim another soul.

In some ways I’m also repeating another older slogan Rust had: “memory safety without garbage collection.” But it’s more like “no UB and no garbage collection.” Gosh, why didn’t we put that on the website? Practically rolls right off the tongue. What makes Rust appealing, and I think especially to the “non-systems” crowd, is that it shares a property of many managed languages, that of “no undefined behavior by default,” with a property of many systems languages, that of “no runtime, as little overhead as possible.” And sure, there’s an escape hatch, but it’s rarely used, and clearly separates the two worlds. This is a fundamentally different way of viewing the world than “unsafe by default.”

Another way in which there’s an interesting split here in language design is between the scripting languages like Ruby, Python, and JavaScript, and languages like Rust, Go, and Java, is that the boundary in the former is purely FFI, while in the latter, there are also unsafe faculties in the host language itself, as well as with FFI. In all three cases, they’re behind either a specific package or unsafe {}, but they give you some interesting tools that don’t require you to use another language to use. I think this is an under-appreciated part of the overall design space.

threads.into_iter().for_each(|t| t.join().unwrap());

So: did Rust focus too much on memory safety? And what should newer languages do in this area?

Well, I think that in some sense, the answer is obviously “no, Rust did not focus too much on memory safety.” Rust has been wildly successful, reaching heights that I only could dream about eleven years ago. Yes, usually I have tried to be fairly tempered when talking about Rust’s successes, and some folks still try to claim Rust is still super niche. But when I think about the ways that Rust touched even this blog post, it’s staggering: I wrote this post in VS: Code, which uses Rust, on Windows, which has Rust in the kernel. I published it to GitHub, which uses Rust for code search. I previewed it in Chrome, which uses some Rust today and may start having more tomorrow. It’s hosted on Vercel, which uses Rust in projects like turbopack, though maybe not in the critical path for this post. When you read it, the bytes probably passed through CloudFlare, which uses a ton of Rust, and maybe you’re reading this in Firefox, which is about 12% Rust. And maybe you’re on Asahi Linux, where the GPU drivers are written in Rust. Rust has Made It, and is here to stay. And the marketing we used got us here. So in some sense, it would be silly to declare said messaging a failure.

However, I also wonder what else could have been. Could Rust have grown faster with a different message? Maybe. That said, what would that message be? “Undefined behavior is scary?” I’m not sure that’s as compelling, even if I think it’s actually a bigger deal. While memory unsafety is only part of undefined behavior, it is the part that is easy to understand: pointers are the bane of every C newbie. It has consequences that are easy to understand: security vulnerabilities are bad. Has Rust really conflated “memory safety” and “safety” that much? I don’t actually think so. People still talk about SQL injection, about Remote Code Execution, about Cross Site Scripting. People don’t really believe that just because it’s written in Rust, bugs are impossible. Sure you can find a few comments on the internet as such, but the big organizations that are using Rust for meaningful work do not take this position. Given Amdahl’s Law, if 70% of security vulnerabilities are related to memory safety, focusing so strongly on it makes sense, as fixing that will give you a much larger impact than other things.

So what should new languages do here? Well, pretty much the only language that is gaining developer mindshare that’s new and is memory unsafe by default is Zig. I think Andrew is very sharp, and a tremendous engineer. I am very interested to see how things shake out there. But as a language, it does not appeal to me personally, because I think we’ve sort of moved past “unsafe by default.” That being said, it’s also not that Zig is exactly like other languages in this space either: if we are willing to go beyond “memory unsafe vs memory safe at compile time” as a strict binary, and instead look at “memory unsafe at compile time” itself, we see a pretty big gradient. Zig is doing a bunch of things to mitigate issues where possible. Maybe “safe enough” truly is safe enough. Time will tell. But beyond that, you also don’t have to go full Rust in the design space. Rust wants to be a language that you can (and at Oxide, we do) use at the lowest levels of the system. But also memory safe. And extremely fast. These constraints mean that Rust takes on a lot of complexity in the type system, where languages that are willing to relax these things wouldn’t have to. Here are three languages that I think are sort of “post-Rust” in a sense, that are learning from Rust but putting their own twist on things:

Hylo (formerly Val) tries to, in their own words, “strengthens [value semantics], while de-emphasizing reference semantics”. This lets them avoid a lot of Rust’s complexity. Hylo looks more explicitly to Swift than Rust for positive inspiration in many cases, but I’m including it here anyway, as I’ve sort of ended up not talking about Swift in this post.
Austral is “if affine types are good for systems programming, linear types are even better.” Austral also believes that parsimony is a virtue in language design, and that linear types can replace a lot of different features in other programming languages, while also eschewing other features that they deem as not carrying their weight.
Mojo, “the usability of Python with the performance of C.” It also has this sort of “lower into a more complex language” idea; you can Just Write Python for the most part and it works, but if you want to get fancier, you can. Lifetimes are a current proposal.

I think there are some other languages that are forming the next big cohort of programming languages, and they’re interesting too, but these three I think have the most explicit connection to Rust in the design and goals of the language itself. Shout out to Nu, Inko, Gleam, Roc, Oil Shell, Unison, and others. Lots of cool stuff going on.

int atexit(void (*func)(void))

There is one way in which the question “is the focus on memory unsafety misguided?” is much more serious than battles in the comment sections of various fora: the government.

We talked about this earlier, but various government organizations have been slowly putting out recommendations that organizations should be moving towards MSLs. And the first reactions to these were always “well they’re not laws so they don’t have teeth” as well as “laws are inevitably coming.” To be honest, I have mostly been in the former camp, and viewed the later as fearmongering. Sure, maybe someday, but like, dipping your toes in does not mean that you’re about to cannonball moments later.

But then I got this interesting reply from David Tolnay on Hacker News. In it, he refers me to ADSP: The Podcast, and more specifically, Episode 160: Rust & Safety at Adobe with Sean Parent. In it, Sean makes reference to two pieces of legislation, one in the US and one in the EU. Now, I am not a politican, and these bills are huge, so I wasn’t able to figure out how these bills do this specifically, but to quote Sean:

The one in the U.S. that’s pending basically says that the Department of Defense is going to within 270 days of the bill passing (and it’s a funding bill which means it will probably pass late this year - early next year) that the Department of Defense will establish guidelines around safety and security including memory safety for software products purchased by Department of Defense. The E.U. has a similar wording in a bill that’s slowly winding its way through their channels. I don’t have insight into when that will pass. The U.S. one will almost certainly pass here within a month or two.

This sounds much more realistic, of course. Now, this does not mean that C++ is outlawed or something silly like that, but it does mean that using Rust could become a more serious competitive advantage when it comes to selling to government: if you don’t need an exception for your product, that’s an advantage over a product which does.

This all also may go the way of another anecdote Sean talks about in the past: when a similar bill tried to mandate POSIX compliance. What I’m saying is, go listen to the podcast. And while this still isn’t something as wild as “not using MSLs is illegal,” it is really shocking to me how quickly things seem to be moving in that vague direction, though of course it will never actually arrive there. There seems to be at least some sort of thinking that’s not immediately lauged out of the room that memory unsafety could be a consumer safety issue in a similar sense that other unsafe materials are a consumer safety issue. Is that right? I’m not sure, but it seems to be where the conversation is heading.

So where does that leave existing widely used memory-unsafe languages? Well, the C++ community has… had a wide variety of discussions on this topic. I should also say I mean “community” at large, and the committee feels like it has a big divide as well. Now I should re-iterate that while I pay attention to the C++ standardization process, I am very much an outsider: I have never attended a meeting, I have no direct involvement in these things, I just read a lot. And while historically most of the community wrote Rust off as a novelty, I think that at this point, those with their head in the sand about Rust’s usage are mostly a fringe minority. But the big question that remains is, what to do?

Two approaches have appeared:

Improve the safety of C++ in some fashion.
Create a “successor language” that will fix C++‘s problems.

What is interesting to me about these approaches is that they are both good, for different reasons. The former is a sort of harm reduction approach: don’t let the perfect be the enemy of the good. What can we do today to improve the lives of a ton of people? But if you want to move into the realm of an MSL, you have to break backwards compatibility. This is tantamount to just doing #2 in the first place, and so some are cutting straight to that approach. But if we look at it in a slightly different way, an interesting thing happens:

breaking backwards compatibility vs not
memory safety by default vs not

This leads to four approaches:

We do not break backwards compatibility, and we ignore the memory safety issue (this is the ‘do nothing’ option).
We do not break backwards compatibility, but we try to make things as memory safe as possible given the constraints we have.
We break backwards compatibility, and we try to make things as memory safe as possible but still remain unsafe.
We break backwards compatibility, and we become memory safe by default.

The first option is seemingly untenable, but is also where I see C heading. There’s seemingly not nearly as much desire for movement in the C space as the C++ space, even though I am following what JeanHeyd Meneide is doing and appreciate it.

The second option is how I view at least one of the possible C++ successor languages, Circle. Circle is incredibly interesting and is worth its own post. But in short, it is a C++ compiler that also implements a lot of extensions, both ones that are proposals in the committee, but also ones that its author is interested in. He has been doing lifetime experiments.

The third option is where the most famous successor languages live: Carbon and cpp2. I haven’t gotten to spend a ton of time with either of these yet, but they explicitly aim to do to C++ what C++ did to C, or what TypeScript did to JavaScript: create a new language for new code, while allowing you to use the older language for older code. This allows them to break backwards compatibility more easily: new code can more easily be written under the new constraints. This gives them a larger degree of freedom to make changes, which may be necessary to move the needle significantly on the safety issue. But to my knowledge, they do not attempt to be memory safe by default. They’re conceptually similar to Zig in this way.

But what about a fourth option? What if someone did a TypeScript for C++, but one that was closer to what Rust does? You might argue that this is basically just Rust? (Interestingly, it is also sort of what Zig is with C: you can use the same compiler to compile a mixed codebase, which is close enough in my opinion.) What makes me worry about option #3 is that it doesn’t pass the seeming test that is coming: memory safe by default. I still think moving to a #3 style solution is better than staying with #1, but is that enough? I don’t know, time will tell. I wonder to what degree you can get “C++ but memory safe” without just being Rust. As we talked about earlier, most languages that are trying to improve on Rust in this space are willing to trade off some core constraints on Rust’s design to get them, but C++ and Rust share the same constraints. That said, it would take a tremendous amount of hubris to declare that Rust is the final, ultimate, best language in the “memory safe without GC” space. Rust has made a lot of mistakes too. Maybe reflecting on those will make for a good twelfth year post. Someone get started on Rust++, please!

The reason that today is my Rust-a-versary is that I usually play with a new language over Christmas break, and in 2012, that language was Rust. Maybe this Christmas, I’ll give cpp2 a try.

In the final minutes before publishing this, I also found another interesting report. This is a report to the CISA director by the Technical Advisory Council on how to engage with the memory safety question. Rust has 17 mentions. I haven’t read it yet, but I figured I’d include it here.

Updating Buck

Mon, 08 May 2023 00:00:00 GMT

Hey there! A shorter post today, but I wanted to continue my series on Buck by going over some things that have changed since this series started.

Updating Buck

Lots of great stuff has been happening since the initial release of buck2, and we’d like to take advantage of that. If we try and update things, though, we’ll get (well, you might not, but I did) an error:

~> cargo +nightly-2023-03-07 install --git https://github.com/facebook/buck2.git buck2
<snipped>
   Replacing C:\Users\steve\.cargo\bin\buck2.exe
error: failed to move `C:\Users\steve\.cargo\bin\cargo-installYqDCSV\buck2.exe` to `C:\Users\steve\.cargo\bin\buck2.exe`

Caused by:
  Access is denied. (os error 5)

(Why yes, my console prompt has changed…)

See on Windows, we can’t replace a program that’s running, while it’s running. On other systems, this may work for you. However, it’s probably a good idea to not take advantage of this particular feature, because of the underlying cause: buck runs a daemon in the background. This daemon is still going. Trying to replace it while it’s running means you’ll still have the old one going around in the back, and while that’s fine (is it? I don’t actually know), best to cleanly shut down first. So do this:

~\Documents\GitHub\buck-rust-hello> buck2 killall
Killed buck2.exe (4788). C:\Users\steve\.cargo\bin\buck2.exe --isolation-dir v2 daemon --dont-daemonize
Killed buck2.exe (40200). C:\Users\steve\.cargo\bin\buck2.exe --isolation-dir v2 daemon --dont-daemonize

buck2 killall will kill every instance of the daemon. As you can see, it found two of mine, and shut them down. As you can see, you want to run this command from within one of your projects.

And now an upgrade works.

Update Reindeer

If you’re building Rust code and using Reindeer, like we talked about in the last post, go ahead and grab that too. Lots of good stuff in there:

~> cargo install --git https://github.com/facebookincubator/reindeer/ reindeer -f

Now that we’re all updated, let’s fix up our project to make use of the latest changes that are relevant.

Update the prelude

I no longer need my weird toolchain hacks to get MSVC Rust working. Thank you dtolnay! 🙏

Let’s pull in changes to the prelude:

~\Documents\GitHub\buck-rust-hello> cd prelude
~\Documents\GitHub\buck-rust-hello\prelude> git fetch origin
remote: Enumerating objects: 28, done.
remote: Counting objects: 100% (28/28), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 17 (delta 12), reused 15 (delta 10), pack-reused 0
Unpacking objects: 100% (17/17), 3.60 KiB | 102.00 KiB/s, done.
From https://github.com/facebook/buck2-prelude
   920d3f2..370cd4d  main       -> origin/main
~\Documents\GitHub\buck-rust-hello\prelude> git reset --hard origin/main
HEAD is now at 370cd4d Http_archive execution platform hack
~\Documents\GitHub\buck-rust-hello\prelude> cd ..
~\Documents\GitHub\buck-rust-hello>

And remove the hack (if you had to do it too) in toolchains\BUCK:

system_cxx_toolchain(
    name = "cxx",
    visibility = ["PUBLIC"],
)

No more link.exe stuff!

Fix some configuration

Some good changes have landed in the buck2 init subcommand that we’ll want to add ourselves.

First up, we should add an empty file named .buckroot to the root of our repository. Because buck works with your filesystem hierarchy, it can and will traverse upwards looking for things at times. Adding this file ensures that if it does so, it will stop before it starts trying to traverse even higher. There’s no need for anything in the file, as the contents are ignored.

I think this kind of change is an interesting way to look at the usability of various systems. Adding an empty file here is sort of “more complex” than say, Cargo. But it’s also more explicit. Which means it can be more… I want to say “legible”, if you also read insufferable books like I do sometimes. And therefore, easier. Anyway, more thoughts on all of this in the future.

Next, we have a change that is demonstrated by this example. Can you guess what it is?

~\Documents\GitHub\buck-rust-hello> buck2 run //src/bin:hello_world
File changed: root//.git/modules/prelude
Build ID: 1a24b418-acdb-4d23-a5e1-6e9b644c01e6
Jobs completed: 83. Time elapsed: 1.9s. Cache hits: 0%. Commands: 3 (cached: 0, remote: 0, local: 3)

There’s no reason for buck to be watching our .git directory for changes. And that’s why it is now put in the default configuration when you buck2 init --git. But we’ve gotta make that change ourselves. Open up your .buckconfig and add this at the bottom:

[project]
ignore = .git

We want it to ignore the .git directory. Seems good.

… I lied though, there’s one more improvement we want to make: we also don’t want buck to bother listening to the target directory either, as those files are for Cargo’s output. So what we actually want is this:

[project]
ignore = .git, target

After doing that we’ll want to buck2 kill to shut the daemon down, so that it can pick up our new configuration on the next boot.

Regenerating reindeer

Since we’ve got new bugfixes in Reindeer too, let’s regenerate our config for our dependencies:

~\Documents\GitHub\buck-rust-hello> reindeer --third-party-dir third-party buckify
[WARN  reindeer::fixups] semver-1.0.17 has a build script, but I don't know what to do with it: Unresolved build script at ..\..\..\..\.cargo\registry\src\github.com-1ecc6299db9ec823\semver-1.0.17\build.rs. Dependencies:

We still have to deal with the build script! We didn’t talk about the contents of third-party\BUCK last time, and we won’t this time either. If you want to see what’s changed, you can take a peek though. One change that we didn’t explicitly talk about before, but you may end up noticing, is that it did not generate a target to try and build our build script.

Let’s try it out now:

~\Documents\GitHub\buck-rust-hello> buck2 run //src/bin:hello_world
File changed: root//BUCK
File changed: root//third-party/BUCK
File changed: root//third-party
Build ID: 2eb0c121-d1fb-43d2-b8a4-f923d8dda657
Jobs completed: 20. Time elapsed: 1.1s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 0, local: 2)

Nice. Also note that we aren’t seeing any more .git or target changes, not that we’ve run anything that would inherently change those files, but go ahead, invoke Cargo or git, and then build again. You shouldn’t see notifications about those directories anymore.

Fixing some breakage

Speaking of invoking Cargo, remember how I said this in the last post?

We do have two different Cargo.tomls now. That is a bit of a bummer. But at least it is easy to determine if there’s a problem: dependency failures are loud, and if you’re building with both in CI, you’ll notice if stuff goes wrong. There also may be a solution to this I’m just not aware of.

Well…

~\Documents\GitHub\buck-rust-hello> cargo run
   Compiling hello_world v0.1.0 (C:\Users\steve\Documents\GitHub\buck-rust-hello)
error[E0432]: unresolved import `semver`
 --> src\bin\main.rs:1:5
  |
1 | use semver::{BuildMetadata, Prerelease, Version, VersionReq};
  |     ^^^^^^ use of undeclared crate or module `semver`

For more information about this error, try `rustc --explain E0432`.
error: could not compile `hello_world` due to previous error

Going back and re-reading my last post, I did have a cargo add semver in there, so maybe I just forgot to commit that in my last post. Just in case, we’ll fix that:

~\Documents\GitHub\buck-rust-hello> cargo add semver
    Updating crates.io index
      Adding semver v1.0.17 to dependencies.
             Features:
             + std
             - serde

With that, cargo build and cargo run are back in business.

We also have… well “breakage” isn’t exactly right, but we have a buck configuration issue. Let’s try to build every target:

~\Documents\GitHub\buck-rust-hello> buck2 build ...
Error running analysis for `root//:build (prelude//platforms:default#fb50fd37ce946800)`

Caused by:
    0: Error looking up configured node root//:build (prelude//platforms:default#fb50fd37ce946800)
    1: `root//src/bin:hello_world` is not visible to `root//:build` (run `buck2 uquery --output-attribute visibility root//src/bin:hello_world` to check the visibility)
Build ID: 51657a07-112c-46b4-a2eb-91d60a4b0aed
Jobs completed: 8. Time elapsed: 0.0s.
BUILD FAILED

We didn’t declare that our binary was visible anywhere, and so when we try and build it, it isn’t happy. We do want this to be public, so change src\bin\BUCK by adding this visibility line near the end. It should look like this:

rust_binary(
    name = "hello_world",
    srcs = ["main.rs"],
    crate_root = "main.rs",
    deps = [
        "//third-party:semver",
    ],
    visibility = ["PUBLIC"],
)

And now that will work:

~\Documents\GitHub\buck-rust-hello> buck2 build ...
File changed: root//src/bin/BUCK
Build ID: 9da23da4-3f99-4e78-8c58-ae5e2f1facaa
Jobs completed: 25. Time elapsed: 0.6s. Cache hits: 0%. Commands: 3 (cached: 0, remote: 0, local: 3)
BUILD SUCCEEDED

Okay, now that we’ve fixed both Cargo and buck… let’s make sure that isn’t gonna happen again. We aren’t testing any of this. So it broke. Just like I said it was easy to not let break. Sigh.

We’re going to use GitHub Actions because this is already on GitHub. I’m sure you can adapt it to your setup of choice.

Put this in .github\workflows\ci.yml:

name: CI

on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    name: ${{matrix.name || format('Rust {0}', matrix.rust)}}
    runs-on: ${{matrix.os || 'ubuntu'}}-latest
    strategy:
      fail-fast: false
      matrix:
        include:
          - rust: nightly
          - rust: beta
          - rust: stable
          - rust: 1.69.0
          - name: Cargo on macOS
            rust: nightly
            os: macos
          - name: Cargo on Windows (msvc)
            rust: nightly-x86_64-pc-windows-msvc
            os: windows
    steps:
      - uses: actions/checkout@v3
      - uses: dtolnay/rust-toolchain@master
        with:
          toolchain: ${{matrix.rust}}
      - run: cargo check
      - run: cargo run

  buck:
    name: Buck2 on ${{matrix.os == 'ubuntu' && 'Linux' || matrix.os == 'macos' && 'macOS' || matrix.os == 'windows' && 'Windows' || '???'}}
    runs-on: ${{matrix.os}}-latest
    if: github.event_name != 'pull_request'
    strategy:
      matrix:
        os: [ubuntu, macos, windows]
    steps:
      - uses: actions/checkout@v3
        with:
          submodules: true
      - uses: dtolnay/rust-toolchain@stable
        with:
          components: rust-src
      - uses: dtolnay/install-buck2@latest
      - run: buck2 build ...
      - run: buck2 run //src/bin:hello_world

      - uses: actions/cache/restore@v3
        id: cache
        with:
          path: ~/.cargo/bin/reindeer${{matrix.os == 'windows' && '.exe' || ''}}
          key: ${{matrix.os}}-reindeer

      - run: cargo install --git https://github.com/facebookincubator/reindeer reindeer
        if: steps.cache.outputs.cache-hit != 'true'

      - uses: actions/cache/save@v3
        if: steps.cache.outputs.cache-hit != 'true'
        with:
          path: ~/.cargo/bin/reindeer${{matrix.os == 'windows' && '.exe' || ''}}
          key: ${{steps.cache.outputs.cache-primary-key}}

      - run: reindeer buckify
        working-directory: third-party
        if: matrix.os == 'ubuntu'
      - name: Check reindeer-generated BUCK file up to date
        run: git diff --exit-code
        if: matrix.os == 'ubuntu'

This is… you guessed it, based off of dtolnay’s CI for cxx. What can I say, he writes good code. This version is a bit stripped down, since this is primarily a project for showing off a build system, rather than a regular project. This has:

Builds with Cargo on Linux against nightly, beta, stable, and 1.69.0 as a sort of MSRV check.
Builds with Cargo on MacOS and Windows against nightly Rust.
Buck on Linux, MacOS and Windows against stable Rust.
On Linux, checks that we have comitted any relevant changes to dependencies.

This does not have:

A “fail if there are any warnings” build
A clippy build
A full matrix of every os/language/build system combination

or other things you may want out of a build. Again, you’ll probably want to customize this heavily, but this is what I’m going to do here.

Wrapping up

And with that, we are done! Next post we’re going to deal with that build script. And use some crates that have more intense dependencies, and get them all working.

As always, you can check out the code at this point if you’d like.

Using Crates.io with Buck

Thu, 27 Apr 2023 00:00:00 GMT

In a previous post, I laid out the basics of building a Rust project with buck2. We compared and contrasted it with Cargo. But what about one of the biggest and best features that Cargo has to offer, the ability to use other Rust packages from crates.io? They don’t use buck, so how can we integrate them into our build?

This post represents how to do this at the time that this was posted; future posts may update or change something that happens here. Here’s a hopefully complete but possibly incomplete list of updates and the posts that talk about it:

build-script-build target is no longer generated, see “Updating Buck”

Depending on semver

Let’s use the semver package example as our program. I am choosing this for a few reasons:

I used to be involved in maintaining it, and I’m sentimental
It’s authored by dtolnay, who authors many great Rust crates.
More specifically than that, his excellent cxx library maintains build rules for Cargo, Buck, and Bazel, and examining how he uses Buck in cxx helped me write both the last post and this one. I wanted to make sure to shout that out.
It’s pure Rust (this is easier than something that depends on C, since as I mentioned I’m still having an issue or two with getting my own C toolchain working so far).
It has one optional dependecy on serde, but no others. Again, this is just easier, not a fundamental limitation.

Here’s the example, from the page on crates.io:

use semver::{BuildMetadata, Prerelease, Version, VersionReq};

fn main() {
    let req = VersionReq::parse(">=1.2.3, <1.8.0").unwrap();

    // Check whether this requirement matches version 1.2.3-alpha.1 (no)
    let version = Version {
        major: 1,
        minor: 2,
        patch: 3,
        pre: Prerelease::new("alpha.1").unwrap(),
        build: BuildMetadata::EMPTY,
    };
    assert!(!req.matches(&version));

    // Check whether it matches 1.3.0 (yes it does)
    let version = Version::parse("1.3.0").unwrap();
    assert!(req.matches(&version));
}

Let’s change src/bin/main.rs to contain this code. What do we need to do to add this with Cargo?

buck-rust-hello〉cargo add semver
      Adding semver v1.0.17 to dependencies.
             Features:
             + std
             - serde
buck-rust-hello〉cargo run
   Compiling semver v1.0.17
   Compiling hello_world v0.1.0 (C:\Users\steve\Documents\GitHub\buck-rust-hello)
    Finished dev [unoptimized + debuginfo] target(s) in 0.71s
     Running `target\debug\main.exe`
buck-rust-hello〉

Easy enough! We expect no output, becuase the asserts should pass. (Note that we didn’t ask for the serde feature, so we aren’t using it, and therefore don’t depend on anything other than semver.)

How Cargo handles this

Before we move on, let’s talk for a moment about what Cargo actually does here.

First, cargo add has added this to our Cargo.toml:

[dependencies]
semver = "1.0.17"

This is the latest release of semver at the time of this writing.

When we cargo build, Cargo will figure out all of the crates we need. It will then check the cache for the source code for that crate, and if it’s not there, download it. On my system, this lives here:

 ~\.cargo\registry\src\github.com-1ecc6299db9ec823\semver-1.0.17\

This means we have a global cache of source code.

Cargo will then compile semver, as you can see in the output above. It places that output in the target directory, more specifically

.\target\debug\deps

This directory will contain libsemver-ded1559592aad8f7.rlib, libsemver-ded1559592aad8f7.rmeta, and semver-ded1559592aad8f7.d. These have hashes embedded in them so that if we had multiple versions of semver in our project, they can be disambiguated. If you’re not familiar with rustc output:

rlib files are simlar to archive files, like Unix ar. The details are not standard, and may change at any time. Object code lives in here, as well as other things.
rmeta files contain crate metadata. This can be used for cargo check, for example.
.d files are a dependency file. This format comes from gcc/make, and is sorta standardized. This is in theory useful for use with other build systems, but we won’t be using this today.

Cargo will then build our project, passing in libsemver-*.rlib as a dependency.

If you’re curious about the exact commands and flags, cargo build -v will show that. Make sure to cargo clean first, or else you’ll get no output, given that we’ve already built this project.

For example, here’s the rustc invocation that Cargo makes for building this step:

rustc --crate-name main --edition=2021 src\bin\main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=c398fba113f38e08 --out-dir C:\Users\steve\Documents\GitHub\buck-rust-hello\target\debug\deps -C incremental=C:\Users\steve\Documents\GitHub\buck-rust-hello\target\debug\incremental -L dependency=C:\Users\steve\Documents\GitHub\buck-rust-hello\target\debug\deps --extern hello_world=C:\Users\steve\Documents\GitHub\buck-rust-hello\target\debug\deps\libhello_world-c5b898f1ca338cca.rlib --extern semver=C:\Users\steve\Documents\GitHub\buck-rust-hello\target\debug\deps\libsemver-ded1559592aad8f7.rlib

Since we are not using Cargo, we need to replace all of that stuff, and get Buck to generate that rustc line, or at least, some equivalent of it.

Let’s talk about the various things we need to do:

The mismatch

Let’s start with one thing that is basically the same: buck-out and target are both directories in our project that cache the output of our build. Yeah the name and details are different, but we’re not going to try and somehow unify these, as they’re both considered implementaiton details of the respective systems, and trying to get them to share is a lot of work for not much gain.

Buck does not have a central registry of packages that we can download code from.

Buck is interested in reproducable builds, and therefore, a global cache of source code doesn’t make as much sense. You want the code stored locally, with your project. The dreaded (or beloved) vendoring.

Buck does not understand the crate index, Cargo configuration for a given package, and other implementation details. As a more general build system, those are pretty much out of scope.

Lucikly, other people have done a lot of work here.

Reindeer

Enter Reindeer. Reindeer is a project that will help us bridge this gap. Here’s how this will work: Reindeer will create and (mostly) manage a third-party directory for us. It will generate BUCK files that we can then use to depend on these external crates. We can even choose to vendor our sources or not.

You can install reindeer through Cargo:

> cargo install --git https://github.com/facebookincubator/reindeer/ reindeer

Let’s set this up. First off, we need to create some files:

> mkdir third-party
> code third-party\Cargo.toml # or vim, or whatever

In that Cargo.toml, we’ll need to put this:

[workspace]

[package]
name = "rust-third-party"
version = "0.0.0"
publish = false
edition = "2021"

# Dummy target to keep Cargo happy
[[bin]]
name = "fake"
path = "/dev/null"

[dependencies]
semver = "1.0.17"

We’re creating a fake package here, so nothing truly matters except the [dependencies] section. Here we depend on semver as usual.

We’re also going to configure Reindeer to not vendor our source code, because that’s how I prefer to do things. Vendoring was the default behavior, with non-vendoring being added very recently, so if you prefer to vendor, that workflow works very well

Anyway put this in third-party/reindeer.toml:

vendor = false

So run this inside of your third-party directory:

〉cd third-party
〉reindeer buckify
[WARN  reindeer::fixups] semver-1.0.17 has a build script, but I don't know what to do with it: Unresolved build script at vendor\semver\build.rs. Dependencies:

Oh no, build scripts! Yeah I said that I picked semver because it should be easy, but it does have a build script, which is another Cargo-specific feature. Now, the semver crate’s build script is used as a feature to support older versions of the compiler; all it does is detect old versions and then spit out some configuration to make sure to not use the newer features of the language. This is why this is a warning, not an error; in this case, we don’t actually need the build script since we are using a new compiler. So, we are going to skip this for now, but we’ll come back and fix it.

At this point, reindeer has generated a BUCK file for semver. Let’s see what targets we have now:

〉buck2 targets //...
Jobs completed: 4. Time elapsed: 0.0s.
root//:build
root//src/bin:hello_world
root//src/lib:hello_world
root//third-party:semver
root//third-party:semver-1.0.17
root//third-party:semver-1.0.17-build-script-build

We have a few new ones! One for semver, one for the specific version of semver, and one for the build script of semver. The general semver is an alias for semver-1.0.17.

Do you know how to modify our build so that buck builds successfully?

Here’s the answer: change src\bin\BUCK:

rust_binary(
    name = "hello_world",
    srcs = ["main.rs"],
    crate_root = "main.rs",
    deps = [
        "//third-party:semver",
    ],
)

And now we can build and run:

〉buck2 run //src/bin:hello_world
File changed: root//src/bin/BUCK
Build ID: b18ba58d-8a77-439a-9d95-6051f3cf21d4
Jobs completed: 26. Time elapsed: 1.9s. Cache hits: 0%. Commands: 3 (cached: 0, remote: 0, local: 3)

Success! Our program has no output, if the assertions failed we would have gotten something, but this is expected given the example code.

Now, whenever we need to add or remove a dependency, we can modify third-party\Cargo.toml, re-run the buckify command, and we’re good.

We do have two different Cargo.tomls now. That is a bit of a bummer. But at least it is easy to determine if there’s a problem: dependency failures are loud, and if you’re building with both in CI, you’ll notice if stuff goes wrong. There also may be a solution to this I’m just not aware of.

If you’d like to see the whole thing at this point, this commit should have you covered.

This should get you going with building more advanced projects in Rust using buck2. In the next post, we’ll talk about fixups, which you’ll need for dependencies that are more complex than semver.

By the way, if you’re interested in this stuff, I’ve made a Discord for buck fans. Come hang out, chat about stuff, ask questions, it’s gonna be a good time.

Using buck to build Rust projects

Thu, 13 Apr 2023 00:00:00 GMT

A few days ago, Facebook/Meta/idk announced that buck2 is now open source.

Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient.

As it just so happens, I have taken an increasing interest in build systems lately. I have mostly ignored the cool kids’ build system things, because I have never worked at a FAANG, or at a true monorepo shop. I also personally try and avoid the JVM wherever possible, and the first generation of these tools were all built on top of it. (Yes, that bias may be outdated, I’m not saying you should avoid the JVM, just stating my own bias up front.)

So this timing was perfect! Let’s explore what using buck looks like.

A brief aside: listen, I have no love for Facebook. I deleted my account almost a dozen years ago. That doesn’t mean I’m not interested in using good tools they produce. If you feel differently, fine, but that’s not what I want to talk about today, so I’m not going to.

Oh, one last bit before I begin: I’m not going to do a lot of motivating on “why would you want to use Buck?” in this post. There’s a few reasons for that, but for now, if that’s what you’re looking for, this post isn’t it. We’re doing this purely for the novelty of trying out some new tech right now. I will probably end up with a post giving better motivations at some point in the future, but I think it makes more sense once you see how it works, rather than starting there.

buck2 init also creates a file named .buckroot, see “Updating Buck”

Getting started with buck2

The Getting Started page will give you instructions on installing buck. As of this moment, the instructions are:

$ rustup install nightly-2023-01-24
$ cargo +nightly-2023-01-24 install --git https://github.com/facebook/buck2.git buck2

This is mega-convenient for me as a Rust user, but probably not if you don’t have Rust installed. That said, this is a first release, and so I don’t expect anything fancier. This is what cargo install is good for!

Let’s make a new directory, hello:

$ mkdir buck-rust-hello
$ cd buck-rust-hello

To initialize a project, we use this command:

$ buck2 init --git

Before we move forward, let’s examine what this generated for us.

Initial project files

We now have this stuff in our directory:

$ git add .
$ git status
On branch main

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
        new file:   .buckconfig
        new file:   .gitignore
        new file:   .gitmodules
        new file:   BUCK
        new file:   prelude
        new file:   toolchains/BUCK

Let’s talk about each of these in turn.

`.buckconfig`

The .buckconfig file is… a configuration file for Buck, go figure. It looks like this:

[repositories]
root = .
prelude = prelude
toolchains = toolchains
none = none

[repository_aliases]
config = prelude
fbcode = none
fbsource = none
buck = none

[parser]
target_platform_detector_spec = target:root//...->prelude//platforms:default

That none = none is kind of amusing. Regardless of that, this file is extremely important: it configures the entire thing. In a sense, it’s like Cargo.toml: just like a package is defined by the existence of a Cargo.toml, a .buckconfig defines the existence of a… cell. Which defines a package. We’ll get there. Point is, this is the top level configuration. We say that the repository root is in the current directory, we’d like to use the default prelude and toolchains.

I uh… I don’t know what the none = none is for. It might just be a bug. I haven’t seen it in some of the other config files I’ve poked at. Let’s just leave that alone for now. I do have a suspicion though… and it involves the next section.

We also have a table for repository aliases. I couldn’t find any documentation on this, but I would imagine this means we could use the name config instead of prelude later. Looks like we don’t have any way to refer to fbcode and fbsource, which makes sense, and same with buck.

(I wonder if this is what the none = none is about above, defining a sort of “none repository” that we can then alias these to.)

Finally, we have a parser table, with one entry, pointing out where a thing exists. I know this configures Buck’s parser, but other than that… I’m sure I’ll figure it out eventually.

`.gitmodules` & `prelude`

We have a git submodule, pointing to https://github.com/facebook/buck2-prelude.git, that lives at the prelude directory. If you poke around in there, you’ll find a bunch of .bzl files that implement useful features for us to use. We’ll get into those in a moment, but the point is that this is sort of like a ‘standard library’ if you will. You could also not use it and define your own. If you’re that kind of person.

`.gitignore`

A very simple .gitignore will be created, that contains one line: /buck-out. This is where buck stores artifacts produced by your builds, so we don’t want that checked into version control.

`BUCK`

Now we get to the good stuff. Here’s the generated BUCK file:

# A list of available rules and their signatures can be found here: https://buck2.build/docs/generated/starlark/prelude/prelude.bzl

genrule(
    name = "hello_world",
    out = "out.txt",
    cmd = "echo BUILT BY BUCK2> $OUT",
)

genrule is like a function, provided by our prelude. If you’re curious, the implementation is in prelude/genrule.bzl. This command, as you may imagine, sets up a rule, named hello_world, that produces a file called out.txt. It does this by running the cmd. Nice and straightforward. We’ll give that a try in a moment, but first, one last file:

`toolchains/BUCK`

This file describes a toolchain. Here’s the contents:

load("@prelude//toolchains:genrule.bzl", "system_genrule_toolchain")

system_genrule_toolchain(
    name = "genrule",
    visibility = ["PUBLIC"],
)

This loads a certain rule from the prelude, and then defines this as a public toolchain. We can define as many toolchains as we want here, for example, if we wanted to build both Rust and Python, we could define both toolchains here for later use.

The “genrule” toolchain is used to generate files from a shell command, as we saw before with our rule that produces out.txt. So, in my understanding, here we are defining that we wish to actually use that. And then, in the BUCK file, we’re using this toolchain to implement our rule.

Invoking our first build

Okay, let’s actually give this a shot. To instruct Buck to build something, we invoke it with the “target pattern” as an argument. Let’s ask Buck what targets it knows how to build. To do this:

C:\Users\steve\Documents\GitHub\buck-rust-hello〉buck2 targets //...                                                     04/10/2023 02:01:50 PM
Build ID: cd778a29-2ba4-484b-8956-dc67f6fc0625
Jobs completed: 4. Time elapsed: 0.0s.
root//:hello_world

The //... is a “target pattern.” The /... means “all build targets in build files in subdirectories”, and / means our root directory, so //... means “all targets in all build files in all subdirectories.” By passing this target to buck2 targets, we can see every target in the project. This shows our one target we’ve defined, root://:hello_world. This name was defined in our BUCK file above. If you change that to

genrule(
    name = "lol",

then buck2 targets //... would show root://:lol.

Let’s actually build our target:

〉buck2 build //:hello_world
File changed: root//BUCK
Build ID: 73f4b797-2238-47bc-8e43-7ffcb2b7d9b7
Jobs completed: 36. Time elapsed: 0.0s. Cache hits: 0%. Commands: 1 (cached: 0, remote: 0, local: 1)
BUILD SUCCEEDED

Okay the build succeeded, but where is our out.txt? We can ask buck!

〉buck2 build //:hello_world --show-output
Build ID: 7ce93845-ab1e-4b0a-9274-51fed9f9e295
Jobs completed: 3. Time elapsed: 0.0s.
BUILD SUCCEEDED
root//:hello_world buck-out\v2\gen\root\fb50fd37ce946800\__hello_world__\out\out.txt

It lives in a deeply nested subdirectory of buck-out, a new top-level directory that was created for us. If you remember from before, this directory is ignored in our .gitignore.

If we look at the file, you can see it contains the text we wanted it to contain.

Let’s build a second time!

〉buck2 build //:hello_world
File changed: root//.git/index.lock
File changed: root//.git
File changed: root//.git/modules/prelude/index.lock
31 additional file change events
Build ID: c00e4bfa-a1f8-40c7-a61c-2a394dca5da5
Jobs completed: 5. Time elapsed: 0.0s.
BUILD SUCCEEDED

Buck has noticed that we’ve changed some files, but since our rule doesn’t depend on any of them, we’re good to go.

Building some Rust code

Okay, echo to a file is fun, but let’s actually build some Rust. Create a file, hello.rs:

fn main() {
    println!("Hello, world!");
}

and then update the BUCK file to this:

rust_binary(
    name = "hello_world",
    srcs = ["hello.rs"],
    crate_root = "hello.rs",
)

This says “hey, we’re building a Rust binary, it has this target name, these source files, and the crate root lives here.” Given we only have one file, there’s some reptition. It happens. Let’s build:

〉buck2 build //:hello_world
File changed: root//BUCK
Error running analysis for `root//:hello_world (prelude//platforms:default#fb50fd37ce946800)`

Caused by:
    0: Error looking up configured node root//:hello_world (prelude//platforms:default#fb50fd37ce946800)
    1: Error looking up configured node toolchains//:cxx (prelude//platforms:default#fb50fd37ce946800) (prelude//platforms:default#fb50fd37ce946800)
    2: looking up unconfigured target node `toolchains//:cxx`
    3: Unknown target `cxx` from package `toolchains//`.
       Did you mean one of the 1 targets in toolchains//:BUCK?
Build ID: f126ce07-efe8-41d3-8aae-8b7d31a4dafc
Jobs completed: 4. Time elapsed: 0.0s.
BUILD FAILED

Oops! We didn’t set up a rust toolchain! Let’s do that now. Edit toolchains/BUCK:

load("@prelude//toolchains:rust.bzl", "system_rust_toolchain")

system_rust_toolchain(
    name = "rust",
    default_edition = "2021",
    visibility = ["PUBLIC"],
)

And… when we build again, the same error. Now. I am not 100% sure what’s going on here, but this is what we need to do:

load("@prelude//toolchains:rust.bzl", "system_rust_toolchain")
load("@prelude//toolchains:genrule.bzl", "system_genrule_toolchain")
load("@prelude//toolchains:cxx.bzl", "system_cxx_toolchain")
load("@prelude//toolchains:python.bzl", "system_python_bootstrap_toolchain")

system_genrule_toolchain(
    name = "genrule",
    visibility = ["PUBLIC"],
)

system_rust_toolchain(
    name = "rust",
    default_edition = "2021",
    visibility = ["PUBLIC"],
)

system_cxx_toolchain(
    name = "cxx",
    visibility = ["PUBLIC"],
)

system_python_bootstrap_toolchain(
    name = "python_bootstrap",
    visibility = ["PUBLIC"],
)

I believe that this is because, to compile the Rust compiler, we need Python and a C++ compiler. Well, I did believe that, but after digging into things some more, it’s that the Rust toolchain from the prelude depends on the CXX toolchain in the prelude, because the Rust toolchain invokes the C compiler to invoke the linker. I’m still not 100% sure why Python needs to be in there. Anyway.

Now, when I run, I got this:

〉buck2 build //:hello_world -v 3
Action failed: prelude//python_bootstrap/tools:win_python_wrapper (symlinked_dir resources)
Internal error: symlink(original=../../../../../../../../../prelude/python_bootstrap/tools/win_python_wrapper.bat, link=C:\Users\steve\Documents\GitHub\buck-rust-hello\buck-out\v2\gen\prelude\fb50fd37ce946800\python_bootstrap\tools\__win_python_wrapper__\resources\win_python_wrapper.bat): A required privilege is not held by the client. (os error 1314)
Build ID: 57a66885-f7e7-474b-a782-b49fc4425be9
Jobs completed: 14. Time elapsed: 0.0s.
BUILD FAILED
Failed to build 'prelude//python_bootstrap/tools:win_python_wrapper (prelude//platforms:default#fb50fd37ce946800)'

I got this becuase I’m on Windows, and Windows restricts the ability to create symlinks by default. Turning on “Developer Mode” (which I’m surprised that I haven’t had to turn on so far yet), I get further:

<whole bunch of output>
  = note: 'clang++' is not recognized as an internal or external command,
          operable program or batch file.

Here’s that “invokes the compiler to get the linker” thing I was referring to above.

Now… by default, the Rust support is for the GNU version of the Windows toolchain. I never use that. Upstream has said that they want everything to be supported, so that change will come at some point, maybe by the time you read this! But in the meantime, I could get my (pure Rust) projects building with two small patches:

diff --git a/prelude/toolchains/cxx.bzl b/prelude/toolchains/cxx.bzl
index c57b7b8..dc14ca7 100644
--- a/prelude/toolchains/cxx.bzl
+++ b/prelude/toolchains/cxx.bzl
@@ -39,7 +39,7 @@ def _system_cxx_toolchain_impl(ctx):
         CxxToolchainInfo(
             mk_comp_db = ctx.attrs.make_comp_db,
             linker_info = LinkerInfo(
-                linker = RunInfo(args = ["clang++"]),
+                linker = RunInfo(args = ["link"]),
                 linker_flags = ["-fuse-ld=lld"] + ctx.attrs.link_flags,
                 archiver = RunInfo(args = ["ar", "rcs"]),
                 archiver_type = archiver_type,
diff --git a/prelude/toolchains/rust.bzl b/prelude/toolchains/rust.bzl
index 8172090..4545d55 100644
--- a/prelude/toolchains/rust.bzl
+++ b/prelude/toolchains/rust.bzl
@@ -23,7 +23,7 @@ _DEFAULT_TRIPLE = select({
         # default when we're able; but for now buck2 doesn't work with the MSVC
         # toolchain yet.
         "config//cpu:arm64": "aarch64-pc-windows-gnu",
-        "config//cpu:x86_64": "x86_64-pc-windows-gnu",
+        "config//cpu:x86_64": "x86_64-pc-windows-msvc",
     }),
 })

Now a build works!

〉buck2 build //:hello_world
File changed: root//BUCK
File changed: root//.git/index.lock
File changed: root//.git
6 additional file change events
Build ID: 65fc80aa-7bfa-433a-bfa7-57919147b550
Jobs completed: 65. Time elapsed: 1.0s. Cache hits: 0%. Commands: 1 (cached: 0, remote: 0, local: 1)
BUILD SUCCEEDED

We can run it to see the output:

〉buck2 run //:hello_world
Build ID: 78b0ca23-2c7c-4c02-a161-bba15e3b38bd
Jobs completed: 3. Time elapsed: 0.0s.
hello world

Same idea as cargo run.

Speaking of cargo run, what might this look like with Cargo? Well, we can create a Cargo.toml:

[package]
name = "hello_world"
version = "0.1.0"
edition = "2021"

[[bin]]
name = "hello_world"
path = "hello.rs"

and try it out. Oh, and you’ll probably want to put target into your .gitignore.

Let’s build. The “benchmark” command in nushell is sort of like time on a UNIX system:

〉benchmark { cargo build }
   Compiling hello_world v0.1.0 (C:\Users\steve\Documents\GitHub\buck-rust-hello)
    Finished dev [unoptimized + debuginfo] target(s) in 0.34s
416ms 490us 100ns
〉benchmark { cargo build }
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s
77ms 317us 200ns

Not too bad, a bit under half a second for the initial build, and near immediate on a subsequent build. What about buck?

〉benchmark { buck2 build //:hello_world -v 3 }
Running action: <snip>
Build ID: 47ebd9f1-3394-4f72-a0fb-02c936035d2b
Jobs completed: 58. Time elapsed: 0.8s. Cache hits: 0%. Commands: 1 (cached: 0, remote: 0, local: 1)
BUILD SUCCEEDED
945ms 557us
〉benchmark { buck2 build //:hello_world -v 3 }
Build ID: 5eed24e8-a540-454f-861a-855464aba3c9
Jobs completed: 3. Time elapsed: 0.0s.
BUILD SUCCEEDED
78ms 274us 100ns

Not too shabby; Buck is a teeny bit slower on the initial build, but when stuff is cached, both are the same speed. All of this is fast enough to qualify as “basically instant.”

Adding a library

Let’s up the complexity a bit, by adding a library that we want to depend on. Here’s a lib.rs:

pub fn print_hello() {
    println!("Hello, world!");
}

We want to change our top-level BUCK to add this:

+rust_library(
+    name = "print_hello",
+    srcs = ["lib.rs"],
+    edition = "2021",
+    visibility = ["PUBLIC"],
+)
+
 rust_binary(
     name = "hello_world",
     srcs = ["hello.rs"],
     crate_root = "hello.rs",
+    deps = [
+        ":print_hello",
+    ],
 )

Here, we make a new library, print_hello, and then make our binary depend on it.

Let’s change the code in main.rs to use the library:

fn main() {
    println!("hello world");
    print_hello::print_hello();
}

And that’s it! Let’s examine our targets:

〉buck2 targets //...
Build ID: 4646f2e7-0ea3-4d59-8590-3da0708ce96e
Jobs completed: 4. Time elapsed: 0.0s.
root//:hello_world
root//:print_hello

They’re both there! We can now build one, the other, or everything:

# build everything
〉buck2 build //...
# build just the library
〉buck2 build //:print_hello
# build 'just' the binary, this will of course end up building the library too
〉buck2 build //:print_hello

Let’s make sure it still prints our output:

〉buck2 run //:hello_world
Build ID: d76c80fb-dd77-463a-86a1-b6443cea10f6
Jobs completed: 3. Time elapsed: 0.0s.
Hello, world!

Nice.

Let’s compare that to Cargo. Modify Cargo.toml:

[lib]
name = "print_hello"
path = "lib.rs"

and build:

〉cargo run
   Compiling hello_world v0.1.0 (C:\Users\steve\Documents\GitHub\buck-rust-hello)
    Finished dev [unoptimized + debuginfo] target(s) in 0.42s
     Running `target\debug\hello_world.exe`
Hello, world!

Nice.

more cargo-ish

Before we move forward, this isn’t exactly an apples to apples comparison; we’ve been doing a lot of configuration for Cargo that we normally wouldn’t have to do, and also, what if you already have a Cargo project, but you want to try out Buck with it?

Do this:

> mkdir src
> mv hello.rs src/main.rs
> mv lib.rs src/main.rs

And delete the configuration from Cargo.toml, leaving just the package table. Finally, we need to change src/main.rs, given that we’re using the default crate name for the library crate, which is hello_world and not print_hello:

fn main() {
    hello_world::print_hello();
}

After this, cargo build works just fine. But what about Buck?

So. We have a bit of weirdness here, and I’m not sure if it’s actually work-around-able in Buck or not, since I’m still learning this myself. But if we do the basic translation, we’ll get an error. Let’s try it. This is how you modify the BUCK file:

rust_library(
    name = "hello_world",
    srcs = glob(["src/**/*.rs"]),
    edition = "2021",
    visibility = ["PUBLIC"],
)

rust_binary(
    name = "hello_world",
    srcs = ["src/main.rs"],
    crate_root = "src/main.rs",
    deps = [
        ":print_hello",
    ],
)

Cargo produces a binary and a library, both called hello_world, but buck doesn’t like that:

〉buck2 run //:hello_world
Error evaluating build file: `root//:BUCK`

Caused by:
    Traceback (most recent call last):
      * BUCK:8, in <module>
          rust_binary(
    error: Attempted to register target root//:hello_world twice, re-run the command with `--target-call-stacks` to obtain a call stack of the first registration
      --> BUCK:8:1
       |
     8 | / rust_binary(
     9 | |     name = "hello_world",
    10 | |     srcs = ["src/main.rs"],
    11 | |     crate_root = "src/main.rs",
    12 | |     deps = [
    13 | |         ":print_hello",
    14 | |     ],
    15 | | )
       | |_^
       |

Build ID: d6a8925d-0180-4308-bcb9-fbc888e7eca1
Jobs completed: 4. Time elapsed: 0.0s.
BUILD FAILED

It’s like hey! You have two targets named hello_world! That’s confusing! It also reveals a difference between Buck and Cargo. With Cargo, if you remember our configuration, we had to point it to the crate root. From there, Cargo just leans on rustc to load up all of the other files that may be required if you have a bunch of modules. But with Buck, we need to tell it up front which files we use. So as you can see above:

    srcs = glob(["src/**/*.rs"]),

We can use the glob command to glob up all of our files, which is nice, but it’s… it’s actually wrong. We want to glob everything except main.rs. If main.rs were to change, this would try and re-build both the binary and the library, in my understanding. So that’s annoying.

It’s not just annoying for Buck, though. Having both a src/main.rs and a src/lib.rs has led to so much confusion from beginners over the years. At some point, someone puts mod lib; into src/main.rs and everything goes to hell. The original intention of this layout, to make simple things simple, is a good idea, but I think that sadly, we missed the mark here. Luckily, Cargo also supports a bin directory:

> mkdir src/bin
> mv src/main.rs src/bin

We can still cargo run and cargo build and this all works just fine. This doesn’t fix our glob issue, though, because src/bin is still inside of src.

I think, in the abstract, I’d prefer a layout like src/{lib,bin}. You want things to not really be nested. So let’s do that. Both Cargo and Buck can handle it! It’s just not as nice as being purely default in Cargo, since that convention is so strong.

# if you didn't do this above
> mkdir src/bin
> mv src/main.rs src/bin

> mkdir src/lib
> mv src/lib.rs src/lib/print_hello.rs

We have to change src/bin/main.rs to use hello_world again

fn main() {
    hello_world::print_hello();
}

And we have to re-add some configuration into Cargo.toml:

[lib]
path = "src/lib/print_hello.rs"

Everything should build just fine. But what about Buck?

So, once you start getting into subdirectories, you can also start using multiple BUCK files. So we can empty out our root BUCK file (I’m leaving it existing but empty, if you want to delete it you can but you’ll recreate it in the next part anyway), and create two new ones. Here’s src/lib/BUCK:

rust_library(
    name = "hello_world",
    srcs = glob(["**/*.rs"]),
    crate_root = "print_hello.rs",
    edition = "2021",
    visibility = ["PUBLIC"],
)

and src/bin/BUCK:

rust_binary(
    name = "hello_world",
    srcs = ["main.rs"],
    crate_root = "main.rs",
    deps = [
        ":hello_world",
    ],
)

We added in a crate_root to the library as well. Okay, let’s try this:

〉buck2 run //:hello_world
File changed: root//src/lib/BUCK
Unknown target `hello_world` from package `root//`.
Did you mean one of the 0 targets in root//:BUCK?
Build ID: d5059fc9-8001-47c4-ba5a-6ba605a4182c
Jobs completed: 2. Time elapsed: 0.0s.
BUILD FAILED

Oops! Since we moved files around, the names of our targets have changed. Let’s examine them:

〉buck2 targets //...
Build ID: c4165964-cb87-49b4-8afe-4a3fc2c526bc
Jobs completed: 4. Time elapsed: 0.0s.
root//src/bin:hello_world
root//src/lib:hello_world

We had only seen very basic target patterns, but this is enough to show off:

root/src/bin:hello_world

can be read as

The “hello_world” target defined in /src/bin/BUCK.

Our target names changing also means we made one mistake in our new BUCK files. Let’s give it a try:

〉buck2 run //src/bin:hello_world
Error running analysis for `root//src/bin:hello_world (prelude//platforms:default#fb50fd37ce946800)`

Caused by:
    0: Error looking up configured node root//src/bin:hello_world (prelude//platforms:default#fb50fd37ce946800)
    1: Cyclic computation detected when computing key `root//src/bin:hello_world (prelude//platforms:default#fb50fd37ce946800)`, which forms a cycle in computation chain: `root//src/bin:hello_world (prelude//platforms:default#fb50fd37ce946800),root//src/bin:hello_world (prelude//platforms:default#fb50fd37ce946800)`
Build ID: 930ab541-c2dd-44f5-aef1-f6658a2b7c53
Jobs completed: 2. Time elapsed: 0.0s.
BUILD FAILED

Right. Our binary depends on :hello_world, which it is itself named hello_world, so there’s a problem. But that’s just it, we don’t want to depend on any old hello_world, we want to depend on our libary. Can you write out the target pattern that should go in src/bin/BUCK?

It looks like this:

    deps = [
        "//src/lib:hello_world",
    ],

“The hello_world target in /src/lib/BUCK. Perfect. And now it works!

〉buck2 run //src/bin:hello_world
File changed: root//src/bin/BUCK
Build ID: c6d2fdaa-298a-425a-9091-d3f6b38c4336
Jobs completed: 12. Time elapsed: 0.5s. Cache hits: 0%. Commands: 1 (cached: 0, remote: 0, local: 1)
Hello, world!

It kinda stinks to have to type all of that out. Luckily, Buck supports aliases for target patterns. Take our top-level BUCK file, and add this:

alias(
    name = "build",
    actual = "//src/bin:hello_world",
    visibility = ["PUBLIC"],
)

And now we can use it:

〉buck2 build
Build ID: a87ed1e2-cfab-47b0-830e-407217997fd7
Jobs completed: 2. Time elapsed: 0.0s.
BUILD SUCCEEDED

Fun.

Okay! This is getting a bit long, so let’s end there. We have more to learn before buck can actually replace Cargo in our normal development workflow, but I hope this helped you see how you could get started with Buck if you wanted to.

If you want to check out this on your own, I’ve published this on GitHub: https://github.com/steveklabnik/buck-rust-hello/tree/024ef54ba45627e87a65aaf2f69c6661198c336c

Next up, we’ll be tackling other features, like “using crates from crates.io.” No promises on when that’ll get published though!

Rust's Golden Rule

Mon, 27 Mar 2023 00:00:00 GMT

I find myself thinking about a particular design principle of Rust today. I’m not sure I’ve ever seen it named specifically before, but it gets referred to from time to time, and I think it’s an under-rated but very important aspect of why Rust works so well. I was going to refer to it as “the signature is the contract” today, but then I ended up changing it. Regardless of that, if someone else has already written this out somewhere, and used a different name, please let me know!

Magic: the Gathering is a really interesting project. I say “project” rather than “card game” because while it is a card game, it also pioneered a whole bunch of incedental other things that had big effects on related hobbies.

I learned MtG in the late 90s. The rules were a bit different then, but many of them are the same. The very first rule I was taught was sort of the “Magic Golden Rule,” though in today’s Comprehensive Rulebook, there are four of them. This one is still the first, though:

The Magic Golden Rules

101.1. Whenever a card’s text directly contradicts these rules, the card takes precedence. The card overrides only the rule that applies to that specific situation. The only exception is that a player can concede the game at any time (see rule 104.3a).

This rule is the most important rule because it kind of creates the spaces of possibilities for cards: many cards exist to tweak, modify, or break some sort of fundamental rule.

That being said, all these years later, this idea is so pervasive in games like this that it’s barely even considered an actual rule. It’s just part of the physics of the genre, it’s how these games work. Yet it’s critical to the entire enterprise.

Rust also has a rule. It’s kinda funny, because in some senses, this rule is almost the opposite of Magic’s, if you can even stretch the comparison this far. Here it is:

Whenever the body of a function contradicts the function’s signature, the signature takes precedence; the signature is right and the body is wrong.

This rule is also so pervasive in Rust that we take it for granted, but it is really, truly important. I think it is also important for Rust users to internalize the implications of this rule, so that they know why certain things work the way that they do.

Here is the most famous implication of this rule: Rust does not infer function signatures. If it did, changing the body of the function would change its signature. While this is convenient in the small, it has massive ramifications.

Consider this example program:

fn foo(x: i32) -> i32 {
    dbg!(x);
    
    x
}

This function prints out x, and then returns it. Nothing fancy going on here, this is just random stuff to make an example. This compiles just fine. But let’s imagine that we have a version of Rust that infers our signatures. So we could type this instead:

fn foo(x) {

This is what a Ruby-ish Rust might look like; we declare the name of our argument but not its type, and we don’t declare a return type. Now, let’s do a small refactoring, we’re gonna comment out the final value there:

fn foo(x) {
    dbg!(x);
    
    //x
}

the final expression has changed; it’s no longer x, but instead is (), which is what dbg!(x); evaluates to. Because of type inference, the inferrred type of foo is now fn(i32) -> (). Our function typechecks! It’s all good, right?

Well, no:

error[E0369]: cannot add `{integer}` to `()`
 --> src/lib.rs:5:11
  |
5 |         y + 1
  |         - ^ - {integer}
  |         |
  |         ()

Wait, what? We don’t have a y + 1 anywhere in our code?! Where’s that error coming from… src/lib.rs:5. What’s at the top of lib.rs?

mod bar {
    fn baz() -> i32 {
        let y = crate::foo(5);
        
        y + 1
    }
}

Oh. Some other code was using foo. When its signature changed, we broke this invocation. It’s nice that alt-Rust caught this for us, but this error is (would be, anyway!) really bad: no longer is it telling us that our code is wrong, but instead points to some other code somewhere else we weren’t even messing with. Sure, we can go “hey why is y ()?” and then figure it out, but compare that to today’s error:

error[E0308]: mismatched types
  --> src/lib.rs:9:19
   |
9  | fn foo(x: i32) -> i32 {
   |    ---            ^^^ expected `i32`, found `()`
   |    |
   |    implicitly returns `()` as its body has no tail or `return` expression
10 |     dbg!(x);
   |            - help: remove this semicolon to return this value

This error is far better: it points out that the body of our function contradicts the signature. It points out what in our body is generating the value that contradicts the signature. And it doesn’t complain about callers.

So sure, this gets us nicer error messages, but is that really a big deal? I think it is, but it’s not the only implication here. I’d like to talk about two more: one that’s clearly an advantage, and one that has led to some pain that people would like to resolve. Balance :)

First, the one that’s an advantage. The advantage is modularity. That makes some forms of analysis much more reasonable, or sometimes even possible, as opposed to super difficult. Because everything you need for memory safety is described in the signature of the function, Rust doesn’t need to examine your entire program to determine if there’s some shenanigans going on elsewhere. This is far, far less work than just checking the signatures of the functions you call. Each function can be checked in isolation, and then assembled together. This is a very nice property.

Second, where this leads to some pain. Users have nicknamed this one “borrow splitting,” or “partial borrowing.” It looks like this:

struct Point {
    x: i32,
    y: i32, 
}

impl Point {
    pub fn x_mut(&mut self) -> &mut i32 {
        &mut self.x
    }

    pub fn y_mut(&mut self) -> &mut i32 {
        &mut self.y
    }
}

I find accessors, and especially mutators, to be where this sort of thing pops up most often. This is the classic example. The above code is fine, but if we try to do this:

// this doesn't work
impl Point {
    pub fn calculate(&mut self) -> i32 {
        let x = self.x_mut();
        let y = self.y_mut();
        
        // yes I picked multiplication because this looks kinda funny
        *x * *y
    }
}

// we would call it like this:
let answer = p.calculate();

We get this:

error[E0499]: cannot borrow `*self` as mutable more than once at a time
  --> src/lib.rs:19:17
   |
18 |         let x = self.x_mut();
   |                 ------------ first mutable borrow occurs here
19 |         let y = self.y_mut();
   |                 ^^^^^^^^^^^^ second mutable borrow occurs here
20 |         
21 |         *x * *y
   |         -- first borrow later used here

However, if we didn’t have these accessors, x and y were instead just public, this very similar free function:

fn calculate(x: &mut i32, y: &mut i32) -> i32 {
    *x * *y
}

// called like this:
let answer = calculate(&mut point.x, &mut point.y);

works just fine. Why? Because of the signatures. This signature:

pub fn calculate(&mut self) -> i32 {

and these signatures:

pub fn x_mut(&mut self) -> &mut i32 {
pub fn y_mut(&mut self) -> &mut i32 {

says “hey, I am going to borrow all of self mutably,” which implies an exclusive reference to self. That the body of calculate borrows two different parts of self independently is 100% irrelevant, that’s what the signature says! And so rustc looks at this and says “hey wait a minute, calling x_mut borrows self mutably, and calling y_mut borrows self mutably. That’s aliasing! Bad programmer!

Whereas in the second example, this signature:

fn calculate(x: &mut i32, y: &mut i32) -> i32 {

says “hey, I have two mutable references, to two different integers.” And at the call site, Rust sees that we’re creating two different borrows to two different integers, even though they’re both part of our point, and so it okays things.

This is kind of a pain in the butt! But what it saves us from is that scary action at a distance in our typecheck example. Imagine that Rust somehow inferred that the first version was okay, due to the body only being disjoint. What happens in the future, when we refactor our function, and the borrows need to change? That would have the same problem as before; we’d get weird errors elsewhere in our code. There have been some proposals over the years to possibly fix this pain point, but it’s a tough one. Putting “here’s the fields I’m borrowing in the body” into the signature, in order to conform with the Golden Rule, looks very odd. Maybe something will happen here, maybe not. I have complicated feelings.

So beyond error messages and making accessors/mutators awkward, how does this affect you day to day? Well, one way is that I find I end up paying more attention to signatures and less attention to bodies, when I’m trying to sort something out. In many cases, what you’re doing in the body is irrelevant, I care only about what your signature affords me. This isn’t quite as true as in some pure functional languages, but it has that vague feel to it. Another way, and a slightly bigger one, has to do with the relationships between types and TDD, but I think I’m going to save that for its own post.

Too many words about Rust's function syntax

Tue, 03 Jan 2023 00:00:00 GMT

There’s a famous quote by Eleanor Roosevelt:

great minds discuss semantics, average minds discuss syntax and small minds discuss syntax of comments

(Apologies, Phil Wadler.)

I’d like to preface this post that I do not consider this to be an actual proposal to change Rust, even a pre-RFC. I don’t think that making changes like this has enough benefits once a language is as established as Rust is. Consider this post more of some musings about some things that there wasn’t really time to truly consider when Rust was younger, so that maybe someone who’s making a new language can learn from those things early enough in its life to do so.

Also, I am not the first person to think about this, but I have lost the previous post to time. There also may be a newfangled language that has this or similar syntax already I’m not aware of. If you know of similar previous proposals or languages, please let me know so that I can link to them here.

Anyway, consider this post a “Steve decided to write up a thing instead of just tweeting, it is not particularly well thought out but at least it’s down on paper” kind of thing. A serious proposal here, even if I thought the idea was good in general, which to repeat again, I do not, would require examining a lot more corner cases than I do in this post.

With all of that out of the way, let’s begin.

The fn syntax

Rust does a really good job of being orthogonal in many ways. While it isn’t a tiny language, that things fit together nicely is one of the things that can make it feel a lot smaller, in my opinion. There’s one thing, though, that is really a bit inconsistent today, and that’s the syntax and semantics of functions.

Let me explain: you declare a function item (read: define a function) like this:

fn foo(x: i32) -> i32 {
    todo!()
}

Simple enough. However, there’s a lot more going on for more complex declarations. Here is the beginning of the current grammar of functions:

Function :
   FunctionQualifiers fn IDENTIFIER GenericParams?
      ( FunctionParameters? )
      FunctionReturnType? WhereClause?
      ( BlockExpression | ; )

I read this like this: “A function consists of some function qualifiers, fn, an identifier, and an optional list of generic parameters. We then have some optional function parameters inside of parenthesis, an optional return type, an optional where clause, and then a block expression or a semicolon.”

This is in sharp contrast to a C-style function definition, which would look like this:

int foo(int x) {
    // todo
}

In general, most new languages have tended to settle closer to Rust’s syntax here than C’s. There are a few reasons, and it’s not like Rust invented this style in the first place. One of the core nice things about Rust style syntax is that fn IDENTIFIER, that is, if you want to find out where foo() is defined, grep for fn foo and you can find it pretty quickly. You can’t do this as well with the int syntax. Some code styles have you write

int
foo(int x) {
    // todo
}

Notably, the GNU project uses this style:

It is also important for function definitions to start the name of the function in column one. This helps people to search for function definitions, and may also help certain tools recognize them.

You instead grep for ^foo, and you can get a similar benefit.

So, what’s bad about this syntax? Well, there are useful ‘modifiers’ to functions, and Rust has added some and may add more in the future. For example, if we want foo to be callable in a const context, we’d make it a const fn:

const fn foo(x: i32) -> i32 {
    todo!()
}

That’s all well and good, we still even retain the “grep for fn” idea here, but these are also useful on arbitrary blocks of code:

let x = const {
    1 + 2
};

This is of course a trivial example that’s not very useful, but it shows off the syntax here. My point is… this starts to feel a bit… nonuniform. In the function delcaration, we put const at the front, but when it’s any other block expression, we put it right before the block. Rust’s function bodies are similar to blocks in other ways, such as the whole “evaluates to the final expression” thing. But they’re different when they’re being used to describe the body of a function. That’s a bit less than ideal.

We could try the naive thing:

fn foo(x: i32) -> i32 const {
    todo!()
}

This is… weird, but it does sort of unify some syntaxes. We now can turn any block of code, including a function body, const by adding the const right before the opening brace. However… I kinda shudder at i32 const. There’s not enough separation between the return type and the modifier to make quick sense of things, in my opinion. However, there’s another small tweak we could make here. But first, a small divergence into another corner of Rust’s syntax: variables.

Variables

Rust also has a different syntax for variable declaration. Like C, it’s similar to its function declaration syntax:

// Rust
let x: i32 = 5;

// C
int x = 5;

We have let instead of fn, we have the name: type instead of type name. We can also declare a variable but not initialize it:

let x: i32;

Rust will make us initialize it before we use it, but we can declare the variable on its own, and then give it an initial value later.

But what’s this has to do with functions?

A marginally better syntax

Functions also happen to have a “declare the signature but not the body” syntax in Rust too, though it’s almost exclusively used in an FFI context. Remember the very end of our grammar?

and then a block expression or a semicolon.

If we have a function foo that we’re calling through FFI, we can define it like this:

#[link(name = "something")]
extern {
    fn foo(x: i32) -> i32;
}

We don’t provide a body, but we do provide a semicolon. This is strikingly similar to the variable syntax. So why not have the regular function definition also follow the variable syntax?

fn foo(x: i32) -> i32 = {
    todo!()
}

We’ve now added one little extra bit to the grammar: a = after the return type, if any. This one little change allows us to unify the rest of the syntax around blocks more easily:

// full signature
fn foo(x: i32) -> i32 = {
    todo!()
};

// empty return type
fn foo(x: i32) = {
    todo!()
};

// const
fn foo(x: i32) -> i32 = const {
    todo!()
};

I happen to really like this. It’s a pretty small tweak but I think it cleans stuff up nicely.

let name = value;
fn name() = value;

Where both could be values or blocks, with let it’s most often a value but with fn it’s most often a block. The symmetry pleases me.

(Sharp readers will also notice the introduction of a semicolon. Leaving it off would be closer to the old syntax, requring it would be similar to the variable syntax, but then that would diverge from the other item declaration syntax… I don’t feel strongly about it but just in case anyone noticed I figured I’d mention that as well, yet another rabbit hole to chase down.)

Going too far

This of course raises another divergence. But I’m not sure that fixing this one is worth it; it starts to get too weird, I think. But maybe I’m wrong. This one is about the types.

One of the things that makes name: type syntax nicer than type name syntax is that it degrades nicely in the presence of type inference:

// Rust
let x: i32 = 5;
let x = 5;

// C23 (https://www.open-std.org/jtc1/sc22/wg14/www/docs/n3007.htm)
int x = 5;
auto x = 5;

Rust’s function syntax isn’t like this though, partially because there’s no type inference on function signatures, so there’s not a lot of pressure for it to degrade nicely. The choice not to offer inference is the right one for Rust, but for a future language where that’s not the case, we could unify them with something like this:

// explicit type
fn foo(x): fn(i32) -> i32 = {
    todo!()
}

// inferred
fn foo(x) = {
    todo!()
}

This one… this may be too far afield. And, while Rust doesn’t have named parameters, and possibly never will, I have no idea how this style of syntax would work with that. There’s also some repetition of fn here that might be worth trying to remove, but I’m trying to make the smallest possible deltas from existing syntax here, and fn(i32) -> i32 is the type of that function in today’s syntax.

What I mostly take away from this part of the exercise is that consistency is a good goal, but it’s not the only goal. Even if Rust didn’t allow for inference, making its function declaration syntax be like this may have simply been a bridge too far for wide adoption. Rust already takes a lot of flak for being weird, and spending some Strangeness Budget to unify these syntaxes probably wouldn’t be the right choice. But maybe I’m too conservative! Given that Rust already takes that flak, maybe a bigger jump would have been okay here. Who knows.

… actually, we have one way to find out, I guess. This is very similar to the syntax that Herb Sutter came up with for cppfront:

foo: (x: i32) -> i32 = {
    // todo
}

There’s no fn preceding. The type syntax works a bit differently. But it’s close. And Herb is targetting the people who I’d be most worried about hating this syntax: C++ folks. Maybe this idea isn’t so farfetched at all. He’s also got an example of how making it any expression after the = can be nice for one-liner functions. If we applied this to our fake Rust syntax:

fn main() = println!("Hello, world!");
fn hello(name): fn(&str) = println!("Hello, {name}!");

That’s kinda nice too.

Are out parameters idiomatic in Rust?

Fri, 16 Oct 2020 00:00:00 GMT

There is a pretty straightforward design pattern called “out parameters” that you may have heard of before. In short, the question is, does your function wear pants… err, I mean, does your function modify data like this:

fn foo() -> i32 {
    // body elided
}

let x = foo();

or like this:

fn foo(out: &mut i32) {
    // body elided
}

let mut x = 0;

foo(&mut x);

That is, does your function return a value, or modify a parameter?

There are a number of different reasons you may or may not use one or the other of these techniques in different languages, and they often depend on the semantics of the language itself.

In Rust, you almost always want to return a value from a function, that is, prefer option one over option two. There are a few reasons for this.

The first is that it’s generally semantically more accurate. Your function is producing a value of some kind, so it should, well, produce a value. Additionally, with Rust’s relatively rich data type support, it’s easy to do things like return a tuple if you need to return more than one thing.

The second is that, in some languages and with some compilers, the out parameter is a performance optimization that you as a programmer do by hand. Imagine if I wasn’t returning an i32, which is just a few bytes. The most straightforward way of implementing the function in assembly is to copy the return value from the function to its parent’s stack. This also may be required by your ABI. Now, I am not an expert here, so I am not 100% sure of the exact current state of things, but at least in theory, optimizations can be done by the compiler. This is a pretty well-known optimization, see Wikipedia for more. But, I think that we do already do this today in some cases, if I compile both of these functions:

fn returns_huge(out: &mut [u64; 1024]) {
    *out = [0; 1024];
}

fn returns_huge() -> [u64; 1024] {
    [0; 1024]
}

I get identical assembly:

playground::returns_huge: # @playground::returns_huge
# %bb.0:
    movl    $8192, %edx                     # imm = 0x2000
    xorl    %esi, %esi
    jmpq    *memset@GOTPCREL(%rip)          # TAILCALL
                                        # -- End function

However, it is possible that this breaks for other examples, and so you may, may, may want to move to an out parameter if you see extraneous copies appear in your performance profiling.

C++ guarantees this optimization in certain circumstances; it would be really nice for Rust to do so as well. My understanding is that’s the intention.

When you should use out parameters

There is one case where it is idiomatic to use an out parameter in Rust though. That’s when you are writing into some kind of buffer, and you want to be able to allow your caller to provide the buffer to you, rather than making a new one. This would allow the caller to re-use the same buffer for multiple operations.

(I say “buffer” here but really, this generalizes to any sort of grow-able container, to be clear. I am using “buffer” in a loose sense here.)

You can see this play out in the Rust standard library. std::io::Read looks like this:

pub trait Read {
    fn read(&mut self, buf: &mut [u8]) -> Result<usize>;
}

This accepts a &mut [u8] rather than returning a new buffer. This allows for significant savings by re-using a buffer. For example, in Chapter 2 of the book, we have this example:

use rand::Rng;
use std::cmp::Ordering;
use std::io;

fn main() {
    println!("Guess the number!");

    let secret_number = rand::thread_rng().gen_range(1, 101);

    loop {
        println!("Please input your guess.");

        let mut guess = String::new();

        io::stdin()
            .read_line(&mut guess)
            .expect("Failed to read line");

        let guess: u32 = match guess.trim().parse() {
            Ok(num) => num,
            Err(_) => continue,
        };

        println!("You guessed: {}", guess);

        match guess.cmp(&secret_number) {
            Ordering::Less => println!("Too small!"),
            Ordering::Greater => println!("Too big!"),
            Ordering::Equal => {
                println!("You win!");
                break;
            }
        }
    }
}

We aren’t quite properly re-using the buffer here, but we could modify this to look like this:

use rand::Rng;
use std::cmp::Ordering;
use std::io;

fn main() {
    println!("Guess the number!");

    let secret_number = rand::thread_rng().gen_range(1, 101);

    let mut guess = String::new();

    loop {
        println!("Please input your guess.");

        guess.clear();

        io::stdin()
            .read_line(&mut guess)
            .expect("Failed to read line");

        let guess: u32 = match guess.trim().parse() {
            Ok(num) => num,
            Err(_) => continue,
        };

        println!("You guessed: {}", guess);

        match guess.cmp(&secret_number) {
            Ordering::Less => println!("Too small!"),
            Ordering::Greater => println!("Too big!"),
            Ordering::Equal => {
                println!("You win!");
                break;
            }
        }
    }
}

Here, we create our string outside of the loop, and then on each iteration of the loop, clear it. Clearing it does not deallocate the backing memory; it only changes the length to zero. When the next line is read in, if it’s smaller than the existing size of the string, it won’t allocate any new memory. This would not be possible if stdin() returned a new String every time, so even though the out parameter is a bit less nice to use, it can be much faster.

It took 26 releases of Rust for std::fs::read and std::fs::read_to_string to be added for this reason; we eventually did because in many cases, you don’t care about re-using the buffer, and these functions are nicer to use, but they weren’t included originally because they’re pretty simple, and are built on top of the re-usable solutions:

pub fn read_to_string<P: AsRef<Path>>(path: P) -> io::Result<String> {
    fn inner(path: &Path) -> io::Result<String> {
        let mut file = File::open(path)?;
        let mut string = String::with_capacity(initial_buffer_size(&file));
        file.read_to_string(&mut string)?;
        Ok(string)
    }
    inner(path.as_ref())
}

That said, I am exceedingly glad we did add them. It is a much nicer user experience.

(You may be wondering “what’s up with that inner function there?”… that maybe deserves another post! It’s not neccesary, strictly speaking, but is another case of “the compiler does not do an optimization yet so write some code in a strange way by hand.”)

The CXX Debate

Sat, 22 Aug 2020 00:00:00 GMT

Let’s talk about safe and unsafe in Rust.

To review, Rust has a concept of “unsafe code.” Unsafe Rust is a superset of safe Rust, letting you do some extra things that are dangerous. It’s your job as a programmer to verify that your code is playing by the rules, because the compiler cannot check them for you.

Imagine we have a C program, add.c that looks like this:

#include <stdint.h>

uint32_t add_one(uint32_t x) {
    return x + 1;
}

We can call this function from Rust like this:

extern {
    fn add_one(x: u32) -> u32;
}

fn main() {
    let x = 5;

    let y = unsafe { add_one(x) };

    println!("{}", y);
}

We need to invoke unsafe here because any extern function doesn’t come from Rust; it comes from another language, and it’s possible that that function does something that would cause undefined behavior. We look at add_one, see that it’s not violating any rules, and we’re good.

However, we know that we can pass any u32 to this function, and it’s safe. We don’t need to example the value of the argument every single time, they’re all fine. So we do this:

mod internal {
    extern {
        pub fn add_one(x: u32) -> u32;
    }
}

fn add_one(x: u32) -> u32 {
    unsafe {
        internal::add_one(x)
    }
}

fn main() {
    let x = 5;

    let y = add_one(x);

    println!("{}", y);
}

This is a classic way that unsafe code gets used in Rust; we have our binding, internal::add_one, and our wrapper, add_one. The wrapper is safe, but uses unsafe internally. This is totally fine! The idea is that this gives us a sort of “seam” in our code. Imagine that add_one would do bad things if we passed it a number less than five. We could use our wrapper to check this:

// we would use a real error type in real code but we're using () here because we don't
// actually care about the error handling here.
fn add_one(x: u32) -> Result<u32, ()> {
    if x < 5 {
        return Err(());
    }

    unsafe {
        Ok(internal::add_one(x))
    }
}

But, add_one doesn’t need to do this. Every u32 we pass is totally fine.

Nothing so far is controversial here, this is just background and setup. The purpose of these examples is for you to read and ask yourself, “Is this okay? How do I feel about this?”

Somewhere after this point, opinions start to diverge.

Our C library is growing new functionality! Here it is:

#include <stdint.h>

uint32_t add_one(uint32_t x) {
    return x + 1;
}

uint32_t add_two(uint32_t x) {
    return x + 2;
}

We can now add two, not only one. Amazing, I know. Let’s write another wrapper:

mod internal {
    extern {
        pub fn add_one(x: u32) -> u32;
        pub fn add_two(x: u32) -> u32;
    }
}

fn add_one(x: u32) -> u32 {
    unsafe {
        internal::add_one(x)
    }
}

fn add_two(x: u32) -> u32 {
    unsafe {
        internal::add_two(x)
    }
}

fn main() {
    let x = 5;

    let y = add_one(x);
    let z = add_two(x);

    println!("{}, {}", y, z);
}

Two functions, both have an unsafe binding, and a safe wrapper.

Is this okay? How do you feel about this?

Now, imagine we had 100 of these functions. Writing this all out ends up being a massive amount of error-prone boilerplate. Heck, I almost forgot to switch to internal::add_two after I copy/pasted the new function there. Rust has a tool to abstract over code like this: macros! So let’s refactor our code with a macro:

macro_rules! call_c {
    // implementation elided
}

call_c! {
    fn add_one(x: u32) -> u32;
    fn add_two(x: u32) -> u32;
}

fn main() {
    let x = 5;

    let y = add_one(x);
    let z = add_two(x);

    println!("{}, {}", y, z);
}

I am not showing you the macro because the exact contents are gnarly and immaterial to the point. I also think that maybe this needs to be a procedural macro not a macro_rules macro but again, those details don’t really matter. The point is, we have a macro, it generates the exact same structure that we had before.

Is this okay? How do you feel about this?

This wrapper is also known as CXX. The syntax is slightly different, but conceptually, this is what it is doing. The reason it is controversial is that some people do not in fact feel that the above macro is okay. The reason that they feel it is not okay is that you do not write unsafe by hand, yet, if you’re not careful, things can go wrong. I think this is a totally understandable and reasonable reaction, but I also don’t think that it makes a ton of sense once you actually dig into the details.

We actually don’t need C to think about this problem, we can do this with unsafe Rust purely. And I think that using unsafe Rust makes it a bit more clear, so let’s talk about this in some more depth, but purely with Rust.

The unsafe keyword has two, complimentary uses. It can be used to require your caller to check invariants, and it can be used to tell the compiler “I have checked the invariants.” In other words:

// I am leaving off parameters and return types because it doesn't matter, but real code
// would of course have them.

// When calling foo, please check some invariants.
unsafe fn foo() {
    // ...
}

// I have checked the invaraints, and this is okay to do.
unsafe {
    foo();
}

The compiler forces these combinations of things; using a feature that generates a requirement to check something will require you to also declare that you’ve checked it. These requirements bubble up, in other words, we can also pass that obligation to our callers:

unsafe fn foo() {
   // ...
}

fn bar1() {
    // I have checked the invariants, and this is okay to do
    unsafe {
        foo();
    }
}

// I have not checked the invariants, so since I'm calling foo, you must check
// the invariants
unsafe fn bar2() {
    foo();
}

Rust’s built-in FFI works like bar2; if the contents of foo are written in another language, it has to be unsafe. This means that, even if it is perfectly safe to call foo with any possible arguments, or in any circumstances, we still need to write a wrapper.

CXX’s take on FFI is that it works like bar1; by using the library we are saying “hey, I know that foo is safe in all circumstances, so please generate the wrapper for me.” In other words, in some sense, it pushes the unsafe block across the other side of the boundary, because that’s where it actually belongs in this case.

This has generated a lot of feelings on both sides. The people who don’t like it are upset that the letters ['u', 'n', 's', 'a', 'f', 'e'] are not present, and feel that this makes it quite easy to make a mistake and accidentally use the macro on some code that is not actually safe to call in all circumstances. The people who do like it argue that when you have a large number of calls that are actually safe, using a macro to abstract them away is fine, like you’d abstract away any other boilerplate. Sometimes, repeating the same thing over and over again makes mistakes more likely, not less.

In the meantime, dtolnay has decided to change the macro and re-add the unsafe token, which helps assuage the fears of the people who are upset. But it also centralizes these sorts of uses of unsafe, which helps achieve the goals of the library in the first place. I am continually impressed by dtolnay’s ability to design libraries, and hope he keeps writing Rust code forever.

Anyway, if you were wondering what this all was about, that’s my take on it. I personally feel that CXX is fine the way that it is, but I’m also happy with this compromise, and hope that others are too.

How often does Rust change?

Sat, 11 Apr 2020 00:00:00 GMT

I’ve been thinking a lot lately about how often Rust changes. There are some people that assert that Rust stays fairly static these days, and there are some people who say Rust is still changing far too much. In this blog post, I want to make a data driven analysis of this question. First, I will present my hypothesis. Next, my methodology. Then, we’ll talk about the results. Afterward, we’ll discuss possible methodological issues, and possible directions for further research.

As said on twitter:

The hypothesis

My personal take on Rust changing: to me, it feels like we had more change in the past than we have today. The changes are also getting more minor. This will be my hypothesis.

The methodology

There are tons of different ways that we could measure the question “how often does Rust change?” So before I started collecting data, I had to decide some details.

I realized that my premise here is about how people feel about Rust. So, how do they tend to understand how Rust changes? Well, the primary way that we communicate change to the Rust community is through release blog posts. So I decided to start there.

What I did was, I opened up every release blog post for Rust, from 1.0.0 to 1.42.0, in a new tab. For fun, I also opened the release notes for 1.43, which will come out soon.

I then looked at every single change announced, and categorized them as follows:

adds syntax?
language
standard library
toolchain
major
medium
minor

I initially included “deprecation” and “soundness” as categories too, but I ended up not including them. More on that later.

From here, I had to decide what these categories meant. What standard would I be using here? Here’s the easier ones:

language changes mean some sort of modification to the language definition. These are additive changes, thanks to our stability policy, but they’re new additions.

standard library means a change to the standard library, primarily, new functions, new types, new methods, stuff like that. This one is pretty simple as a criteria but there’s some small interesting methodological issues we’ll get into later.

toolchain changes related to cargo, rustup, new compiler target support, stuff like that. Not part of the language proper, but part of the Rust distribution and things Rust programmers would use.

Now, for the harder ones:

major/medium/minor changes are how big of a change I think that the change is. There’s a few interesting parts to this. First of all, this is, of course, very subjective. I also decided to try and evaluate these as of today, that is, there are some changes we maybe thought were major that aren’t used all that often in practice today, so I categorized those as medium rather than major. This felt more consistent to me than trying to remember how I felt at the time.

adds syntax changes sound easy, but are actually really tricky. For example, consider a feature like “dotdot in patterns”. This does change the grammar, for example, so you could argue that it adds syntax. But as a programmer, I don’t really care about the grammar. The summary of this feature is “Permit the .. pattern fragment in more contexts.” It goes on to say:

This RFC is intended to “complete” the feature and make it work in all possible list contexts, making the language a bit more convenient and consistent.

I think these kinds of changes are actually very core to this idea, and so I decided to categorize them according to my opinion: this is not a new syntax change. You already had to know about ...

I believe this may be the most controversial part of my analysis. More on this later, of course.

Okay, so that’s the stuff I covered. But there’s also stuff I didn’t cover. I was doing enough work and had to draw the line somewhere. I left this stuff out:

compiler speedups. This is interesting with numbers, but that means actually compiling stuff, and I don’t have time for that. This is a whole study on its own.
Documentation work. This tends to be not tracked as new features, but sometimes it appears in release notes, for bigger things. Better to just leave it out. I also don’t think that it affects my hypothesis at all.
Library types implementing new traits. These can be hard to number, especially with generics involved. I decided that it would be better to leave them off.
compiler internal news. We sometimes talk about things like “MIR exists now!” or “we ported the build system from make to cargo!” but similar to documentation, we talk about it very little. It’s also not a change that affects end users, or rather, it affects them more directly by stuff that is counted. MIR enabled NLL, and NLL was tracked as a language feature.

The results & analysis

I am not as good with Google Sheets as I thought, so I asked for help. Big thanks to Manish for helping me get this together. I should note that he set me up, but then I tweaked a ton of stuff, so any mistakes are mine, not his. For example, I got a little lazy and didn’t realize that the colors aren’t the same across the various charts. That’s my bad.

Here’s some quick totals of how many changes and of each kind I found, with some sparklines. You can click on any of the images in this post to enlarge them:

And here’s some charts, per release:

First of all, it is fairly clear that, at least numerically speaking, the standard library is the part of Rust that changes most often. It’s a clear outlier by volume. I find this result kind of hilarious, because Rust is well-known for having a small standard library. We’ll talk more about why I think this is in the “Issues and further research” section below.

Let’s look at the chart without the standard library:

You can see there’s a ton of toolchain changes early on. We had a lot of work to do early on, and so made a ton of changes! It’s calmed down in recent times, but toolchain changes happen almost twice as often as language changes. I should also note that this reduction in toolchain changes may be due to methodology; this is talked about later in the post.

So, the core question here: it looks like the language is changing more, recently, rather than less. I think it’s pretty clear that this graph is not exactly what I would imagine, that is, I expected a nice gentle curve down, but that’s not what has happened. I want to dig into this a bit more, but first, let’s look at some other graphs:

Changes that add syntax are a subset of language changes. There hasn’t been a whole ton of this, overall. Rust 2018 was the large bump. Half of our releases do not add syntax, though 10 out of 43 did introduce language changes. Our first 29 releases, skipping 1.26, had around one or two changes on average, but every since, it’s been between three and four. I believe this has to do with my methodology, but at the same time, this pretty resoundingly refutes my hypothesis. Very interesting!

Here’s major/medium/minor:

There’s a peak around Rust 2018, and one from 1.12 to 1.19, but otherwise, it’s been pretty steady lately, in terms of overall changes. If we look at just major changes:

Rust 2018 had a huge bump. After 1.0, and after Rust 2018, it calmed. I think this chart is very interesting, and demonstrates the 3 year edition cycle well. We ship an edition, stuff calms down, things slowly build back up, we ship an edition, they calm down again.

So, why was my hypothesis wrong? I think this conclusion is fascinating. I think part of it has to do with my methodology, and that maybe, that difference explains some of the ways people feel about Rust. You see, even if we release extremely often, and a lot of releases don’t have a ton of changes in them (as you can see, it’s vaguely around 8 or 10, when you exclude the standard library, on average), the very fact that we do release often and have a decent, but fairly small amount of change means that things get into the release notes that may not if we had the exact same amount of change, but did yearly releases.

So for example, in 2019, we released Rust 1.32 to 1.40. That involved 35 language changes. Would I have included, say, “you can now use #[repr(align(N))] on enums” if I had been writing a post about all of these changes for the whole year? Probably not. But because there were only eight total changes in that release, with half of those being language changes, it made sense to include in the 1.37 release post.

Does this mean that folks who read the release posts and think Rust changes a lot are wrong? No, it does not. For those of us in the thick of it, a lot of the smaller changes kind of blur into the background. I actually initially typed “#[repr(N)] on enums” when writing the above sentence, because this change isn’t relevant to me or my work, and is pretty small, and makes things more orthogonal, and so it is just kind of vague background noise to me. But to someone who isn’t as in-tune with the language, it’s harder to know what’s big and what’s small. The endless release posts with tons of stuff in them makes it feel like a lot is happening. But the same amount of stuff may be happening in other languages too, you just don’t see them in release posts, because they only put out one once a year. You both see them less often, as well as see less in them.

I don’t think this means that the Rust project should change its release schedule, and I’m not entirely sure if this means we should change how we write release posts. However, maybe there’s a better way to surface “these are big changes” vs “this is a small thing,” or something like that. I’m not sure. But I do think this helps me understand the disconnect a lot better.

Issues and further research

There are some big issues with this methodology. No analysis is perfect, but I want to be fair, so I’m calling out all of the stuff I can think of, at least.

The first of which is, this is inherently subjective. There’s different degrees of subjectivity; for example, “total changes in the notes” is fairly objective. But not completely; folks have to decide what goes into the release notes themselves. So there’s already a layer of filtering going on, before I even started looking at the data. Doing this analysis took me two days of real time, but only a few hours of actual work. Something that analyses the git repo in an automated fashion may see other results. I argue that this level of subjectivity is okay, because we’re also testing something subjective. And I think I designed my methodology in a way that captures this appropriately. On some level, it doesn’t matter if the “real” answer is that Rust rarely changes: folks still get this info by reading the posts, and so I don’t think saying that “oh, yeah you feel that way, but this graph says your feelings are wrong” is really going to help this debate.

The second problem expands on this. We sort of had three eras of release note writing: the first few posts were written by Aaron and Niko. As of Rust 1.4, I took over, and wrote almost every post until 1.33. I then stepped back for a while, though I did help write the 1.42 post. The release team wrote 1.34 onward, and it was more of a collaborative effort than previous posts. This means that the person doing the filter from release notes -> blog post changed over time. This can mess with the numbers. For example, there are some blog posts where I would have maybe omitted a small family of features, but the folks writing that post included them all, individually. This may explain the uptick in language changes lately, and why many of them were not rated as “major.” Additionally, the people who did the filter from “list of PRs” to “release notes” also changed over time, so there’s some degree of filtering change there, as well.

To complicate things further, starting with Rust 1.34, Cargo lost its section in the release blog posts. Big changes in Cargo still made it into the text, but the previous posts had more robust Cargo sections, and so I feel like post-1.34, Cargo was under-counted a bit. Cargo changes are toolchain changes, and so the recent calm-ness of the toolchain numbers may be due to this. This still is okay, because again, this analysis is about folks reading the release notes, but it’s worth calling out. Likewise, I think the standards for writing the library stabilizations changed a bit over time too; it’s gone from more to less comprehensive and back again a few times.

This was mentioned above a bit, but for completeness, the structure of Rust releases means there’s a variable criteria for what makes it into a release post. If it’s a smaller release, smaller changes may get featured, whereas if those same feature were in a bigger release, they may not have been. I think this aspect is a significant factor in the rejection of my hypothesis, as I mentioned above.

I think there’s interesting future research to be done here. I initially tried to track deprecations and soundness fixes, but soundness fixes happened so little it wasn’t worth talking about. There were 7 in those 42 releases. This is another weakness / area for future research, because I did not include point releases, only 1.x.0 releases. This would have added a few more soundness fixes, but it also would have added a lot of random noise where not much happened in the graphs, so that’s why I left those out. They also don’t happen on the same regular schedule, so time would get a bit funny… anyway. Deprecations was also just hard to track because there weren’t a lot of them, and sometimes it was hard to say if something was deprecated because of a soundness issue vs other reasons.

Is my critera for adding syntax correct? I think a reasonable person could say “no.” If you have something that’s special cased, and you’re making it more general, there’s a good argument that this is removing a restriction, rather than adding a feature. But reasonable folks could also say you’ve added something. I don’t think that this part is core to my analysis, and so I think it’s fine that I’ve made this decision, but you may not care for this section of results if you disagree.

This analysis doesn’t get into something that I think is a huge deal: the ecosystem vs the language itself. I only track the Rust distribution itself here, but most Rust programmers use many, many ecosystem libraries. When people talk about churn in Rust, do they really mean the ecosystem, not the language? In some sense, doing so is correct: Rust having a small standard library means that you have to use external packages a lot. If those churn a lot, is it any better than churn in the language itself?

Does Rust have a small standard library? There were 962 changes in 42 releases, that’s almost 23 changes per release. A lot of them are small, but still. Maybe Rust has a small, but deep standard library? Due to coherence, I would maybe think this is true. Just how big are standard libraries anyway? I’ve only seen discussion of this concept in terms of design philosophy. I’ve never seen a numerical analysis of such. Is it out there? Do you know of one? I’d love to hear about it!

I think there’s a number of reasons that the standard library changes dominate the total change rate. Primarily, this section of the blog post tends to be the most complete, that is, out of the total amount of changes, more standard library changes make it into blog posts than other kinds of changes. Why? Well, it’s really easy to measure: count up stuff stabilized and put it in a big list in the post. It’s also about how these changes are measured; when a method is added to every number type, that’s u8 + u16 + u32 + u64 + u128 = 5 changes, even if conceptually it’s one change, in a different sense.

This leads to another question about those standard library changes: what is up with that weird peak around Rust 1.33? Well, we added the const fn feature in Rust 1.31. After that landed, we could const-ify a number of functions in the standard library. The initial huge bulk of changes here landed in 1.33. It had 138 changes, but only 13 were not “this function is now const.” And the feature continues to drive change outside of 1.33; the initial landing of const fn in 1.31 was very limited, and as it expands, more functions that already existed can be made const.

In conclusion, even though I still believe that Rust has slowed down its rate of change a lot, I think that it makes total sense that not everyone agrees with me. By some metrics, I am just flat-out wrong.

A sad day for Rust

Fri, 17 Jan 2020 00:00:00 GMT

actix-web is dead.

This situation is bad, from all sides. When Rust was a tiny, tiny community, I thought to myself, “wow, I wonder how long this will last? Probably 1.0 will kill it.” Sort of playing off of Eternal September, I assumed that over time, the community would grow, and we’d encounter problems.

Today is the first day where I say to myself, okay, has that happened?

This story is not super clear-cut. I’m not going to link to a dozen citations, or try to prove that I’m some sort of neutral party here. I’m going to give you account of this story as I remember it and as I felt it. Because this isn’t really about playing judge. This is about thinking about the future.

It’s been very clear from the beginning that the Rust project saw Rust as more than just the language. The community and the people mattered. From the earliest days, leadership explicitly took the position that it wasn’t just the code, but the people around the project were important. Of course, people are also people, and so this wasn’t perfect; we’ve made several fairly large mis-steps here over the years. But Rust has been an experiment in community building as much as an experiment in language building. Can we reject the idea of a BDFL? Can we include as many people as possible? Can we be welcoming to folks who historically have not had great representation in open source? Can we reject contempt culture? Can we be inclusive of beginners?

Rust has a necessary feature, unsafe, that allows you to escape Rust’s guarantees. This is a really important part of Rust, but also a very dangerous one, hence the name. But one of the things that’s important about it is that, since unsafe means “I have checked this code, and it upholds all of the guarantees the compiler cannot check,” you can build a safe interface with unsafe guts.

Interacting with anything outside of the language, for example, using the operating system to print to the screen, or write a file, has to be unsafe, and so without this property, all programs would be unsafe. This premise is treated skeptically by many, but has generally been demonstrated to have worked out. Several folks are in the process of providing tooling to prove that your unsafe code is correct, and provided proofs for a bunch of the standard library’s unsafe code. It not only found a few places where there were bugs, but also a few places where the restrictions were too tight!

But “generally” is doing a lot of work in that previous sentence. This setup creates a gnawing fear in the back of people’s minds: what if some foundational package uses unsafe, but uses it incorrectly? What happens when this causes problems for every package that uses that package? This relationship between safe and unsafe is also a bit hard to understand, and so, when bugs are found around unsafe, people outside of Rust often use it to suggest that all of Rust is a house of cards. While Rust has “made it” in many senses, it has not quite yet in many others, and so I think this produces a fundamental anxiety in the community.

Speaking of making it, actix-web is a good web framework. It came onto the scene, and people generally liked it. It also performed extremely well. Its score on Techempower in particular was impressive, and got more impressive over time. Its author also works at Microsoft, and suggested that Microsoft is using actix-web in production. This was really exciting to people. This was the second project using Rust at Microsoft, and so this seemed really fantastic.

But then, the anxiety.

Before we get into that, though, there’s also the matter of Reddit. The team has basically rejected Reddit for said social reasons. We provide alternate forums for folks, but Reddit is a huge place. The Rust subreddit has ~87,000 subscribers, partially thanks to this. And, for a while, the Rust reddit was a decent place. I still think it’s better than most reddits, but it’s degraded much more quickly than the community at large. “Why Reddit is like this” is a whole other essay; I think it’s built into reddit’s structure itself. But the point is, while Reddit is not official, and so not linked to by any official resources, it’s still a very large group of people, and so to suggest it’s “not the Rust community” in some way is both true and very not true. For the purposes of this story, I think it has to be included in the broader community, and I think that this situation has brought new questions about the relationship of the project and Reddit, though this is already far too long.

So, someone takes a peek under the covers, and it turns out actix-web is using a lot of unsafe code. That’s not inherently a problem. The problem is, a lot of that unsafe code is not actually needed. This is where things start to go wrong.

The author of actix-web basically says “nah it’s fine.” This is basically the perfect storm of unsafe anxiety: a big, important, visible library, a bunch of unsafe, and an unresponsive author.

Now, it’s important to mention that people were not just complaining: they were writing code. It is easy to frame this as a case of open source entitlement, and it still very well may be, but this isn’t just empty complaints. Patches were included, they just weren’t welcome. And that’s also not inherently wrong either; a project maintainer shouldn’t be required to accept a patch just because it exists…

Reddit whips itself into a frenzy. Lots of ugly things are said. More issues are opened, and closed, and locked. Eventually, the author accepts some code that drops the total unsafe count significantly and has more sound usage of what’s left.

People were not happy about how all this played out, for a variety of reasons and in a variety of ways. I haven’t spoken to actix-web’s author, but I can’t imagine that he was very happy about it either.

And then it happens all over again. More unsafe, more patches rejected, more Reddit, more bad blood. I thought that at this point, the author said he was quitting, but that may be faulty memory, or maybe he changed his mind. Regardless, how responsive would you be, if you were the maintainer, after the shit-show of last time?

People were not happy about how all this played out, for a variety of reasons and in a variety of ways.

All was quiet for a while, and actix-web kept climbing the Techempower benchmarks, reaching the top of most of them by a lot. It is wicked fast.

And then, yesterday, a post titled Smoke-testing Rust HTTP clients gets published. It focuses on a, well, smoke test of Rust HTTP clients, but it also comments a lot on the complexity of each project, and the amount of unsafe. The author found and filed a lot of bugs.

But:

A quick glance at the dependencies reveals that it relies on actix-service, which underpins all of Actix and has a bespoke and unsound Cell implementation. For example, this method violates memory safety by handing out multiple mutable references to the same data, which can lead to e.g. a use-after-free vulnerability. I have reported the issue to the maintainers, but they have refused to investigate it.

This causes the now-usual Reddit uproar. It’s extra nasty this time. Some people go far, far, far over the line.

And now the maintainer has quit.

This means, on some level, this situation is over: there will not be a fourth huge actix-web drama. But it’s really left a bad taste in my mouth. It’s unfortunate that a well-known project had soundness issues. It’s unfortunate that the maintainer wasn’t receptive to fixing them. At the same time, I’ve been there: being a maintainer is tough. It’s also unfortunate that there’s this style of response, and kind, and volume. It is true that the situation was dangerous. But did it have to be handled this way? You have to understand a lot of nuance here to even know the main points of the story. One version of this story that will certainly be told is “The Rust community says they’re nice but they will harass you if you use unsafe wrong.” Is that what we want? I, for one, do not. If you’re a part of Rust, you gotta ask yourself: are you happy with this outcome? How did your actions (or lack thereof) contribute to it? Could you have done something better?

EPILOGUE

actix-web has new maintainers, and will now live on: https://github.com/actix/actix-web/issues/1289

Rust has finally outgrown me

Fri, 29 Mar 2019 00:00:00 GMT

Today is a very interesting day. It’s giving me a lot of feels. Today is Rust Latam, a Rust conference in Montevideo.

And I’m not there.

This is significant because, as far as we know, I’m the only person who has been to every Rust conference in existence so far. I went to RustCamp, all three RustConfs, all five RustFests so far, all three Rust Belt Rusts. One RustRush. Am I forgetting any? Thirteen Rust conferences in the past four years.

I really wanted to attend Rust Latam. The main organizer, Santiago, has been a friend for a very long time. It’s amazing that we’re getting a new conference in this part of the world. There’s been a lot of energy with Rust there for quite a while. It would have been nice to meet all those people whose tweets I’ve seen.

But in the end, I’m still between jobs (news on that soon, hopefully!), and spending that money would be irresponsible. I’m just getting settled in Austin, and it’s nice to stay in one place for a little while. I didn’t only attend Rust conferences last year, I attended many more. I missed Thanksgiving with my friends, because I was at a conference. I made five trips in October last year; Eastern Europe, then San Francisco, then Michigan, then New York (for a conf, not to stay at home), then Berlin. I did three more in November. I did about twenty conferences, and a few personal trips, last year. I am very tired.

Don’t get me wrong, I love travel, and plan to do more of it in the future. But I’ve been taking a bit of a breather; I’ve only done four trips so far this year, twice to Europe, twice in the US. But it’s been nice to take a break. For now.

More importantly, it’s time to let go of the pressure I felt to keep that streak up. Some of those trips really took a toll on me, mentally and physically. Nobody should ever fly from New York to Belarus for 30 hours, then immediately fly to San Francisco for three days, then back to New York. I’m not old, but I’m not as young as I once was, either.

It’s also time to acknowledge that I cannot do it all on my own. This conference streak is one thing, but Rust has outgrown me in other ways. I used to be on #rust every day, answering almost every question. Then it was both #rust-beginners and #rust. Not to mention all the other channels. I’m not even on IRC anymore. I was on the core team when the core team approved every RFC; now my team very rarely has RFCs. I used to read every single GitHub email on rust-lang/rust, and then rust-lang/cargo, and then all their other libraries. Now I skim rust-lang/rust, and have unsubscribed from a lot of the other repos. I used to know every single production user of Rust. Those times are long gone; I’m starting to get used to hearing “oh yeah our team has been using Rust in production for a while now, it’s great.” The first time that happened, it felt very strange. Exciting, but strange.

Back when I first decided I wanted to devote my time to Rust, I used to submit every single blog post about Rust to Hacker News and Reddit, and then answer any question anyone had about things. I diligently did this for years. I’m slowly letting go of this one, too. We’ll see.

All of these things are excellent. This is the future I wanted. This is why I spent so much time over the years doing all of this stuff. I did it, mission accomplished. It feels good. But it also feels a little weird. That Rust is bigger than me is an amazing, wonderful thing.

I wonder what the next stage of Rust’s growth will feel like.

Learning Ada

Tue, 26 Mar 2019 00:00:00 GMT

I decided to try and learn Ada. I love programming languages. This post documents my attempt, and explains what I learned. This is pretty stream of consciousness and basically is a bunch of random notes, so if you hate that, better close the tab. I typed my thoughts up pretty raw, so I may be incorrect in places too. Let me know if you know the answer to something I don’t!

The first step, of course, is to google:

Great! I know that Adacore are sort of the stewards of Ada? Vaguely, that might be wrong. Let’s look at all three of those.

Learn.adacore.com is an interactive learning platform designed to teach the Ada and SPARK programming languages.

Ah, that’s awesome! No installation needed.

The second one looks extremely like spam. It does have a page of learning materials that look legit enough. I wonder if this is a low-effort attempt to make money through ads. Or maybe it’s actually good and real and feels outdated, it does have a high google ranking. Upon doing some more research, it seems like the latter. Oops…

The third one is a parked domain, it apparently expired. Oops.

Anyway, I’m going to pursue learn.adacore.com

learn.adacore.com

Here’s the initial code presented:

with Ada.Text_IO; use Ada.Text_IO;

procedure Learn is

   subtype Alphabet is Character range 'A' .. 'Z';

begin

   Put_Line ("Learning Ada from " & Alphabet'First & " to " & Alphabet'Last);

end Learn;

Okay! This clearly has some style to it. Seems like Pascal_Snake_Case. I’m not super hyped about this, but it’s fine. Syntax discussions are like the tabloids of programming language discussion. They’re fun, but they don’t really have any educational comment. One thing I will point out: are apostrophes used for method calls? Otherwise, this seems to be fairly Pascal like, which I think makes sense given what I know about Ada.

I can click run and see it execute right there! Awesome.

There seems to be three sections here: About, Courses, and Books. About is a single page of information that reinforces what I already know:

The Ada programming language was designed from its inception to be used in applications where safety and security are of the utmost importance.The SPARK programming language is a formally verifiable subset of the Ada language which allows developers to mathematically prove program correctness through static means.Founded in 1994, AdaCore is the leading provider of commercial and open-source software solutions for Ada

Cool.

There are two courses, introduction to Ada, and introduction to SPARK. I think SPARK is cool but I’ll learn that after I learn Ada itself. I also do like books, but let’s do that first.

I’m not going to write everything about the book as I read it, but I will say that I like the basic structure here: first some history, then imperative programming stuff, then talking about types. After that it gets into packages, more type stuff, privacy, generics, exceptions, tasks, interfacing with C, the standard library, and some appendices. This feels pretty logical. Time to do some reading, I’ll jump back here at random points with notes.

The first Ada standard was issued in 1983; it was subsequently revised and enhanced in 1995, 2005 and 2012, with each revision bringing useful new features.

I heard that Ada 2020 is a thing; seems like they’re on a roughly 7-8ish year timeframe generally. Cool.

In terms of modern languages, the closest in terms of targets and level of abstraction are probably C++ and Rust.

We got a shout-out! That’s nice.

Readability is more important than conciseness. Syntactically this shows through the fact that keywords are preferred to symbols, that no keyword is an abbreviation, etc.

This is interesting; it seems to suggest that readability and verbosity are synonymous. I often find concise things easier to read, personally. This is, of course, subjective, and I wonder how I’ll feel after I learn more of the language.

Ada is a multi-paradigm language with support for object orientation and some elements of functional programming, but its core is a simple, coherent procedural/imperative language akin to C or Pascal.

This makes a lot of sense, especially given the time it was created.

I’m finding it a little hard to understand what main is. The docs say

Greet is a procedure, and the main entry point for our first program. Unlike in C or C++, it can be named anything you prefer. The builder will determine the entry point. In our simple example, gprbuild, GNAT’s builder, will use the file you passed as parameter.

It will use it as … parameter? Like, is the fact that we called the file greet.adb the reason it calls Greet? It’s not clear to me at the moment.

Integer’Image is a function that takes an Integer and converts it to a String. It is an example of a language construct known as an attribute, indicated by the “’” syntax, which will be covered in more detail later.

Ah, it’s not a method call, it’s an attribute.

In Ada, an integer type is not specified in terms of its machine representation, but rather by its range. The compiler will then choose the most appropriate representation.

Very interesting. I wonder how well-defined these mappings are, for example, if you need to call a C function, you’re gonna need to give it the correct sized type… I’m sure I’ll get to that.

Mainly for efficiency reasons, while machine level overflow always results in an exception, type level overflows will only be checked at specific boundaries, like assignment:

Okay, this is the good stuff. Ranged integers are something people always say Ada is better at than Rust; I wondered if this was a static or dynamic check. It is a dynamic check. Cool. I’ve always wanted to know!

As mentioned previously, every “built-in” type in Ada is defined with facilities generally available to the user.

Pretty cool!

While unconstrained arrays in Ada might seem similar to variable length arrays in C, they are in reality much more powerful, because they’re truly first-class values in the language. You can pass them as parameters to subprograms or return them from functions, and they implicitly contain their bounds as part of their value. This means that it is useless to pass the bounds or length of an array explictly along with the array, because they are accessible via the ’First, ’Last, ’Range and ’Length attributes explained earlier.

Awesome. This is the way to do it, for sure.

Here is how the string type is defined in Ada:type String is array (Positive range <>) of Character;

Hmm, what’s a Character though? Let’s try to mess with the example:

with Ada.Text_IO; use Ada.Text_IO;

procedure Greet is
   Message : String (1 .. 2) := "😊";
   --        ^ Pre-defined array type.
   --          Component type is Character
begin
   for I in reverse Message'Range loop
      --    ^ Iterate in reverse order
      Put (Message (I));
   end loop;
   New_Line;
end Greet;

Here’s the output:

Run...
greet.adb:4:04: warning: "Message" is not modified, could be declared constant
greet.adb:4:33: warning: wrong length for array of subtype of "Standard.String" defined at line 4
greet.adb:4:33: warning: "Constraint_Error" will be raised at run time
raised CONSTRAINT_ERROR : greet.adb:4 length check failed

Hm, wrong length. What is the right length? This particular emoji is four bytes, so let’s try to set it to 5, since we start at 1. Nope that doesn’t work either. What about 4?

Run...
the machine running the examples is not responding, please try again later

Very suspicious. Anyway, I’m guessing it’s ASCII, maybe it’ll tell me later.

One last feature of Ada arrays that we’re going to cover is array slices. It is possible to take and use a slice of an array (a contiguous sequence of elements) as a name or a value.

Hell yeah, I love slices.

Ada has multidimensional arrays, which are not covered in this course. Slices will only work on one dimensional arrays.

Ah, bummer. I guess I’ll have to read up on those separately.

Returning variable size objects in languages lacking a garbage collector is a bit complicated implementation-wise, which is why C and C++ don’t allow it, prefering to depend on explicit dynamic allocation / free from the user.The problem is that explicit storage management is unsafe as soon as you want to collect unused memory. Ada’s ability to return variable size objects will remove one use case for dynamic allocation, and hence, remove one potential source of bugs from your programs.Rust follows the C/C++ model, but with safe pointer semantics. However, dynamic allocation is still used. Ada can benefit from an eventual performance edge because it can use any model.

Hmmm, I wonder how it is doing this. Doesn’t that still dynamically allocate in Ada?

Ada doesn’t have a tuple construct and does not allow returning multiple values from a subprogram (except by declaring a full-fledged record type). Hence, a way to return multiple values from a subprogram is to use out parameters.

I love tuples, oh well.

While reading an out variable before writing to it should, ideally, trigger an error, imposing that as a rule would cause either inefficient run-time checks or complex compile-time rules. So from the user’s perspective an out parameter acts like an uninitialized variable when the subprogram is invoked.

Hmmmmmmmmmmmmmmmmmm. This seems extremely against Ada’s safety focus.

with Ada.Text_IO; use Ada.Text_IO;

procedure Outp is
   procedure Foo (A : out Integer) is
   begin
      Put_Line (Integer'Image (A));
   end Foo;
   A : Integer := 12;
begin
   Foo(A);
end Outp;

gives

Run...
outp.adb:4:19: warning: formal parameter "A" is read but never assigned
outp.adb:10:08: warning: "A" modified by call, but value might not be referenced
32514
2 errors.

Each time I run it, I get a different number. Seems bad?

There are two ways in which Ada helps shield programmers from the dangers of pointers:One approach, which we have already seen, is to provide alternative features so that the programmer does not need to use pointers. Parameter modes, arrays, and varying size types are all constructs that can replace typical pointer usages in C.Second, Ada has made pointers as safe and restricted as possible, but allows “escape hatches” when the programmer explicitly requests them and presumably will be exercising such features with appropriate care.

Seems great! I’m very here for this.

… but they have null. Of course they do, given the timeframe Ada was developed. Oh well.

Okay, so new is a keyword, and is in the language. But freeing is an unsafe standard library API? I guess this makes sense, similarly to how leaking memory isn’t unsafe in Rust. Feels weird, though.

I guess, reading more, the idea is that you should basically never need to do this yourself. I think this is where I was getting caught up earlier with the “return an unknown type” bit too. It is allocating, but since you’re not typing new, you’re not responsible, and so that’s safer. That makes sense. I hope my understanding is correct here, but I’m not totally sure.

This does mean that it feels like allocations exist, but are hidden from you, the user. That’s okay, but it’s very different than Rust.

Ada’s variant records are very similar to Sum types in functional languages such as OCaml or Haskell. A major difference is that the discriminant is a separate field in Ada, whereas the ‘tag’ of a Sum type is kind of built in, and only accessible with pattern matching.There are other differences (you can have several discriminants in a variant record in Ada). Nevertheless, they allow the same kind of type modeling as sum types in functional languages.

This is pretty cool.

The “Character Types” section still does not explain to me what valid characters are. Oh well. Googling it gives me this which says

The predefined type Character is a character type whose values correspond to the 256 code positions of Row 00 (also known as Latin-1) of the ISO/IEC 10646:2003 Basic Multilingual Plane (BMP). Each of the graphic characters of Row 00 of the BMP has a corresponding character_literal in Character. Each of the nongraphic positions of Row 00 (0000-001F and 007F-009F) has a corresponding language-defined name, which is not usable as an enumeration literal, but which is usable with the attributes Image, Wide_Image, Wide_Wide_Image, Value, Wide_Value, and Wide_Wide_Value; these names are given in the definition of type Character in A.1, “The Package Standard”, but are set in italics.

Ah. Yeah so that’s why the emoji didn’t work. Did that crash the compiler earlier, is that what was happening?

Generic subprograms or packages can’t be used directly. Instead, they need to be instantiated, which we do using the new keyword, as shown in the following example:

Wait, is this the same new keyword as before? Do generics require heap allocation? That is a reasonable design, I’m just a bit surprised.

Ada exceptions are not types, but instead objects, which may be peculiar to you if you’re used to the way Java or Python support exceptions.

I found this sentence confusing; aren’t the objects in Java or Python as well? When I was reading this sentence, halfway through, I thought to myself “oh, like the stuff I know, cool” and then the second half was “this is unlike the stuff you know”. Hrm.

A task can be thought as an application that runs concurrently with the main application. In other programming languages, a task can be called a thread, and tasking can be called multithreading.

Cool, this is a big topic I’ve been wondering about.

As we’ve just seen, as soon as the main task starts, its subtasks also start automatically. The main task continues its processing until it has nothing more to do. At that point, however, it will not terminate. Instead, the task waits until its subtasks have finished before it allows itself to terminate.

Ah interesting, this is different than Rust. Well, Rust’s standard library, anyway.

I wish it explained if tasks are system threads, or if they’re green threads.

I did some googling, and apparently this is implementation-defined. GNAT uses system threads for some of the same reason that Rust decided to expose system threads. Seems good.

The pragma Assertion_Policy statement is used to force the compiler to generate code to check the precondition.

Design by contract works at runtime. This is something I always thought was true, but was never actually sure of. Cool.

To interface with C’s built-in types, we use the Interfaces.C package, which contains most of the type definitions we need. For example:

Ah ha, this answers my question from earlier. Cool.

Object-oriented programming (OOP) is a large and ill-defined concept in programming languages and one that tends to encompass many different meanings because different languages often implement their own vision of it, with similarities and differences from the implementations in other languages.However, one model mostly “won” the battle of what object-oriented means, if only by sheer popularity. It’s the model used in the Java programming language, which is very similar to the one used by C++. Here are some defining characteristics:

Spicy! But not incorrect.

Dot notation exists! But only if you’re actually using objects, er, “tagged types”.

Thoughts on Rust in 2019

Thu, 27 Dec 2018 00:00:00 GMT

Each year, we ask the community to write blog posts about what they want to see on Rust’s roadmap for the next year. This is my post for Rust in 2019.

Rust 2021: Maturity

This year is also a bit special; in 2018, we introduced “editions” to Rust, on a roughly three-year schedule. So now is not just a good time to think about 2019, but about 2020 and 2021 as well. Rust 2015 was about “stability”. Rust 2018 was about “productivity.” I’d like Rust 2021 to be about “maturity.”

In order to get there, here’s what we need in 2019.

No new features

Emphasis on “new” here. What do I mean by this? Well, there are a few features that are in the pipeline that I do think should land:

async/await
GATs
const generics

And possibly

Specialization

None of these features are new; we’ve got their basic designs in place. These features are also significant and foundational; we need async/await (and to some degree, GATs) for a great networking story, we need const generics for an excellent numerics story.

But after that? I’d like to see a moratorium on major features until 2020, and I’d prefer if we limited 2020 to two or three, if any.

We’ve hit a sweet spot. We’ve always said that Rust 1.0 was about stability, not completeness. I think we’re fast approaching completeness.

That said, I don’t think that the language team should be disbanded or anything; I think their work should transition to working on specifying what we already have, in more detail. I’m not sure if we can have the reference be done in 2019 (more on that later), but I’d like it to be much further along. That can only happen with the help of the language team, and they can only do that work if they have the time.

Refine the RFC process

The RFC process needs to be significantly re-vamped. Niko wrote a great post on this back in June, and I think it’s really, really, really important. I’d like to work on an RFC to propose this, so if you’re interested, we should talk.

Niko already makes the case and lays out some foundations, so I won’t say more there.

Pay down organizational debt

Consider this section an endorsement of everything boats said. I cannot say it better, so I will just leave it at that.

Figure out documentation sustainability

This year was a bad year for the docs team. The book got shipped, and that was great. We had some people pitch in on the reference, and their work was amazing. Some of the other documentation writers moved on to work on rustdoc, which was important.

But a lot of the goals we wanted to do about writing more docs themselves never came to fruition. For example, contributing to major ecosystem crates? Never happened. The cookbook isn’t done. Rust by Example still languishes. The standard library could use a lot of love.

We just don’t have the people to make this happen. We’ve tried, but nothing has worked. It’s possible that this is simply unfixable, after all, most programmers don’t like writing docs. But I also don’t want to give up on it either. I’m not sure what to do here, but I do know that it is a major problem.

Conclusion

There’s a lot of work to do, and I’m very excited to do it. I think Rust is in an excellent place, and with some work, we can make Rust even more amazing in a year’s time.

Six years with Rust

Fri, 21 Dec 2018 00:00:00 GMT

Today is six years since I first heard of Rust. I wrote a post last year about it.

This past year was… intense. Rust 1.31 was basically Rust 2.0, at least in the marketing sense. I burned myself out getting the first edition of the book together for Rust 1.0, and I burned myself out getting the edition shipped.

Let’s talk about the bad and the good. Bad first so we end on the high notes.

The Bad

This year, the docs team sort of… died. Not entirely, but it’s… different now. We used to have a single docs team that handled most everything docs related. The core of that for the past few years has been me and two others. But early this year, we split the sub-teams into fewer individual teams. The old docs team became the docs team, the rustdoc team, the Rust By Example team, and the Reference team. But we didn’t really attract many more people. We did to some degree! The reference team did a ton of amazing reference work. RBE has still struggled. I stayed on the docs team, and the other two went to the rustdoc team. That’s fine, and they’re doing great work, but it’s a little lonely over here now. I can’t decide how much of this issue is “it’s really hard to get people to volunteer to write docs” and how much of this issue is “I mis-managed the team(s) somehow and so it’s my fault.”

Shipping the edition was a monumental effort that caused a lot of debt. We did a good job at avoiding technical debt, but we have a lot of social and organizational debt. See here for more. This has to be paid off now…

There’s a few other things, but let’s stop there.

The Good

We shipped the edition! In fact, we invented the entire edition system and then shipped our first one. Whew! This is a massive, massive accomplishment. Figuring out how we are going to manage Rust’s evolution decades into the future is, frankly a pretty amazing thing to do in twelve months. Non-lexical lifetimes finally landed. Procedural macros finally landed. Rocket is almost on stable. async/await is almost here, and Futures are finally where they should be after two years of work. We did so much good work this year, technically speaking. I’m very proud.

We shipped the new website! This was met with some controversy, but every single website redesign in the history of redesigns has been met with controversy. Doing so in less than a year is also an incredible accomplishment. It almost always takes way longer. We now have the foundations for Rust web properties going forward. This will also pay off a ton later.

We made wasm a thing! I really really wanted wasm to be a thing. A lot of people put in a ton of work, and Rust’s wasm support is bar-none best-in-class. I think this is still a massive growth area for Rust, and we’re almost a full year ahead of almost everyone else on this.

Speaking of growth, Rust’s commercial uptake skyrocketed this year. I met someone for the first time who told me “I learned Rust because it’s what we use at work.” A ton of companies, big and small, have started deploying Rust in critical places, and report good things. Out of the FAANGs, we’ve got Facebook, Amazon, and Google. Three out of five ain’t bad. This tweet is wild:

Introducing #HyperledgerSawtooth version 1.1, an enterprise-grade #blockchain platform with support for #WebAssembly smart contracts and great performance with @rustlang. Empower your business. https://t.co/wl9OYVnUsR #HyperledgerForum pic.twitter.com/5vurpbVMP1 — Intel Security (@IntelSecurity) December 12, 2018

So is this one:

Discover Rust, a C-like language that supports imperative, functional, and object-oriented programming. https://t.co/fCgY4x6zqc — IBM Developer (@IBMDeveloper) June 10, 2018

There’s still lots more to do, but things are headed in the right direction.

Speaking of heading in the right direction, tomorrow will be the compliment to this post; my entry for #rust2019. Catch you then!

You can't "turn off the borrow checker" in Rust

Fri, 14 Sep 2018 00:00:00 GMT

Every once in a while, someone will talk about unsafe in Rust, and how it “turns off the borrow checker.” I think this framing leads to misconceptions about unsafe and how it interacts with safe code.

Here’s some code that causes a borrow checker error:

fn main() {
    let mut x = 5;
    
    let y = &x;
    let z = &mut x;

    println!("{}", y);
}

And the error:

error[E0502]: cannot borrow `x` as mutable because it is also borrowed as immutable
 --> src/main.rs:5:18
  |
4 |     let y = &x;
  |              - immutable borrow occurs here
5 |     let z = &mut x;
  |                  ^ mutable borrow occurs here
6 | }
  | - immutable borrow ends here

Rust won’t let us have a &T and a &mut T to the same T at the same time.

If unsafe “turned off the borrow checker, we’d expect this code to work:

fn main() {
    let mut x = 5;
    
    unsafe {
        let y = &x;
        let z = &mut x;

        println!("{}", y);
    }
}

However, we get the same error, plus a new warning:

error[E0502]: cannot borrow `x` as mutable because it is also borrowed as immutable
 --> src/main.rs:6:22
  |
5 |         let y = &x;
  |                  - immutable borrow occurs here
6 |         let z = &mut x;
  |                      ^ mutable borrow occurs here
7 |     }
  |     - immutable borrow ends here

warning: unnecessary `unsafe` block
 --> src/main.rs:4:5
  |
4 |     unsafe {
  |     ^^^^^^ unnecessary `unsafe` block
  |
  = note: #[warn(unused_unsafe)] on by default

So what does unsafe actually do? Unsafe Rust is a superset of Safe Rust. Unsafe never changes the semantics of Rust code. It instead adds new features that you can only use inside an unsafe block.

One of those features is the ability to de-reference a “raw pointer”. This code does compile:

fn main() {
    let mut x = 5;
    
    let y = &x as *const i32;
    let z = &mut x as *mut i32;
    
    unsafe {
        println!("y: {} z: {}", *y, *z);
    }
}

y and z here have the types *const i32 and *mut i32, both kinds of raw pointers. In order to print y and z, we must de-reference them, which means that we need an unsafe block.

Incidentally, this code is safe. Raw pointers are allowed to alias. And we have no &T or &mut Ts here. So we’re good.

Why does this matter? Well, a lot of people think that as soon as you drop into unsafe, you’re missing the point of Rust, and that you lose all of its guarantees. It’s true that you have to do a lot more work when writing unsafe, since you don’t have the compiler helping you in certain situations, but that’s only for the unsafe constructs.

For example, let’s look at Rust’s standard library, and the LinkedList that it contains. It looks like this:

pub struct LinkedList<T> {
    head: Option<NonNull<Node<T>>>,
    tail: Option<NonNull<Node<T>>>,
    len: usize,
    marker: PhantomData<Box<Node<T>>>,
}

I’m not gonna go into too much detail here, but it has a head pointer, a tail pointer, a length. NonNull is like *mut T, but asserts that it will never be null. This means that we can combine it with Option<T>, and the option will use the null case for None:

fn main() {
    println!("{}", std::mem::size_of::<*mut i32>());
    println!("{}", std::mem::size_of::<Option<std::ptr::NonNull<i32>>>());
}

prints this on my system:

8
8

This optimization is guaranteed due to guarantees on both Option<T> and NonNull<T>.

So now, we have a sort of hybrid construct: Option<T> is safe, and so we can do some operations entirely in safe code. Rust is now forcing us to handle the null checks, even though we have zero runtime overhead in representation.

I think a good example of how this plays out in practice is in the implementation of append, which takes two LinkedList<T>s and appends the contents of one to the end of the other:

    pub fn append(&mut self, other: &mut Self) {
        match self.tail {
            None => mem::swap(self, other),
            Some(mut tail) => {
                if let Some(mut other_head) = other.head.take() {
                    unsafe {
                        tail.as_mut().next = Some(other_head);
                        other_head.as_mut().prev = Some(tail);
                    }

                    self.tail = other.tail.take();
                    self.len += mem::replace(&mut other.len, 0);
                }
            }
        }
    }

Here’s some comments to help understand what’s going on:

    pub fn append(&mut self, other: &mut Self) {
        // do we have a tail?
        match self.tail {

            // If we don't, then we have no elements, and so appending a
           // list to this one means that that list is now this list.
            None => mem::swap(self, other),

            // If we do have a tail...
            Some(mut tail) => {

                // the first thing we do is, check if the other list has
                //  a head. if it does...
                if let Some(mut other_head) = other.head.take() {

                    // we set our tail to point at their head, and we 
                    // set their head to point at our tail.
                    unsafe {
                        tail.as_mut().next = Some(other_head);
                        other_head.as_mut().prev = Some(tail);
                    }

                    // We set our tail to their tail
                    self.tail = other.tail.take();

                    // finally, we add their length to our length, and 
                    // replace theirs with zero, because we took all
                    // their nodes!
                    self.len += mem::replace(&mut other.len, 0);

                } // there's no else case for that if let; if they don't
                  //  have a head, that list is empty and so there's
                  //  nothing to do.
            }
        }
    }

We only need unsafe when mutating the next of the tail of the first list, and the prev of the head of the second list. And that’s even only in the case where the tail exists in the first place. We can do things like “set our tail to their tail” in safe code, because we’re never de-referencing pointers. Safe Rust is helping us manage many, but not all, of our guarantees. We could move the unsafe block to encompass the entire body of this function, and Rust would still make us check those Option<T>s. It would still check that self and other are not aliased by &T or &mut Ts in the body of the function.

unsafe is a powerful tool, and should be used with care. But that doesn’t mean that you throw out many of the things that make Rust what it is.

Is WebAssembly the return of Java Applets & Flash?

Wed, 25 Jul 2018 00:00:00 GMT

In my last post on WebAssembly, I made the following claim:

Some have compared WebAssembly to Java applets; in some ways, they’re very right, but in some ways, they’re very wrong. Eventually I’ll write a post about the wrong, but for now, the right: in some sense, WebAssembly is a different way of accomplishing what the JVM set out to do: it’s a common virtual machine that can be used to build very cross-platform software.

A lot of people expressed interest in me elaborating, so let’s get to it! For this post, I’m going to make three comparisons: to Flash, to Java Applets, and occasionally to PNaCL. Secondly, this post is going to focus on the web use-case for WebAssembly, even though the previous post was about non-web uses. We’ll make that comparison in the next post. Finally, this post is kind of like eating tapas, there’s a bunch of little sections. I feel like it’s a bit short, but at the same time, I’m trying to get back into the habit of blogging, and if I try to keep expanding it, it’ll take forever, so it is what it is.

For what it’s worth, I think this comparison is very natural; that is, when you hear this pitch for WebAssembly:

WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications.

It sounds quite a bit like previous technologies. Asking how they relate, or assuming that they’re a slightly different take on the problem makes a lot of sense. But WebAssembly is significantly different, for a few reasons.

Wasm won

The first reason that WebAssembly is different is that it’s succeeded, and those technologies did not. When I say this, I mean it in a very specific way; if you look at the number of Flash applications out there, and compare it to the number of WebAssembly apps out there, Wasm loses, for sure. Those of us that are excited about WebAssembly still have a lot of work to do before it’s widely used.

When I say “WebAssembly won”, I mean that it’s become a part of the Web Platform. Flash, Java Applets, and PNaCL used the browser’s plugin functionality to work. They were, in a sense, outside of the web ecosystem. They were never standardized, and that matters a lot.

In some sense, this is the final point, really. But just saying that it’s true doesn’t explain why. The rest of this post digs into why this occurred, and those points sort of sum up to this one: other technologies didn’t make it, but WebAssembly did. There’s a lot of overlap here, but I’m trying to reasonably separate it out into individual points.

Other technologies didn’t fit nicely into the platform

Remember this?

Or this?

What about this?

If you built an applet in one of these technologies, you didn’t really build a web application. You had a web page with a chunk cut out of it, and your applet worked within that frame. You lost all of the benefits of other web technologies; you lost HTML, you lost CSS, you lost the accessibility built into the web. These platforms never gained the ability to interact with the rest of the platform provided by your browser. Well, technically they might have, but that’s not how these technologies were used in practice.

How could the web sustain an ecosystem that doesn’t integrate with it? It was never going to happen. And users ended up solidly rejecting it. Outside of games, users hated Flash. Java Applets were heavy and slow. Enshrining these technologies into the web platform wasn’t going to work.

WebAssembly, on the other hand, is much closer to JavaScript. It doesn’t inherently require taking over a part of your screen. It doesn’t expect to be its own little closed off world. Via JavaScript today, and on its own in the future, it’s able to interact with the surrounding environment. It just… fits.

Other technologies were owned by companies

Java was owned by Sun Microsystems, Flash was owned by Adobe. But why does this matter?

Corporate influence on the web is a complicated topic. But in general, the web runs on a psuedo-consensus model. Java and Flash, on the other hand, were controlled by their respective companies. These companies have significant motive to make profit, not to make the web better. And that’s partially what led to the above situation; these companies didn’t care that they didn’t integrate properly with the rest of the platform. Why would they? It’d be much better for business if you got locked into their platform, and abandoned the rest of the web entirely. The incentives are fundamentally misaligned.

WebAssembly is a joint venture between Mozilla, Google, Apple, Microsoft, and others. It’s not trying to push anyone’s particular platform, but instead represents the shared interests of a wide number of stakeholders, both corporate and individual.

WebAssembly followed the web process

This also meant that these technologies never really followed the process that we use to standardize the web. That being said, the web standards process is healthier than its ever been, and at the time these technologies were big, things worked a bit differently. Regardless, WebAssembly followed the way that we do things on the web.

First, asm.js was created, proving that we could do more impressive things with the web. Its performance was good enough to demonstrate what could be done, though it wasn’t fantastic. The secret sauce of asm.js was that it was just JavaScript; it was fully backwards compatible with the existing ecosystem. “Don’t break the web” is a very, very important rule for browser vendors.

WebAssembly was sort of asm.js done better. After building some degree of consensus around asm.js, a variety of people came together to make WebAssembly a reality. Given that it wasn’t just JavaScript, it had to go through the usual process for browsers to implement it, and it did so. Now a W3C spec, it went with the grain of web standards, not against them.

Corollary: other technologies were too big and unstable

There’s another reason Wasm succeeded: it’s tiny. Wasm takes the approach of other web technologies: start small, and build on those foundations. Stay backwards compatible. These technologies also didn’t follow that particular social convention. For Flash and Java Applets, you’d be downloading the particular runtime you needed, and so there was no need for compatibility. PNaCl built on top of LLVM IR, which is completely unstable. They were going to make it a stable subset, but then when LLVM introduces changes, it would have ended up diverging, which isn’t a great situation to be in.

These technologies were also far too large to ever make it through the stabilization process. Can you imagine the big four browser vendors specifying the JVM in full and then agreeing to those semantics for all time? For all of ActionScript, itself a similar-to-but-not-quite version of ECMAScript? For all of LLVM-IR?

Large technologies introduce an unacceptable amount of risk in this situation. It’s an all-or-nothing kind of deal, and the safe bet is to go with “nothing.” WebAssembly, on the other hand, does almost nothing. It’s pretty much math and calling into JavaScript. This means it’s much easier to develop consensus around.

Collary: other technologies required an entire separate virtual machine

I haven’t mentioned Dart in this post, but it sort of fits here too. One big issue with “let’s just put the JVM in every browser” is that browsers already contain a language runtime: JavaScript’s. It’s hard enough to maintain one runtime, let alone two. And then how do you integrate them?

WebAssembly, on the other hand, was designed to be a small extension to existing JavaScript virtual machines. While you can implement your own WebAssembly VM, and non-web use cases often do, for browsers, the maintenance cost is much, much lower.

Other technologies were too specific

WebAssembly is fundamentally language-agnostic. Flash and Java Applets were built first and foremost to run ActionScript and Java. They’re deeply tied to their relative semantics. Even PNaCl suffers from this to some degree; LLVM is really designed for C-like languages, though not quite the same amount.

Do we really want to christen one language as the next language of the web? We already have JavaScript. Are we someday going to introduce a third language? A fourth? An agnostic approach is significantly better for longevity’s sake.

Wasm has a strong approach for safety

Java Applets and Flash were security nightmares. Even if there were attempts to secure them, in reality, they constantly had problems.

WebAssembly, on the other hand, piggybacks on that JavaScript VM again; all of the effort going into creating its sandbox also applies to Wasm. Wasm is missing some features of regular assembly language that can cause security vulnerabilities, like stack smashing. Wasm is memory-safe, which is huge!

Additionally, WebAssembly was designed for validation in mind; its fully typed, and can be checked without running any of its code. It spec includes instructions on how to do validation. This stuff is quite useful!

Conclusion

One thing I’ll say about this post is that it’s quite implementer focused. But implementers are important; they control the web. Providing a solution that works with their goals is just as important as something that works for users. In a future post, I’d like to try to go over user concerns some as well. I have lots of posts to write!

At the end of the day:

Other technologies were not integrated into the platform, and commercial interests didn’t point towards that happening
Other technologies required too much; too much to agree on, too much to implement
Wasm has a really strong sandboxing and verification story that others simply did not.

WebAssembly is more than just the web

Fri, 13 Jul 2018 00:00:00 GMT

While WebAssembly was given its name for good reason, I also think it’s quite the misnomer. When you hear about WebAssembly, don’t just think of the web: think bigger.

Many programmers know that WebAssembly is a technology that lets you run non-JavaScript code in the browser. That is true, but there’s a lot more to it. You see, the spec is written in a very interesting way: there’s the core, what WebAssembly actually is. Here’s the opening section:

To support the embedding of WebAssembly into different environments, its specification is split into layers that are specified in separate documents.

Core specification - Defines the semantics of WebAssembly modules and its instruction set, independent from a concrete embedding.API specifications - Define application programming interfaces enabling the use of WebAssembly modules in concrete embedding environments.

It goes on to say:

Currently, two APIs are specified: JavaScript API … Web API.

So, WebAssembly itself, the core, is completely independent of the web. The web is a particular environment that Wasm is embedded in, but not actually required to use wasm. Other embedding APIs can be created, with their own interfaces, to do various things.

Let’s make this concrete: on the web, we can write a program that draws graphics using WebGL. But we could also write a desktop program that embeds a WebAssembly interpreter, and provides an OpenGL (or whatever else) API to render to the screen. And now our WebAssembly program is running on the desktop.

Why would you want to do this? Well:

I wonder if wasm, in being sufficiently minimal, well defined, practical and widely supported, will wind up as an archival format of choice for emulation-based software preservation. https://t.co/33OUOrqiWg — Graydon Hoare (@graydon_pub) July 11, 2018

These emulators, with a teeny bit of work, could be made to run on the desktop as well. “Cross-platform” doesn’t just mean “Windows, Linux, and MacOS”, but can also mean running to the web as well.

There’s so many more things that can be done, too. All it requires is a bit of re-framing of WebAssembly in your mind:

WebAssembly is not a way to run code other than JavaScript on the web; WebAssembly is a tiny, embeddable language that you can use almost anywhere.

Some have compared WebAssembly to Java applets; in some ways, they’re very right, but in some ways, they’re very wrong. Eventually I’ll write a post about the wrong, but for now, the right: in some sense, WebAssembly is a different way of accomplishing what the JVM set out to do: it’s a common virtual machine that can be used to build very cross-platform software.

There’s a couple of other interesting applications of WebAssembly outside of the browser; one of my personal favorites is Nebulet:

A microkernel that implements a WebAssembly “usermode” that runs in Ring 0.

I’d describe it like this:

Nebulet is an operating system that runs WebAssembly programs.

None of this has anything to do with a web browser; just like ELF is the standard binary format on many Linuxes, WebAssembly is the standard binary format on Nebulet. It’s a new implementation of a quite old idea, but I’m really excited to see where it goes.

A second interesting example is Parity. I’m a cryptocurrency skeptic, but from a tech perspective, this is quite cool. Parity is a company who is building a VM for the Ethereum cryptocurrency. Etherium’s VM has a programming language, Solidity, that lets you script their blockchain. Writing a programming language is a tough task on its own, and Solidity has had some issues, leading to pretty serious bugs. And those bugs are affecting people’s money.

So, rather than keep their custom language, Parity is experimenting with embedding WebAssembly into the Ethereum VM instead. This gets them a well-tested language that has pretty decent toolchain support, and lets them use a variety of other languages to do said scripting. This is very cool!

These are a few of the possibilities for WebAssembly, and I’m really excited about where it could go in the future.

"The Expressive C++17 Coding Challenge (in Rust)" revisited

Wed, 14 Feb 2018 00:00:00 GMT

In October of last year, I wrote a post, “The Expressive C++17 Coding Challenge (in Rust)”. For various reasons, it got brought up again in the D world, and seb has written a new post. It’s good, you should check it out!

However, it links to my gist, not my blog post. As I said back then:

I held myself to the same constraints as the original contest; no external packages is a bit painful in Rust, but it’s not too bad. Mostly it would let me eliminate boilerplate while also improving correctness, and making the code a bit shorter.

So, that got me thinking: What would this look like if I could use external packages? I took about an hour, and knocked it out. I have two versions to show you today, one where I pay no attention to allocations, and one where it’s zero-allocation.

First, the “whatever just allocate” version:

extern crate csv;
#[macro_use]
extern crate serde_derive;
extern crate serde;
#[macro_use]
extern crate structopt;

use std::error::Error;
use std::fs::File;
use structopt::StructOpt;

#[derive(Debug,Serialize,Deserialize)]
struct Record {
    name: String,
    surname: String,
    city: String,
    country: String,
}

#[derive(StructOpt, Debug)]
#[structopt(name = "basic")]
struct Opt {
    filename: String,
    column_name: String,
    replacement: String,
    output_filename: String,
}

fn run(args: &Opt) -> Result<(), Box<Error>> {
    let input = File::open(&args.filename)?;
    let output = File::create(&args.output_filename)?;

    let mut rdr = csv::Reader::from_reader(input);
    let mut wtr = csv::Writer::from_writer(output);

    for result in rdr.deserialize() {
        let mut record: Record = result?;

        let replacement = args.replacement.clone();

        match &*args.column_name {
            "name" => record.name = replacement,
            "surname" => record.surname = replacement,
            "city" => record.city = replacement,
            "country" => record.country = replacement,
            _ => panic!("incorrect column name"),
        }

        wtr.serialize(record)?;
    }

    wtr.flush()?;

    Ok(())
}

fn main() {
    let opt = Opt::from_args();

    if let Err(err) = run(&opt) {
        eprintln!("error: {}", err);
        std::process::exit(1);
    }
}

This uses the [csv crate](https://crates.io/crates/csv/1.0.0-beta.5) to do the heavy lifting with regards to the CSV stuff, and the [strucopt crate](https://crates.io/crates/structopt) for argument parsing. Not only is this code much shorter than my “no crates” version, but it’s also got more features!

There’s a few things that aren’t the best here: first, the match. Rust doesn’t have reflection, exactly, so saying “hey I have this string and I want to index it in this struct” is not as nice. I’m just panic-ing here, but it’d be nicer to validate it up front, and possibly even do some custom derive shenanigans so that I don’t have to write it myself, but I wanted to present a fairly realistic version of what I’d write today. This is good enough.

Second, there’s that one clone. We need it because we’ve said that our Record type holds Strings, and so has ownership over the string. We end up copying London each iteration of the loop. Is copying a city name a problem? Not really, in the context of this example. But what if it was? Fundamentally, there’s no reason to do any allocation here: We’re reading from a file, and we’re writing the same string over and over again.

Enter version 2:

extern crate csv;
#[macro_use]
extern crate serde_derive;
extern crate serde;
#[macro_use]
extern crate structopt;

use std::error::Error;
use std::fs::File;
use structopt::StructOpt;

#[derive(Debug,Serialize,Deserialize)]
struct Record<'a> {
    name: &'a str,
    surname: &'a str,
    city: &'a str,
    country: &'a str,
}

#[derive(StructOpt, Debug)]
#[structopt(name = "basic")]
struct Opt {
    filename: String,
    column_name: String,
    replacement: String,
    output_filename: String,
}

fn run(args: &Opt) -> Result<(), Box<Error>> {
    let input = File::open(&args.filename)?;
    let output = File::create(&args.output_filename)?;

    let mut rdr = csv::Reader::from_reader(input);
    let mut wtr = csv::Writer::from_writer(output);

    let mut raw_record = csv::StringRecord::new();
    let headers = rdr.headers()?.clone();

    while rdr.read_record(&mut raw_record)? {
        let mut record: Record = raw_record.deserialize(Some(&headers))?;

        match &*args.column_name {
            "name" => record.name = &args.replacement,
            "surname" => record.surname = &args.replacement,
            "city" => record.city = &args.replacement,
            "country" => record.country = &args.replacement,
            _ => panic!("incorrect column name"),
        }

        wtr.serialize(record)?;
    }

    wtr.flush()?;

    Ok(())
}

fn main() {
    let opt = Opt::from_args();

    if let Err(err) = run(&opt) {
        eprintln!("error: {}", err);
        std::process::exit(1);
    }
}

This is slightly more involved, as you no longer get access to the nice iterator, and have to do things a little more manually. But it took me about five minutes to port this bit over, so it’s not too onerous.

So, as you can see, once you let people use external packages, the Rust gets significantly nicer. The code pretty much mirrors the task: open the two files, iterate over them, replace the column, write it out. Nice.

I’m not going to do any performance comparisons here because that’s not really what the contest was about. I am mildly curious… if I really wanted to optimize performance, I’d also skip the string stuff and go for bytes directly. This isn’t a super tough transformation, but moves things to be slightly lower level.

If you manipulate CSVs often, you should check out xsv, a command-line tool in Rust that lets you slice and dice these files with speed and ease.

I should also mention that several people have been working on quicli, a package to make making CLIs super easy. One of its authors posted a gist of what this looks like with quicli, you should check that out too!

Looking back at Rust in 2018

Tue, 09 Jan 2018 00:00:00 GMT

A careful reader will note the date of this post; it’s fundamentally in response to this call for blog posts, with a rhetorical device of looking backwards as if my personal plans for Rust came true. If you’re reading this in 2019 or beyond, this may be a work of fiction.

It’s been an exciting year for Rust. Well, it depends on what you mean by “exciting”, exactly. In a word, this year has been about polish. I find this kind of thing exciting! It’s a very different kind of exciting than the “new and shiny” exciting. It’s “take what we have and make it shiny.”

Let’s recap the major achievements of the year.

Rust 2018

First of all, we shipped our first epoch! If you type cargo new today, you’ll see a new shiny entry in your Cargo.toml:

[package]
name = "new-program"
version = "0.1.0"
authors = ["steveklabnik <steve@steveklabnik.com>"]

epoch = "2018"

[dependencies]

What does this mean in practical terms? Well, a few things. First, let’s get the big one out of the way: Try to compile this code with no epoch, or with epoch = "2015":

fn main() {    let catch = 5;}

You’ll see this:

> cargo build
   Compiling new-program v0.1.0 (file:///C:/Users/steve/tmp/new-program)
warning: `catch` will become a keyword in the 2018 epoch
 --> src\main.rs:2:9
  |
2 |     let catch = 5;
  |         ^^^^^
  |
  = note: #[warn(new_keyword)] on by default
  = note: to avoid this warning, consider using `_catch` instead

And if you change it to epoch = "2018", you’ll see this:

> cargo build
   Compiling new-program v0.1.0 (file:///C:/Users/steve/tmp/new-program)
error: expected pattern, found keyword `catch`
 --> src\main.rs:2:9
  |
2 |     let catch = 5;
  |         ^^^^^

error: aborting due to previous error

Success! However, the specifics here aren’t as interesting as what this change means:

First, this is one of only two changes that relies on the epoch. When we announced our plans for epochs, many people were skeptical that we would do as we said, and only included small, parser-level changes here. But we did, and will continue to do so. catch and dyn Trait are the only two changes here.

Second, we’ve demonstrated that this doesn’t split the ecosystem. All of the crates that had no epoch are treated as 2015, and everything still works. Mixing 2015 and 2018 code works just fine.

Finally, mrustc has not had trouble implementing this split. [mrustc is an alternative implementation of Rust](https://github.com/thepowersgang/mrustc), and some of the worries about epochs would be that they would unnecessarily burden alternative implementations. This fear has not borne fruit yet though.

We did ship some new stuff

Even though this year was about polish, that doesn’t mean that we had no new features. The key is, these features were all designed in 2017 or before. That is, features like non-lexical lifetimes, impl Trait, and Macros 2.0 didn’t land in stable until 2018, but they’ve been worked on for years. At the end of 2017, we had preliminary implementations already landed, it was just about polishing them up so we could ship.

Cargo, however, did gain some new abilities. Cargo was sort of neglected in 2017, or at least, wasn’t worked on as much as the rest of the project was. Cargo grew the ability to use alternative registries, and at least one company is offering “crates.io as a service” and “crates.io behind your firewall”. Cargo also became easier to use offline, and the “integrate Cargo into other build systems” work bore some more fruit.

WebAssembly and Embedded platforms

At the end of 2017, Rust was poised to take advantage of WebAssembly. And take advantage, we did. WebAssembly has not taken over the world yet, but that was also never its intention. It is a common aspect of the Web Platform these days, and Rust’s support for it is so excellent that it’s the primary language with which you write new code for WebAssembly today. Tons of JavaScript programmers use Rust for their various systems-level tasks on the web, and we’re just starting to see completely new kinds of applications form. Adding Rust to your Node or front-end project is trivial, thanks to excellent integration from JavaScript’s build tools, or as we now call them, “compilers”.

An aside: in the 2014-2017 era, even though JavaScript was JIT compiled, many programmers used various “build tools” or “transpilers” to do things like minify and bundle their JavaScript together. This was useful for things like “let me use new ES2017 features today but transpile to ES5.” In 2017, lots of JavaScript programmers, through tools like Rollup, started to appreciate that what they’re doing, on a fundamental level, is compiling. Once these tools realized this, they started taking advantage of decades of existing compiler research, and these tools started doing amazing things. Oddly enough, this makes JavaScript feel more like Java: you compile to a thing that ends up being JIT-ed. And that’s not even half of it; stuff like Ember’s GlimmerVM took this approach to new heights.

Furthermore, it’s considered standard practice to test your crate against the WebAssembly target, just like you would Windows, Linux, and macOS, so the vast majority of the crates.io ecosystem Just Works.

But this work on wasm has a good side effect: wasm is basically a very minimal embedded platform. By doing the legwork to make wasm awesome, we also made embedded platforms of various kinds awesome. There’s also specific work that had to be done, for example, remember xargo? Cargo just does that now. We’ve elevated ARM to a Tier 1 platform these days, and provide easy instructions for getting a new architecture going.

This also implies stabilizing some of the lower-level things that are needed to do OSdev work. You can’t write a “real” OS in stable Rust yet, but you can write toy ones, and real OSes should be usable by next year.

Stabilization

One way in which we did polish was to ship a ton of stuff that was previously in flight. I mentioned non-lexical lifetimes earlier, as well as some other features, but there’s also a sort of “long tail” of stabilizations that we addressed.

At the start of the year, the teams went through each of their feature gates, and voted: stabilize, put on hold, or remove. Things put on hold this year will get decided next year, but cannot be put on hold for another year. Of course, there were some small course-corrections over the year, but in general, it let us go “oh hey, slice patterns has been unstable since before Rust 1.0, let’s finally ship it.”

Bugs & new contributors

Rust’s bugtracker was in a much better state in 2017 than in 2016, but got even better in 2018. 2017 saw the infrastructure team actually ensure that all bugs get triaged, that is, they all have tags within two days of being opened. In 2018, however, there was a focus to knock out a lot of the long-standing, papercut style bugs. At the start of 2018, there were ~3100 bugs on the tracker, but today, there are only ~1500. That’s four bugs per day! We couldn’t have gotten there without all of the new contributors and team members we’ve added this year; in 2017, we had about 75 people on our various teams, and now we have 100. Your average Rust release in 2017 had about 120 contributors, it’s 180 for 2018.

Community and Job growth

It’s not just the teams that have grown, but also the community. For example, in 2016, crates.io had ~7500 crates. In 2017, ~13,000. In 2018, we’re up to 35,000. That’s a lot of packages! One major driver of this growth is jobs; once REDACTED announced that they’d been using Rust in a production setting, and that compared to the old system written in REDACTED, it saved them millions of dollars in server costs per year, a ton of companies started using Rust. It’s not super easy to get a Rust job, but they’re out there. We’ve also seen our meetups continue to grow, RustConf had two tracks this year, and there’s a new RustFest in South America, with rumors of a conference in China next year.

IDEs

While I don’t really use IDEs, we put a lot of work into making them awesome, and 2018 was the first time that people who do use IDEs love using one. Specifically, they love using the new IDE that JetBrains has released for Rust. In 2017, JetBrains was sponsoring the plugin for IntelliJ, but in 2018, they’ve released a whole product. Other IDEs are coming along swimmingly; Visual Studio now has really excellent support, for example.

Documentation

I work on docs, so I’m going to give this a section, even though maybe other things are more “exciting”.

2018 saw the release of $REDACTED, a sort of “next-generation rustdoc.” It focused on not just API docs, but all of your docs holistically; you can publish guides as well, structure things how you’d like, etc. In some ways, it’s “rustdoc + mdbook” in a nice integrated package. This has led to even more people writing even better docs, since it’s so nice to use.

There were two other major initiatives this year: focusing on intermediate documentation, and helping major ecosystem crates get excellent docs. In 2017, there was sort of a “what do I do after TRPL” situation, but now we’ve got some great resources to help you grow as a Rust programmer. Additionally, many of the more popular crates have docs that rival the standard library’s, and smaller crates have decent docs.

Oh, and TRPL saw its initial paper release early in the year. Yay!

Epilogue

Whew, that’s a lot of stuff! What does 2019 hold for Rust? That post is for another time.

Five years with Rust

Thu, 21 Dec 2017 00:00:00 GMT

Today is five years since I first heard of Rust. I wrote a post last year about it. This year, I have something slightly different in mind.

Five years is a long time in tech. It’s also a long time for me to be focused on one specific thing; other than the pizza job I held in high school and college, I’ve never stayed in a single job for five years. (This is true today as well; it’s not like I got a job with Rust the first day I learned of its existence.) I started with Ruby and Rails in 2008/2009, and my final commits to Rails were in August/September of 2013. So at this point, my career has basically been split in half, rather than dominated by Ruby.

Five years ago tomorrow, I registered http://www.rustforrubyists.com/.

Yesterday, I had lunch with someone I’m very honored to call a friend. We talked about a lot of things, but also about Rust’s past, present, and future. One thing I’ve always felt a little awkward about is that there was a crucial point back in 2011/2012 where I wasn’t sure what to do with myself. I was very frustrated by the industry, and considered doing other things instead. But, as they and I talked about, sometimes, it’s not about your interests. If what you want to do is change the world, caring a lot isn’t enough; you have to analyze what you can actually do. While I think our industry is pretty broken, I at least have the ability to try and improve it, thanks to my stature. (God, even just typing that makes me feel gross.)

I decided to put my time into Rust because I believed, and still believe, that it’s an important thing. And I saw the work that needed to be done, and saw that it’s the kind of work that I could do. And so, it’s maybe not something bigger, but I resolved myself to doing the work.

Five years in, I think my journey has been successful so far. I’m proud of how far Rust has come. That said, there’s a lot more work to do. I’m still committed as ever to getting that work done, and feel proud to be able to work alongside people as amazing as the Rust community.

Here’s to five more years. ?

"The Expressive C++17 Coding Challenge" in Rust

Wed, 25 Oct 2017 00:00:00 GMT

A few weeks ago, two C++ bloggers initiated a challenge (here and here): use the shiny new features of C++17 to write a small program that manipulates CSV files. The link is to the results; you can see the examples people came up with.

I’ve written a follow-up post too!

I decided to implement it in Rust for fun. You should give it a try, in whatever language you use, it’s a nice sample problem. I held myself to the same constraints as the original contest; no external packages is a bit painful in Rust, but it’s not too bad. Mostly it would let me eliminate boilerplate while also improving correctness, and making the code a bit shorter.

Anyway, here it is:

https://gist.github.com/steveklabnik/ad0a33acc82e21ca3f763e4278ad31a5

I implemented From for everything so I could use ?, but then main just has a ton of unwraps. I normally would remove those too, but the challenge does not include good error handling, so I decided to just stick to the text of the challenge.

I’m not going to say that this is the best Rust code in the world, nor compare it directly to the C++ examples. But it was a fun hour or two to knock it out, and figured I’d at least share my results.

Want to join the Rust docs team?

Fri, 24 Mar 2017 00:00:00 GMT

One of Rust’s major goals this year is to provide mentoring at all levels of the project. This is a goal I’m very on-board with. There’s lots of ways to get involved with Rust, but I’d like to talk about the bit I’m most involved in: documentation.

There’s many ways to contribute to Rust’s docs. Some people contribute occasional PRs, some people end up contributing a lot. No matter what your interest, I’d like to help. But before I get into those details, why should someone get involved with documentation? I have my own reasons, but I also have a pitch for you:

Working on documentation is a fantastic way to learn something in a deeper way.

In order to document something, you have to understand it, even if only a little bit! It’s a good excuse to dig into the details. And I’ve found that many people are eager to help explain something to you in return for you writing that down and synthesizing it into documentation.

If that sounds intriguing to you, let’s talk! There’s a number of different ways to do this, and I’m interested generally in helping anyone work on Rust’s documentation, in any form. However, if you don’t have something in mind, I do have a suggestion.

Back in October of 2015, I made a big GitHub issue to track giving the whole standard library documentation a once-over. I then made sub-issues for each main module. Slowly, myself and others have been knocking them out, but there’s still a number of issues open.

Today, I went through all of the remaining ones, and made fine-grained bullet points of what needs to be done. I’ll admit that some of them may not be as explicit as they could be, but it at least gives a starting point to talk about. Here’s all the sections that are currently open:

Some of the work that needs to be done is very small: add a link here, re-word a sentence there. Other bits are large; “this whole thing needs re-written.”

So here’s my pitch to you: have you ever wanted to learn a bit more about Rust? Do any of those names sound interesting to you? Maybe you’ve wanted to learn a bit more about threads in Rust. Check out that ‘thread’ issue above, poke around, learn some Rust, and then let’s make the docs better! We can work together on the issues, through email, or on IRC. Whatever works. Maybe you’ll find it as addicting as I do. And I find that people who are non-experts write way better docs than experts. So don’t worry about that!

Worst case, you’ll have learned a bit of the standard library a bit better, and that’s always a good thing, too.

An overview of macros in Rust

Tue, 24 Jan 2017 00:00:00 GMT

Rust has had a lot of stuff going on with “macros”. There are a few, interconnected aspects of these features, many different names for the same thing, and a lot of development work going on. This post will give an overview of the past, present, and future of macros in Rust.

Ultimately, macros allow you to write things like this:

let map = hashmap!{
    "a" => 1,
    "b" => 2,
};

#[derive(Serialize, Deserialize)]
struct S {
    #[serde(rename = "a")]
    f: i32,
}

That is, some macros look like functions: foo!, but allow you to write nearly arbitrary syntax inside of the ()s. Some let you control the ability to write custom attributes. Both generate some other kind of valid Rust code for the compiler to interpret. In the first case, the hashmap! macro will expand to a call to HashMap::new and a bunch of insert calls, and the second case, the Serialize and Deserialize traits will be automatically implemented for this struct, along with a bit of custom logic where the field f maps to the name a in the data that’s being {de,}serialized.

These features are extremely powerful, but also must be done right: metaprogramming is very useful, but can also make code inscrutable, or give poor error messages.

Oh, and I’d like to give a shout-out to The Little Book of Rust Macros, which is basically the resource for doing this kind of work. If you are interested in writing macros, you should check it out.

Pre-1.0

As Rust developed, two major forms of macros evolved. One was called “macros” and the other was called “compiler plugins”. Both of these systems were pretty good, but both of them also had a lot of problems. We’ll get into the details of each of these kinds of macros below, but there’s one thing we should talk about first. On the way to 1.0, it was decided to stabilize one kind of macros, but not the other. From Stability as a Deliverable: (also, this uses some terms that we will cover shortly)

After extensive discussion, we plan to release globs and macros as stable at 1.0. For globs, we believe we can address problems in a backwards-compatible way. For macros, we will likely provide an alternative way to define macros (with better hygiene) at some later date, and will incrementally improve the “macro rules” feature until then. The 1.0 release will stabilize all current macro support, including import/export.On the other hand, we cannot stabilize syntax extensions, which are plugins with complete access to compiler internals. Stabilizing it would effectively forever freeze the internals of the compiler; we need to design a more deliberate interface between extensions and the compiler. So syntax extensions will remain behind a feature gate for 1.0.Many major uses of syntax extensions could be replaced with traditional code generation, and the Cargo tool will soon be growing specific support for this use case. We plan to work with library authors to help them migrate away from syntax extensions prior to 1.0. Because many syntax extensions don’t fit this model, we also see stabilizing syntax extensions as an immediate priority after the 1.0 release.

We’ll discuss in each section how this ended up working out, but it’s important to remember that only one form of these macros was available on stable Rust for 1.0, and in fact, as of the time of this writing. That’s going to change soon…

It’s also important to note that we weren’t entirely happy with macros. They were fine, but they have some flaws. We wanted to fix them, but we also didn’t want to ship 1.0 without macros. Language design is hard. I’ll talk about this more in the appropriate section.

Declarative Macros

The first form of Macros in Rust, and the one that’s most widely used, are called “declarative macros”, sometimes “macros by example”, sometimes “macro_rules macros”, or sometimes just “macros”. At their core, declarative macros allow you to write something similar to a Rust match statement:

match x {
    4 => println!("four!"),
    5 => println!("five!"),
    _ => println!("something else"),
}

With match, x’s structure and value will be evaluated, and the right arm will execute based on what matches. So if x is five, the second arm happens. Etc.

These kinds of macros work in the same way: you set up some sort of pattern, and then, if that pattern matches, some code is generated. One important difference here is that in match, x gets evaluated. With macros, x does not get evaluated.

There are two kinds of declarative macros: ones defined with the macro_rules! keyword, and ones defined with the macro keyword.

`macro_rules!`

These are most commonly known as “macros” in Rust world, and are the only thing that’s stable in Rust at the time of this writing. Here’s a macro you’ll see when using Rust:

let x: Vec<u32> = vec![1, 2, 3];

This macro creates a new vector, with three elements inside. Here’s what the macro could look like, written with macro_rules!:

macro_rules! vec {
    ( $( $x:expr ),* ) => {
        {
            let mut temp_vec = Vec::new();
            $(
                temp_vec.push($x);
            )*
            temp_vec
        }
    };
}

Whew! That’s a bunch of stuff. The most important line is here:

    ( $( $x:expr ),* ) => {

This pattern => block looks similar to the match statement above. If this pattern matches, then the block of code will be emitted. Given that this is the only pattern in this macro, there’s only one valid way to match; any other will be an error. More complex macros will have more than one rule.

The $x:expr part matches an expression, and gives it the name $x. The $(),* part matches zero or more of these expressions. Then, in the body of the macro, the $()* part is generated for each part that matches, and the $x within is replaced with each expression that was matched.

These macros are fine, but there’s a number of bugs and rough edges. For example, there’s no namespacing: if a macro exists, it’s everywhere. In order to prevent name clashes, this means that you have to explicitly import the macros when using a crate:

#[macro_use]
extern crate serde;

Otherwise, you couldn’t import two crates that had the same macro name. In practice this conflict doesn’t come up much, but as we grow, it becomes more likely. The hygiene of macro_rules is there, but not perfect. (Only local variables and labels are hygienic…) Et cetera. The plan is that with the new macro systems, you’ll be able to use the regular module system, and use macros just like you use functions today.

Even with these problems, macros were useful enough to include in Rust 1.0. But plans for a better macro system were already being worked on, and so, they were given the name macro_rules! so that we could eventually deprecate all of this. This is one of the few features that we decided to do this with Rust 1.0; a vestigial thing that was always planned for obsolescence. Doing it hurts, but not having any metaprogramming at all would hurt much more.

`macro`

As such, the macro keyword was reserved for a new macro system. We’re still not 100% sure if we’re calling these “macros by example” (which you’ll sometimes see with a “2.0” at the end) or “declarative macros”. I prefer the latter, because it pairs nicely with “procedural macros.” It seems like most agree with me. We’ll see.

These macros are an incremental improvement over the existing macro_rules! macros: they’re fundamentally the same thing, but with some nicer features.

At the time of this writing, an initial RFC is in FCP, laying out some high-level aspects of this feature. An initial implementation is underway, but it hasn’t landed yet, so you can’t use macro in real code yet.

Procedural Macros

In opposition to the pattern-based declarative macros, the second form are called “procedural macros” because they’re functions: they accept some Rust code as an input, and produce some Rust code as an output. I say “code” but I don’t mean that literally; we’ll get into the exact details in each section.

Syntax Extensions

Syntax extensions, also sometimes just called “procedural macros” or “compiler plugins”, were not stabilized for Rust 1.0. Here’s (part of) a syntax extension that implements roman numeral literals:

#![feature(plugin_registrar, rustc_private)]

extern crate syntax;
extern crate rustc;
extern crate rustc_plugin;

use syntax::parse::token;
use syntax::tokenstream::TokenTree;
use syntax::ext::base::ExtCtxt;
use rustc_plugin::Registry;

fn expand_rn(cx: &mut ExtCtxt, sp: Span, args: &[TokenTree])
        -> Box<MacResult + 'static> {
    // implementation goes here
}

#[plugin_registrar]
pub fn plugin_registrar(reg: &mut Registry) {
    reg.register_macro("rn", expand_rn);
}

You’d use it like this:

#![feature(plugin)]
#![plugin(roman_numerals)]

fn main() {
    assert_eq!(rn!(MMXV), 2015);
}

That is… a lot of code. And frankly, probably doesn’t even work on today’s Rust right now. Why not? Well, that’s the big problem with syntax extensions:

extern crate syntax;
extern crate rustc;
extern crate rustc_plugin;

This literally loads up bits of the Rust compiler and uses it as a library. Then, it uses Rust’s internal AST to do the processing. This is extremely powerful, but also extremely unstable. We don’t and can’t freeze Rust’s internal AST, or any of its other internal APIs, at least not at this time. Not to mention that this means any alternative Rust compiler would need to either use our AST or map ours to theirs, which puts a damper on alternative Rust implementations. This means that syntax extensions haven’t had a good path towards stability. It’s also why they couldn’t be made stable as-of Rust 1.0. But “run arbitrary Rust code to produce more Rust code” is a useful feature, and so, some projects needed them. In addition, while declarative macros can produce function-style foo! calls, they cannot do the custom attribute stuff. And those are very useful. We could make this work, but there hasn’t been a real reason to, nor much demand.

A side note: there’s also the term “compiler plugin.” Syntax extensions are a particular kind of compiler plugin; there are others, like lint passes, LLVM pass plugins, MIR plugins… we’re only talking about syntax extensions in this post, however.

The current plan is to never stabilize syntax extensions, and instead, stabilize “procedural macros.” Again, this isn’t for all compiler plugins, only syntax extensions.

Procedural Macros

Accepted in RFC 1566, procedural macros will give you the same power as compiler plugins, without needing to reach into the guts of rustc. How? Well, they look like this:

#[proc_macro]
pub fn foo(TokenStream) -> TokenStream {
    // do stuff to implement the foo macro
}

This creates a foo! procedural macro. You’ll note that it accepts a TokenStream and produces a TokenStream. To quote the RFC:

The first argument is the tokens between the delimiters in the macro use. For example in foo!(a, b, c), the first argument would be [Ident(a), Comma, Ident(b), Comma, Ident(c)].The value returned replaces the macro use.

Why tokens? Well, while an AST can change, the actual tokens will not. This approach will allow us the flexibility to change rustc’s internal AST, while giving us the full power of compiler plugins.

So, how do you work with one of these kinds of macros? You can of course work with the token stream directly, but we also expect to see libraries that will provide easier-to-use interfaces on top of this. You could imagine a package that converts a TokenStream to its own AST, and then provides an interface to that AST. This is nicer to use than working with tokens directly. There are already some libraries that do similar kinds of things, like syn. More on that in a moment.

Function-like vs attribute like

The above is for a macro that gives you a foo! macro, that is, one that looks like a function. What about one that gives you an attribute? That’d look like this:

#[proc_macro_attribute]
pub fn foo(Option<TokenStream>, TokenStream) -> TokenStream {
    // implementation goes here
}

To quote the RFC:

The first argument is a list of the tokens between the delimiters in the macro use. Examples:#[foo] => None#[foo()] => Some([])#[foo(a, b, c)] => Some([Ident(a), Comma, Ident(b), Comma, Ident(c)])The second argument is the tokens for the AST node the attribute is placed on.

Internally, it’s the same thing: use a TokenStream, produce a TokenStream.

“Macros 1.1”

You may have also heard a lot of commotion about “Macros 1.1.” In fact, they will be stable as of the next release of Rust, 1.15. Macros 1.1 are the Pareto principle applied to procedural macros: as it turns out, the vast majority of people who use procedural macros today use it for “custom derive”, that is, the ability to say #[derive(MyTrait)] and implement a trait automatically. As such, a plan was hatched: we could stabilize just the very very very basics of procedural macros, enough to get the ability to write custom derives, and stabilize that, without needing to implement everything. What does that look like? Here’s one that implements #[derive(Double)]

extern crate proc_macro;

use proc_macro::TokenStream;

#[proc_macro_derive(Double)]
pub fn double(input: TokenStream) -> TokenStream {
    let source = input.to_string();

    // Parse `source` for struct/enum declaration, and then build up some new
    // source code representing a number of items in the implementation of
    // the `Double` trait for the struct/enum in question.
    let source = derive_double(&source);

    // Parse this back to a token stream and return it
    source.parse().unwrap()
}

You’ll notice that this still looks like TokenStream -> TokenStream, but there’s something more subtle here: input.to_string(). That is, for Macros 1.1, TokenStream has two methods; all it has is the ability to be converted to a string, with the original Rust code inside of it, and a parse method, which can take a string of Rust code and produce another TokenStream. What you do with that String is your own business.

What most people will do is use syn to parse the Rust code, and quote to generate Rust code from that. These libraries will make it feasible to build real-world custom derive code, without us needing to stabilize anything we don’t want to. Of course, once full procedural macros are implemented, it’s not likely that people will want to turn TokenStreams into and parse TokenStreams from strings, but it’s not totally unheard of, and will be the tiniest wart. It’s worth it to move a significant chunk of the ecosystem onto the stable compiler.

For more details, see RFC 1681.

“Macros 1.2” ?

There’s also been some small talk about possibly doing a “macros 1.2”, that is, another step towards full procedural macros without needing the whole thing. This would be “procedural macros without hygiene”, though it’s still early days for this idea; there isn’t even an RFC yet. We’ll see if this happens. There’s still some controversy around this idea.

Announcing Rust Contributors

Thu, 12 Jan 2017 00:00:00 GMT

When I was working with Ruby on Rails, someone once sent me this link:

http://contributors.rubyonrails.org/

In some ways, this link changed my life. You see, I respond really strongly to integers going upwards. I am a sucker for points, hits, whatever. That web page, however, puts that integer towards something useful: open source contributions. It’s fairly well known that open source contribution is a long tail, but that’s also perfect for this kind of gamification. One contribution, as of today, will put you in 2461th place. Your second? 1473! That’s almost a thousand places! Just one more contribution puts you at number 1114. Still three hundred spots. Your forth commit will let you get in the top 900, #893. You get the idea. (I ended up at #47.)

Today, I’m working on Rust. And one of the things that I do is write the release announcements, or at least, their first draft. And one of my favorite things to do is the last bit: thanking all of the people who have put in work toward this release. It’s one of my favorite things because I really do appreciate all of the patches, but also because I remember how hooked I was when people gave me a little bit of credit for my contribution. Thanking people is powerful.

So to write those blog posts, I have to run some git commands. And I always forget exactly what I have to write, given our branching strategy. So I wrote a little Rakefile almost a year ago that will generate a new post with the appropriate thank yous and statistics. But I never made it general enough; I have to tweak the numbers every release. It’s annoying. I’ve always wanted to fix it up. There’s one more way in which I don’t like this setup: we only thank people who contribute to rust-lang/rust. Even those with a commit to Cargo don’t make it in there. It’s a historical artifact of the release process, but I don’t like it. We should be thanking those people too.

While all of this has been rolling around in the back of my brain, I’ve been thinking about contributors.rubyonrails.org. I’ve said for a while I wanted it in Rust. So, over the last few days, I hacked one together.

https://rust-contributors.herokuapp.com/

Right now, it only replicates what we say in the blog posts. It doesn’t show commit counts, or order by it, or anything. But the foundation is there. Also, since this is a fun hack, I didn’t take too much care when writing it: there’s a number of low-hanging fruit kinds of things to do. I didn’t use a fancy web framework. I didn’t focus on refactoring relentlessly. I didn’t write any tests. The error handling is basically non-existent. I just knocked it out as fast as I can. But it’s a web app, written in Rust, and that’s fun.

If you’d like to help me hack on this, head over to https://github.com/steveklabnik/contributors and take a look at the issues. I’ve already identified a number of things, and if you do web work but have been meaning to mess with Rust, this might be a good chance to give it a shot.

Thank you to everyone who submits patches to Rust; you’re all great, and I’m lucky to be able to work with you all.

Fire Mario not fire flowers

Thu, 29 Dec 2016 00:00:00 GMT

My post yesterday got a lot of great responses. Two of them are from people I greatly respect, Dave Herman and Graydon Hoare. You should go read those two posts; this one is in response to them. I’m going to respond to both at the same time by mixing them together.

Graydon says:

Safety in the systems space is Rust’s raison d’être. Especially safe concurrency (or as Aaron put it, fearless concurrency). I do not know how else to put it.

“Fearless concurrency” is a different way to put it! That is, to use Dave’s metaphor, “safety” is the fire flower. But “fearless concurrency” is Fire Mario. This is where I think Dave misses the mark slightly as well. I agree with his metaphor, but disagree with his conclusion.

I’m not saying that we get rid of fire flowers. I’m saying that we focus on “Rust makes you Fire Mario,” not “Rust is made of fire flowers”, when talking to people about Rust.

Dave also misses my mark slightly, but in a different way:

I don’t think the Rust community should worry about shying away from talking about safety, but I hope we all remember that it’s in service of something much more exciting: making more badass systems programmers!

We are so close here that it feels like nitpicking, but I think the difference is important. I think one of Rust’s greatest potentials, like Dave does, is bringing a ton of new people into systems programming. But my point is this: “memory safety without garbage collection” doesn’t speak to that at all. That is, memory safety without garbage collection is the mechanism of how that’s accomplished, but it’s not the accomplishment itself. It’s the Fire Flower, not Fire Mario.

I’d like to make sure we’re talking about being Fire Mario, rather than being Fire Flower enthusiasts.

Rust is more than safety

Wed, 28 Dec 2016 00:00:00 GMT

Rust is most well-known for its features around writing low-level code in a safer way than its predecessors. But it’s also a lot more than that. The Rust community is slowly evolving the way that we talk about Rust to people who haven’t yet given Rust a try, but it can take time. But how did we get here?

In Rust’s initial release announcement, Graydon used a phrase that is amazing:

[Rust is] Technology from the past come to save the future from itself

He goes on to elaborate a bit:

Many older languages [are] better than new ones. We keep forgetting already-learned lessons.

This is one of the many things that the “Rust” name evokes. Rust was not intended to be a cutting-edge PLT (programming language theory) research language. It was challenging an assumption that I think many of us hold dear: that “progress” means that things always get better. Reality is a bit more complex than that. Yes, we’ve improved on some things, but there are some older things that were actually “better” in some sense, but never quite made it for other reasons. I’ve always phrased this sentiment like this:

Let’s look at the initial release dates of some of the top programming languages:

C: 1972
C++: 1983
C#: 2000
Java: 1995
JavaScript: 1995
Python: 1991
PHP: 1995
Ruby: 1995

The first thing you’ll notice is: what the hell were they putting in the water in 1995?

The second is that, other than C# in 2000, 1995 or earlier is where most of our programming languages come from. Have we really not made any advancements significant enough in the last 20 years to have gotten people’s attention?

Now, there are a multitude of newer languages, and people do use them. And these languages have had more releases since then. But none of these languages has fundamentally changed. They’re all pretty much the same, but better. And specifically, you’ll notice that many of these languages are scripting or higher-level languages: if we restrict ourselves to “what most operating systems are programmed in”, we haven’t had a new generation of programming languages in thirty or forty years!

PLT research and industry live in two different worlds. Industry is effectively using ancient PLT tech. There’s lots of good reasons for this too, mind you. But if you’re looking to make a new hit programming language, you don’t actually have to be innovative in the PLT sense. You can take PLT tech from the 2005-2010 era and you’ll still be ten years ahead!

So in some senses, this is what Rust is trying to do: not be a cutting-edge programming language in a PLT sense, but advance the industry by mining “older” research.

However, plans don’t always work out the way you want to. As Rust evolved, many of its original bits were thrown out, and replaced with new ones. Rust has advanced a small bit of the universe in PLT with the ownership/borrowing system. Like most research, it’s built on top of older work. But that’s exciting!

Another thing about the borrow checker is that it’s “unique” about Rust as well. So if someone asks you, “hey, what’s cool about Rust?”, it’s extremely tempting to answer with the thing that makes it most clearly different than other programming languages. And for some people, this pitch is compelling! But not for everyone…

Shifting priorities

Marketing is hard. But there’s one thing that I’ve learned about it that I think applies to Rust:

Your marketing should be focused on what problems your product solves for the consumer.

Rust’s marketing, historically, hasn’t been as good at this as it could have been. That is, we’ve been too focused on the details, and not enough on the big picture. We’ve been worried about what’s interesting in a PLT aspect, and not what problems Rust solves for users. I can think of two big examples where this has happened:

Rust is safer than C or C++. (“I never have problems with safety in the C++ code I write” is a common reply.)
Rust is as low-level of a programming language as C! (“I never write C. Rust is not for me.”)

Both of these pitches for Rust focus on the how, and not on the problem itself. And that means that sometimes people draw the opposite inference as to what you were intending with the pitch in the first place.

I have some ideas on a better pitch for Rust, and I even had a draft of it here, but I’ve decided to delete it for now. Instead, I’d like to ask all of you who love Rust to give this some thought, and write your own blog posts about what problems Rust solves for you. I’ll edit this post and link to yours if you ask me to. I’ve seen several people do a good job at articulating this kind of thing before. And I’ll work on my own post as well, as a follow-up.

Let’s figure out the best way to share our love of Rust with the rest of the programming world.

Here’s some of the responses generated from this:

I’ve written Fire Mario, not Fire Flowers, as a response to two responses to this post.

The /r/rust, proggit, and orange website threads are full of good replies.

Four years with Rust

Wed, 21 Dec 2016 00:00:00 GMT

Today is four years since I first learned about the existence of Rust. I know this because Rust 0.5 was the first release I used. Rust has changed a lot in that time. For a brief overview of its history, watch this talk of mine. But for today, I thought it would be fun to check out the release announcement and see what’s the same, and what’s changed.

rust-dev

First up, the announcement itself. The rust-dev mailing list used to be your one-stop shop for talking about Rust in a persistent form. But in January of 2015, we decided to close it down. Why? Let’s see what Brian said at the time:

You likely have already noticed, but traffic to rust-dev has decreased dramatically in recent months. This was a result of natural changes in project coordination at first, and then an intentional effort to phase out the list.In the beginning of the project there were only two places to talk about rust: #rust and rust-dev, and they served us well for all purposes for several years. As Rust grew though the coordination of the project moved to different venues, conversations were happening in a number of other places, and the purpose of rust-dev became less clear. At the same time there were also a number of heated and essentially unmoderatable discussions that caused many to lose faith in the effectiveness of the mailing list.

I love mailing lists, but I know I’m fairly unique in that. But they do have one big problem, which is that the tools for moderating them are extremely thin. You can pretty much ban someone, and that’s about it. No removing only certain posts, no cooldown periods, no shadow banning. While Rust has a generally positive reputation for community and moderation, that doesn’t mean things were always perfect. We learned some lessons the hard way.

More than that, mailman represents something of the “old guard” of open source. People see it as a sign of something backwards. Very few people want even more email. The culture is changing. We recognized this, and decided to move to Discourse instead. rust-dev was survived by two children, users and internals. This created a clear split between “discussion about Rust” and “discussions about building Rust.” It’s still free/open source software, it has good moderation tools, and you can even have it send you email!

We also don’t make release announcements on these forums; we do it on our blog.

Let’s dig into those notes!

900 changes, numerous bugfixes

900 changes is a lot! At the time, Rust was on a roughly six-month release schedule, so that’s about five patches merged per day. Rust 1.14 will have 1230 patches over its six-week release cycle: that’s just over 29 per day. That’s a much, much higher velocity. And that only counts the main repository; we’ve since added many more, like Cargo and crates.io. This is actually something I’ve been worried about for a while now; we only credit people for commits to rust-lang/rust in release announcements, basically due to this history, but there’s a lot more work that goes on here these days. I want Rails Contributors, but for the Rust project. Anyone want to work on it with me sometime?

Syntax changes

Removed <- move operator

Rust went through several iterations of “when do you move, when do you copy” over its lifetime. Today, that distinction is made between types that implement Copy and ones that don’t. But older Rusts had various schemes. Given that this one was removed in the first version of Rust I used, I don’t exactly remember this, but I think it was

x <- y; // move
x = y; // copy

Rust also had this at one point in time, I believe:

x = move y; // move
x = y; // copy

We decided to unify the two syntactically for two reasons: one, the annotations felt like busywork, the compiler would tell you what to put if you got it wrong. Secondly, at least in today’s Rust, move and copy are identical operations, with the exception of what you can do with the older variable after the move. Sharing the syntax reinforces that.

Niko’s blog has a lot of great history in it, and this topic is no exception.

Completed transition from #fmt extension syntax to fmt!

Long ago, Rust had a “no keywords can have more than five letters” rule, and so many things were abbreviated. This particular extension lives on today, but with the longer name format!.

Removed old fixed length vector syntax - [T]/N

Rust’s vector and array types, as we know them today, went through a lot of internal iteration, across many axes. The moral equivalent of [T]/N today is [T; N], but I’m sure there are/were lots of subtle differences between the two. I can’t quite remember.

New token quasi-quoters, quote_tokens!, quote_expr!, etc.

These live on in some sense; they’re still not stable, so much so that they’re note even shown in the official docs. Manish hosts a copy though. These are tools for “syntax extensions”, Rust’s most powerful form of metaprogramming. A final design for them was just accepted eight days ago, so it’ll be a while still before you see this on stable Rust.

Macros may now expand to items and statements

I don’t know why macros couldn’t do this before, but they still can today, and it’s very useful.

a.b() is always parsed as a method call, never as a field

This is a fun edge case. Here’s the problem:

struct Env<F: Fn(i32)> {    f: F,}let e = Env { f: |i| println!("Hello, {}", i) };e.f(); // what does this do?

According to this entry, this would be a method call. In today’s Rust, this would be an error, you need to write (e.f)(); At least the error tells you exactly what to do!

error: no method named `f` found for type `Env<[closure@<anon>:6:22: 6:50]>` in the current scope
 --> <anon>:8:7
  |
8 |     e.f(); // what does this do?
  |       ^
  |
note: use `(e.f)(...)` if you meant to call the function stored in the `f` field
 --> <anon>:8:7
  |
8 |     e.f(); // what does this do?
  |       ^

Eq and IterBytes implementations automatically generated with #[deriving_eq] and #[deriving_iter_bytes] respectively

We now have a more generic way to derive traits: #[derive(Eq)], for example. Eq is still here today, but IterBytes is not. I can’t recall exactly what it did.

Today, the list of traits you can derive is still only blessed ones by the compiler, and it’s a huge cause of people using nightly today, for better ergonomics with libraries like Serde and Diesel. But as of Rust 1.15, this restriction will be lifted, and one of the largest blockers of people using stable Rust will be eliminated! ???

Removed the special crate language for .rc files

The moral equivalent of .rc files today are .crate files, which Cargo will upload to crates.io. But unlike in the 0.5 days, you almost never need to worry or think about them, since Cargo makes it Just Work (tm).

Function arguments may consist of any irrefutable pattern

This is true today, and is a little known fact! It works like this:

struct Point {    x: i32,    y: i32,}fn takes_point(Point {x, y}: Point) {    println!("({}, {})", x, y);}fn main() {    let origin = Point { x: 0, y: 0 };    takes_point(origin);}

That is, arguments to functions are PATTERN: TYPE, not NAME: TYPE. Inside of takes_point, x and y are in scope, rather than a whole point.

Semantic changes

& and ~ pointers may point to objects

Rust did have objects, long ago, but I thought they were removed at this point. This one is unclear to me.

Worth mentioning that ~ is Box<T> now, though. That’s a long story…

Tuple structs - struct Foo(Bar, Baz). Replace newtype enums.

These are still in Rust today.

Enum variants may be structs

This is true today:

enum Foo {
    Variant { x: i32, y: i32 },
}

Destructors can be added to all nominal types via Drop trait

This is still true!

I think the “nominal” here is alluding to something older in Rust’s type system; we don’t make these kinds of distinctions today, as IIRC, all types are nominal. Not 100% sure.

Structs and nullary enum variants may be constants

This is true today.

Values that cannot be implicitly copied are automatically moved without writing move explicitly

I alluded to this above with <-.

&T may now be coerced to *T

*T is *const T or *mut T today, but this is still true:

let x = &5;
let y: *const i32 = x;

Coercions happen in let statements as well as function calls

We’ve gone through many different iterations of “what coerces and what doesn’t” over the years; I actually expected the example with *const i32 above to require an as.

use statements now take crate-relative paths

This is true today. I really like it, but a lot of people find it confusing. Basically, “use” always starts from the root of the crate. So

mod bar {    struct Foo;    mod baz {        use bar::Foo; // from the root        use super::Foo; // if you want to start from the parent        use self::super::Foo; // self makes it be relative. Not even sure if self + super works, but you get the idea.    }}

The module and type namespaces have been merged so that static method names can be resolved under the trait in which they are declared

Not sure on this one, it’s too in the weeds.

Improved support for language features

Trait inheritance works in many scenarios

This has always been a bit of a misnomer, inheritance means:

trait Foo: Bar {
}

If you implement Foo, you must also implement Bar. That’s it. This line makes me smile: “it works more often.”

Support for explicit self arguments in methods - self, &self @self, and ~self all generally work as expected

Today’s Rust doesn’t just support this; it’s required! However:

self is still self
&self is still &self
@self is Rc<self>, sorta kinda.
~self is Box<self>

@self was supposed to be a garbage-collected type, but was never really more than refcounting.

In addition, the last two don’t actually work like that: they would be self: Rc<Self> and self: Box<self>. The Box version will compile today, but the Rc version does not, because Box is magical. Eventually this will be rectified.

Static methods work in more situations

Another “yay more features work” line. :)

Experimental: Traits may declare default methods for impls to use

This works today, and is very useful. Consider Iterator: all of those methods have default implementations, so you only need to define next(), and you get the rest automatically.

Libraries

New condition handling system in core::condition

Conditions were a Lisp way of handling errors that Rust supported for a while. They were really cool, but nobody really knew how to effectively use them, so they never got used, so they’re gone now.

Timsort added to std::sort

std::sort doesn’t exist anymore, though we do have a sort function on slices. We don’t declare it to have a particular algorithm, though we do say “This sort is stable and O(n log n) worst-case but allocates approximately 2 * n where n is the length of self.”

Recently it was improved a ton, by a first-time contributor, no less! Extremely well-done. In the PR, he explains the algorithm:

However, if we take the main ideas from TimSort (intelligent merging strategy of sorted runs) and drop galloping, then we’ll have great performance on random inputs and it won’t be bad on partially sorted inputs either.

TimSort is still around, kicking.

New priority queue, std::priority_queue

Today this is std::collections::binary_heap.

Pipes for serializable types, `std::flatpipes’

No idea where this went.

Serialization overhauled to be trait-based

This is true today. See the above comments about Serde, though. rustc-serialize got to cheat though, and the compiler understands RustcEncodable and RustcDecodable. Can’t wait for Rust 1.15.

Expanded getopts definitions

We’ve moved getopts out of tree, but it’s still there.

Moved futures to std

Hoo boy! I forgot that we used to have futures in the standard library. Futures are one of the hottest topics in Rust today. Tokio is one of the most anticipated releases of all time in Rust world. For a taste, watch this talk. A little birdie told me that Tokio should have a 0.1 release soon…

More functions are pure now

Rust no longer has a notion of purity, though const fn (still unstable) sorta feels like it in some senses. For some history on purity in Rust, see Niko’s blog, or this explanation by Graydon on why it was removed.

core::comm renamed to oldcomm. Still deprecated

Very, very long gone.

rustdoc and cargo are libraries now

Rustdoc still exists, though not really as a library. Cargo however, is not the cargo you’re thinking of. It’s a totally different one. Rust went through many iterations of package managers before coming up with today’s Cargo; there was also rustpkg, for example, which was after the Cargo referenced here, but before the Cargo we’re using today.

Misc

Added a preliminary REPL, rusti

We removed the repl from the tree, as it never really worked, and was a maintenance nightmare. This one is the only one I know of today. Some people ask about one, but nobody has done the work to get a good one together yet. It’s tough!

License changed from MIT to dual MIT/APL2

This is still true today, with no plans on changing.

Contributors to Rust 0.5:

We had 41 people contribute to 0.5. 1.14 will have 144 contributors, and that’s around average for a release these days. Out of those 41 people, at a glance, I recognize the names of about 17 individuals, and roughly six of them are still involved with Rust and Servo today. What about the other eleven? Some were interns that got jobs at other places and can’t work on Rust anymore, some are contributors that left for various reasons, and then of course, there’s Graydon.

I personally opened my first pull request six days after discovering it. You’ll note this PR wasn’t merged due to a procedural issue: I sent it into the wrong branch! Today, GitHub lets you change this, but I had to open a new PR instead.

As I said in that PR:

I’ve just started using Rust, and I really like it. One of the things that makes Rust hard to use is its lack of documentation, which is obviously fine, given that Rust isn’t production-ready yet.I’d like to help change that.To get started, I’ve just modified this one little description in the rustdoc. If this works out, and you all like my changes, I’ll start working on more docs like this for the rest of core, and maybe the stdlib after that.

You might say it did work out. I’m now on the core team, I’ve sent in 866 PRs since then, and that first patch landed in Rust 0.6. Rust 0.10 is the only release since then that I’ve not gotten at least one patch into, and today, I write the release announcements.

I hope you’ve enjoyed this little trip down memory lane! I’m still really loving Rust and its community. Here’s to many more years ?♥

What's new with "The Rust Programming Language"

Mon, 15 Aug 2016 00:00:00 GMT

It’s been a while since I’ve mentioned my work on “The Rust Programming Language”, and I wanted to give you all an update. Before we get into the news, though, I’d like to take a moment to reflect on where the book has been, and then we’ll talk about it’s future.

I previously wrote about my history with Rust and docs. As I said back then:

But my first area of focus will be on the area of Rust’s documentation that’s the most weak, and simultaneously the most important: the tutorial.

“The tutorial” eventually turned into “The Rust Programming Language.” But there’s a funny thing about writing a book for a language that isn’t done yet: you don’t really know how to write a good book on something that doesn’t exist, because, well, you haven’t played with it yourself. I could make decent guesses, given that Rust was evolving incrementally, but that’s still very different. I think “technical debt” is a great metaphor here: the evolution of the language meant that I had to keep up with the work of many, many people to try and update the documentation. Furthermore, since nobody had tried to learn Rust 1.0 yet, since it didn’t exist, I couldn’t write a book that would do an amazing job of teaching them, since I hadn’t used it either!

In some ways, this mirrors the compiler itself, which accrued a massive amount of technical debt until 1.0. And so, the story since 1.0 has largely about been paying off this debt; culminating in MIR. By the same token, since 1.0, a lot of people have read the book, and a lot of people have tried to learn Rust. I now know so much more about how people go about doing so, and so am much, much better situated to write something that’ll be great for them.

And so, now that time has passed, re-writing the book is what I’ve been spending a lot of my time on. I’ve been mostly quiet about this work, since I’ve wanted to get a chunk finished before starting to talk about it, and so today is finally that day.

Let’s talk about what’s going on with the book today.

A new coauthor.

The first bit of news is something that I’ve been wanting to tell you all for a few weeks now, and is the bit that I’m most excited about: I’m bringing Carol (Nichols || Goulding) on as a coauthor. TRPL is an incredibly personal work for me, one that I’ve poured my life into over the last few years. Carol and I have been friends for a long time, and there’s nobody else I’d rather share an authorship with. Not only is Carol a friend, but she’s been doing Rust longer than most. You might know her from Rustlings, or the Rust Belt Rust conference, or as a cofounder of Integer32, the first Rust-focused consulting company. Carol knows her stuff, and I’m proud to have her name next time mine on the cover. The book will be far better through her influence.

A new beginning

One of Carols’ first tasks was taking a bunch of the stuff I’d already written and checking it out herself, making improvements. She’s taken the first six chapters of the book and whipped them into shape. Some of the content you might recognize from the first edition of TRPL, but much of it is brand-new.

As an example of how much better this book is, can, and will be than the first edition, compare the Ownership chapter to the current one ( plus borrowing ). Instead of an abstract set of rules, we took one of the biggest things facing a new Rustacean, strings, and used them as a way to make these concepts concrete.

If you have the time, please check it out, and file issues with any suggestions!

More to come

If you’d like to see chapters as we work on them, just follow the repo: everything is open source. Initial PRs are already up for the start of the next three chapters, and hopefully we’ll get them merged and move forward soon. I’m really proud of how the book is shaping up, and hope you enjoy it too.

Structure literals vs. constructors in Rust

Tue, 26 Jul 2016 00:00:00 GMT

Learning the basics of a language and its syntax is easy. Learning how all those bits fit together is a bit harder. There’s a neat intersection between three of Rust’s features that I’ve seen people use, but never seen written down. I was explaining this technique to someone in #rust-beginners the other day, and thought I’d write it down in case it helps you, too.

A small review: If you have a struct in Rust, like this:

struct Point {
    x: i32,
    y: i32,
}

You can use ‘struct literal syntax’ to create a new instance of the struct:

let origin = Point { x: 0, y: 0 };

However, this syntax only works if you have the proper access to the struct and its members via Rust’s privacy rules.

mod foo {
    pub struct Point {
        x: i32,
        y: i32,
    }

    pub fn foo(x: i32, y: i32) -> Point {
        Point { x: x, y: y } // this is fine, as we're in the same module
    }
}

fn main() {
    let origin = foo::Point { x: 0, y: 0 }; // this is not fine
}

We can’t use struct literal synatx in main because x and y are also not public. But within the same module, we have access, so it works. So how would we let main instantiate a Point if we can’t use the literal syntax? Well, our foo function does this, so we could expose it. More conventionally, we’d make it an associated function and call it new:

mod foo {
    pub struct Point {
        x: i32,
        y: i32,
    }
    
    impl Point {
        pub fn new(x: i32, y: i32) -> Point {
            Point { x: x, y: y } // this is fine, as we're in the same module
        }
    }
}

fn main() {
    let origin = foo::Point::new(0, 0);
}

Great. But what if we wanted x and y to be public for some reason, yet we still wanted to force people to use the new function to create Points? Maybe the initial creation does some sort of side effect that’s important. If we changed our code to this:

mod foo {
    pub struct Point {
        pub x: i32,
        pub y: i32,
    }
    
    impl Point {
        pub fn new(x: i32, y: i32) -> Point {
            Point { x: x, y: y } // this is fine, as we're in the same module
        }
    }
}

fn main() {
    let origin = foo::Point::new(0, 0);

    // but so does this:
    let origin = foo::Point { x: 0, y: 0 };
}

By making all of the elements of Point public, we’ve re-enabled the literal syntax, which isn’t what we wanted. So what do we do?

Fixing this requires two insights. The first is “zero-sized types”. In Rust, certain types only have values that don’t require any storage. Take, for example, (), the “unit” type. It only has one possible value, also (). Since it only has one value, there’s no need to actually store anything in memory to represent (); if we have a valid value, we already know what it is. That means that once we compile down to actual assembly, () just goes away entirely. So we can do this:

mod foo {
    pub struct Point {
        pub x: i32,
        pub y: i32,
        _secret: (),
    }
    
    impl Point {
        pub fn new(x: i32, y: i32) -> Point {
            Point { x: x, y: y, _secret: () }
        }
    }
}

fn main() {
    let origin = foo::Point::new(0, 0);
}

Now, we have a new, non-public field, _secret. I’ve given it a name that starts with an underscore because we don’t intend to use it for anything, and so Rust won’t warn us about it. _secret has the type of (), and so it’s entirely a compile-time construct; it doesn’t materially affect Point’s representation. But it being private does affect how we’re allowed to construct our Points. main can no longer use the struct literal syntax, since not all of the fields are public.

However, remember that privacy is a module-level thing in Rust. Therefore, we can still use the struct literal syntax inside of the foo module:

mod foo {
    pub struct Point {
        pub x: i32,
        pub y: i32,
        _secret: (),
    }
    
    impl Point {
        pub fn new(x: i32, y: i32) -> Point {
            Point { x: x, y: y, _secret: () }
        }
    }

    fn foo() -> Point {
        Point: { x: 0, y: 0, _secret: () } // this is still allowed!
    }
}

fn main() {
    let origin = foo::Point::new(0, 0);
}

To prevent foo from being able to use the literal syntax, we need one more concept: pub use. Check this out:

mod foo {
    mod point {
        pub struct Point {
            pub x: i32,
            pub y: i32,
            _secret: (),
        }
    
        impl Point {
            pub fn new(x: i32, y: i32) -> Point {
                Point { x: x, y: y, _secret: () }
            }
        }
    }

    pub use foo::point::Point;

    fn foo() -> Point {
        Point::new(0, 0) // must use `new` here, as we're no longer in the same module!
    }
}

fn main() {
    let origin = foo::Point::new(0, 0);
}

By giving Point its own module, everything that’s private to it is private. But typing foo::point::Point would be redundant and an ergonomic regression; pub use saves the day! We re-export the Point structure into foo, so we can still use foo::Point, but since one of its members is private, literal syntax isn’t allowed.

To me, understanding things like this is when I really start to feel like I’m getting to know a language: putting three or four disparate concepts together to achieve some goal. It’s when a language stops being a bunch of disjoint parts and starts becoming a cohesive whole.

"The Rust Programming Language" will be published by No Starch Press

Tue, 08 Sep 2015 00:00:00 GMT

I’m happy to bring you some news: “The Rust Programming Language” is going to be published, in physical form, by No Starch Press. I am really excited to work with No Starch on this: they’ve published some of my favorite programming books, like “If Hemingway Wrote JavaScript”, “Ruby Under a Microscope”, and “Ruby Wizardry”.

Another reason I’m keen to work with No Starch on this is that they believe in publishing freely available material: we’re going to take the existing manuscript, whip it into shape, make a nice paper book out of it, and merge the changes back into the official documentation. So the improvements will get back to everyone, even those who can’t afford a physical copy.

The authorship credit is to “The Rust Project Developers,” as other people have contributed significantly: we even have an active PR to re-write an entire chapter. Rust is a team effort.

Oh, and one last thing: the profit from the book won’t go to me, it will go to OpenHatch, “a non-profit dedicated to matching prospective free software contributors with communities, tools, and education,” to use their words about it.

I’ll let you know when we’re closer to actually shipping!

Community Versions for Rust

Sun, 23 Aug 2015 00:00:00 GMT

Rust has been through a lot of changes in recent years. As such, tutorials get out of date. We’ve occasionally said that including rustc -v in your blog posts would be helpful, but today, I saw this tweet go by:

Tired of guessing what versions of Ember work with a given blog post? Check out http://t.co/6joJiNdJQ5 #feedbackplz pic.twitter.com/YToBduBVtx — mixonic (@mixonic) August 23, 2015

After some quick hacking, I’ve modified it to work with Rust. You can find the project here: https://github.com/steveklabnik/rust-community-versions

TL;DR: Put this into your blog post, substituting the appropriate date and title:

<iframe
  width="178" height="24" style="border:0px"
  src="http://steveklabnik.github.io/rust-community-versions/2013/10/18/pointers-in-rust-a-guide.html">
</iframe>

Then, make a pull request adding a new post with some front-matter:

---
layout: post
url: /writing/pointers-in-rust-a-guide/
title: "Pointers in Rust: a Guide"
date: 2013-10-18
start_version: 0.8
end_version: 0.9
---

(end_version is optional)

And bam! You get a nice looking badge. Check out out on this blog post of mine: /writing/pointers-in-rust-a-guide/

What’s nice about this approach is that, if your post ever gets out of date, someone else can modify the YAML in the repo, and your badge will automatically update to include the end date. No remembering to go back and fix it.

Let’s see how this goes!

A new introduction to Rust

Fri, 27 Feb 2015 00:00:00 GMT

Lately, I’ve been giving a lot of thought to first impressions of Rust. On May 15, we’re going to have a lot of them. And you only get one chance at a first impression. So I’ve been wondering if our Intro and Basics are putting our best foot forward. At first I thought yes, but a few days ago, I had an idea, and it’s making me doubt it, maybe. So instead of re-writing all of our introductory material, I’m just going to write the first bit. A spike, if you will. And I’d like to hear what you think about it. This would take the same place as 2.4: Variable bindings in the existing structure: They’ve installed Rust and gotten Hello World working.

$ rustc --version
rustc 1.0.0-dev (dcc6ce2c7 2015-02-22) (built 2015-02-22)

Hello, Ownership

Let’s learn more about Rust’s central concept: ownership. Along the way, we’ll learn more about its syntax, too. Here’s the program we’re going to talk about:

fn main() {
    let x = 5;
}

This small snippit is enough to start with. First up: let. A let statement introduces a variable binding. Bindings allow you to associate a name with some sort of value.

Why ‘variable binding’? Rust draws heavily from both systems languages and functional programming languages. The name “variable binding” is a great example of this. Many systems languages let you declare a variable. These variables are called by that name because they can change over time, they’re mutable. Many functional languages let you declare bindings. These bindings are called by that name because they bind a name to a value, and don’t change over time. They’re immutable.

Rust’s variable bindings are immutable by default, but can become mutable, allowing them to be re-bound to something else. In other words,

fn main() {
    let x = 5;
    x = 6; // error: re-assignment of immutable variable

    let mut y = 5;
    y = 6; // just fine
}

You won’t be typing mut that often.

In any case, there’s one way in which let bindings work just like variables in other languages, but they’re the key insight into ownership. As you know, a computer program is executed line by line. At least, until you hit a control flow structure, anyway. Let’s give our program line numbers, so we can talk about it.

// 1
fn main() {    // 2
               // 3
    let x = 5; // 4
               // 5
}              // 6
               // 7

Line one is before our program starts. The endless void. Not actually endless, though, as line two is where we start main. This is the first line that is actually executed, and kicks off our program. Great. Line three is blank, so nothing happens, just like one. Line four is where the first actually interesting thing occurs: we introduce a new variable binding, x. We set x’s initial value to five, allocated on the stack. If you don’t know what that means, we’ll talk about it right after ownership. For now, x is five. No big deal. Line six has a closing curly brace, and so main, and thus, our program, is over. Line seven, the void returns.

This is basically the same thing as many programming languages. But let’s point out an aspect you, as a programmer, probably take for granted. Scoping. If I asked you, “Is x valid on line one?” you would say “no.” “Three? Seven?” “Nope, nada. x is valid from line four, where it was declared, to line six, where it goes out of scope.” This illustrates the idea. There is a certain scope, a certain set of lines, where x is a valid identifier. That scope starts from where it was declared, and goes until the end of the block. We can look at this scope in two ways: for the first, imagine this source code printed on a piece of paper. You highlight lines four three six. In some sense, this is a distance: three lines of code, rather than three meters. But if we imagine the computer running this program, this scope represents a time: three statements of processor execution. Even though that number is actually different based on the assembly, but at our level of abstraction, three units.

In Rust, we have names for these concepts, which are implicit in other languages. The thing that introduces a new scope is called the ‘owner.’ It’s in charge of the data it’s bound to, so we say that it ‘owns’ that data. The length of a scope is called a ‘lifetime,’ taken from that idea of time passing as your program executes. But you can also think of it as a segment of lines of code.

So if other programs do this, why does this make Rust special? Sure, Rust gives these concepts specific names, but names themselves aren’t significantly different. The difference is that Rust takes this concept and cranks it up to 11. Where most programming languages only keep track of how long variables are in scope, Rust knows how to connect the scopes of variables that are pointing to the same thing, as well as how to know the scope of things that are more than just stack-allocated memory.

Let’s talk more about this connecting of scopes. Here’s another Rust program:

fn main() {         // 1
    let x = 5;      // 2
                    // 3
    {               // 4
                    // 5
        let y = &x; // 6
                    // 7
    }               // 8
                    // 9
}                   // 10

In line four, we use an open curly brace to create a new scope. This scope, like in many languages needs to be closed before the scope of main gets closed. It’s nested inside of it.

In this new scope, on line six, we declare a new binding, y, and we set it equal to &x. This reads as “a reference to x,” as & is the reference operator in Rust. References are Rust’s version of ‘pointers’ from other systems programming languages. Here’s a quick introduction, if you haven’t had to deal with pointers before.

By default, in Rust, data is allocated on “the stack.” Your program is given a chunk of memory, and it’s able to allocate information there. The stack starts at the bottom address of memory, and then grows upwards. For example, at the start of our program, the stack has nothing in it:

Untitled

At line2, we introduce x, and it stack-allocates the number five, like this:

Untitled

We’ve got an address in memory, the value at that address, and the name we’ve used in our program. It starts at the bottom because remember, the stack grows upward. Finally,on line six, we have y:

Untitled

Instead of being a value itself, y is a pointer to another location in memory, which holds a value. Because y = &x, we take the memory location of x and store it into y.

On line eight, the inner scope ends, and y goes out of scope. Its lifetime is over. It gets deallocated. Our memory is back to this:

Untitled

Finally, on line ten, our program is over. x goes out of scope, and we look like this:

Untitled

Everything is gone, all is well.

Pointers can point other places than the stack: they can also point to ‘the heap.’ The heap starts at the top of your program’s memory, and grows down:

Untitled

The heap starts at 0x00 and grows down, and the stack starts at 0xff and grows up. While the stack contains the data that the programming language has allocated for our bindings, our program can request dynamic amounts of memory from the heap. Here’s how to do this in C:

#include<stdio.h>
#include<stdlib.h>

int main() {
        int *x = malloc(sizeof(int)); // 1
        *x = 5;                       // 2
                                      // 3
        printf("%i", *x);             // 4
}

The call to malloc (‘memory allocate’) on line 1 requests some memory from the heap. We get a pointer, int *x, to that memory. Our memory looks like this:

Untitled

We have x, which is a variable on the stack, as a pointer to our dynamically allocated memory, which is located on the heap at 0x00. But at that location, we haven’t done anything, so who knows what the value is? This means that after line one is done executing, x will be pointing to garbage memory. Hopefully we set it equal to a value before we try to use it, or bad things could happen.

On line two, we set that memory location to five. Whew! Memory looks like this:

Untitled

Everything looks good. On line four, we have to print out *x, not x: x is the address of the memory, so we’d get 0x00 instead of 5. After line four, x goes out of scope, and gets deallocated. Memory looks like this:

Untitled

… wait a minute, we never took care of that 5! We left some memory ‘dangling.’ In this program, that’s okay, because now that it’s over, the operating system reclaims all this memory, and so we don’t have any problems. But we do have a bug. We just don’t know it. This bug will sit there until something changes, and the bug is found. But we’ll get to that.

A slightly more responsible program looks like this:

#include<stdio.h>
#include<stdlib.h>

int main() {
        int *x = malloc(sizeof(int));
        *x = 5;

        printf("%i", *x);

        free(x);
}

The call to free releases the memory pointed to by x, and all is good.

Let’s compare this to an equivalent Rust program:

fn main() {
    let x = Box::new(5);

    println!("{}", x);
}

It looks similar at first, with Box::new filling in for malloc, But where’s the free? Does this have the same bug?

No. This Rust is in fact equivalent to the one with the malloc and free. You see, when we introduce our binding, x, Rust knows that x has ownership over the memory in the Box. and so, at the end of the program, when x goes out of scope, Rust also ensures to free the memory that the Box allocated. It happens automatically. The programmer can’t forget it. And the allocation itself was a little easier, because Rust used the type system to figure out how many bytes to allocate, you didn’t need to calculate it with sizeof. Allocation also set the value to 5 for you, which is convenient and less error-prone. When printing out x, you could just use the binding name itself, and not need to dereference it with *.

C++ programmers have ways of doing this kind of thing, with something called “RAII.” As promised before, Rust knows this concept at a deeper, language level. So it can be safe in places that RAII cannot. Here’s an example, in C++:

#include<iostream>
#include<vector>
#include<string>

int main() {
    std::vector<std::string> v;

    v.push_back("a");

    std::string& x = v[0];

    v.push_back("b");

    std::cout << x;
}

This program creates a vector of strings, called v:

Untitled

v is a ‘fat pointer,’ three pointers in one. The first is a pointer to the data, the second is the length of the array, how many elements are in it. The third is the capacity of the array, how much memory is allocated for this vector. The amount of initial capacity is up to the implementation, I’ve chosen one here. Because this is a vector of strings, we need to allocate a single string, so in line with our capacity. Strings are also a fat pointer, and so we allocate three bytes, starting at 0x00.

The program then calls push_back() on the vector, which pushes a string onto the end of the vector:

Untitled

We’ve now replaced our initial, GARBAGE, string with a real one. Therefore, our data, length, and capacity are updated: 0x03 is the next free location in the heap, so it gets our signle character, A. We have a length of one, and a capacity of one, so the other two values are set appropriately.

Next, it creates a pointer to the 0th element of that vector:

Untitled

x is allocated on the stack, and points to the first element of v, which, as we know, is located at address 0x00, as its data pointer, 0xff, shows us. So x has the initial value of 0x00.

Next is where everything goes wrong. It tries to push a string onto the end of that vector. The problem is, we only allocated one element. So we need to allocate more. After this is done, memory looks like this:

Untitled

Inside push_back(), there’s a call to malloc, and that is assigned to some temporary value. We’ve chosen 0xfb and (temp) to identify it here. It points to the newly allocated data, at 0x04.

Now that we’ve got space for two strings, let’s copy the data from the original place to the new one, from 0x00 to 0x04:

Untitled

and then free that original string:

Untitled

Then, we have to update v to point at the newly allocated vector, at 0x04:

Untitled

This step is where the bug is! You see, x was pointing at the same data that v was pointing at. This is called ‘aliasing.’ Because C++ does not have the concept of ownership and lifetimes, it doesn’t understand that the validity of x and the validity of v are closely intertwined. Thus, we’re allowed to do this reallocate-and-copy dance with what v points to, but x has no idea. Look at the diagram above: it still has x pointing to 0x00. But we just deallocated that memory! This is when the bug occurs, but it’s a silent bug. A time bomb, ticking away.

Let’s see how it explodes. Our code goes on, oblivious.

Our new string at 0x04 still says that we have one capacity, one length, and that data is at 0x03. That’s all still accurate, so we’re all good. Next, we need to deal with our new string, “B.” We have the space, so we do so:

Untitled

We’ve placed our new string at 0x07 to 0x09, with its data pointer, length, and capacity. Our data is at 0x0A, which has the value B stored at it. All is good, and so we return from push_back. (temp) is deallocated, and memory looks like this:

Untitled

On the final line, we print out the value that x points to. As we can see above, x points to 0x00, which has nothing saved at it! x is invalid memory. Using x is a bad idea. But the programming language can’t save us, even though the bug happened many lines ago.

How are you supposed to deal with this problem? Here’s the documentation for push_back:

If the new size() is greater than capacity() then all iterators and references (including the past-the-end iterator) are invalidated.

In other words, when you call push_back, it’s your responsibility to make sure to not use anything that points to the underlying data, because they are now invalid. The documentation has explained our bug perfectly! And while this is a short example, these problems can crop up in huge codebases, and the errors can be very hard to track down.

Let’s look at the same problem, in Rust:

fn main() {
    let mut v = vec![];

    v.push("Hello".to_string());

    let x = &v[0];

    v.push("world".to_string());

    println!("{}", x);
}

This program has identical semantics to the C++ version presented above. (If we weren’t trying to get the exact same semantics, we could leave off the .to_string() calls, and it would still work. But let’s ignore that for now.) So does it have the same bug? Well, it would, but it doesn’t compile:

main.rs:8:5: 8:6 error: cannot borrow `v` as mutable because it is also borrowed as immutable
main.rs:8     v.push("world");
              ^
main.rs:6:14: 6:15 note: previous borrow of `v` occurs here; the immutable borrow prevents subsequent moves or mutable borrows of `v` until the borrow ends
main.rs:6     let x = &v[0];
                       ^
main.rs:11:2: 11:2 note: previous borrow ends here
main.rs:1 fn main() {
...
main.rs:11 }
           ^

When you call the method push() on a vector, it ‘borrows’ the vector. Just like we wrote let y = &x before, to borrow the value of x. Rust now understands that the scope of x and the scope of v are connected. There’s just one problem: because push allocates more memory, it can’t just do a regular borrow, it needs a mutable borrow. If &x borrows x, &mut x mutably borrows x. Like we discussed previously, mutability means that you can reassign the value of that binding. But when we mutably borrow something, we make a guarantee: this is the only reference to this memory that exists. That way, we don’t have the problem of the C++ code, where a reference became outdated. If we’re the only reference, we are free to change what we’re pointing to, becuase that won’t cause any problems.

So when we create a borrow with &, it lasts for the scope that it exists, like this:

fn main() {
    let mut v = vec![];

    v.push("A");   

    let x = &v[0];     // <+
                       //  |
    v.push("B");       //  |
                       //  |
    println!("{}", x); //  |
}                      // <+

The line indicates the scope, or lifetime, of the borrow. But when we call push with B, we have to get a mutable borrow, in order to allocate the new memory. But because &mut is a promise that there are no other references, this is a violation of the rules. This is the point at which the bug happened in the C++ example, if you remember. And so it’s the place where the Rust compiler throws its error message. It might make more sense this time around:

main.rs:8:5: 8:6 error: cannot borrow `v` as mutable because it is also borrowed as immutable
main.rs:8     v.push("world");
              ^
main.rs:6:14: 6:15 note: previous borrow of `v` occurs here; the immutable borrow prevents subsequent moves or mutable borrows of `v` until the borrow ends
main.rs:6     let x = &v[0];
                       ^
main.rs:11:2: 11:2 note: previous borrow ends here
main.rs:1 fn main() {
...
main.rs:11 }
           ^

We cannot borrow v as mutable, with push(), becuase it is also borrowed as immutable, by x. That’s exactly the problem, and Rust solves it.

There’s more to both ownership and Rust syntax, so let’s cover some syntactical things, and then talk more about ownership.

I’m fairly confident with the memory layouts involved, though the addresses are of course not exact. I’m also not sure that this mega deep detail dive is the correct way to do an introduction either, it’s also thorough, but maybe in a bad way? It’s almost the exact opposite of the way that I’ve done things at present. Also, apparently I visuallize things in reverse, with 0x00 to 0xFF as going down, but others think the opposite, which may be confusing. Please send me a tweet or an email with your thoughts. Thanks!

Rust's documentation is about to drastically improve

Mon, 16 Jun 2014 00:00:00 GMT

Historically, Rust has had a tough time with documentation. Such a young programming language changes a lot, which means that documentation can quickly be out of date. However, Rust is nearing a 1.0 release, and so that’s about to change. I’ve just signed a contract with Mozilla, and starting Monday, June 23rd, I will be devoting forty hours per week of my time to ensuring that Rust has wonderful documentation.

A year and a half ago, I was in my hometown for Christmas. I happened upon a link: Rust 0.5 released. I’ve always enjoyed learning new programming languages, and I had vaguely heard of Rust before, but didn’t really remember what it was all about. So I dug in. I loved systems programming in college, but had done web-based stuff my entire professional life, and hadn’t seriously thought about pointers as part of my day-to-day in quite some time.

There was just one problem: Rust was really difficult to learn. I liked what I saw, but there was very little of it. At the same time, I had been working on some ideas for a new toolchain for my book on hypermedia APIs, but wanted to try it out on something else before I took the time to port the content. And so, Rust for Rubyists was born. I decided that the best way to teach people Rust was to mimic how I learned Rust. And so, as I learned, I wrote. It ended up at about fifty pages of two weeks of work. I never contributed it to the main repository, because for me, it was really about learning, both about my ebook toolchain as well as Rust itself. I didn’t want the burden that came with writing an official tutorial, making sure that you cover every single feature, pleasing every single Github contributor…

After learning Rust, I decided that I really liked it. No other language provided such a promising successor to C++. And I really missed low-level programming. So I kept evangelizing Rust, and every so often, contributing official documentation. I figured that even if my low-level chops weren’t up to the task of writing actual patches, I could at least help with my writing skills. I’d previously contributed lots of documentation to Ruby and Rails, so it was something that was very familiar to me. I’ve often found that I start with documentation and then move into contributing code, once I get my head wrapped around everything. Writing is part of my own process of understanding.

Rust for Rubyists was a great hit, even amongst non-Rubyists (damn my love of alliteration!). Six months ago, on the eve of the first anniversary of the initial release of Rust for Rubyists, I gave a talk at the Bay Area Rust meetup, specifically on the state of Rust’s documentation. In it, I laid out a plan for how I envisioned docs looking in the future. In the last six months, a lot has improved, but a lot hasn’t. But no more! I’m now going to be able to allocate a significant amount of my time on getting documentation done.

I’m also pleased in a meta-sense. You see, by contracting someone to work on documentation full-time, Mozilla is indicating that they take Rust and its future very seriously. You can (and I do) talk a lot of trash on Microsoft, but one of the reasons that the Microsoft platform is used by so many people around the world is that Microsoft products often have excellent documentation. I often find that open source ‘products’ are technically superior, but are often much harder to use, because they’re built by a community, for free, and very few people want to write documentation for free. Combined with the work that Tilde is doing on Cargo, Mozilla is investing a significant amount of effort and dollars into ensuring that Rust will be a fantastic platform for those developing on it. Since I love Rust, this makes me very, very happy.

Forty hours a week is a significant amount of documentation, and I have a lot of work in front of me. But my first area of focus will be on the area of Rust’s documentation that’s the most weak, and simultaneously the most important: the tutorial. I tackled the first tip of that iceberg with my 30 minute introduction, and I’d like to tweak it too. The main tutorial, however, is the first place where people go to really learn about Rust and how it works. And the current tutorial is largely the same as the 0.5 days, back when I first learned Rust. It suffers from receiving patchwork contributions from a variety of people, rather than having one unifying vision. It’s also much more of a list of features and how to use them than a coherent tutorial.

I’m really excited to get to work. Let’s all make Rust 1.0 a fantastic release.

A 30 minute introduction to Rust

Sat, 11 Jan 2014 00:00:00 GMT

I recently gave a proposal for Rust’s documentation. An important component of my overall proposal is a short, simple introduction for people who may have heard of Rust, so that they can figure out if Rust is right for them. The other day, I saw this excellent presentation, and thought it might serve as a great basis for this introduction. Consider this an RFC for such an introduction. Feedback very welcome on the rust-dev mailing list or on Twitter.

This tutorial has now become the official one.

Rust is a systems programming language that focuses on strong compile-time correctness guarantees. It improves upon the ideas other systems languages like C++, D, and Cyclone by providing very strong guarantees and explicit control over the life cycle of memory. Strong memory guarantees make writing correct concurrent Rust code easier than in other languages. This might sound very complex, but it’s easier than it sounds! This tutorial will give you an idea of what Rust is like in about thirty minutes. It expects that you’re at least vaguely familiar with a previous ‘curly brace’ language. The concepts are more important than the syntax, so don’t worry if you don’t get every last detail: the tutorial can help you out with that later.

Let’s talk about the most important concept in Rust, “ownership,” and its implications on a task that programmers usually find very difficult: concurrency.

Ownership

Ownership is central to Rust, and is one of its more interesting and unique features. “Ownership” refers to which parts of your code are allowed to modify various parts of memory. Let’s start by looking at some C++ code:

int *dangling(void)
{
    int i = 1234;
    return &i;
}

int add_one(void)
{
    int *num = dangling();
    return *num + 1;
}

This function allocates an integer on the stack, and stores it in a variable, i. It then returns a reference to the variable i. There’s just one problem: stack memory becomes invalid when the function returns. This means that in the second line of add_one, num points to some garbage values, and we won’t get the effect that we want. While this is a trivial example, it can happen quite often in C++ code. There’s a similar problem when memory on the heap is allocated with malloc (or new), then freed with free (or delete), yet your code attempts to do something with the pointer to that memory. More modern C++ uses RAII with constructors/destructors, but it amounts to the same thing. This problem is called a ‘dangling pointer,’ and it’s not possible to write Rust code that has it. Let’s try:

fn dangling() -> &int {
    let i = 1234;
    return &i;
}

fn add_one() -> int {
    let num = dangling();
    return *num + 1;
}

When you try to compile this program, you’ll get an interesting (and long) error message:

temp.rs:3:11: 3:13 error: borrowed value does not live long enough
temp.rs:3     return &i;

temp.rs:1:22: 4:1 note: borrowed pointer must be valid for the anonymous lifetime #1 defined on the block at 1:22...
temp.rs:1 fn dangling() -> &int {
temp.rs:2     let i = 1234;
temp.rs:3     return &i;
temp.rs:4 }
                            
temp.rs:1:22: 4:1 note: ...but borrowed value is only valid for the block at 1:22
temp.rs:1 fn dangling() -> &int {      
temp.rs:2     let i = 1234;            
temp.rs:3     return &i;               
temp.rs:4  }                            
error: aborting due to previous error

In order to fully understand this error message, we need to talk about what it means to “own” something. So for now, let’s just accept that Rust will not allow us to write code with a dangling pointer, and we’ll come back to this code once we understand ownership.

Let’s forget about programming for a second and talk about books. I like to read physical books, and sometimes I really like one and tell my friends they should read it. While I’m reading my book, I own it: the book is in my possession. When I loan the book out to someone else for a while, they “borrow” it from me. And when you borrow a book, it’s yours for a certain period of time, and then you give it back to me, and I own it again. Right?

This concept applies directly to Rust code as well: some code “owns” a particular pointer to memory. It’s the sole owner of that pointer. It can also lend that memory out to some other code for a while: the code “borrows” it. It borrows it for a certain period of time, called a “lifetime.”

That’s all there is to it. That doesn’t seem so hard, right? Let’s go back to that error message: error: borrowed value does not live long enough. We tried to loan out a particular variable, i, using Rust’s borrowed pointers: the &. But Rust knew that the variable would be invalid after the function returns, and so it tells us that: borrowed pointer must be valid for the anonymous lifetime #1... but borrowed value is only valid for the block. Neat!

That’s a great example for stack memory, but what about heap memory? Rust has a second kind of pointer, a ‘unique’ pointer, that you can create with a ~. Check it out:

fn dangling() -> ~int {
    let i = ~1234;
    return i;
}

fn add_one() -> int {
    let num = dangling();
    return *num + 1;
}

This code will successfully compile. Note that instead of a stack allocated 1234, we use an owned pointer to that value instead: ~1234. You can roughly compare these two lines:

// rust
let i = ~1234;

// C++
int *i = new int;
*i = 1234;

Rust is able to infer the size of the type, then allocates the correct amount of memory and sets it to the value you asked for. This means that it’s impossible to allocate uninitialized memory: Rust does not have the concept of null. Hooray! There’s one other difference between this line of Rust and the C++: The Rust compiler also figures out the lifetime of i, and then inserts a corresponding free call after it’s invalid, like a destructor in C++. You get all of the benefits of manually allocated heap memory without having to do all the bookkeeping yourself. Furthermore, all of this checking is done at compile time, so there’s no runtime overhead. You’ll get (basically) the exact same code that you’d get if you wrote the correct C++, but it’s impossible to write the incorrect version, thanks to the compiler.

You’ve seen one way that ownership and lifetimes are useful to prevent code that would normally be dangerous in a less-strict language, but let’s talk about another: concurrency.

Concurrency

Concurrency is an incredibly hot topic in the software world right now. It’s always been an interesting area of study for computer scientists, but as usage of the Internet explodes, people are looking to improve the number of users a given service can handle. Concurrency is one way of achieving this goal. There is a pretty big drawback to concurrent code, though: it can be hard to reason about, because it is non-deterministic. There are a few different approaches to writing good concurrent code, but let’s talk about how Rust’s notions of ownership and lifetimes can assist with achieving correct but concurrent code.

First, let’s go over a simple concurrency example in Rust. Rust allows you to spin up ‘tasks,’ which are lightweight, ‘green’ threads. These tasks do not have any shared memory, and so, we communicate between tasks with a ‘channel’. Like this:

fn main() {
    let numbers = [1,2,3];

    let (port, chan)  = Chan::new();
    chan.send(numbers);

    do spawn {
        let numbers = port.recv();
        println!("{:d}", numbers[0]);
    }
}

In this example, we create a vector of numbers. We then make a new Chan, which is the name of the package Rust implements channels with. This returns two different ends of the channel: a channel and a port. You send data into the channel end, and it comes out the port end. The spawn function spins up a new task. As you can see in the code, we call port.recv() (short for ‘receive’) inside of the new task, and we call chan.send() outside, passing in our vector. We then print the first element of the vector.

This works out because Rust copies the vector when it is sent through the channel. That way, if it were mutable, there wouldn’t be a race condition. However, if we’re making a lot of tasks, or if our data is very large, making a copy for each task inflates our memory usage with no real benefit.

Enter Arc. Arc stands for ‘atomically reference counted,’ and it’s a way to share immutable data between multiple tasks. Here’s some code:

extern mod extra;
use extra::arc::Arc;

fn main() {
    let numbers = [1,2,3];

    let numbers_arc = Arc::new(numbers);

    for num in range(0, 3) {
        let (port, chan)  = Chan::new();
        chan.send(numbers_arc.clone());

        do spawn {
            let local_arc = port.recv();
            let task_numbers = local_arc.get();
            println!("{:d}", task_numbers[num]);
        }
    }
}

This is very similar to the code we had before, except now we loop three times, making three tasks, and sending an Arc between them. Arc::new creates a new Arc, .clone() makes a new reference to that Arc, and .get() gets the value out of the Arc. So we make a new reference for each task, send that reference down the channel, and then use the reference to print out a number. Now we’re not copying our vector.

Arcs are great for immutable data, but what about mutable data? Shared mutable state is the bane of the concurrent programmer. You can use a mutex to protect shared mutable state, but if you forget to acquire the mutex, bad things can happen.

Rust provides a tool for shared mutable state: RWArc. This variant of an Arc allows the contents of the Arc to be mutated. Check it out:

extern mod extra;
use extra::arc::RWArc;

fn main() {
    let numbers = [1,2,3];

    let numbers_arc = RWArc::new(numbers);

    for num in range(0, 3) {
        let (port, chan)  = Chan::new();
        chan.send(numbers_arc.clone());

        do spawn {
            let local_arc = port.recv();

            local_arc.write(|nums| {
                nums[num] += 1
            });

            local_arc.read(|nums| {
                println!("{:d}", nums[num]);
            })
        }
    }
}

We now use the RWArc package to get a read/write Arc. The read/write Arc has a slightly different API than Arc: read and write allow you to, well, read and write the data. They both take closures as arguments, and the read/write Arc will, in the case of write, acquire a mutex, and then pass the data to this closure. After the closure does its thing, the mutex is released.

You can see how this makes it impossible to mutate the state without remembering to aquire the lock. We gain the efficiency of shared mutable state, while retaining the safety of disallowing shared mutable state.

But wait, how is that possible? We can’t both allow and disallow mutable state. What gives?

A footnote: unsafe

So, the Rust language does not allow for shared mutable state, yet I just showed you some code that has it. How’s this possible? The answer: unsafe.

You see, while the Rust compiler is very smart, and saves you from making mistakes you might normally make, it’s not an artificial intelligence. Because we’re smarter than the compiler, sometimes, we need to over-ride this safe behavior. For this purpose, Rust has an unsafe keyword. Within an unsafe block, Rust turns off many of its safety checks. If something bad happens to your program, you only have to audit what you’ve done inside unsafe, and not the entire program itself.

If one of the major goals of Rust was safety, why allow that safety to be turned off? Well, there are really only three main reasons to do it: interfacing with external code, such as doing FFI into a C library, performance (in certain cases), and to provide a safe abstraction around operations that normally would not be safe. Our Arcs are an example of this last purpose. We can safely hand out multiple references to the Arc, because we are sure the data is immutable, and therefore it is safe to share. We can hand out multiple references to the RWArc, because we know that we’ve wrapped the data in a mutex, and therefore it is safe to share. But the Rust compiler can’t know that we’ve made these choices, so inside the implementation of the Arcs, we use unsafe blocks to do (normally) dangerous things. But we expose a safe interface, which means that the Arcs are impossible to use incorrectly.

This is how Rust’s type system allows you to not make some of the mistakes that make concurrent programming difficult, yet get the efficiency of languages such as C++.

That’s all, folks

I hope that this taste of Rust has given you an idea if Rust is the right language for you. If that’s true, I encourage you to check out the tutorial for a full, in-depth exploration of Rust’s syntax and concepts.

Rust is surprisingly expressive

Sat, 28 Dec 2013 00:00:00 GMT

Do you remember the first time you saw Rails’ ActiveSupport library? I do: it totally blew my mind. The only dynamic language I had used up to that point was Perl, and it had a similarly mind-blowing effect on my young self. I thought that dynamic typing was mostly a requirement towards making this kind of expressiveness possible, but it turns out Rust can be just as expressive, while retaining type safety and static dispatch.

There was a time in the early days of Rails when its evangelists would show off snippets of what it let you do. These were often ActiveSupport extensions to the core language itself. It’d go something like this:

Hey, have you tried Rails? Check this out:

irb(main):002:0> Time.now - 2.days
=> 2009-12-26 09:57:02 -0800
irb(main):003:0> 2.days.ago
=> 2009-12-26 09:57:04 -0800

Did I just blow your mind???

At the time, I enthusiastically replied “Yes, yes you did!” and immediately typed rails new foo. And all was well and good in the world.

Fast forward a few years. I now know that static typing is not equivalent to public static void main(String [] args) {. I’ve seen the Hindley-Milner light. Yet, for all practical purposes, I’ve still been hacking in Ruby for these past few years. For the most part, I’d found the complaints about dynamic typing from the static typing camp to be very FUD-y. I very rarely get TypeErrors in my Ruby code. Refactoring generally isn’t very difficult.

But slowly, this has started to change. I think that it’s that I tend to do more infrastructure and plumbing work now. I’ve run into more and more situations where types would be helpful, or picking the mutable state path has caused headaches.

Then, I wake up to an IM from Yehuda:

bro, did you see my tweets?

Basically, this:

➜  active_support git:(master) cat main.rs
extern mod active_support;
use active_support::Period;
use active_support::Time;

fn main() {
  let time = Time::now();
  println!("{:?}", time);
  println!("{:?}", 2.days().from_now());
  println!("{:?}", 2.weeks().from_now());
  println!("{:?}", 2.months().from_now());
  println!("{:?}", 2.years().from_now());
}
➜  active_support git:(master) ./main 
active_support::time::Time{date: active_support::date::Date{year: 2013, month: 12u, day: 28u}, hours: 0u, minutes: 59u, seconds: 6u, nanos: 242647081u}
active_support::time::Time{date: active_support::date::Date{year: 2013, month: 12u, day: 30u}, hours: 0u, minutes: 59u, seconds: 6u, nanos: 243287552u}
active_support::time::Time{date: active_support::date::Date{year: 2014, month: 1u, day: 11u}, hours: 0u, minutes: 59u, seconds: 6u, nanos: 243347757u}
active_support::time::Time{date: active_support::date::Date{year: 2013, month: 2u, day: 28u}, hours: 0u, minutes: 59u, seconds: 6u, nanos: 243388962u}
active_support::time::Time{date: active_support::date::Date{year: 2015, month: 12u, day: 28u}, hours: 0u, minutes: 59u, seconds: 6u, nanos: 243427393u}

Whaaaat? Yup. To compare side-by-side:

# Ruby
2.days.from_now
// Rust
2.days().from_now()

Awesome. Please note that in Rust, this is fully type-checked, safe, and statically dispatched. No slow reflection techniques used here!

Now, I should say that I almost never use 2.days, but the point is that it’s a benchmark of expressiveness: if you can write it, you can do all sorts of other flexible things. Thinking back to my ‘libraries vs. application code’ confusion over types, I’m reminded by something Gary Bernhardt once said about Ruby vs. Python. I’m paraphrasing, but as I remember it:

Python is a beautiful, clean language. But the same restrictions that make it nice and clean mean that it’s hard to write beautiful, clean libraries. Ruby, on the other hand, is a complicated, ugly language. But that complexity allows you to write really clean, nice, easy-to-use libraries. The end result is that I prefer Python the language, but I find myself preferring programming with Ruby.

I’m starting to wonder if maybe I feel the same way about static vs. dynamic typing: I like the simplicity and the speed of programming in a dynamic language, but maybe, given the right static language, I’ll end up preferring to program in $LANGUAGE better.

And that language might just be Rust.

You can check out Yehuda’s rust-activesupport on GitHub, if you’d like to see how this is accomplished. He’s also told me that he will write a post explaining its implementation in technical detail, I’ll link to it here when he’s written it.

Pointers in Rust, a guide

Fri, 18 Oct 2013 00:00:00 GMT

Rust’s pointers are one of its more unique and compelling features. Pointers are also one of the more confusing topics for newcomers to Rust. They can also be confusing for people coming from other languages that support pointers, such as C++. This tutorial will help you understand this important topic.

You don’t actually need pointers

I have good news for you: you probably don’t need to care about pointers, especially as you’re getting started. Think of it this way: Rust is a language that emphasizes safety. Pointers, as the joke goes, are very pointy: it’s easy to accidentally stab yourself. Therefore, Rust is made in a way such that you don’t need them very often.

“But tutorial!” you may cry. “My co-worker wrote a function that looks like this:

fn succ(x: &int) -> int { *x + 1 }

So I wrote this code to try it out:

fn main() {
    let number = 5;
    let succ_number = succ(number);
    println!("{}", succ_number);
}

And now I get an error:

error: mismatched types: expected `&int` but found `<VI0>` (expected &-ptr but found integral variable)

What gives? It needs a pointer! Therefore I have to use pointers!”

Turns out, you don’t. All you need is a reference. Try this on for size:

fn main() {
    let number = 5;
    let succ_number = succ(&number);
    println!("{}", succ_number);
}

It’s that easy! One extra little & there. This code will run, and print 6.

That’s all you need to know. Your co-worker could have written the function like this:

fn succ(x: int) -> int { x + 1 }

fn main() {
    let number = 5;
    let succ_number = succ(number);
    println!("{}", succ_number);
}

No pointers even needed. Then again, this is a simple example. I assume that your real-world succ function is more complicated, and maybe your co-worker had a good reason for x to be a pointer of some kind. In that case, references are your best friend. Don’t worry about it, life is too short.

However.

Here are the use-cases for pointers. I’ve prefixed them with the name of the pointer that satisfies that use-case:

Owned: ~Trait must be a pointer, becuase you don’t know the size of the object, so indirection is mandatory.
Owned: You need a recursive data structure. These can be infinite sized, so indirection is mandatory.
Owned: A very, very, very rare situation in which you have a huge chunk of data that you wish to pass to many methods. Passing a pointer will make this more efficient. If you’re coming from another language where this technique is common, such as C++, please read “A note…” below.
Managed: Having only a single owner to a piece of data would be inconvenient or impossible. This is only often useful when a program is very large or very complicated. Using a managed pointer will activate Rust’s garbage collection mechanism.
Borrowed: You’re writing a function, and you need a pointer, but you don’t care about its ownership. If you make the argument a borrowed pointer, callers can send in whatever kind they want.

Five exceptions. That’s it. Otherwise, you shouldn’t need them. Be skeptical of pointers in Rust: use them for a deliberate purpose, not just to make the compiler happy.

A note for those proficient in pointers

If you’re coming to Rust from a language like C or C++, you may be used to passing things by reference, or passing things by pointer. In some languages, like Java, you can’t even have objects without a pointer to them. Therefore, if you were writing this Rust code:

struct Point {
    x: int,
    y: int,
}

fn main() {
    let p0 = Point { x: 5, y: 10};
    let p1 = transform(p0);
    println!("{:?}", p1);
}

I think you’d implement transform like this:

fn transform(p: &Point) -> Point {
    Point { x: p.x + 1, y: p.y + 1}
}

// and change this:
let p1 = transform(&p0);

This does work, but you don’t need to create those references! The better way to write this is simply:

struct Point {
    x: int,
    y: int,
}

fn transform(p: Point) -> Point {
    Point { x: p.x + 1, y: p.y + 1}
}

fn main() {
    let p0 = Point { x: 5, y: 10};
    let p1 = transform(p0);
    println!("{:?}", p1);
}

But won’t this be inefficent? Well, that’s a complicated question, but it’s important to know that Rust, like C and C++, store aggregate data types ‘unboxed,’ whereas languages like Java and Ruby store these types as ‘boxed.’ For smaller structs, this way will be more efficient. For larger ones, it may be less so. But don’t reach for that pointer until you must! Make sure that the struct is large enough by performing some tests before you add in the complexity of pointers.

Owned Pointers

Owned pointers are the conceptually simplest kind of pointer in Rust. A rough approximation of owned pointers follows:

Only one owned pointer may exist to a particular place in memory. It may be borrowed from that owner, however.
The Rust compiler uses static analysis to determine where the pointer is in scope, and handles allocating and de-allocating that memory. Owned pointers are not garbage collected.

These two properties make for three use cases.

References to Traits

Traits must be referenced through a pointer, becuase the struct that implements the trait may be a different size than a different struct that implements the trait. Therefore, unboxed traits don’t make any sense, and aren’t allowed.

Recursive Data Structures

Sometimes, you need a recursive data structure. The simplest is known as a ‘cons list’:

enum List<T> {
    Nil,
    Cons(T, ~List<T>),
}
    
fn main() {
    let list: List<int> = Cons(1, ~Cons(2, ~Cons(3, ~Nil)));
    println!("{:?}", list);
}

This prints:

Cons(1, ~Cons(2, ~Cons(3, ~Nil)))

The inner lists must be an owned pointer, becuase we can’t know how many elements are in the list. Without knowing the length, we don’t know the size, and therefore require the indirection that pointers offer.

Efficiency

This should almost never be a concern, but because creating an owned pointer boxes its value, it therefore makes referring to the value the size of the box. This may make passing an owned pointer to a function less expensive than passing the value itself. Don’t worry yourself with this case until you’ve proved that it’s an issue through benchmarks.

For example, this will work:

struct Point {
    x: int,
    y: int,
}

fn main() {
    let a = Point { x: 10, y: 20 };
    do spawn {
        println(a.x.to_str());
    }
}

This struct is tiny, so it’s fine. If Point were large, this would be more efficient:

struct Point {
    x: int,
    y: int,
}

fn main() {
    let a = ~Point { x: 10, y: 20 };
    do spawn {
        println(a.x.to_str());
    }
}

Now it’ll be copying a pointer-sized chunk of memory rather than the whole struct.

Managed Pointers

Managed pointers, notated by an @, are used when having a single owner for some data isn’t convenient or possible. This generally happens when your program is very large and complicated.

For example, let’s say you’re using an owned pointer, and you want to do this:

struct Point {
    x: int,
    y: int,
}
    
fn main() {
    let a = ~Point { x: 10, y: 20 };
    let b = a;
    println(b.x.to_str());
    println(a.x.to_str());
}

You’ll get this error:

test.rs:10:12: 10:13 error: use of moved value: `a`
test.rs:10     println(a.x.to_str());
                       ^
test.rs:8:8: 8:9 note: `a` moved here because it has type `~Point`, which is moved by default (use `ref` to override)
test.rs:8     let b = a;
                  ^

As the message says, owned pointers only allow for one owner at a time. When you assign a to b, a becomes invalid. Change your code to this, however:

struct Point {
    x: int,
    y: int,
}
    
fn main() {
    let a = @Point { x: 10, y: 20 };
    let b = a;
    println(b.x.to_str());
    println(a.x.to_str());
}

And it works:

10
10

So why not just use managed pointers everywhere? There are two big drawbacks to managed pointers:

They activate Rust’s garbage collector. Other pointer types don’t share this drawback.
You cannot pass this data to another task. Shared ownership across concurrency boundaries is the source of endless pain in other languages, so Rust does not let you do this.

Borrowed Pointers

Borrowed pointers are the third major kind of pointer Rust supports. They are simultaneously the simplest and the most complicated kind. Let me explain: they’re called ‘borrowed’ pointers because they claim no ownership over the data they’re pointing to. They’re just borrowing it for a while. So in that sense, they’re simple: just keep whatever ownership the data already has. For example:

use std::num::sqrt;

struct Point {
    x: float,
    y: float,
}

fn compute_distance(p1: &Point, p2: &Point) -> float {
    let x_d = p1.x - p2.x;
    let y_d = p1.y - p2.y;

    sqrt(x_d * x_d + y_d * y_d)
}

fn main() {
    let origin = @Point { x: 0.0, y: 0.0 };
    let p1     = ~Point { x: 5.0, y: 3.0 };

    println!("{:?}", compute_distance(origin, p1));
}

This prints 5.83095189. You can see that the compute_distance function takes in two borrowed pointers, but we give it a managed and unique pointer. Of course, if this were a real program, we wouldn’t have any of these pointers, they’re just there to demonstrate the concepts.

So how is this hard? Well, because we’re igorning ownership, the compiler needs to take great care to make sure that everything is safe. Despite their complete safety, a borrowed pointer’s representation at runtime is the same as that of an ordinary pointer in a C program. They introduce zero overhead. The compiler does all safety checks at compile time.

This theory is called ‘region pointers,’ and involve a concept called ‘lifetimes’. Here’s the simple explanation: would you expect this code to compile?

fn main() {
    println(x.to_str());
    let x = 5;
}

Probably not. That’s becuase you know that the name x is valid from where it’s declared to when it goes out of scope. In this case, that’s the end of the main function. So you know this code will cause an error. We call this duration a ‘lifetime’. Let’s try a more complex example:

fn main() {
    let mut x = ~5;
    if(*x < 10) {
        let y = &x;
        println!("Oh no: {:?}", y);
        return;
    }
    *x = *x - 1;
    println!("Oh no: {:?}", x);
}

Here, we’re borrowing a pointer to x inside of the if. The compiler, however, is able to determine that that pointer will go out of scope without x being mutated, and therefore, lets us pass. This wouldn’t work:

fn main() {
    let mut x = ~5;
    if(*x < 10) {
        let y = &x;
        *x = *x - 1;

        println!("Oh no: {:?}", y);
        return;
    }
    *x = *x - 1;
    println!("Oh no: {:?}", x);
}

It gives this error:

test.rs:5:8: 5:10 error: cannot assign to `*x` because it is borrowed
test.rs:5         *x = *x - 1;
                  ^~
test.rs:4:16: 4:18 note: borrow of `*x` occurs here
test.rs:4         let y = &x;
                          ^~

As you might guess, this kind of analysis is complex for a human, and therefore hard for a computer, too! There is an entire tutorial devoted to borrowed pointers and lifetimes that goes into lifetimes in great detail, so if you want the full details, check that out.

Returning Pointers

We’ve talked a lot about funtions that accept various kinds of pointers, but what about returning them? Here’s the rule of thumb: only return a unique or managed pointer if you were given one in the first place.

What does that mean? Don’t do this:

fn foo(x: ~int) -> ~int {
    return ~*x;
}

fn main() {
    let x = ~5;
    let y = foo(x);
}

Do this:

fn foo(x: ~int) -> int {
    return *x;
}

fn main() {
    let x = ~5;
    let y = ~foo(x);
}

This gives you flexibility, without sacrificing performance. For example, this will also work:

fn foo(x: ~int) -> int {
    return *x;
}

fn main() {
    let x = ~5;
    let y = @foo(x);
}

You may think that this gives us terrible performance: return a value and then immediately box it up?!?! Isn’t that the worst of both worlds? Rust is smarter than that. There is no copy in this code. main allocates enough room for the @int, passes it into foo as x, and then foo writes the value into the new box. This writes the return value directly into the allocated box.

This is important enough that it bears repeating: pointers are not for optimizing returning values from your code. Allow the caller to choose how they want to use your output.

Rust-y Scraps: iterating with a step

Thu, 30 May 2013 00:00:00 GMT

This blog post works with Rust v 0.6, and may change in the future :).

A common need in various algorithms is to iterate with some sort of step. A simple way to do this in Rust is with unit::range_step:

let nums = [1,2,3,4,5,6,7];
for uint::range_step(0, bytes.len() - 1, 2) |i| {
    println(fmt!("%d & %d", nums[i], nums[i+1]));
}

This yields

1 & 2
3 & 4
5 & 6

Of course, this is awkward: I wish I had some code to do something like:

let nums = [1,2,3,4,5,6,7];

for nums.in_groups |i, j| {
  println(fmt!("%d & %d", i, j));
}

But I think this is just out of my abilities. The issue is that the closure needs to take the same number of arguments as the step, and I’m not sure that I have the Rust-fu to make it work yet.

After a discussion in the IRC room, Rust doesn’t have the Rust-fu yet either; varargs aren’t there, so this style is a no-go for now.

Oh, and since there were no examples of this yet, I added some to the docs.

Upgrading from Rust 0.5 to 0.6

Sun, 07 Apr 2013 00:00:00 GMT

Here are some error messages that I got when updating Rust for Rubyists from Rust 0.5 to the new 0.6, and how to fix those errors.

Error:

rust.rs:1:4: 1:16 error: unresolved name
rust.rs:1 use task::spawn;
              ^~~~~~~~~~~~
rust.rs:1:4: 1:16 error: failed to resolve import: task::spawn
rust.rs:1 use task::spawn;
              ^~~~~~~~~~~~
error: failed to resolve imports
error: aborting due to 3 previous errors
make: *** [all] Error 101

rust.rs:1:4: 1:16 error: unresolved name
rust.rs:1 use io::println;
              ^~~~~~~~~~~~
rust.rs:1:4: 1:16 error: failed to resolve import: io::println
rust.rs:1 use io::println;
              ^~~~~~~~~~~~
error: failed to resolve imports
error: aborting due to 3 previous errors
make: *** [all] Error 101

The fix:

Imports are now more explicit, so you can’t just import things from core without specifying that you want to any more.

+ use core::task::spawn;- use task::spawn;

+ use core::io::println;- use io::println;

etc.

Error:

error: main function not found
error: aborting due to previous error

The fix:

When I used to write TDD’d Rust, I wouldn’t often write a main function until I had done a bunch of tests. This was okay, but now, you need one.

+ fn main() {
+ }

Error:

rustc rust.rs --test
rust.rs:5:2: 5:6 error: unresolved name: `fail`.
rust.rs:5   fail;
            ^~~~
error: aborting due to previous error
make: *** [build_test] Error 101

The fix:

fail was turned into a macro. It now needs to be passed an owned pointer to a string:

- fail;
+ fail!(~"Fail!");

Error:

rust.rs:5:18: 5:26 error: unresolved name: `int::str`.
rust.rs:5           println(int::str(num))
                            ^~~~~~~~
error: aborting due to previous error
make: *** [build] Error 101

The fix:

int::str is now int::to_str

- int::str
+ int::to_str

Error:

rust.rs:5:12: 5:18 error: cannot determine a type for this local variable: unconstrained type
rust.rs:5     let mut answer;
                      ^~~~~~
error: aborting due to previous error
make: *** [build] Error 101

The fix:

You have to tell it what kind it is. In my case, it was a string:

- let mut answer;
+ let mut answer = "";

Error:

rust.rs:3:11: 3:21 error: expected `;` or `}` after expression but found `is_fifteen`
rust.rs:3     assert is_fifteen(15)
                     ^~~~~~~~~~
make: *** [build_test] Error 101

The fix:

This happens because assert is now a macro.

- assert is_fifteen(15)
+ assert!(is_fifteen(15))

Error:

rustc rust.rs
rust.rs:11:10: 11:24 error: the type of this value must be known in this context
rust.rs:11 chan.send(10);
^~~~~~~~~~~~~~
error: aborting due to previous error
make: *** [build] Error 101

The fix:

Rust 0.6 got rid of ‘capture clauses.’ Don’t worry about it.

- do spawn |chan| {
+ do spawn {

Error:

rust.rs:10:17: 10:21 error: expected `,` but found `chan`
rust.rs:10   do spawn |move chan| {
                            ^~~~
make: *** [build] Error 101

The fix:

move is gone. Just remove it.

- do spawn |move chan| {
+ do spawn |chan| {

Note that this diff wouldn’t actually make it work, you’d still run into the issue above. But it gets rid of the move error.

Error:

rust.rs:13:6: 13:12 error: unexpected token: `static`
rust.rs:13       static fn count() {
                 ^~~~~~
make: *** [build] Error 101

The fix:

static was removed. Any method that doesn’t take self is static.

- static fn count() {
+ fn count() {

Error:

rust.rs:11:30: 11:37 error: obsolete syntax: colon-separated impl syntax

The fix:

As it says, the colon syntax is gone. Replace it with for:

- impl float: Num
+ impl Num for float

Error:

rust.rs:5:18: 5:22 error: use of undeclared type name `self`
rust.rs:5       fn new() -> self;
                            ^~~~

The fix:

This happens when you’re making a trait, and you want the implementations to return whatever type they are. The type name is now Self:

- fn new() -> self;
+ fn new() -> Self;

I hope you found that informative! Happy hacking!

Steve Klabnik — Rust

Arguing about arguments

Named parameters

Optional and/or Default arguments

Function Overloading

These features make me uneasy

I’m okay with named parameters now

There’s tons of practical issues though

In conclusion

Thirteen years of Rust and the birth of Rue

Is Rust faster than C?

Inline Assembly

Similar code, different results

Social Factors

Compile time vs runtime?

My Conclusions

Rust 1.0, ten years later

Thinking like a compiler: places and values in Rust

Lexical Analysis (aka ‘scanning’ or ‘tokenizing’)

Syntactic Analysis (aka ‘parsing’)

Semantic Analysis (aka ‘wtf does this mean’)

So… how do I understand this code?

Statements

Expressions

Places and Values

Deref

Conclusion

Does unsafe undermine Rust's guarantees?

Memory safe languages and their implementations

Unsafety in implementation

Extending languages and unsafety

So how is this memory safe?

Programs, properties, and proofs

Aristotelian syllogisms

Logic and programs

Hoare Logic

Separation Logic

Global vs local analysis

Global analysis

Local analysis

Composability of proofs

Rustbelt

Weaknesses

The theoretical vs the empirical

When things go wrong

How can we check that things haven’t gone wrong?

Do things go wrong?

A Happy Day for Rust

When should I use String vs &str?

Level 1: Don’t think about it at all

Level 2: prefer &str for function parameters

Level 3: return &str sometimes

Level 4: When to use &str in a struct

Memory Safety is a Red Herring

ThreadId(1)

ThreadId(2)

ThreadId(3)

ThreadId(4)

ThreadId(5)

ThreadId(6)

threads.into_iter().for_each(|t| t.join().unwrap());

int atexit(void (*func)(void))

Updating Buck

Updating Buck

Update Reindeer

Update the prelude

Fix some configuration

Regenerating reindeer

Fixing some breakage

Wrapping up

Using Crates.io with Buck

Depending on semver

How Cargo handles this

The mismatch

Reindeer

Using buck to build Rust projects

Getting started with buck2

Initial project files

.buckconfig

.gitmodules & prelude

Level 2: prefer `&str` for function parameters

Level 3: return `&str` sometimes

Level 4: When to use `&str` in a struct

`.buckconfig`

`.gitmodules` & `prelude`

`.gitignore`

`BUCK`

`toolchains/BUCK`

`macro_rules!`

`macro`