As for Rust ’s relevance to Influx, I have a dream of creating a Rust -based implementation of Flux (our new scripting & query language) that uses the C++ Apache Arrow libraries, that would be embeddable in other systems (like Spark, Kafka Streams or other places). So those are what motivated me to finally take the leap, but I soon found out that those wouldn’t be the only things that make Rust a compelling language to work with.
I’ve written an interpreter before, I’m very familiar with Go, and the nice thing about this project is that it limits the scope of what I’d need to learn. Thorsten’s book comes with all the code and extensive tests so it’s easy to make sure things are correct.
I started out by reading The Rust Programming Language, which is part of the free online documentation. Reading through this particular book made me appreciate early one of the Rust community’s strengths: documentation is built into everything.
Docs for the standard library are online, or you can bring them up locally with a single command (something that’s great when you’re learning on a plane). The other amazing thing about the documentation in comments is that the examples you put into your docs will actually get built during testing, so code in documentation never falls out of sync with the actual library definitions (example from my project).
These little touches all combine to build what I think is a very solid foundation for future Rust library authors, and more importantly, users. The order you implement everything is this: first the lexer, then the parser, then the interpreter, and then go back to add features to the language in all three areas.
Converting the lexer over to Rust was a fairly straightforward process and didn’t present too many challenges beyond initial struggles with the borrow checker and compiler. Since a big part of learning a new thing is rote memorization, the mechanical aspects of building a lexer and parser are actually a good repetitive task to start hammering the syntax and vocabulary into your brain.
It’s also fun and satisfying to create a test, see it fail, then write a bit of code to make it pass. Specifically, I had to figure out how to do recursion and a nested tree structure (the AST) without having the Rust borrow checker yell at me.
I Googled around a bit and re-read some sections of the book, but I obviously needed a deeper understanding of the borrow checker. I also read other people’s accounts of “hitting a wall” in the process of learning Rust, so I figured extra effort would lead me to scaling it.
Contrary to popular belief, I don’t think great programmers have some innate ability that makes learning to code easy. I think they just fight through the hurdles in search of those few moments when things “just work” and you feel a deep sense of satisfaction.
My sense is that if you’re going to learns, you need to prepare yourself for this level of effort and frustration, but if you do, I think the payoff is worth it. This one covers some in-depth stuff about how memory is organized and frequently references C++ code in comparison, but knowledge of C++ isn’t a prerequisite for getting significant value out of it.
I tried to stick pretty close to Thorsten’s Go implementation, but I did bring in some Rust specific things. As you can see, I kept a pretty faithful port with the function names and basic code structure being largely the same.
The experienced Rust developers will probably notice my inconsistent use of moves or borrows when passing a token to a method (I should refactor). I think Rust has the most elegant patterns for error handling of any language I’ve worked with.
I suppose I did take a little time to learn how to implement local scopes and closures in the language. I had to pick up RC (a reference counted pointer) and Recall (for dynamically mutating state in the Environment associated with a function).
Although this design led to memory leaks in my implementation, that’s because it creates a cycle of reference counted structs due to the Monkey language’s closures, so they don’t get freed up. I’m wondering if there’s a way to design and structure the code around this or if I need to implement a basic garbage collector.
I’ve opened up an issue to discuss a GC for Monkey Rust if anyone is up for giving me a few pointers in the right direction. Load balancers, proxies, operating systems, databases, network queues, distributed systems, machine learning libraries, the underlying implementation for higher level languages, and probably countless other things that are not coming immediately to mind.
Contrast this to Rust, which has significantly more syntax, a model for working with memory management that few if any programmers are familiar with, and a compiler that can be more strict than even the worst disciplinarian. However, there are compelling advantages that Rust can boast, which I think warrant the initial learning curve.
We’ve had a number of production bugs in Influx due to data races that were only caught under heavy load. With Rust, the compiler forces you to “do the right thing”, which is great because then you won’t have to worry about it slipping past a code review.
In Rust there are entire classes of bugs that are simply impossible to create because of the compile time guarantees. This is the payoff, and despite the strictness of the compiler, Rust ’s bet is that if you learn the way you can be as productive (or more) than with another language.
I have more miles to travel on my Rust journey before I can validate Bland and Sendoff’s proposition, but I intend to give it real effort. After that I’ll pick up a small prototype project to create a web service that uses a C++ storage library.
After all of this, I’ll be in a good place to make real efforts on some projects I envision for Rust at Influx. I think an embeddable implementation of Flux would be a fantastic thing to create in Rust.
I think there’s a very real possibility that mountains of C and C++ code gets rewritten over the next fifty years leading to more secure systems' software built around Rust ’s guarantees. And if Rust ’s radical wager turns out to be a winning bet, it’ll become my de facto first choice for servers and systems.
Indeed, when I learned it, I considered it to be the hardest programming language up to that time I’ve met. Looking back, I’m not sure I was correct, C++ is probably harder but it was distributed into much longer time than learning Rust.
Also, I’ll claim this upfront I don’t consider it necessarily a bad thing for a language to be hard to learn, not when I get something in return for the investment. On the other hand, if you declare upfront that your language needs to be able to solve any hard problem anyone thinks of, run fast and be safe to use then you’ll probably get your wish, but it’ll hardly be simple.
However, this is difficulty that anyone trying to enter the low-level world of system programming should expect. When the language designers want to solve a problem, there are two general approaches.
The other approach is being honest about the complexity of the problem, admit it, but arm the programmer with tools to handle it. This means the beginner must grasp the complete complexity of the problem, but makes it easier in the long term simply because after solving the problems, there’s no beginner anymore and because sometimes having the needed tools and not fighting the language to get to the problem is just easier.
One of the problems why I found Rust hard to learn was that it looked similar to other imperative languages on one side, but introduced a lot of novel concepts. Put some callbacks here and there… and then the compiler complained, because such model doesn’t fit.
I tried to force it and after a time I got a really hairy API I didn’t like a bit. It is not clear upfront the language is different from whatever else one might already know and that the other knowledge needs to be put on a shelf for a while.
There are many books about good design in the OOP world, many books and classes how to structure a program in pure functional world. Books about best practices for a world based on ownership and composition.
All things together, Rust insists that your program will be correct or it won’t compile. Strict typing makes you think about the relations in your program.
It makes sure your programs are better than of the competition (if you manage to compile them, of course), that they are less likely to crash and burn or something even worse. It makes the language easier to use (imagine if you had to actively think about all these things instead of relying on the compiler to check it for you welcome to C).
If you just learned about how to write a function, it doesn’t feel very encouraging when the compiler starts to talk about data races (what are these, anyway, you ask at the time). A program that is subtly wrong would be fine at that time at least it would run.
Today, there’s a lot of very good documentation about this it at least explains why such things are incorrect and how to fix them. The fact they learned in times when there was no or little documentation doesn’t change a thing here.
Some frustration that his or her programs don’t compile on the first (or tenth) attempt. Becoming a better programmer, even in other languages (for example, ownership is just formalization what most people do in languages with manual memory management anyway, but Rust makes sure it becomes second nature).