Cartesi - The Road to Descartes

cartesi (42)in #blog • 4 years ago (edited)

Cartesi’s Descartes SDK, just launched, is the first software development kit for DApps that exposes Cartesi’s core technology through a programming interface that DApp developers will find easy to use. With Descartes, DApp developers can quickly and easily understand what it means to augment their Ethereum contracts with gas-free, unlimited, unmetered, and Linux-based off-chain functions.

Cartesi’s journey, culminating so far in this landmark software release, has been a most interesting one. This article tells a story (from my particular point of view) of how the Descartes SDK came to be, and it also explores some of what lies beyond Descartes. In the future, we will publish articles that go deeper into the technical details of the Descartes architecture.

Cartesi’s Proof-of-concept: Creepts

In Q1 2020, Cartesi released Creepts, which was, and still is, one of the most advanced and ambitious DApps ever developed on Ethereum. The Creepts milestone included not only the release of all source code required to build all of Creepts, but also the hosting of several publicly playable versions of decentralized tower defense tournaments.

The release of Creepts proves that the Cartesi vision is real. Creepts is a complete and complex DApp that was co-evolved with the Cartesi Core, which is the stack of software libraries and engines that implements the mathematical magic at the heart of Cartesi, and that powers the Creepts code. Creepts would be impossible to do as a regular Ethereum DApp without Cartesi, as the gas cost would be astronomical and the Ethereum network would simply be unable to handle even a vanishing fraction of the total load of one game session.

Although Creepts worked and proved the real-world benefits for Linux-based off-chain computing in blockchain-based DApps, the entire software stack of Creepts was not easy to document, that is, to explain. The development process of Creepts was the development process of an application using solid base libraries, but no pre-existing middleware that had its own development process, with its own requirements. The intermediary components of the Creepts architecture are not optimized for reducing the cognitive load at the DApp layer, which is an explicit goal you need when developing middleware for mass adoption by the mainstream programmer.

Thus, in Q2 2020, as we were documenting the existing Cartesi API for external developers, we were faced with unexpected difficulties in explaining it to external developers. Somehow, the Cartesi SDK’s API works. It is powerful, flexible, customizable, and modular, with a pretty solid internal design that has its own clear evolutionary path. But when we try to put it in written words, into didactic material, then, suddenly, it looks like massive work, and it doesn’t quite click. So we had to stop for a bit and think about what was going on.

Too much genius

As a software developer, that is not the first time I have seen this phenomenon. That is a very particular and specific problem that may be quite common at the cutting edge of software engineering.

For some reason, I have always ended up working with people who are obscenely intelligent and far more capable than me: the professionals at the very edge of our industry. Maybe that is because I am attracted only by the very edge of possibility in software in general, and in distributed systems in particular.

It is entirely natural that the middleware may be the last component to appear when a stack is developed from scratch. First, you focus on the core, the foundation. Then, you feel the urge to stretch it all the way up, to the application. It is easy to imagine applications, and it is easy to know what the foundation is, but the middle parts are a mystery because there are several ways to tackle them, and the middle parts are where all of the configurations and domain-specific customization and all of the opinionated design decisions are made. Middlewares are generalizations of an application domain — that, in the case of the general domain of DApps, is still in a formative phase.

The middleware is the fork in the road. That’s where you decide who you are, as an organization, as a distributed systems project. Foundational components are mathematical challenges. Applications are communication challenges. And middleware is a combination of both. It is where the soft and the hard discipline meet, and where you need to perform alchemy, to combine them. A middleware picks and chooses from the array of advanced weaponry that’s made available by foundational components, and builds a battle plan to support a given class of applications and of development organizations: to fight a specific battle.

But the genuine mistaking of “middleware that works” with “middleware that is wieldable” is an unmissable hallmark of genius. Because, to the true genius, usability problems don’t exist, as everything appears easy to use or, at least, the complexity is sufficiently manageable. So, when a genius finds some software interface slightly inconvenient to use, it is time to wrap it with an “Easy API” or to scrap it, because odds are that regular people will not be able to use it!

If you think I’m exaggerating with this talk of “genius,” consider for a moment that the Cartesi company was founded by a couple of professors from the Institute for Pure and Applied Mathematics (IMPA), which is basically the Brazilian version of the Massachusetts Institute of Technology (MIT). And just like the MIT, the IMPA is also name-dropped in popular fiction. IMPA is mentioned in the first season of the Netflix series “Reality Z,” as a way to establish that a character is intelligent. So, yes, this is not actually hyperbole. It is an actual condition. And like with every other condition in the world, it is a buff, but also a nerf. A gift, and a curse. And if you go through the whole roster of Cartesi, the overall situation does not get any better.

From the Cartesi SDK to the Descartes SDK

So, the company decided that “the [Cartesi] SDK” would instead become a family of SDKs, each with its selection of foundational and middleware components to serve a specific applicability profile. And the first one would be baptized after mathematician and philosopher René Descartes, after which the company is named.

Although the Cartesi SDK as it exists in the open-source repositories today can be adopted and integrated by any well-funded company that wishes to build a state-of-the-art product on top of Cartesi, it is not really accessible to mainstream development efforts. Thus, the Descartes SDK offering is laser-focused on accessibility, on being immediately learnable and useful. And the usual tradeoff for developer accessibility is made: the middleware layer encroaches significantly less into the application space, leaving more things to be done by the application (or higher-level middleware) programmer.

In other words, Descartes is less opinionated about how you should build your DApp’s entire stack than the original SDK is. However, the Descartes SDK is already fit as a foundation for many real-world solutions. It is the easiest and fastest path to imbuing your smart contracts with Cartesi’s unlimited-computing power. If you already are an Ethereum blockchain engineer you should be able to pick up Descartes pretty quickly.

The problem, however, was in building an entirely new SDK in under two months and documenting it, to make it to the Q2 roadmap deadline. That was, obviously, an impossible task. But the Cartesi team did it. Because of course. At this point, I’m sure I don’t need to comment further on how capable this team is, as the output speaks for itself.

What the Descartes SDK is, and isn’t

The central design and implementation challenges for any rendition of middleware for any Cartesi-based stack lie in deciding on how to deal with the “off-chain code.”

By off-chain code it is not meant the deterministic code that runs inside of the Cartesi Machines. Although Cartesi Machines run code “off-chain” by definition, that is not, particularly, the central challenge when writing a “software development kit” (that is, the application-developer-facing interface of the middleware), and so that is not really what is meant here.

The problem is in how to offer a clean interface for application programmers to specify control code for on-chain events, be these events originating from Cartesi’s own core smart contracts, or the application’s own contracts. In that sense, the application code that is packed into a Cartesi Machine does not suffer from this problem, given that the Cartesi Machine abstraction, being a full machine abstraction, is pretty much the naturally ideal solution to expressing application code as a black-box to a middleware that’s supposed to run it.

The remaining off-chain code, which is written by the application, becomes responsible for orchestrating all Cartesi-based computation dispute resolution, requesting the execution of Cartesi Machines, and feeding results to Cartesi components whenever necessary, which are comprised of both on-chain and off-chain components. It sits at a nexus of multi-dimensional complexity: between the application and the middleware, between smart contracts and native code compiled to a real operating system, between the user’s machine and every other user’s machine and between verifiable (Cartesi Machine) and non-verifiable computing. And, of course, let’s not forget all of the security and billing (who-pays-for-whose-gas) implications.

The Descartes SDK is designed to minimize the surface of this “off-chain” code. In fact, it has completely eliminated it for the application developer. Descartes does not have an opinion, at all, regarding the application-layer off-chain code. And, in doing so, Descartes can shield its own off-chain code completely from the application.

Descartes-based applications write Linux-based software, which is packaged inside Cartesi Machines, and their own smart contracts. They do not have to write any of that glue code that has to teach a comprehensive list of simple choices to several different components, where the actual information that is contributed by the application is far simpler than the cognitive load required to understand the language in which that information is being communicated. The middleware needs very little information or coordination from the application. The downside is that the application developer has to write more code if the default behavior offered by Descartes does not cover their needs.

The Technical Road Ahead

The Descartes SDK is just the beginning. Descartes will grow and expand, while Cartesi also launches other SDK and platform offerings, with ever-increasing coverage of the application space. The Descartes platform will be the entry-point into the world of Cartesi for developers. But when you need more, Cartesi will be there to give you more. Much, much more.

Besides the refinement of the core vertical stack that solves the problem of scalable consensus computing, Cartesi will also work on its entire horizontal ecosystem of solutions. For example, applications need data availability, and so the Cartesi side-chain is coming. Meanwhile, the Cartesi Core is also growing, with more powerful, game-changing mathematical magic already in the pipeline.

Conclusion

Software, like any engineering discipline, is a mixture of theory and practice. Engineering is always resource-constrained, and it always lives and dies by the collection of the innumerous design choices that go into the development of a family of components or products. From the sprawling possibility space, one has to find guiding principles: a theme, a story, a purpose. Just like individuals do, companies and organizations are also perpetually engaged in self-discovery, because they are living organisms in a living environment, both of which are always evolving.

Cartesi has a long road ahead of itself, and it is ready to go all the way. We know where we are, and we have a pretty good theme for what lies at the far horizon: the realization of the potential presented by the blockchain revolution, whatever that may turn out to be. And what drives us is the passion for empowering as many people as we can to travel that road with us, be them developers or end-users.

And so, all that is left now is to figure out what lies in the middle of that road. We will cross all of its bridges as we get to them, with determination and dedication, but also with joy and fulfillment.

The Cartesi team is made of (metaphorical) rocket scientists, not only for the exploration of the outer space but also of the inner one. And so we are aware of the massively privileged position that we are in. We are privileged to be able to serve in this new revolution in computing, which will again multiply the relevance of Information Technology to society at large. We are thankful for that opportunity every day, and there’s nothing else on our agenda other than to continuously learn and evolve, to be able to continually surprise you (and ourselves!) with just how much Consensus Technology can deliver.

A big Thank You to all of our supporters, and… let’s keep traveling that road!