Oct ‘20 update: this essay and others are now available on my urbit-centric blog, subject.network
it’s been a long time since my last post. in the intervening year i have remained very interested in urbit, completed the introductory hoon course, and began organizing a regional meetup group. i’ve also acquired a better technical understanding of the system design. i wrote this essay for our meetup group’s inaugural presentation; you can watch it on youtube here.
i’ve gotten pretty positive feedback from urbit novices about this, so i hope it’s helpful for anybody else who stumbles upon it. if you notice something that needs a correction, please reach out to me!
urbit: an introduction
by ~sitful-hatred; written for urbit dfw meetup 2020.1
introduction
urbit is a personal server. that is to say, it’s a computer acts as an agent on your behalf to talk to other computers on the internet. right now, we all use computers that belong to megacorporations and serve their interests; when you log into facebook and talk to your friend, all the real computation and storage is done on servers that belong to facebook. in a very real sense, they own your data. it’s very difficult to avoid at this point. the goal of urbit is to address the technical issues that led to this state of affairs, so that everybody can run and control their own software. there are a lot of projects that have tried to address one or two of the many problems that urbit tries to solve, but i doubt there is any other current project as ambitious or farsighted. urbit is an open source project primarily developed by a company called tlon, in san francisco, and there’s another company in austin also working full-time to develop it, called urbit.live.
urbit is famously difficult to describe; this is due to a combination of complexity, an opaque naming scheme, and its genuinely novel ideas and clever recombinations of old ones. it can seem like it leaked into our world from an alternate timeline. thinking about urbit requires you to modify a lot of conceptual models that you’re used to using to think about computers and the internet. i’m not any kind of expert on any on these subjects but i’ve spent a decent amount of time trying to digest the technical parts of urbit, and my goal here is to convey that understanding to you, and connect the technical decisions and constraints to the problems being solved. this presentation is aimed at someone who has heard of urbit but doesn’t quite get it yet. so, to begin, here is a list of questions:
what if your computer was deterministic – a minimal, strictly defined mathematical function that given an event sequence, always ended up in the same state? how would it be different from the computers we’re used to, and what advantages might it have? what would a network of these computers look like? what if that computation was standardized, so that an app i write today runs on another arbitrary computer in ten years, or the computer i’m using now still works when i pass it down to my kids?
why can’t i take my social graph with me when i leave facebook? why isn’t facebook just a protocol? why has the internet transformed from a peer-to-peer assemblage into a surveillance panopticon? why are we doing all our computation on some corporation’s mainframe as if it’s the 70’s, while they build dossiers on us? what happened to the dream of using networked computers to liberate individuals?
how do you grow social software to internet-scale without running into the obstacles produced by the conflicting values of different communities? how do we preserve the freedoms of association and speech while still allowing for the enforcement of values within groups? how do you disincentivize spam and bad actors in an open system?
as you may guess, some of the goals of urbit are to answer these questions and challenges in an elegant way. i will give you a cursory overview of the system and then talk more about the ideas it exists to implement.
the stack
these problems are very difficult to address within the constraints of the current internet, but urbit is an entire networked computing stack from scratch. the core thesis of urbit is that all of these are technically solvable problems if you approach them at the right scale. i’ll try working from the bottom up here and give a cursory overview without getting too bogged down in details. i apologize if this is a little dense, but bear with me and it should cohere. the second half of this is less technical.
nock is a turing-complete function. you can think of it as the functional assembly code that underpins the entire urbit stack. its formal spec is typically described as “small enough to fit on a t-shirt”, and it has 12 opcodes, or basic instructions. this is a couple of orders of magnitude smaller than the x86_64 instruction set that your pc uses. nock is pared down to the minimum elements of what might make up a useful computer; for instance, it defines a single recursive datatype, a binary tree of integers; and its only arithmetic operator is increment.
but a combinator whose only arithmetic operator is increment sounds unusable as a computer; how do you subtract, for instance?
as you may know, if your computer can increment, you can derive all of the rest of the math you’d like out of that operation – for instance decrementing x by incrementing up to x-1. this is extremely inefficient, but urbit has a cool hack for this: the standard library has all the operations you would expect implemented as “jets”, written in optimized c, which manipulate values in a formally equivalent manner to a specification written in nock. that is to say, if you ran an incredibly slow version of your urbit without jets that did all its math the hard way but otherwise had the same inputs, its state would end up identical to the jetted version.
nock is virtualized with an interpreter programmed in c called vere. vere sits between your urbit and your computer, handing over keyboard and packet inputs to your urbit, and outputs back to your unix computer to be displayed in its terminal or sent off through its network interface. right now urbit runs as a unix process, but one can imagine urbit on hardware meant to run nock and jets directly. for now, vere is what runs your urbit on your macbook or vps.
hoon is the system programming language for the urbit operating system, arvo. hoon is a strongly-typed, functional higher-level language that compiles to nock; it’s effectively a collection of nock macros. expressions are denoted with ‘runes’, which are digraphs of ascii special characters. hoon code is made up of these expressions and as a result can be kind of scary to look at, but i promise it’s not as hard to use as you may think!
arvo is the urbit operating system. it has a minimal kernel, around 1000 lines of code, plus some kernel modules, called vanes. arvo’s vanes handle the networking, apps, a web server, a version-controlled file system, etc. your urbit instance’s arvo begins with a blank slate, then processes any events (packets and keystrokes), writes them to a log, and outputs its effects, producing new events and its new state. if you play back this log, your urbit will end up in an identical state.
urbit is deterministic. your urbit’s state is one giant binary tree that is modified by an event log. nock takes your state and some new event, applies the event to the state, and outputs a new state and a list of effects. arvo orchestrates events between vanes.
the urbit network protocol & kernel module is called ames. ames is peer-to-peer and encrypted, and runs over udp. it has a global immutable namespace; an instance’s name is permanent, and is both an identity and routing address. urbit’s address space is 128-bit, just like ipv6. the total number of addresses (or ‘ships’) is effectively unlimited. however, urbit’s address space is also partitioned into classes with different privileges. 8-bit address ships are called galaxies, which are core infrastructure nodes that sign software updates. underneath galaxies are stars, 16-bit address ships that act as routing infrastructure. beneath stars are planets, which are ships meant for individual humans. planet addresses are 32-bit like ipv4, so there’s about 4.2 billion of them. galaxies issue stars, and stars issue planets.
urbit uses a base-256 phonemic scheme for pronouncible numbers called @p. galaxies have single-syllable names like ~zod; stars are two-syllable, like ~marzod. planets have two two-syllable names, like ~dalsum-simreg. this is your identity – think of it as a combination ip & email address. the words are human-memorable and sound like cool sci-fi names.
how do you own these names? urbit’s public key infrastructure is a set of ethereum contracts collectively called azimuth. azimuth acts as an ownership registry for ships and consensus mechanism for network governance. if you receive a ship, your proof of ownership is an erc-721 token, which is an ethereum standard for non-fungible tokens. you own this token like you own bitcoin; as long as you have the master password, it belongs to you forever.
azimuth technically exists outside of urbit as a general-purpose pki; urbit just uses it as a source of truth. if you dislike ethereum for political or technical reasons, keep in mind that it’s not a permanent intrinsic part of how urbit works, it’s just the current scheme – though in my opinion it’s a pretty elegant solution.
philosophy
urbit has very strong opinions, and some design decisions that might not make immediate sense. this is a system meant to replace the internet we’ve spent half a century building; it needs some really compelling wins to be serious.
for instance, why make a computer with such a minimal core? so you can freeze it and standardize it.
a basic idea underlying the design of urbit is that many of the problems we have today with the way networked computers work is that the technologies composing them accrued in layers atop each other in ways that produce unforeseen conflicts. we have a million identity systems because each layer reinvents everything in the layers below it in mutually inoperable ways. a universal identity system, a secure peer to peer networking stack transparent to the software using it, and key management and cryptocurrency management baked into the core allows developers to offload those problems onto the operating system without worrying about re-implementing them at the app level and fretting over security, dependencies, translating data between layers, etc. these features are basic traits of the platform, available to all software on the system to use.
urbit approaches the problem of cascading system complexity by introducing a hard brake on the core components, called kelvin versioning. instead of incrementing version numbers, the versions of nock and hoon count down to absolute zero, at which point they can never be changed again. nock is at 4k at the moment, pretty close to its final state, and hoon is around 150. this is meant to ensure future interoperability and compatibility, one of the most technically ambitious parts of urbit. this is a system designed so that your kids can execute code you write today on their future computers, or even boot up your entire ship after an arbitrary amount of time offline or on different hardware. nock’s frozen status means everything above it can be upgraded on the fly, including live upgrades of the kernel, and ensures architectural compatibility forever. i should also mention that urbit’s codebase is much, much more compact than the computers you use every day; the whole stack is somewhere in the neighborhood of 50k lines of code. the linux kernel alone is somewhere around 12 million.
standardization also allows for commoditization. any computer that can run an urbit can run any other urbit and all its software. the platform is an open standard, which allows commodity providers to compete over performance without users being locked in or out by proprietary changes or interoperability problems. in practice, this will look something like urbit hosts running your instance on their hardware for a fee, but you being able to download your instance and all your data to run on another provider, or even your personal computer if you decide to do so. it can also look like hardware implementations of nock, and urbit-native computers manufactured by anybody who wants to, like pc clones.
as you’re probably aware, the internet of today is built on top of the client-server model. you want to connect to other computers, but you don’t or can’t leave your computer on all the time, so you offload that duty onto a dedicated server run by somebody else. this model grew out of the timesharing mainframes of yesteryear, and it has served us well, but the limitations have become clear; he who runs the server, owns the data. google runs some excellent email servers and facebook is fabulous at showing you pictures of people you’ve dated, but it’s 2020 and i don’t want to be spied on anymore; why don’t we just run this stuff ourselves? first, it’s hard, and second, network effects.
running a personal server today is technically possible. you can rent a vps, set up and maintain an email server, throw a mastodon instance on it, connect a bunch of chat services, etc, but it sucks and is a huge pain to maintain and troubleshoot. worse, you have to be significantly more technically inclined than average to get that far. there is a reason almost nobody does this. unix is industrial machinery; urbit is meant for individuals. your urbit is intended to be manageable like an iphone, through a simple user interface that allows you to configure the software on it without having to touch the command line (though it’s there if you prefer). unlike an iphone, it is open source and extensible, and will likely develop an ecosystem of competing interfaces. today, the interface is called landscape, a minimal and attractive web ui that you control via your browser.
network effects on the other hand are a tricky thing to overcome; why should we be confident that urbit will be preferable to facebook? why would anybody use a hypothetical twitter-like service on urbit instead of actually-existing twitter, or an alternative like mastodon? because, these things are not mutually exclusive. urbit is a computer, not just a social network. existing networks and services have api’s that allow you to access them via third party clients. urbit is a general purpose computer that you can straightforwardly program to scrape, display, and store data. why not program your urbit to scrape your all your feeds and messages in one place? because urbit is an entire computing platform, every piece of functional utility added to it by software increases the network effect of urbit; once enough people are using urbit for whatever disparate purposes they desire, be it running a bitcoin-trading bot or collated twitter feeds, they will all find themselves on a decentralized network with a universal identity system and basic messaging utilities. when you and your friends get hooked into some urbit functionality or another, the external megacorp services are redundant – so why bother?
urbit’s model is everyone running their own server. a hypothetical reddit or usenet-like service on urbit is an app you install, that communicates with your friends’ urbits, which are also running the app. existing social platforms can be conceived of as a relatively simple set of rules for displaying content, comments and messages; the hard part is getting everyone in one place, which urbit’s identity and network layers provide for anything built on top of it. it’s exciting to imagine the possibilities for rapid experimentation that it makes possible.
urbit presents itself as a critique of the internet as it works today; this isn’t purely on technical grounds, but also a vision of a more pro-social internet. the legacy internet was taken over quite quickly by spammers and hackers. it was designed originally to trust anyone by default. if a person sends you mean emails or a remote code exploit, there’s little you can do in response – IP addresses change and email accounts are free and unlimited. urbit’s solution to this is in its finite address space.
i mentioned earlier that there are 4.2 billion planets, or addresses meant for humans. your proof of ownership of your address is a cryptographic token. right now these sell for about $20, though the price is ultimately determined by the market. however, the fact that they cost money is one half of the solution. the other half is that infrastructure nodes can blacklist bad actors. if it costs ten or twenty dollars to get an urbit identity, and you start spamming or abusing people, people may ask the operators of the routing nodes above you to stop routing your traffic. the cost of a planet is ideally cheap enough to not be a burden, but expensive enough that you can’t make up the cost from spamming or be worth burning because you’re angry at someone.
one can also imagine reputation systems, where people individually set rules to automatically block crowdsourced lists of known bad actors. i should note that this system is theoretical at the moment; urbit is designed to make this kind of thing easy to do, but right now it’s small enough that everybody is nice to each other.
to briefly recap the design of the network, at the top of the routing hierarchy are galaxies, which sign software updates for all the ships below them and issue stars. the star is expected to perform peer discovery and nat traversal for the planets beneath it. stars can issue planets, which are meant for people. the connections your server makes with your friend’s are directly peer-to-peer where possible, and mediated by a star when it isn’t.
an important qualification to understand here is that all traffic on urbit’s network is encrypted. your host star tells you where to send the packets or gets them through the firewall for you, but nobody but the recipient can read them.
a simple illustration to understand routing: you buy a planet from a star. after your planet is up and running, you want to talk to your friend’s planet. your planet asks your star for its ip address. your star asks its galaxy, which asks your friend’s galaxy, which asks you friends star, and the answer comes back to you the same way. now that you have his ip address, you send your encrypted packets back and forth directly. [note: i’m not sure if this is a very precise description but my understanding is that it’s something like this]
the star above you is called your sponsor. i described how stars could block abusive urbits from reaching other ships, but that runs both ways. if your star is unfair or malicious, or you just don’t like the guy running it, your planet can ‘escape’ to another star if it will have you. stars provide services to planets and in the long run will probably be mostly operated as businesses, and it’s a good idea to keep your customers happy.
this is a system of voluntary relations; your star has to like you enough to route your traffic, and you have to like him enough to trust his services and potentially pay him. this creates pressure on both sides to maintain good behavior, and gives a release valve for irreconcilable differences. the sponsor relationship also applies between stars and galaxies. in the long run, this also allows the network to fragment gracefully if it needs to.
one of the major development milestones coming up in 2020 is the integration of native bitcoin and ethereum support into the urbit kernel. this is basic wallet support and api calls for full nodes, but big things have small beginnings. what new possibilities are opened up by having secure digital identities and simple system calls for trustless money? here’s an easy example. write a program that watches for bitcoin transactions to a wallet you designate, grants access permissions to media in your filesystem for the urbit id’s associated with the senders, then sends them a message. you now have a decentralized patreon.
this is a bit of a personal angle, but i am very attached to the aesthetics and design that tlon has brought to bear on this project. an example is the sigil system. in the same way that @p converts routing addresses into pronounceable names, sigils convert urbit names into visual symbols you can recognize. they look kind of alien and mysterious. my girlfriend gave me a painting of mine that i put above my computer.
all of this aims at a terminal goal of taking away the necessity of having other people’s servers between you and your friends & family. when urbit conquers the world, your digital life will be fully contained on a device you control, running on an open standard computing model. all your traffic will be private and encrypted, and all traffic meant for you will be addressed to you, personally. you will easily experiment with new social platforms with your friends, which will simply be protocol exchanges between your devices and theirs, and you won’t be beholden to or spied on by the googles of the world. conflicts will be mediated within the groups you belong to instead of from above. your home iot devices will run child ships issued by your planet and be entirely under your control. you will be able to run the device orchestrating your social and financial data online in a high-assurance data center or your home pc. more than anything, your computer and data will belong to you in a way that it simply cannot in today’s world.
there’s a good deal i’ve skipped over and a few things i’ve undoubtedly gotten a little wrong in this already long presentation, but i hope you find this as exciting as i do.
getting involved
urbit is young but moving fast. there’s plenty to dip your feet into if you find this stuff exciting.
- read the docs
urbit’s documentation is significantly more clear and robust today than when i first tried learning about it. there’s a huge amount of stuff to read on the website, and you could do worse than looking through the glossary; one of the hurdles to learning about this is that it has a good deal of jargon. there’s a blog that’s had some really excellent posts about design decisions and constraints of various parts of the stack posted recently. there’s also a whitepaper from a few years ago if you like reading those.
- hang out in u-h
the de facto community hangout spot is on ~dopzod/urbit-help. it’s meant for technical discussion of urbit, and it’s a great place to ask questions, but it’s also a pretty typical friendly chat room.
- hoon school
learning to program with hoon is the single best thing you can do to understand how things work under the hood if you’re technically inclined. you can do this yourself by reading documentation and asking questions in u-h, but if you’re like me you may benefit from structure. tlon organizes informal beginner and an advanced hoon school sessions. you can sign upx for the waitlist on the website.
hoon is pretty weird, especially if you don’t have any experience with functional programming. don’t let me saying that discourage you though – i completed the 101 course with almost no programming experience. i won’t pretend it was a piece of cake, but it probably took 5-10 hours a week for me.
- meetups
there are meetup groups all over the world. if you’re interested in helping me organize and promote the dfw meetup, feel free to talk to me about it.
- grants
tlon in incentivizing the community development of urbit by granting address space in exchange for features. for instance, different components of bitcoin support are being contributed in exchange for grants of stars. this meetup is also part of the grant program. it’s worth taking a look!