Scuttlebutt Genesis

@bobhaugen hey, I first became interested in p2p a few years back. I was a bit burnt out from the previous job and was living in a sailboat in a swamp in Auckland, NZ. Playing around with frontend code (because I usually did backend) I realized that there was a disconnect between the way that frontend works (change events moving between layers) and backend (request/response to the database) why couldn't it all be change events?

Then I realized that git essentially modeled change events, and that a system based on replication would work really well offline because you'd have a valid copy of the data always at hand (pretty useful in a sailboat). So i started looking into git and implemented a replication prototype https://github.com/dominictarr/snob I spoke about this at my local javascript meetup and got a very enthusiastic response! I hadn't had a good response for one of my "crazy" ideas before, so I decided I should just work on that and see where it took me.

Important influences here where the amazon dynamo paper, CRDTs, git and couchdb. Couchdb's most interesting feature is the changes feed which allows you to "tail" the database and synchronize changes. I don't think couchdb's replication is very good, but it still raised the bar.

After a while my experiments distilled into https://github.com/dominictarr/scuttlebutt which should now be called "insecure scuttlebutt". That work got me invited to speak at conferences, particularily one in Ireland, nodedublin (now nodeconf.eu). nearform (organizers of nodedublin) where also interested in in database replication, I started working with them to develop a new replicatable database product. In the end, nearform pivoted away from that idea, but everything I had done was opensource and that introduced me into leveldb, and learned me a lot about databases.

Camlistore was an important influence - it is immutable, with hashes for keys, but you can query the database with search instead of the keys, which overcomes awkwardness of not being able to choose the keys.

I didn't like camlistore's replication protocol. It was too batch-mode, and not particularily efficient. I did some experiments looking for a general purpose replication protocol but decided it wasn't possible. I saw cryptosphere which is a distributed filesystem project, but uses replicated logs to record which peers are cooperative, basically a trust network. https://github.com/cryptosphere/cryptosphere/wiki/Philosophy Replicating logs is easy, so that is good, but instead of building a boring filesystem on top of a trust network... why not just expose that directly to the humans? I realized that all the web2.0 apps are basically just replicated logs, and basically implicit trust networks, and that this was a particularily easy architecture to decentralize.

What makes it easy is that since you more or less explicitly "follow" or "friend" people that defers the spam prevention to the humans in a way that seems natural. Contrast with email, where spam prevention is a messy soft-AI problem.

This also dovetailed with earlier ideas about trust networks, that I didn't know how to persue at the time. For example, experts have reputations, and can teach or certify students, but are disincentiveized from misvouching because they will loose their reputation if thier students are found incompetent. This was while I was at university and felt that a degree stamped by the university didn't mean very much, and was also very restrictive in terms of subject matter etc. Building a decentralized trust network seemed like a very good idea. One of the key insights of web 2.0 is that users respond to UI, making them aware of an option means they can do it. Users and UI flow together. A certain UI design may influence user behavior, and this also works at higher layers, certain application technology influences possible applications and certain economic models influences business models. I had worked witnessed some terrible software in my day, but recoginized that it was terrible for social reasons more than technical... I seek more equitable social arrangements because I think it will create better software. Everything is becoming software right? so if more equitable circumstances produces better software then the good software can kill the bad software.

By this time I had figured out the basic architecture and was starting to work on it. I had noticed @paul's work, and recoginized it's importantance as another part of the puzzle. I wanted an open platform that anyone could build things on (as twitter was originally pitched, but has closed off they try to monetize etc). Also, we couldn't realistically plan to just sit down and create an app that everyone wants to use, we need many experiments so that one can succeed, therefore we need an decentralized application platform more than we need any given a decentralized application.