ecton.dev

Back to Blogging

I've been wanting to get back to writing, and I wanted a more personal location to reflect on a more regular basis than I have been on my community forums. Although I try to maintain a fairly informal style, I still feel like everything I put there needs a focus on the idea of making Khonsu Labs what it is. While that's a vision of what I want to build, in my head I want messaging on the "Khonsu Labs" level to be more certain than my day-to-day thoughts are. That's why there's been a draft post in the works for weeks, and I still haven't published it. The goal of this post is to unblock me from writing about the accomplishments I've had on the progress towards Cosmic Verge.

Right now, I'm doing something that is by all-accounts absolutely crazy. I'm writing my own database engine. As someone who is trying to write a game, and who already has a rendering engine project, what in the world am I thinking? Those are the thoughts I wanted to explore.

It all started when I found a second contributor

Back in February, I had made the decision to open-source Cosmic Verge. I had been exploring my motivations in life, and I felt like the only true way to ever feel good about letting anyone else work on the project with me is if I tried to build it as a community project. I would still plan on funding artwork and music that we couldn't source as a community, but I would specifically license it in an open way. I have since ironed out a lot more of the details, but at the time I was just making the decision of the direction I wanted to take the project.

One fateful evening, I uploaded the license file at 9:41PM my time, changed the access permissions, and went to sleep without posting it anywhere. At 11:42PM a message arrived on my Discord channel from daxpedda introducing himself and saying he might be interested in contributing. It's been an incredibly productive and interesting time ever since we began talking and working together.

Rust or bust

Right away, our love of Rust was at the forefront of many of our conversations. He would push me to reconsider choices to try to have a better solution and also a way to use Rust to its advantages. We also decided that we wanted to try to have a better deployment story than what I currently had, and that became his primary focus. One stipulation I kept coming back to was: We must have PostgreSQL.

The reason in my mind for PostgreSQL is that if you're going to spend the effort to set up an highly-available (HA) database cluster, you might as well spend the effort to get a first-class database in the process. I'm a firm believer in its abilities to perform incredibly well in many workflows, but I also recognize it has its limits. One of them is just how difficult it is to maintain an HA cluster. Even if you try to use tools to automate the solutions, you must be aware of them so that in situations where failures are happening you can diagnose and correct the issues. This is true of every technology you are maintaining yourself as part of your stack, not something unique to databases.

The reason that PostgreSQL was what I kept insisting on is that it's what I know. It's painful to go through the processes to get a cluster set up, but I've done it. I know how to troubleshoot WAL-shipping. But, I didn't really want to.

In our many, multi-hour Discord calls, we explored many ideas for other storage technologies. We still needed data that would live on each "system server" that we didn't want to replicate to central storage. In evaluating various storage models, I became aware of how sled uses btrees for the storage model.

It hit me one night. While I had a love/hate relationship with CouchDB from the process of building my last business atop of it, I have to say that overall it was a great database to use. It's very clever in how it's implemented, but at the core is that it leverages a map/reduce implementation that sits atop of a btree view system to provide a programmable database. So, what if I skipped the "hard" part of implementing a fast storage layer by using sled and just built a database inspired by CouchDB atop it?

I began having interesting thoughts of where this could go eventually with WASM -- imagine a database where you can write stored procedures in any language that compiles to WASM, and know that you're getting very good performance? Or, what if I explored building an app engine atop this database using WASM similar to how CouchDB allows you to host apps written in HTML/Javascript.

Visions of Grandeur

While those thoughts are compelling, the thoughts that exicted me were these goals:

  • Make it easy to start with an embedded database and switch to a separate database server.
  • Make it easy to set up read-replicas to allow for a simple leader/follower setup for when you want a little redundancy in your infrastructure, or when you want move data onto servers in a local, read-only cache.
  • Make it easy to deploy quorum-based, ACID-compliant databases.
  • Give enough power to enable complex placement algorithms (written in Rust of course).
    • An example would be for ensuring geo-redundancy across multiple availability zones.

Each time I go to deploy any idea, I always have to answer, "What database?" It's time to try to solve that using Rust.

But, to me this isn't that grandious of an idea. By utilizing sled to provide an ACID transactional storage layer, all of the problems become focused on moving data around. One of the side effects of using CouchDB is that you become aware of how it works because there's really no other way to use it. That knowledge allowed me to make significant progress on PliantDb in a little over a week.

PliantDb's Status

Yesterday, I decided to implement deleting documents. After I finished and was reviewing the list of open tasks, I realized I had hit an interesting milestone: I had all the components necessary to implement database replication.

I'm really proud of the state of the codebase at this point in time. It has:

Right now it has local-database functionality with no networking, but the Connection trait abstracts all of the logic behind an async interface so that soon the same code that works with a local database can also work with a remote database.

I haven't spent any time optimizing the database. Currently, I know its write performance will probably not be up to snuff, mostly because it flushes after every write. However, I'm not very worried about that -- there are multiple strategies if that is an issue, but I think for mose people the current performance characteristics should be perfectly fine as long as you're able to batch your writes.

Now that that's off my chest

I'm going to be trying to do a bit more frequent of updates here. I'm not sure the exact schedule, but I mostly want to talk about what I've accmoplished and any updates to goals or my plans. While I try to plan ahead, I'm also living each day as I take it. My vision for the game is getting honed while I work on these other projects, and I'm excited to get back into it. However, I'm even more excited at the prospect of having this type of database available.

I'm planning on pursuing PliantDb to the point that it has clustering support before returning to Cosmic Verge. The problems don't seem that complicated in my head, so I'm hopeful that it could be as early as May that I could reach that goal.

My most immediate goals are:

  • Implement a Replicator flow that allows replicating between two Connections -- this design means that you can replicate between local databases and remote databases without the replicator caring.
  • Implement a multi-database Client-Server API.
  • There are some other issues that I've been tracking in GitHub, but once I feel confident enough that my storage format isn't changing I'll be updating Khonsubase to using it and will finish working on a few more features that will allow me to use that for my multi-project issue tracker.
  • After that, I'm hoping to return to writing code for Cosmic Verge.

So, a bit of a hiatus from the game, but I am hopeful it's for a good cause. Before then, though, I'm hoping to start blocking time each week dedicated to flushing out a game design wiki/document. I was starting on that before I had the idea for PliantDb.

Until next time!