I made some significant changes to how Cosmic Verge is deployed and wanted to talk a bit about what I did. But, I figure a good place to start is explaining why I decided to work on this now.
I had been a grateful recipient of some credits to Amazon Web Services, but those credits were running out, and I felt like I could save a fair amount of money by changing solutions. While it may seem like I just picked these solutions for Cosmic Verge a month ago, that’s far from the full story – the deployment strategy I picked was an evolution of what I was using for ncog last fall. Cosmic Verge only came together to what it is so quickly because it was largely copied and pasted from previous projects in the past year.
With the realization that my credits were running low, I began investigating various options. I wanted:
- Managed PostgreSQL hosting
- No requirement to deal with patches on servers
- No public access to the services without going through the load balancers
The requirement for managed PostgreSQL limited my choices, but in the end, I picked DigitalOcean’s managed Kubernetes offering. I’ll explain why I switched away from my requirement of managed PostgreSQL further below, but let’s first talk about the success of lowering my hosting bill!
As of today, I’m much happier with my hosting setup, and the footprint is a mere $30/mo currently. To run the setup I plan on running once I’m accepting payments (quite some time away), I plan on those costs increasing a little to help ensure as much reliability as possible. The current setup is mostly “highly-available,” although I have several tasks on my to-do list to test those scenarios more thoroughly.
DigitalOcean provides managed Kubernetes, which solves one of my desires for the infrastructure I have deployed: I don’t want to need to remember to run server patches. With their service, you also don’t need to pay for “master nodes” like you would on many other hosting providers.
In the end, my bill for Cosmic Verge is:
- 2x$10/mo for 2 “Droplets” with 1 vCPU and 2GB of RAM each
- 1x$10/mo for 1 “small” managed load balancer
That feels pretty good to me, and I want to investigate using CloudFlare’s Access to expose some private services like PgAdmin. If I figure out how to add that, the same tool that provides those tunnels can enable using Cloudflare as a LoadBalancer. It’s a strange concept, and it would only reduce the cost by $5/mo, but the performance might be equivalent to the current setup.
Staying a database administrator
As a part of being the person who designed and managed my last company’s infrastructure from the non-raided single server setup through some pretty fun dedicated server setups to the eventual migration to AWS, I’m no stranger to administrating PostgreSQL. I ran a replicating 9.x cluster and did it across multiple hardware upgrades without some of the amazing tools that exist today. I was so happy when I could simply pay AWS for an RDS instance and not worry about the database anymore.
However, as I was researching, I ran into an open-source project called CruncyData. Today’s project was to start playing around with it, and it works incredibly well, and the documentation is spectacular. I still want to test some of the failure conditions, but all of the annoying parts of database management seem to be automated by this tool.
The net effect was that my hosting bill didn’t need to include the smallest PostgreSQL hosted plan at DigitalOcean, which was $15/mo. While you can easily argue that the $15/mo is not worth the time I already spent learning, the costs scale as the database requirements increase. Additionally, over the 10 years of running Core-apps, I grew to enjoy the infrastructure side, so this diversion was a fun experience.
Future infrastructure ideas
I’ve been thinking over the last few weeks that Redis really is redundant. I should be able to have the servers talk to each other directly. In the previous hosting environment, I didn’t have a good way to let each node know about each other. However, in Kubernetes, you can discover available services via DNS.
If I can look up at least one other node to talk to via DNS, I can ensure that multiple servers can automatically find each other and begin talking. Pair that with one of the excellent consensus algorithm implementations like Raft, and instead of using Redis, the webservers can share the needed state directly with each other.
This does increase the server’s complexity, but it means that there are a few key points where Redis is currently used to synchronize state that could be eliminated (or at least be replaced by waiting for Raft to receive the changes).
This server-to-server API could also be used for direct pub-sub by broadcasting published messages to all known peers.
I’m excited about the prospects, but I will be switching my focus to continuing working on the game itself. I accomplished my main goal of reducing my expenses, and I’ve documented enough of the process that I can come back to it easily enough in the future.