JH0project

Home
Art Gallery
Infrastructure
DN42

Infrastructure

JH0project operates on a global distributed infrastructure, across multiple regions and continents including Japan, US West, and UK.

Architecture

The infrastructure is designed to be uniform, without any dedicated server for specific services. Same set of containerized services are deployed across all servers, this ensure that all services are design to be stateless, can be scaled horizontally, and same configuration can be applied to all servers, making it easier to manage and maintain.

Software Stack

Infrastructure Management

All service are containerized and running on barebone Docker, managed by Coolify. The reason behind this is that the regional latency is too high (max regional latency is 266ms) for Kubernetes, more specificly etcd won't able to elect leader, nor sync.

latency Regional latency show in CockroachDB Console

Besides there's only two servers in each region that are powerful enough to run Kubernete (API server), so we can't form a cluster per region. Not to mention we only have few services to manage, so the overhead of Kubernetes plus other services like FluxCD/ArgoCD, CSI, Seal Secret for managing it is not worth it.

Luckly, when we're looking for a solution Coolify released multiple server support (In Beta). Coolify is a really cool app you should it check out.

Databases

We're using CockroachDB a distributed SQL database that is compatible with PostgresDB, as out primary database. The reason for choosing CockroachDB out of researched database is that, it able to distributed across multiple regions in active-active (eliminated PostgreSQL), with PrismaORM support (eliminated YugabyteDB), and its a CP database (eliminated Cassandra).

We are also planning on hosting Redis cluster for caching, mainly for Next.js. Since everytime we deploy new version or restart the Next.js website, it will clear the cache, including the cache for the computationally expensive image optimization. Not only that, we would like to share the cache between all servers, so each server don't need to render the image again.