Hello there again. It’s been a while. Last time, I wrote about ouchan, our monorepo for Go-based code. In that article, I briefly mentioned mochi*, which is what we call our Kubernetes clusters where most of our backend services are running on. We use GKE to run these clusters across three zones in Tokyo region. At the moment, we have three clusters, each for development, staging, and production. Each environment is mapped to a specific branch: master, qa, and production, respectively.

overview

Mochi contains all the YAML files, environment variables, credentials, and other configuration files for all of our ouchan services. It also has its own build system which is responsible for recreating and tearing down these clusters, which really is the main point of mochi: having the ability to tear down and recreate our clusters with minimal to no intervention.

Normally, we try stuff first on master branch, which is our development cluster. Then we merge these changes to the staging branch, and eventually, to production. In CI, mochi’s build system does a cluster blue/green deployment: create a new cluster and deploy all services to that cluster. After some checks, traffic is rerouted from the current cluster to the newly-created one. Finally, if successful, build system deletes the old cluster. This also means mochi always has all of ouchan services’ latest container image information for deployment. This is part of ouchan’s build process in which all built services update their equivalent deployment files with something like:

# Update the deployment file in mochi:
$ sed -i -e "s/image\\:\\ asia.gcr.io\\/project\\/container\\:.*$/image\\:\\ asia.gcr.io\\/project\\/container\\:tag/g" yaml-file

# Update the running deployment:
$ kubectl set image deployment <deployment-name> <container=image[:tag]

We actually started mochi around December last year using AWS and the awesome kops tool. We had it going for a couple of months but in the end, cluster management was too much for us so we moved to GKE even though we are still pretty much an AWS shop. The only services we use in GCP are GKE, PubSub, and StackDriver.

Thanks for reading. See you in the next post.

Best,

@flowerinthenight
Engineering Lead

PS:
mochi* - named after Mochi-san, Mobingi’s employee #1

---
For any comments, questions, or feedback, please reach out to us @MobingiTech.