Go: Achieving 70 Million Sets To Aerospike Database In 30 Minutes

In continuation of my previous article where I’ve detailed on the limitations of goroutines, in this article, I would like to share how we achieved 70 million sets to Aerospike Database in 30 minutes by leveraging the entire system Cores with the help of Go Worker Pool Pattern.

Usecase Insights

• Currently, at GoIbibo we had close to 70 million users and we had a set of categories that each user belongs to. The categories describe if the user is new | fraud |regular transacting user | user who didn’t transact in last year etc and so on …
• User Categories change dynamically based on their behaviour and we have a generation task where we gather the data from multiple sources and save to our Database with Key as UserID and Value as List of Categories and the generation has to happen for all 70 million users every day.

Database Insights

• For those who are not aware of Aerospike, it’s a multi-threaded and distributed Key-Value store with storage based out of SSD.
• We have 3 instances of Aerospike where keys get split and stored across instances with the desired replication factor.

Advantages over Redis

• This being a multi-threaded Key-Value Store doesn’t block all the requests in case if there is any slow query unlike Redis, that blocks all the requests if 1 request gets slowed down
• SSD being cheaper than RAM and being a distributed database we always have a chance to horizontally scale, unlike Redis where we need to scale vertically with growing data.

Drawbacks

• This doesn’t support batch set which means if we need to insert 70 million key-value, we need to make 1 call to DB per key which equals 70 million DB Calls.

Approaches, Pitfalls and TakeOver’s

We aggregate data from multiple sources and store it in memory as a map data structure with the key being userID and value is the list of categories. We initially started with a synchronous approach by iterating through the map wherein we are setting one record to the database at a time. When we tried to calculate the estimated time, the above solution resulted to be the worst possible way. Why?

We are running on 8 Core CPU and a database that supports multi-threading but from the application, we are still processing sequentially. We aren’t utilizing the system cores nor the DB Resources. Let’s calculate the total time taken for a sequential generation.

70,00,00,000 keys * 2 ms(average set speed) = 140,00,00,000 ms = 39 hours

What the hell did I just saw? A task that has to be generated twice every day is taking a whopping 40 hours which is close to 2 days. Is this feasible? Hell no.

This is where we analyzed that sequential processing doesn’t work and we have to parallelize the task by using the entire system & DB Resources and that is when we came across the Go Worker Pool Pattern.

Let’s first get to have a touch base on the code and I will get you through each line and also provide a detailed explanation on how context switching happens.

Now let’s get through each line of code.

First, we get the aggregated data with keys as userIDs and values as a slice of categories.

Worker Pool Pattern (Pseudo Code)

• First, we advertise saying at any given point of time we will have a maximum of 100 jobs via a buffered job channel.
• We then recruit 60 workers who keep waiting for jobs.
• Once we’ve fed the workers with all the jobs with nothing left in our plate, we can successfully terminate the workers.

In-depth understanding of code.

When we made a call to createWorkers fn, it internally spawned 60 workers a.k.a goroutines uniformly over 8 core CPU. Every worker that we spawned keeps listening to an event from the jobs channel and until it receives an event the code inside the for loop (AerospikeWrapper.Set) doesn’t get executed.

Now that we’ve had enough workers in the pool, it’s time for us to feed the workers with some work. In our case, the feed to the worker should contain an userID, categories (passed via userData struct) of a particular user which can then be sent to DB.

To represent the scenario in a picture, it looks like below.

As we spawned 60 workers across 8 cores, each core is responsible to handle close to an average of 7–8 goroutines. Each of these goroutines keeps listening to the jobs channel for a new job and as soon as it got a job, it then performs the database set sequentially. But here is a catch.

We all know that a CPU core can almost execute only one instruction at a time, does that mean we can parallelise the process to only * no of CPU cores? Hell no…

This is where the Context Switching comes into the picture.

The first goroutine handled by the first core gets an event and as soon as we start a database call inside the goroutine, it will be pushed to the process queue as database call being asynchronous and the next goroutine will start listening to the jobs channel and this cycle repeats until one of the items in process queue is done executing.

Once the item in the process queue is done executing the synchronous network | I/O operation, the CPU Core will again pick this item in process queue and execute this until it hits an I/O or Network Call again. If it doesn’t it will be done executing that job and move on to the next.

If you look at the sequence of actions that we did, at any given point of time we are making around 60 DB calls in parallel which is directly proportional to the number of workers that we had in the pool.

Though the process of context switching is way too complex than what I say, I just wanna give a top-level overview of how it happens and how each core is responsible for handling a set of goroutines.

With this, we achieved close to 10000 Database Sets per Second to Aerospike/Instance and we have 3 DB instances.

So 7,00,00,000 / 30,000 = 35 minutes.

Hope I’ve conveyed what I intended to. In case of any question do post a comment and I will surely try to reply.

In the next article, I will detail on visualising the CPU Utilisation using HTOP.

Stay Tuned