Think about scalability before writing code

Building Ape Mining Club was an interesting expereince and had a few areas that took some planning. The game was modelled after another game called Mine Rocks. This game had numerous difficulties and I believe even one pool was shut down causing a lot of frustration with it's users.

You can from the screenshot the game is very similar. I took this idea and made a lot of changes. The most critical change was using Hive as there is no transaction fees and no need to "approve" each pool with a significant transaction fee as well as transaction fees for every action including claiming rewards.

I was originally going to have the ability to claim rewards as they build up, similar to haow you claim rewards on Hive. I felt like this feature would be spammed everytime someone earns 0.01 rewards, so I opt'd to do a daily payout instead. This drastically reduces the amount of transactions created by the game on the blockchain. It also improves the user experience by having a consistent hands off reward system. I also wanted to take advantage of real-time UI to show pending rewards updating in real-time as the community interacts with the game.

The real problem came when someone purchased equipment in the game. In Mine Rocks, you purchased rocks in a vein, when you did a transaction went out to all token holders immediately. This is something I would have loved to mimic, but without smart contracts this would be a nightmare of transaction spam.

There are currently around 160 players of Ape Mining Club and that would mean everytime someone purchased equipment it would result in 160+ transactions, potentially multiple times in the same 3 second block. By pending rewards until a daily payout, this eliminated this bloat, but it did not eliminate the high resource cost of this operation.

Because 10% of every equipment purchase goes out to all APE token holders, I need to get an updated list of token holders and then update the database with their pending rewards. With a typical token with typical distribution, it can take 6-8 seconds or more just to get this list of token holders. This is too slow when potentially doing more than one call in a 3 second period.

Here is what happens when someone purchases equipment in a pool, this is the same process if they order 1 unit or 100.

Most of this is relatively easy to do, the split 10% with all token holders is very time consuming though. I had to find a solution to this problem before launching. It would have been fine initially, but even within 24 hours when we broke over 100 users it would have started showing problems.

The solution I went with is maintaining my own state of token holders in real time. This means I had to watch the blockchain for all operations that can change a users balance of APE. This includes:

• Market Orders
• Transfers (including game operations)

If you read yesterday's post you would know I made sure I processed all transactions in a block at once, either all or nothing. This ensures I know for a fact I have processed all transactions once and only once and in order.

info: -----Processing block: 9903961-----
info:
    BEGIN;
call updatetokenbalance('ecoinstant', -25);
call purchaseMiner('ecoinstant', 'gpu', 25);
call updateenginelastblock(9903961);
    COMMIT;

info: Processing Transfer
info: Processing Game Ops
info: -----Committing block: 9903961-----

As you can see here, whenever a transaction is made I immediately update their balance. This means when someone purchase equipment, I simply update one table based on the state of another table, something Postgres can do in microseconds. As a result, the Buy Equipment operation completes in less than 200 ms, instead of 2-20+ seconds. This is a massive improvement and allows the game to scale to thousands of users without self destructing.

Premature optimization is the root of all evil.

I am well aware of this popular developer mantra, but in this case this situation would have arrised rather quickly and was very predictable. I had to find a solution that worked without smart contracts and I had to do it before opening the game to the public.

While I firmly believe in developing first and solve performance problems when they become an issue, you need to address ones you know will happen and will cause mission critical issues when they do.

Sometimes the solution isn't what you think, sometimes you have to address the problem from a completely different angle to solve it.