Reconfiguration notice for api.hive.blog API node infrastructure

As I mentioned in my last post on development progress at BlockTrades, we’ve been asking Hive-based apps to change the way they broadcast transactions. Transactions are used to add user-generated operations such as voting, posting, transferring funds, delegating HP, etcetera to the Hive blockchain.

<p dir="auto">In this post I’m going to briefly describe why we’re requesting this change and also describe a change we’ve made to our API node infrastructure to better handle the increased traffic from bots playing Splinterlands with “bad” API calls. This latter information will likely be interesting to other API node operators. <h1>Apps should use broadcast_transaction call <p dir="auto">Most Hive-based apps previously used the call <code>broadcast_transaction_synchronous to broadcast transactions. This call, as the name implies, waits for a transaction to be included into the blockchain before it returns to the calling application. The problem with this call is that we’ve measured it takes about 3 seconds on average to complete on our API node infrastructure even under normal loading conditions. That’s a long time to a computer. <p dir="auto">So if one of our hived nodes gets a lot of these calls, the calls keep all of that hived’s worker threads busy, effectively slowing down all API calls made to that hived server. <p dir="auto"><span>By adding additional logging to one of our hived nodes, we were able to observe that even read-only API calls arriving at the loaded hived node could wait 2 seconds (or even more under very heavy loads) before they got assigned a worker thread to process them. A few days ago, this was noticeable as a slowdown on not only web sites that broadcast transactions such as hive.blog, ecency, and peakd, but even on read-only Hive-based sites (e.g. Hive block explorers such as <a href="https://hiveblocks.com" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://hiveblocks.com). <p dir="auto">The ideal solution to this problem is for apps to replace all these slow calls with the newer, faster <code>broadcast_transaction call. This call doesn’t wait for the transaction to be included into the blockchain, and it completes on api.hive.blog within an average of 0.027 seconds (more than 100x faster than the synchronous version). <h2>Most hive apps have already migrated to the better API call <p dir="auto">Most of the major Hive apps have moved to the faster broadcast call within the past few days, as app devs saw how much more responsive their app became with the new calls (i.e. they became faster than they ever were before, even when we had less loading on the Hive network). <p dir="auto">And I expect the few remaining big apps, such as hive.blog, to convert to using the new calls within the next couple of days (we have an engineer working now to fix one known issue preventing us from rolling out the latest version of condenser with the broadcast_transaction fix). <h1>Mitigating bad traffic from misbehaving bots <p dir="auto">But despite movement of the major Hive apps to the faster call, we still see a lot of broadcast_transaction_synchronous traffic on our node being generated by presumably home-grown bots playing Splinterlands for their users. I suppose these bot devs will eventually fix their bots, but in the meantime, we have no easy way to contact them, so we’ve made a change to our API node infrastructure so that this “bad” traffic doesn’t impact the apps generating “good traffic”. <p dir="auto">We have redirected all incoming broadcast_transaction_synchronous traffic to a single hived node that only processes this type of API traffic, and all other API calls (including the “good” broadcast_transaction) are routed to our other hived nodes. This means that apps using the proper calls will not be slowed down by the bad traffic. And it will probably ultimately encourage the bad traffic generators to change their bots as well, although I’m not holding my breath for when that will happen. <h1>Add an extra consensus hived to manage broadcast_transaction_synchronous traffic <p dir="auto">If other API nodes want to be capable of serving all the traffic from the Hive network right now, here’s the relatively easy way to do it: <ul> <li>add one additional consensus (not account history) hived node to your server. Since consensus nodes don’t require much memory (around 4GB), the main additional resource cost is around 370GB of disk space for the additional block_log file. You only need a consensus node because it is only going to be processing broadcast calls and nothing else. <li>Modify your jussi config file to redirect all below types of traffic to your consensus node:<br /> ** steemd.network_broadcast_api.broadcast_transaction_synchronous<br /> ** appbase.condenser_api.broadcast_transaction_synchronous<br /> ** appbase.network_broadcast_api.broadcast_transaction_synchronous <p dir="auto">With the above steps, your regular account_history node(s) will process all the good traffic and the bad traffic will be offloaded to the light weight consensus node. Also, if you want to further improve quality of service for the bad traffic, you can increase the web-threads setting in your consensus node’s configuration file from the default value of 32 to 64 (or higher), at the cost of increased memory usage. <h1>Quick Update <p dir="auto">We swapped over hive.blog to use the new calls earlier today and we swapped over wallet.hive.blog about an hour ago. The UX feels much more responsive now.