Finally found the main culprit

Welp, this happened to one of the 4 RAM modules in my main build that runs techcoderx.com Hive API.

For weeks I have been figuring out the ongoing instability issues on the node, including performing several OS re-installations, using state snapshots of mine that are known to be functional, and even running everything at stock speeds and not using mixed RAM.

After all that, I ran memtest86 on all the RAM modules one by one running in different DIMM slots just to realize that the 16GB RAM module that was in the 32GB RAM kit that I purchased late last year are failing badly, as shown above. The remaining modules passed the test.

As memory errors can potentially cause permanent file corruptions including the OS, everything would need to be setup again using the remaining RAM modules before I have to RMA the affected kit. This time I have not experienced any weird issues at any point, and managed to complete a hived replay from scratch and halfway through Hivemind sync as I'm writing this. At the moment, techcoderx.com is serving data from hived only and redirecting Hivemind traffic until this process completes in a few days time.

On the increasing activity

As many would have probably know, Hive has been breaking records on Blocktivity nearly on an everyday basis, and ~90% of that are custom_json operations.

For those who are curious, this is how the block time offsets look like on a node located in South-East Asia with decent internet connection (sub 5ms ping), what it used to hover around -50ms to 100ms a few weeks ago. Perhaps more proofs that we need more nodes around here.

For reference as to how bad it is, my producer node is better than that and would still produce empty blocks (or very few transactions) half of the time. Not sure exactly how it happened as nothing unusual has been seen in the logs. My guess would be CPU struggling to keep up with incoming transactions, I guess would have to move to a better server.

Witness performance

Back to the usual report.

Current rank: 91st (active rank 82nd)
Votes: 4,267 MVests
Voter count: 165

Producer rewards (7 days): 21.191 HP
Producer rewards (30 days): 112.642 HP
Missed blocks: 6

Server resource statistics

This section will be present in every witness update logs (if any of my nodes are online) to provide new witnesses (and/or node operators) up-to-date information about the system requirements for running a Hive node.

hived (v1.25)

block_log file size: 367 GB
account-history-rocksdb-storage folder size: 498 GB
shared_memory.bin size: 19 GB
blockchain folder size: 883 GB
RAM usage: 9.5 GB

---