It's incredible to think that we have been running the mainnet for almost 2 months! In that time we have been able to gain a much better understanding of the performance of the EOS application.
<h3>Full Nodes <p dir="auto">To date, the most problematic node type for the network has been Full nodes. These are the public facing RPC API nodes that all users and dApps interact with. There is a lot of strain on these nodes as they store and process more data than any other node type and are computationally expensive due to the <code>nodeos app being single threaded. <p dir="auto">One of the main culprits for Full node failure is down to the use of the <code>filter-on: * config. With the wildcard, the node will attempt to store all smart contract data. There are a couple of contracts which are generating large amounts of data (spam?) into the network. Take a look at this chart: <p dir="auto"><img src="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmd7acdRRYuP1G3xb89by15Jm4Aj7R5LEinExhnmAzqmrs/Screen%20Shot%202018-08-02%20at%2010.28.55.png" alt="Screen Shot 2018-08-02 at 10.28.55.png" srcset="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmd7acdRRYuP1G3xb89by15Jm4Aj7R5LEinExhnmAzqmrs/Screen%20Shot%202018-08-02%20at%2010.28.55.png 1x, https://images.hive.blog/1536x0/https://cdn.steemitimages.com/DQmd7acdRRYuP1G3xb89by15Jm4Aj7R5LEinExhnmAzqmrs/Screen%20Shot%202018-08-02%20at%2010.28.55.png 2x" /> <p dir="auto">This chart shows the top contracts with their associated actions. Notice how the top two have more activity than the <code>eosio system contract actions! Well, by using the <code>filter-on wildcard, all of this data will be taking up precious RAM and processing resource on the Full nodes. <p dir="auto">If we remove <code>blocktwitter and <code>gu2tembqgage from this chart, the landscape looks more healthy: <p dir="auto"><img src="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmWfWKFNbf9Gcm9vRw3u3E53KRojACLvybXXJiUpu54Emq/Screen%20Shot%202018-08-02%20at%2010.30.04.png" alt="Screen Shot 2018-08-02 at 10.30.04.png" srcset="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmWfWKFNbf9Gcm9vRw3u3E53KRojACLvybXXJiUpu54Emq/Screen%20Shot%202018-08-02%20at%2010.30.04.png 1x, https://images.hive.blog/1536x0/https://cdn.steemitimages.com/DQmWfWKFNbf9Gcm9vRw3u3E53KRojACLvybXXJiUpu54Emq/Screen%20Shot%202018-08-02%20at%2010.30.04.png 2x" /> <p dir="auto">The problem is that using a whitelist of contracts for <code>filter-on could result in legitimate new dApps not having their contract data available via major Full nodes throughout the chain. To combat this, a new config switch has been devised: <code>filter-out. This will allow BP's running Full nodes to maintain a wildcard for all contracts, but to specifically ignore contract actions from known spam accounts. <p dir="auto">This certainly helps with system resources but it's a tricky subject, as this decision is down to the discretion of the BP's and there is no consensus methodology in place for who should be added to this list. <h3>Performance Tuning <p dir="auto">At Block Matrix, we have been A/B testing various difference system configurations to determine what increases performance and/or lowers system resource utilisation. There is one configuration improvement that has had more immediate impact than any other: <strong>separate disk mount for data directories. <p dir="auto">Across all of our nodes, we now have a separate mount for the <code>blocks and <code>statedata directories. We moved from storing these directories on our <a href="https://en.wikipedia.org/wiki/Ext4" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Ext4 O/S partition, to having a dedicated <a href="https://en.wikipedia.org/wiki/XFS" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">XFS mount which we found gave us the best combination of performance and reliability. <p dir="auto">We initially moved 50% of our 6 node set across to this setup, and it's easy to see the impact this had: <p dir="auto"><img src="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmZGznWv1tDwm1MRyBCdUqvEcc2vkNNa17qxcppaD457dN/Screen%20Shot%202018-08-02%20at%2010.13.33.png" alt="Screen Shot 2018-08-02 at 10.13.33.png" srcset="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmZGznWv1tDwm1MRyBCdUqvEcc2vkNNa17qxcppaD457dN/Screen%20Shot%202018-08-02%20at%2010.13.33.png 1x, https://images.hive.blog/1536x0/https://cdn.steemitimages.com/DQmZGznWv1tDwm1MRyBCdUqvEcc2vkNNa17qxcppaD457dN/Screen%20Shot%202018-08-02%20at%2010.13.33.png 2x" /> <p dir="auto">We use <a href="https://www.influxdata.com/time-series-platform/telegraf/" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Telegraf agent on each node which bubbles up stats to <a href="https://grafana.com/" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Grafana via InfluxDB, this allows us to monitor all system resources but it's easily configurable for application monitoring too. <p dir="auto">After 2 weeks of testing, we are now confident in moving all our remaining nodes to this configuration, if you give this a try or already have a configuration like this in place, we'd love to hear about it, especially the filesystem you're using! <p dir="auto">For the techies out there, here is our current <code>fstab: <pre><code>/eosdata xfs noatime,nodiratime,attr2,nobarrier,logbufs=8,logbsize=256k 0 0 <h3>Moving Forward <p dir="auto">There are many other improvements and techniques that we have planned for testing, it is extremely important that we don't rely on scaling vertically to handle the ever increasing demands from the network. Emerging tech such as <a href="https://www.intel.co.uk/content/www/uk/en/architecture-and-technology/intel-optane-technology.html" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Optane drives is exciting, but it is imperative that the <code>nodeos application is continually improved to use all available resources on the host machine. <hr /> <p dir="auto"><a href="https://blockmatrix.network" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Block Matrix are currently a paid standby BP for the EOS network. We are super passionate about the EOS project and are focussed on creating robust infrastructure and open sourcing everything we build to support the network and the wider community. <p dir="auto"><a href="https://github.com/BlockMatrixNetwork" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Github<br /> <a href="https://t.me/blockmatrixnetwork" target="_blank" rel="nofollow noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Telegram
It would require a little more setup, but you could minimize ram requirements and processing if you ran one nodeos dedicated to RPC requests (minus push_transaction) with the
<p dir="auto">And ran another for handling push_transaction without the <code>filter-on = *. This of course would require a proxy to route the requests. <p dir="auto">Update: Actually a proxy is not necessary as nodoes will forward on the transaction to other nodeos.filter-on = *for history_plugin andread-mode = head.Oh cool, will give this a go - thanks for the info!
"we now have a separate mount for the blocks and statedata directories"
so blocks and statedata share the same mount?
would it improve even farther to have blocks and statedate on separate mounts?
Yes, absolutely! This has been tested by some of my fellow BP's and works well.