<p dir="auto">After going live with the <a href="https://gridcoin.science/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">gridcoin.science website in support of the research blockchain, I decided to take another step to support my favourite things: Steemit, Gridcoin and science. Today, I added a new witness node to the network named <strong>gridcoin.science, embodying all three in a single endeavor to the mutual benefit of our communities. <h2>A Little About Me <p dir="auto">I am Tim, a biomedical engineer living in Christchurch, New Zealand. I have a Bachelor's Degree in Mechanical Engineering, and am currently 6 months from completing a PhD. This doctorate is focused on building computer models of the brain to better understand neurodegenerative diseases like Alzheimer's and Parkinson's. If you would like to learn more about me, I invite you to read my <a href="https://steemit.com/introduceyourself/@dutch/hello-steemit-dutch-biomedical-engineer-checking-in" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">introductory post. <p dir="auto">Some of what I get up to on Steemit: <ul> <li><p dir="auto">I am an active moderator for the <a href="https://utopian.io" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Utopian.io<a href="https://utopian.io/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Utopian.io homepage.<span> platform, created by <a href="/@elear">@elear. In this capacity I curate Steemit content to reward high quality contributions to open-source communities. The project has already released over USD$227,506 in contributor rewards, and can be explored further on the <li><p dir="auto">I am an active Steemit community member. My blog aims to represent, inform and progress the distributed compute communities surrounding Gridcoin and BOINC. These topics are close to my heart as a medical researcher, where my progress is often slowed by compute limitations. I aim to facilitate a cross-over between the Steemit community and the scientific community to the benefit of both. <li><p dir="auto">I have been involved with the ongoing debate and analysis around vote and flagging weights, linear vs <code>n^2 rewards, and vote buying. As the platform moves to address these issues for the betterment of the whole, I'd like to be involved in both the search and the implementation of solutions to these challenges. <h2>Meet Nick <p dir="auto"><span>Running this witness with me is my close friend Nick (<a href="/@deltik">@deltik), a software and systems engineer living around the world – usually in Austin, Texas, or Vienna, Austria. The time difference between him and myself ensures round-the-clock support if anything were to happen to the witness server. <p dir="auto">Nick's professional experience began with web hosting back in 2008. Since then he has branched out into managed support, custom software solutions, and cloud virtualization. He is currently working as a converged infrastructure engineer, while contributing significant resources to open-science BOINC projects like <a href="http://srbase.my-firewall.org/sr5/index.php" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">SRBase and <a href="https://www.cosmologyathome.org/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Cosmology@home. <p dir="auto">Despite the ability to copy conventional digital data bit for bit, that data is easily corrupted and lost. <p dir="auto"><a href="https://xkcd.com/1683/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link"><img src="https://images.hive.blog/768x0/https://imgs.xkcd.com/comics/digital_data.png" alt="If this text is showing instead of an image, that means XKCD URLs don't last forever, either." srcset="https://images.hive.blog/768x0/https://imgs.xkcd.com/comics/digital_data.png 1x, https://images.hive.blog/1536x0/https://imgs.xkcd.com/comics/digital_data.png 2x" /> <p dir="auto">Nick is fascinated by blockchains because in contrast, they are ideal for preserving data integrity. One such blockchain implementation is Steem, and he believes in the potential of this backend for the social aspect rather than it merely being a transaction ledger. <p dir="auto">For this reason, Nick has joined forces with me to support the Steem blockchain with a new witness! <h2>Witness Tech Specs <p dir="auto">We've implemented the witness daemon a bit differently. We wanted the foundation to be rock-solid, so we started with dedicated hardware: <blockquote> <p dir="auto">CPU: Intel® Xeon® Processor D-1521 @ 2.40GHz<br /> Memory: 64GB DDR4-2666 ECC<br /> Disk: 2×480GB SSD and 2×2TB HDD<br /> Bandwidth: 250Mbps<br /> Operating System: Ubuntu 16.04 LTS <h3>Storage <p dir="auto">On the disks, we did a custom installation of <a href="https://www.ubuntu.com/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Ubuntu 16.04 LTS on a <a href="https://github.com/zfsonlinux/zfs/wiki" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">ZFS root. There are two pools that mirror each type of disk (<code>fastpool made from the SSDs and <code>slowpool made from the HDDs). We chose ZFS because of its unparalleled data reliability features. <p dir="auto">What does ZFS do for us? <ul> <li>The Steem blockchain is quite compressible. Since ZFS supports native compression, we were able to achieve a 37% reduction in disk usage. <li>ZFS lets us take regular snapshots, which means we can roll back the witness in case of a screw-up. Hopefully there won't be any incidents, but it's best to be prepared. <li>Snapshots also allow us to send backups as a stream (<code>zfs send/<code>zfs receive) into flexible backup destinations. We have a 500GiB NFS backup datastore right now, which is more than enough for the witness, and we're exploring more resilient disaster recovery solutions for the future. <li>The ZFS Adjustable Replacement Cache (ARC) boosts performance of the ZFS datasets, which make input-output (I/O) operations on our witness faster. <li>Unlike standard RAID, ZFS is aware of the data it stores and checksums all of it. Combined with the disk mirror (analogous to <a href="https://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">RAID 1), we are even protected from <a href="https://en.wikipedia.org/wiki/Data_corruption#SILENT" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">silent data corruption, and regular scrubs heal any inconsistent data automatically. <h3>Virtualization <p dir="auto">As for the witness itself, we put it in a <a href="https://www.linux-kvm.org/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">KVM virtual machine on this dedicated server. Here are the specs of that virtual machine: <blockquote> <p dir="auto">CPU: 4 logical cores of Intel Xeon D-1521<br /> Memory: 16GiB<br /> Disk: 400GiB SSD-backed<br /> Bandwidth: Unmetered<br /> Operating System: Ubuntu 16.04 LTS <p dir="auto">Virtualization has several advantages: <ul> <li><strong>Isolation: The witness runs in its own environment so that it can't interfere with the operational infrastructure below. <li><strong>Security: If the witness is compromised by some unforeseen vulnerability, we can stop it in its tracks, go back in time, and rectify the vulnerability before it's exploited again. <li><strong>Overhead: Virtual machines don't have the hardware overhead of dedicated servers, so starting up is much faster. <li><strong>Portability: In combination with ZFS snapshots, we've opened the possibility of migrating the witness to another physical server if we ever need to upgrade. <p dir="auto">Why not containerization? <p dir="auto">Although <a href="https://github.com/steemit/steem/blob/master/Dockerfile" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link"><code>steemd can make use of operating-system-level virtualization through the use of Docker, we chose full virtualization for greater isolation and to explore how far we could go with zram. <h3>Memory <p dir="auto">There's a problem. As of January 2018, <code>steemd loads over 20GiB into its memory-mapped file, <code>shared_memory.bin, but we only gave 16GiB of RAM to the virtual machine. Doesn't that mean we run out of memory or have to do unacceptably slow <a href="https://en.wikipedia.org/wiki/Paging" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">paging to swap? <p dir="auto">Actually, no. We discovered that this witness runs comfortably with the help of <a href="https://www.kernel.org/doc/Documentation/blockdev/zram.txt" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">zram. <code>steemd's entire memory-mapped file, no matter how large the <code>config.ini <code>shared-file-size is set, compresses to around 8.2GiB (as of January 2018) in zram. <p dir="auto">Thanks to the speed of zram, there is a negligible loss in Steem witness performance. <p dir="auto">The Steem blockchain is going to keep growing, though, which means that someday, the shared memory file will be too large to fit in the 16GiB allocated to the witness virtual machine. Fortunately, the dedicated server is fully under our control and has 64GB of RAM, so we will have room to scale vertically for a while. <h3>Room for Improvement <p dir="auto">If the community supports this witness, we will match the commitment by developing a high availability infrastructure. <p dir="auto">Our goal will be to minimize downtime. There are existing ways to make witnesses more robust, of which we will make use, but we would like to go above and beyond. We are considering: <ul> <li>The possibility of an active/active configuration, allowing two or more witnesses to run at the same time without stepping on each other <li>Shared storage, enabling live migrations so that physical servers can be taken down while the virtual machines stay up <li>Faster ways to redeploy <code>steemd, maybe obviating the need for <code>--replay-blockchain <h2>Closing Remarks <p dir="auto">If you have made it this far - thank you for reading! <p dir="auto"><strong>gridcoin.science to the box at the bottom of the page, click vote, and authorise using your Active Key.<span>To vote, visit <a href="https://steemit.com/~witnesses" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://steemit.com/~witnesses and add <p dir="auto"><center><img src="https://images.hive.blog/768x0/https://steemitimages.com/DQmcdJwtxDGhAPeB2oGn2F8K4nQKaogVpQfd4oXoQDLn9uZ/image.png" srcset="https://images.hive.blog/768x0/https://steemitimages.com/DQmcdJwtxDGhAPeB2oGn2F8K4nQKaogVpQfd4oXoQDLn9uZ/image.png 1x, https://images.hive.blog/1536x0/https://steemitimages.com/DQmcdJwtxDGhAPeB2oGn2F8K4nQKaogVpQfd4oXoQDLn9uZ/image.png 2x" /> <p dir="auto"><span>Wrapping things up, I'd like to invite you to ask <a href="/@deltik">@deltik or me anything in the comments below. Let's get to know each other better! <hr /> <p dir="auto"><a href="https://steemit.com/@dutch" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link"><img src="https://images.hive.blog/0x0/https://steemitimages.com/DQmZ2WQ4aAcT7qRe5hizgmAUK4dDGh79xBD6CmXM5AwWpjv/dutch.gif" alt="dutch.gif" />