How to setup and run the EOSIO Genesis Snapshot Generator

These instructions and acompanying video provide detailed steps for creating a genesis snapshot used to bootstrap an EOS.IO blockchain that will reflect the contributions of stakeholder in the EOS tokensale.

<blockquote> <p dir="auto"><em><strong>Update 15th Feb 2018<br /> We've just heard from the EOSIO development team that there may not be a 'genesis block' as such with the new EOS BIOS process so don't invest too much time with the process we documented here until we all get the update. The GitHub repository is also being updated with new code so the procedure in this post uses now used the <code>0.2.1 Testnet Release of the snapshot generator. <p dir="auto"><span>We are using the software provided by the EOS.IO development team at <a href="https://github.com/EOSIO/genesis" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://github.com/EOSIO/genesis and expanding on the high-level installation guide, so any developer or enthusiast can create a snapshot, genesis file and so validate the tools and process. <p dir="auto"><strong>As we get closer to a community blockchain launch, it’s important for as many people as possible to validate this code, it’s output and ideally develop alternate implementations so we have certainty in the starting stake of the EOS community in any EOS.IO based blockchain. <p dir="auto">At a very high level, the snapshot generator queries the Ethereum EOS tokensale contracts to determine all contributions and balances and then outputs this to a comma seprated file <code>snapshot.csv along with a snapshot meta-data file <code>snapshot.json which contains some high-level stats and checksums. <p dir="auto"><span>These snapshot files, can then be used to create an EOS.IO compatible genesis file using the 'Genesis Block Generator' at <a href="https://eosio.github.io/genesis/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://eosio.github.io/genesis/ <blockquote> <p dir="auto">The EOS.IO development team have told us to expect a significant update to the Genesis Snapshot Generator and accompanying documentation at some point. We'll update the post and video when it's available. <p dir="auto">Let’s get started… <hr /> <h2>How to setup and run the EOSIO Genesis Snapshot Generator <p dir="auto">Prepare a fresh VM, physical or cloud server with the following specifications: <p dir="auto">Ubuntu 16.04 (LTS) Server, 4 CPU cores , 8GB memory and at least 120Gb of SSD storage. <p dir="auto">Update unbutu with the latest repository package lists<br /> <code>sudo apt-get update <h3>Install screen <p dir="auto">Let's install screen which allows us to keep our processes running with output even if you get a dropped connection. <p dir="auto"><code>sudo apt-get install screen <p dir="auto"><code>screen –S eos <blockquote> <p dir="auto"><span>If you've never used screen before, check out this quick tutorial <a href="https://www.mattcutts.com/blog/a-quick-tutorial-on-screen/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://www.mattcutts.com/blog/a-quick-tutorial-on-screen/ <p dir="auto">Exit the eos screen by typing <code>Ctrl-a then <code>d. <h3>Install parity.io and full sync the ethereum blockchain <p dir="auto"><code>bash <(curl https://get.parity.io -Lk) -r stable <p dir="auto">Run parity in it’s own ‘screen’ so we can monitor it easily <p dir="auto"><code>screen –S parityio <p dir="auto"><code>parity --no-warp <p dir="auto">It’s very important to use the <code>--no-warp option with parity to be able to create accurate snapshot data <blockquote> <p dir="auto">warping will skip over historical data that the snapshot requires, tricking web3 into thinking the chain is synced when it is not. <p dir="auto">The current ethereum full blockchain is over 50GB in size and will likely take a few days to sync! <p dir="auto">Again, exit the screen by typing <code>Ctrl-a then <code>d, leaving parity to sync. <h2>EOS.IO Genesis Snapshot Build Process <h3>Clone the EOS.IO genesis GitHub respository <p dir="auto"><code>git clone https://github.com/EOSIO/genesis<br /> <code>cd genesis<br /> <code>git checkout 0.2.1 <h3>Install MySQL <p dir="auto"><code>cd<br /> <code>sudo apt-get install mysql-server <p dir="auto">When prompted be sure to enter a root password for the mysql root account <p dir="auto">Test that the mysql service is running... <p dir="auto"><code>mysqladmin -u root -p version <h3>Create the mysql <code>eos_snapshot database and load the schema <p dir="auto">Log into mysql command line<br /> <code>mysql -u root -p <p dir="auto">Create the <code>eos_snapshot database<br /> <code>CREATE DATABASE eos_snapshot;<br /> <code>exit <p dir="auto">Create the <code>eos_snapshot database schema by importing the EOS.IO developed schema definitions<br /> <code>mysql -u root -p eos_snapshot < ./genesis/bin/schema.sql <p dir="auto">Check that the mysql schema import was successful <p dir="auto">Login.<br /> <code>mysql -u root -p eos_snapshot<br /> and<br /> <code>SHOW TABLES; <p dir="auto">You should see the following output: <pre><code>+------------------------+ | Tables_in_eos_snapshot | +------------------------+ | buys | | claims | | keys | | reclaimables | | registrations | | snapshot | | state | | transfers | | wallets | +------------------------+ 9 rows in set (0.00 sec) <p dir="auto">exit the mysql client. <h3>Install Node.js and NPM <p dir="auto">Ensure your ubuntu installation can access the correct Node version repositories, this will enable us to install Node.js v6.x<br /> <code>curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash - <p dir="auto">Install nodejs (JavaScript Runtime)<br /> <code>sudo apt-get install -y nodejs <p dir="auto">Change to the EOS.IO genesis root directory<br /> <code>cd genesis <p dir="auto">Install the 'build-essentials' so we can compile the scrypt 'cryptographic hash' npm dependancy<br /> <code>sudo apt-get install build-essential <p dir="auto">Now install/compile the required npm dependencies for the genesis snapshot software that are referenced in <code>package.json (npm is a package manager for JavaScript) <p dir="auto"><code>sudo npm install <h3>Set your snapshot generator defaults by creating a <code>config.js <p dir="auto"><code>cp config.default.js config.js <p dir="auto">edit the config file using your favourite editor<br /> <code>nano config.js <p dir="auto">Remember to change the <code>eth_node_path to <strong>your home folder / parity defaults. <pre><code>module.exports = { period: 1, include_b1: true, cache: false, fallback: false, overwrite_snapshot: false, recalculate_wallets: false, author: "info@eosphere.io", //ETH node eth_node_type: 'ipc', eth_node_path: '/home/rdold/.local/share/io.parity.ethereum/jsonrpc.ipc', //redis redis_host: "localhost", redis_port: 6379, mysql_db: "eos_snapshot", mysql_user: "root", mysql_pass: "eos", mysql_host: "localhost", mysql_port: 3306 } <p dir="auto">This snapshot config file will create a snapshot for Period's 0 and 1, without the experimental fallback support. <h3>Run the snapshot <p dir="auto">From the genesis directory run<br /> <code>nodejs snapshot.js <pre><code>:~/genesis$ nodejs snapshot.js prompt: Config file detected, load it?: (true) .--------------------------------------------------------------------------------. | Settings | |--------------------------------------------------------------------------------| | period | 1 | | include_b1 | true | | cache | false | | fallback | false | | overwrite_snapshot | false | | recalculate_wallets | false | | author | info@eosphere.io | | eth_node_type | ipc | | eth_node_path | /home/rdold/.local/share/io.parity.ethereum/jsonrpc.ipc | | redis_host | localhost | | redis_port | 6379 | | mysql_db | eos_snapshot | | mysql_user | root | | mysql_pass | 42 | | mysql_host | localhost | | mysql_port | 3306 | '--------------------------------------------------------------------------------' Starting in 5 seconds. sequelize deprecated String based operators are now deprecated. Please use Symbol based operators for better security, read more at http://docs.sequelizejs.com/manual/tutorial/querying.html#operators node_modules/sequelize/lib/sequelize.js:242:13 MySQL: Connected Web3: Connected Web3 is Still Syncing (At Block #5041978). trying again in 30 seconds Web3: Synced Truncating tables, starting clean --------------------- INFO: Period Map needs to be Updated - not synced, syncing to Period (Today is Period 230) Generating new row(s) for Period Block Map (Up to Period 1) This will take a little while --------------------- Added period 0 to the Period Map :) { begin: 3904416, end: 3957867 } Added period 1 to the Period Map :) { begin: 3957868, end: 3962664 } Syncing Contract State between block #3904416 & 3962664 Syncing Contracts between block #3904416 and #3962664, this may take a while. .-----------------------. | 3904416 ~> 3958215 | |-----------------------| | Transfers | 3648 | | Buys | 19967 | | Claims | 2917 | | Registrations | 6537 | | Reclaimables | 0 | '-----------------------' Started: 29.86 seconds ago, Average: 55.50 ms/pass finished .-----------------------. | Complete: 3904416 ~> 3962664 | |-----------------------| | Transfers | 14127 | | Buys | 22442 | | Claims | 8044 | | Registrations | 7981 | | Reclaimables | 5 | '-----------------------' Started: 40.30 seconds ago, Average: 69.12 ms/pass Syncing Wallets ... ... ... .... { period_map: [ { begin: 3904416, end: 3957867 }, { begin: 3957868, end: 3962664 } ], block_begin: 3904416, block_end: 3962664, sync_contracts: { buys: 22442, claims: 8044, registrations: 7981, transfers: 14127, reclaimables: 5 }, total: 15992, tests: { total_supply: true, daily_buys: true, negative_balances: true, validation_balance: true } } .------------------------. | Tests Passed | |------------------------| | total supply | ✓ | | daily buys | ✓ | | negative balances | ✓ | | validation balance | ✓ | '------------------------' Fallback: Skipping Fallback Snapshot Table Synced 5229 Records Saved to CSV Generating snapshot.json Generating DB Table Checksums Snapshot meta written to snapshot.json Snapshot for Period #1 Completed. <h3>Genesis <p dir="auto">If all has run well, you'll have a 'snapshot.csv' that identifies all the EOS ERC-20 stakeholder addresses along with their registered EOS public key and snapshot EOS balance. <p dir="auto">You can find these files in <code>$HOME/genesis/data/ with a folder for each period snapshot you constructed. To see the Period 1 snapshot files we just created :<br /> <code>ls $HOME/genesis/data/1 <p dir="auto"><span>We'll leave actually creating a genesis block file for another tutorial, but if your keen head over to <a href="https://eosio.github.io/genesis/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://eosio.github.io/genesis/ and use the files you just created as input. <p dir="auto">As always, leave any feedback or suggestions/requests for other walkthroughs in the comments. <hr /> <p dir="auto"><em>EOSphere are passionate group of EOS enthusiasts and data centre professionals setup to help in the global community launch of an EOS.IO blockchain later this year. Based in Australia, EOSphere are initially focused on bootstrapping the Australian EOS community by hosting a series of meetups in each state capital. <p dir="auto"><span>Follow us on steemit <a href="/@eosphere">@eosphere and on twitter @ <a href="https://twitter.com/eosphere_io" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://twitter.com/eosphere_io