Hivemind Easy Deployment With hivemind-docker

<h2>Project Information <p dir="auto"><br /> Project Name: hivemind-docker<span>Repository: <a href="https://github.com/Jolly-Pirate/hivemind-docker" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://github.com/Jolly-Pirate/hivemind-docker <h2>Introduction <p dir="auto"><span>Hive (<a href="https://github.com/steemit/hivemind" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://github.com/steemit/hivemind) is a "consensus interpretation" layer for the Steem blockchain, maintaining the state of social features such as post feeds, follows, and communities. Written in Python, it synchronizes an SQL database with chain state, providing developers with a more flexible/extensible alternative to the raw steemd API. <p dir="auto">I'm using docker-compose for this project because of its flexibility in managing multiple containers, and to minimize the use of long complicated docker cli commands. With dependency checks in the scripts, I covered many possibilities to make a hivemind deployment easy. The tricky and frustrating part was to get postgres to use a mapped local database outside of docker volumes. The reason for that is to have better control of the database and to simplify redeployment to other servers, instead of having to redownload/reimport a dump which takes hours. Postgres is picky about permissions, plus its native command <code>initdb was causing errors when synchronizing an imported hivemind dump. After solving the problems with some workarounds and thoroughly testing, hivemind-docker was finally ready. <hr /> <h3><center>An elephant and a whale can be friends <p dir="auto"><center><span><img src="https://images.hive.blog/768x0/https://info.crunchydata.com/hubfs/DockerNPostgresLogos.png" srcset="https://images.hive.blog/768x0/https://info.crunchydata.com/hubfs/DockerNPostgresLogos.png 1x, https://images.hive.blog/1536x0/https://info.crunchydata.com/hubfs/DockerNPostgresLogos.png 2x" /> <hr /> <h2>Features <ul> <li>High flexibility with simple commands <li>Local storage of the database for easier redeployment in an infrastructure <li>Cli commands to control the containers <h2>Requirements <ul> <li>Docker Engine 18.06.0+ <li>PostgreSQL 10+ <li>2.5GB of memory for hivemind synch process <li>250GB storage for the database <h2>Git Installation <pre><code>git clone https://github.com/Jolly-Pirate/hivemind-docker.git cd hivemind-docker chmod +x run.sh <h2>Preparation <p dir="auto">Do the following steps sequentially: <ol> <li>Create a .env file from the example and secure it<br /> <code>cp .env.example .env<br /> <code>chmod 700 .env <li>Carefully edit all the variables in the .env file and save it<br /> <code>nano .env <li>Preinstall tools and NTP synchronization<br /> <code>./run.sh preinstall <li>Install docker and docker-compose<br /> <code>./run.sh installdocker <li>Build the needed containers<br /> <code>./run.sh build <h2>Running Postgres <p dir="auto">Hivemind requires a postgres backend. Start postgres with these two commands consecutively: <p dir="auto"><code>./run.sh initdb (will initialize a fresh database cluster)<br /> <code>./run.sh start postgres (will start postgres using the credentials from .env) <h2>Importing database dump <p dir="auto">For an efficient way to get hivemind going with a short DB synchronization, download a database dump (from a daily postgres snapshot), and import it. The dump was done with PostgreSQL 10.8. ETA depends on your internet speed, storage and CPU. <p dir="auto">NOTE: The download and import will run in screen sessions and automatically exit when complete. After the dump file is fully read, the import may seem stalled, but it's actually creating the indexes. Do not stop the process with <code>ctrl-c or close the session . You can detach from the session with <code>ctrl-a-d. <p dir="auto"><code>./run.sh importdb <p dir="auto"><strong>ETA ~3h <h2>Running Hivemind <p dir="auto">After the DB import, start hivemind to synchronize the missing blocks. <p dir="auto"><code>./run.sh start hive <p dir="auto"><strong>ETA ~1h <h2>Running Hivemind and Postgres simultaneously <p dir="auto">If you didn't do the 3 steps above, hivemind can be ran from scratch, the script will create a new postgres database, then synchronize it with an endpoint RPC server. However be advised that this approach is lenghthy, depending on your machine specs. For this <em>plug-n-play solution, run the following command: <p dir="auto"><code>./run.sh start all <p dir="auto"><strong>ETA few days <p dir="auto">To stop the hivemind and postgres containers <p dir="auto"><code>./run.sh stop all <p dir="auto">If you already imported the database, doing <code>./run.sh start all will resume synchronization from the last processed block. <h2>Testing Hivemind <p dir="auto">Once hivemind is fully synchronized, you can test it by querying it on the port you defined in <code>.env, for example <code>HIVEMIND_PORT=8080: <p dir="auto"><code>./run.sh testhive <p dir="auto">which runs this command: <p dir="auto"><code>curl -s --data '[{"jsonrpc":"2.0", "method":"condenser_api.get_follow_count", "params":{"account":"initminer"}, "id":1}]' http://localhost:8080 | jq -r <p dir="auto">and gives this result: <pre><code>[ { "jsonrpc": "2.0", "result": { "account": "initminer", "following_count": 0, "follower_count": 15 }, "id": 1 } ] <h2>Running Jussi <p dir="auto"><span>Jussi is an optional reverse proxy, its configuration won't be covered in this guide. You can check it out at <a href="https://github.com/steemit/jussi" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://github.com/steemit/jussi <p dir="auto">Edit and place the DEV_config.json file in the hivemind-docker folder, then start it with: <p dir="auto"><code>./run.sh start jussi <p dir="auto">You can test it with <p dir="auto"><code>./run.sh testjussi <h2>Checking the logs <p dir="auto">At any time, you can check the logs with: <p dir="auto"><code>./run.sh logs <p dir="auto">Press <code>ctrl-c to stop following the logs. <h2>Project command options <p dir="auto">The commands for managing the docker project are listed by typing: <p dir="auto"><code>./run.sh <p dir="auto">Here's a summary of the available commands: <pre><code> preinstall - preinstall tools and NTP synchronization installdocker - install docker and docker-compose build - stop the running containers and (re)build all the images initdb - initialize database cluster (e.g. postgresql database) importdb - download and import the database dump start|stop|restart (e.g. start all) all - initdb+postgresql+hivemind postgres - postgresql container (with initdb dependency) hive - hivemind container (with postgresql dependency) jussi - jussi reverse proxy enter - enter a container with bash shell; e.g. enter hive logs - live logs of the running containers status - check the containers status testhive - test a hive API call to hivemind testjussi - test a steemd API call to jussi dbsize - check the database size dbactivity - check the database activity <h2>Proof of work done <p dir="auto"><span>Github account: <a href="https://github.com/Jolly-Pirate" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://github.com/Jolly-Pirate <h3>Acknowledgment <p dir="auto"><span>Thanks to <a href="/@emrebeyler">@emrebeyler. His guide on setting up hivemind was an inspiration for this project. <ul> <li><span><a href="https://steemit.com/hivemind/@emrebeyler/steps-to-restore-a-hivemind-database-snapshot" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://steemit.com/hivemind/@emrebeyler/steps-to-restore-a-hivemind-database-snapshot <li><span><a href="https://steemit.com/hivemind/@emrebeyler/update-to-daily-hivemind-snapshots" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://steemit.com/hivemind/@emrebeyler/update-to-daily-hivemind-snapshots <p dir="auto">Also, thanks for providing the community with a daily hivemind database snapshot. <p dir="auto"><center><sub><span>Posted via <a href="https://www.palnet.io" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://www.palnet.io<br /> <a href="https://www.palnet.io" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link"><img src="https://images.hive.blog/0x0/https://cdn.steemitimages.com/DQma2sXTD5C79ByLLUddrjzyztXK2xCbzz2xPo4FtZfVpxP/plcoin-onblack_gif.gif" /> <hr /> <p dir="auto"><center> <h4>Available & Reliable. I am your Witness. I want to represent You. <h5>🗳 If you like what I do, consider voting for me 🗳 <p dir="auto"><a href="https://steemit.com/~witnesses" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link"><img src="https://images.hive.blog/0x0/https://steemitimages.com/DQmfRx7jNSrFAnRZPrKRHLDxR22gRFYTBnKr2uLNdVty3MG/drakos-witness-thanks.gif" alt="Vote" /> <h6>Check my guide about <a href="https://steemit.com/witness-category/@drakos/tips-and-guidelines-voting-for-witnesses" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Voting for Witnesses. <h6><span>Go to <a href="https://steemit.com/~witnesses" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://steemit.com/~witnesses. Next to my name, click the <img src="https://images.hive.blog/768x0/https://steemitimages.com/DQmYq3yGqzvbK4JQFudeXoDp6PEtdny8vSvX5swqW5sA2fE/chevron-up.png" srcset="https://images.hive.blog/768x0/https://steemitimages.com/DQmYq3yGqzvbK4JQFudeXoDp6PEtdny8vSvX5swqW5sA2fE/chevron-up.png 1x, https://images.hive.blog/1536x0/https://steemitimages.com/DQmYq3yGqzvbK4JQFudeXoDp6PEtdny8vSvX5swqW5sA2fE/chevron-up.png 2x" /> once. <h6>Alternatively you can use SteemConnect to <a href="https://app.steemconnect.com/sign/account-witness-vote?witness=drakos&approve=1" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">vote for me or <a href="https://app.steemconnect.com/sign/account-witness-proxy?proxy=drakos&approve=1" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">set me as proxy <p dir="auto">