Steem Pressure #8: Power Up Your dApp. RethinkDB? KISS my ASAP.

While building your Steem blockchain dApp, you will soon realize that it’s not that easy to get the data you are looking for quickly, unless there’s already an API that gives you what you need.

<p dir="auto"><center> <p dir="auto">We have several reliable, general purpose, “full” API nodes, provided by Steemit Inc. and community members: <div class="table-responsive"><table> <thead> <tr><th>Steem API endpoint<th>owner <tbody> <tr><td><span><a href="https://api.steemit.com" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://api.steemit.com<td>Steemit Inc. <tr><td><span><a href="https://anyx.io" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://anyx.io<td><span><a href="/@anyx">@anyx <tr><td><span><a href="https://api.steem.house" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://api.steem.house<td><span><a href="/@gtg">@gtg <tr><td><span><a href="https://steemd.minnowsupportproject.org" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://steemd.minnowsupportproject.org<td><span><a href="/@followbtcnews">@followbtcnews <tr><td><span><a href="https://steemd.privex.io" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://steemd.privex.io<td><span><a href="/@privex">@privex <p dir="auto">You can use them for development or even (if you are brave enough) for production.<br /> If your dApp is small and simple, it might work. And yes, in many cases it will.<br /> But is it optimal? No. <p dir="auto">In previous episodes, I described different types of nodes and their names and how this can be misleading when it comes to the so-called “full” nodes. <p dir="auto">TL;DR<br /> A full node means here something different than in the Bitcoin realm, where Bitcoin full node is pretty much something that a Steem consensus node can do.<br /> Here, on Steem, the word “full” doesn’t refer to anything related to the blockchain - it refers to the fully featured set of APIs enabled within steemd. <p dir="auto">Do you really need a full set of API calls?<br /> In most cases you don’t. To get an idea about the information you can get from common Steem APIs, take a look at the devportal: <p dir="auto"><span><a href="https://developers.steem.io/apidefinitions/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://developers.steem.io/apidefinitions/ <p dir="auto"><sup>By the way, please consider <a href="https://steemconnect.com/sign/update-proposal-votes?proposal_ids=[63]&approve=true" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">voting for <a href="/@inertia/proposal-dev-portal-documentation-q1-2020">Steem.DAO @inertia’s proposal, who already greatly contributed to the Dev Portal Documentation and is eager to do more. <p dir="auto">For example, if you need the information returned by <code>get_market_history, take a look at <code>condenser_api.get_market_history and at its underlying api, in which case you will find: <blockquote> <p dir="auto">Also see: <code>market_history_api.get_market_history<br /> Which means that the API node you need should be running a <code>market_history plugin. <p dir="auto">Another example is a situation in which you want to find who “Resteemed” (a.k.a. “Reblogged”) a given post.<br /> Here, you are lucky, because there’s a method for that called <code>get_reblogged_by and provided by the plugin <code>follow. <p dir="auto">Maybe you want to get tags used by a specific author? That’s easy too, there’s the <code>get_tags_used_by_author method provided by - yes, you guessed correctly - <code>tags plugin. <p dir="auto">There’s a pretty big chance that if your app needs to use <code>get_market_history, it doesn’t need <code>tags or <code>follow plugins, because it’s not blogging oriented, and vice-versa - if you deal with <code>tags and <code>follow, you probably don’t care about the internal market. <p dir="auto">Such knowledge lets you optimize the API infrastructure needed for your dApp.<br /> For instance, you can run a “fat node” without <code>market_history, <code>account_history(see <a href="/@gtg/steem-pressure-6-mira-ymmv-rtfm-tldr-lgtm">Steem Pressure #6) and use <a href="https://github.com/steemit/hivemind" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">hivemind to get the <code>tags and <code>follow features. <p dir="auto">That’s not the end yet. You get a lot more data than you’ve asked for, but still that may not be what you need. <p dir="auto">Do you know why? If not, then please, read the first paragraph again. <p dir="auto">How many blocks has the witness <code>gtg missed so far?<br /> That’s easy.<br /> Run: <pre><code>curl -s --data '{"jsonrpc":"2.0", "method":"database_api.find_witnesses", "params": {"owners":["gtg"]}, "id":1}' https://api.steem.house <p dir="auto">You will instantly get your answer, because the <code>find_witnesses method returns this information… among many pieces of information that you don’t need. <p dir="auto">Now, try something more complicated: <h3>Problem to solve <p dir="auto"><br /><br /> And how many blocks did he produce that contained more than 50 transactions?<br /> And what percentage of blocks produced in 2018 had more non-virtual operations than transactions?<br /> How many blocks had only one transaction and which of them had the highest number of non-virtual operations? What is the id of the transaction that contains them?<span>How many blocks did <a href="/@gtg">@gtg produce until block 40000000?<span> And how many blocks did <a href="/@gtg">@gtg produce in August 2017? <p dir="auto">You should be able to answer all these questions, all the data are in the blockchain after all. The problem is that steemd has no suitable API to answer these questions quickly.<br /><a href="https://steemsql.com" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">SteemSQL “SQL server database with all Steemit blockchain data”, which is available in subscription model), or look into blocks to find your answer.<span> This means that you either have access to a third party service / API that can answer these questions (see <a href="/@arcange">@arcange’s <p dir="auto">In the latter case, you will have to inspect all the blocks ever produced using the <code>get_block method.<br /> The good news is that every consensus node can run the <code>block_api plugin.<br /> The Steem blockchain has recently passed the 40 million blocks mark.<br /> This means that in order to answer the first question, you will need to send more than 40 million requests to the API endpoint.<br /> Each time this type of question is asked you will need to repeat the procedure, unless you… <h3>Build and run your own custom Steem API <p dir="auto"><center><br /> <img src="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmS3Nrmnqbo21TPLNSjrrGDeWrd63KArjUXyAgn3u7pFP2/steemit-meme-api.jpg" alt="steemit-meme-api.jpg" srcset="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmS3Nrmnqbo21TPLNSjrrGDeWrd63KArjUXyAgn3u7pFP2/steemit-meme-api.jpg 1x, https://images.hive.blog/1536x0/https://cdn.steemitimages.com/DQmS3Nrmnqbo21TPLNSjrrGDeWrd63KArjUXyAgn3u7pFP2/steemit-meme-api.jpg 2x" /><br /> <br /> <sub>TL;DR in Memeglish so that project managers can get it. <p dir="auto">No. I’m not going to build an API in this post.<br /> It is supposed to be <strong>your own API, solving <strong>your problems, optimized for <strong>your use case.<br /> Instead, I’m going to play with some blockchain data to give you some tips. <p dir="auto">To illustrate this, let’s analyze the problem described above. .<br /> Let’s ignore incoming blocks and keep our focus in the range up to 40000000 blocks (catching up with the head block and dealing with micro-forks is a good topic for another episode; in the meantime, you can take a look at how hivemind does this).<br /> Let’s assume that you can’t or don’t want to extend steemd with your custom <code>fancy_plugin or a <code>fancy_plugin_api that does the job. <h1>Keep It Simple Stupid <p dir="auto">We don’t need information about virtual operations, so instead of using <code>get_ops_in_block, which requires <code>account_history plugin and API, we can use <code>get_block, which saves a lot of resources, if you want to run your own node. <h4>How does a request look? <pre><code>curl -s --data '{"jsonrpc":"2.0", "method":"block_api.get_block", "params":{"block_num":197696}, "id":1}' <h4>How does a response look? <pre><code>{ "jsonrpc": "2.0", "result": { "block": { "previous": "0003043f5746b4d9d06932467ad852ac5d71231c", "timestamp": "2016-03-31T13:56:18", "witness": "sminer22", "transaction_merkle_root": "63321e3f001bed17d301399c5eeaa9b37a0bf74b", "extensions": [], "witness_signature": "1f64e94d51baef0b84be04644fafc0b05959e80db64de7b099d00499587f23f97216acd3d8df3c3b81f672b24f8cf47167720d205693d3f5713a65bcf004cffef3", "transactions": [ { "ref_block_num": 1087, "ref_block_prefix": 3652470359, "expiration": "2016-03-31T13:56:45", "operations": [ { "type": "vote_operation", "value": { "voter": "proskynneo", "author": "proskynneo", "permlink": "steemit-firstpost-1", "weight": 10000 } } ], "extensions": [], "signatures": [ "202765ea6b688c2ce968ff2b3f1f6715fddd67374f409a202ae720ca072cb6a919175c27921edfc24c5e3d898a221e88395af31a155b1d60b2bd5475d30a30e226" ] } ], "block_id": "00030440d0fbfc8323a1388cd009f2a7f9f43162", "signing_key": "STM6tC4qRjUPKmkqkug5DvSgkeND5DHhnfr3XTgpp4b4nejMEwn9k", "transaction_ids": [ "679bcc6414b2f2bd16818a83f020bb9a478a84c4" ] } }, "id": 1 } <p dir="auto"><sup>Well, when pretty printed for your convenience, API normally returns a compact version, which is just one line without extra whitespaces. <h4>Caveat <p dir="auto">As you can see, the resulting data don’t contain <code>block_number, but you can use the <code>id that equals the block number you are asking for. <p dir="auto">We need to get 40M blocks.<br /> That’s a lot, but fortunately you need to do this only once.<br /> Irreversible blocks are not going to change because… well, I hope you can guess why ;-)<br /> Even if you use <a href="https://developers.steem.io/services/#json-rpc-batch" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">jussi’s batch requests, throwing that many requests at public API nodes will quickly trigger rate limiting, and even if it doesn’t, it will take a significant amount of time. <h4>Your own consensus node for simple tasks <p dir="auto">You can use your own consensus Steem node for that. Local means faster.<br /> Think about latency and how any latency (<code>1ms? <code>2ms?) is no longer negligible if you multiply it by 40000000.<br /> Such a node is not resource hungry, and with MIRA you can easily run it on a 8GB RAM machine with 500GB storage.<br /> A consensus node currently needs <code>245GB for <code>block_log and <code>56GB for the state file.<br /> Besides, your dApp will benefit from such a node also for broadcasting purposes.<br /> Configuration is as simple as that:<br /> <center><span><img src="https://images.hive.blog/0x0/https://cdn.steemitimages.com/DQmfJxAnAk2jvJMEoo3U3ivLDhie3vPvXxmSU8RfFLMrYcQ/config-ini.gif" /> <h4>Use someone else’s hard work <p dir="auto">There are already people (like me) who <a href="https://gtg.steem.house/get/blockchain/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">serve the <code>block_log to speed up replay times.<br /> If only there were a place where one could bulk download JSON files with Steem blocks...<br /> Well, if there’s a need for it, I can make it available with 1M block packages every time we reach 5 or 10 million.<br /> It will save a lot of time for all microservices that require block processing (the last mile can be synced using traditional methods) <h4>Bulk <p dir="auto">You’ve already seen how a single block looks in JSON format.<br /> Obviously, we are using compact output to save space.<br /> The whole data set (40M blocks in the uncompressed JSON format) takes 580GB.<br /> For our needs, we need only a small subset of such data. <p dir="auto">To get what’s needed we can use <code>jq: <pre><code>jq -c ' { id: .id, block_id: .result.block.block_id, timestamp: .result.block.timestamp, witness: .result.block.witness, txs: .result.block.transactions | length, ops: [.result.block.transactions[].operations[]] | length, txids: .result.block.transaction_ids }' <p dir="auto">After processing our blocks we get 46GB of uncompressed json data. <h4>Everything Should Be Made as Simple as Possible, But Not Simpler <p dir="auto">A flat file? Are you kidding me?<br /> Use your favorite database to store it.<br /> (Unless your favorite database is MongoDB, then use something else.) <h4>Or maybe RethinkDB? <p dir="auto"><span>Some time ago <a href="/@someguy123">@someguy123 told me about some fancy database designed to store JSON. <blockquote> <p dir="auto"><em>”RethinkDB is the first open-source, scalable JSON database built from the ground up for the realtime web. It inverts the traditional database architecture by exposing an exciting new access model - instead of polling for changes, the developer can tell RethinkDB to continuously push updated query results to applications in realtime. RethinkDB’s realtime push architecture dramatically reduces the time and effort necessary to build scalable realtime apps.” <p dir="auto">So this not only means effective and effortless storage of JSON data, but also sounds like a perfect choice for many Steem dApps.<br /> But is it? Maybe. For some needs. It’s not perfect for sure, but no solution is.<br /> I’m going to give it a try.<br /> Do your own research. Maybe it will fit your needs. If not, look for something else. <p dir="auto">Let’s see how easy it is to store data we need in RethinkDB <h5>Step 1: Install RethinkDB <p dir="auto">Check the RethinkDB webpage for <a href="https://rethinkdb.com/docs/install/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">installation instructions suitable for your server.<br /> For Debian or Ubuntu, it’s enough to do the usual: <h6>Add RethinkDB repository <pre><code>echo "deb https://download.rethinkdb.com/apt `lsb_release -cs` main" | sudo tee /etc/apt/sources.list.d/rethinkdb.list wget -qO- https://download.rethinkdb.com/apt/pubkey.gpg | sudo apt-key add - sudo apt-get update <h6>Install RethinkDB package <pre><code>sudo apt-get install rethinkdb <p dir="auto">That’s all. No questions asked. <h6>Install the RethinkDB python driver <p dir="auto">Because the <code>rethinkdb import subcommand has - surprisingly - an external dependency on the python driver, depending on the way you installed RethinkDB, you might need to get the RethinkDB python driver too.<br /> For that, you need to have <code>pip, and, ideally, <code>virtualenv, so if you don’t then: <pre><code>sudo apt-get install python3-pip python3-venv <p dir="auto">and once you have them, install the RethinkDB python driver into your virtual environment: <pre><code>python3 -m venv ./rethink source rethink/bin/activate pip install rethinkdb <h5>Step 2: Run the server <p dir="auto"><code>rethinkdb<br /> Yes. That’s it for now. You can play with more advanced features later.<br /> It enables an administrative console available through HTTP on <code>localhost:8080 <h5>Step 3: Load your data. <p dir="auto">Throw your JSON data at RethinkDB instance by running:<br /> <code>rethinkdb import -f blockstream.json --table steem.blocks<br /> Where <code>blockstream.json is your big 46GB file with data from 40M blocks.<br /> It goes to a table called <code>blocks within a database called <code>steem. <pre><code> [========================================] 100% 40000000 rows imported to 1 table in 3615.68 secs Done (3615 seconds) <p dir="auto">Not bad, on my hardware it consumes json stream a little bit faster than what a 100Mbps link could provide. <p dir="auto"><code>rethinkdb_data takes 66GB <p dir="auto"><center><br /> <img src="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmbkyKyE3kzUy75eCBaGQUzSDfgokiMBtnqpy8RDUXqEi1/rethinkdb.png" alt="rethinkdb.png" srcset="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmbkyKyE3kzUy75eCBaGQUzSDfgokiMBtnqpy8RDUXqEi1/rethinkdb.png 1x, https://images.hive.blog/1536x0/https://cdn.steemitimages.com/DQmbkyKyE3kzUy75eCBaGQUzSDfgokiMBtnqpy8RDUXqEi1/rethinkdb.png 2x" /><br /> <h5>Step 5: Make use of the data <p dir="auto">You can use the Data Explorer feature from the administrative console, or use Python, Javascript or Ruby to get the data you are looking for. <p dir="auto">If you want to learn more look at the <a href="https://rethinkdb.com/docs" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">RethinkDB documentation <p dir="auto">Depending on your needs, you can create and use secondary indexes to speed up your queries. <h4>Or... <p dir="auto">Use a different set of tools and solutions that meet your needs.<br /> A less exotic solution is to use MySQL that supports a native JSON data type, which combined with generated columns might be a good alternative.<br /> (I’m going to try that too.) <h1>Previous episodes of Steem Pressure series <p dir="auto"><center> <p dir="auto"><a href="/@gtg/introducing-steem-pressure-1">Introducing: Steem Pressure #1<br /> <a href="/@gtg/steem-pressure-2-toys-for-boys-and-girls">Steem Pressure #2 - Toys for Boys and Girls<br /> <a href="/@gtg/steem-pressure-3-steem-node-101">Steem Pressure #3 - Steem Node 101<br /> <a href="/@gtg/steem-pressure-the-movie">Steem Pressure: The Movie ;-)<br /> <a href="/@gtg/steem-pressure-4-need-for-speed">Steem Pressure #4 - Need for Speed<br /> <a href="/@gtg/steem-pressure-5-run-block-run">Steem Pressure #5 - Run, Block, Run!<br /> <a href="/@gtg/steem-pressure-6-mira-ymmv-rtfm-tldr-lgtm">Steem Pressure #6 - MIRA: YMMV, RTFM, TLDR: LGTM<br /> <a href="/@gtg/steem-pressure-7-go-fork-yourself-step-by-step-guide-to-building-and-setting-up-a-mira-powered-hf21-ready-steem-consensus-node">Steem Pressure #7 - Go Fork Yourself!<br /> Stay tuned for next episodes of Steem Pressure :-) <hr /> <p dir="auto"><sup><br /> If you believe I can be of value to Steem, please vote for me (<a href="/@gtg"><strong>gtg) as a witness.<br /> Use <a href="https://steemitwallet.com/~witnesses" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Steemit's Witnesses List or <a href="https://steemconnect.com/sign/account-witness-vote?witness=gtg&approve=1" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">SteemConnect<br /> Or if you trust my choice you can set <a href="/@gtg"><strong>gtg as a proxy that will vote for witnesses for you.<br /> <strong><em>Your vote does matter!<br /> You can contact me directly on <a href="https://steem.chat" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">steem.chat, as <a href="https://steem.chat/direct/gandalf" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Gandalf<br /> <br /> <center><br /><br /> <a href="/">Steem On<br /> <span> <img src="https://images.hive.blog/768x0/https://steemitimages.com/DQmSheuDfCHizk1xvHPcrFjQNKfBzgun9UXDxdEp6JJCum9/steem_wide.png" srcset="https://images.hive.blog/768x0/https://steemitimages.com/DQmSheuDfCHizk1xvHPcrFjQNKfBzgun9UXDxdEp6JJCum9/steem_wide.png 1x, https://images.hive.blog/1536x0/https://steemitimages.com/DQmSheuDfCHizk1xvHPcrFjQNKfBzgun9UXDxdEp6JJCum9/steem_wide.png 2x" />