A Hacky Guide to Hive (part 2.2.1: blocks)

Context

<p dir="auto">In the <a href="/@felixxx/a-hacky-guide-to-hive-part-21-yo-broadcast">previous post, I made a <a href="https://hivehub.dev/tx/eb025cf797ee5bc81d7399282268079cc29cc66d" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">special transaction.<br /> I broadcasted a <code>custom_json transaction of the <code>type: <code>YO.<br /> This information will forever be stored in block <strong>89040473 of Hive's blockchain.<br /> To <strong>get to this information again, I could querry a Hive node's: <ul> <li><a href="https://developers.hive.io/apidefinitions/#block_api.get_block" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">block_api.get_block, by blocknumber <li><a href="https://developers.hive.io/apidefinitions/#transaction_status_api.find_transaction" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">transaction_status_api.find_transaction, by transaction ID <p dir="auto">If I don't know those 2 parameters, but want to find <strong>my move, I could use: <ul> <li><a href="https://developers.hive.io/apidefinitions/#account_history_api.get_account_history" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">account_history_api.get_account_history, by account name... <p dir="auto">...you can access blockchain data many different ways, use the above enpoints with Beem or <a href="https://github.com/emre/lighthive" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">lighthive... <p dir="auto">I demonstrated, how anyone can YO, now I want to show a method, to get to all YOs.<br /> It could be any custom_json. Or a different event. It's just an example. It could be a <em>move in a blockchain game, or you could go as far as trying to build your own little hive engine.<br /> You might want to observe votes or comments as they come in, and store some, so you don't have to look them up again later, maybe for a notification system... <hr /> <p dir="auto"><img src="https://images.hive.blog/768x0/https://files.peakd.com/file/peakd-hive/felixxx/23u5SmjF6Ya2X8TzxdWqXjc4Q93v812HtAGqFF29S5VR3EFBFPvRV1apReyuY5XPXx8Vd.png" alt="FELIXBOXVHSTAPE400YO.png" srcset="https://images.hive.blog/768x0/https://files.peakd.com/file/peakd-hive/felixxx/23u5SmjF6Ya2X8TzxdWqXjc4Q93v812HtAGqFF29S5VR3EFBFPvRV1apReyuY5XPXx8Vd.png 1x, https://images.hive.blog/1536x0/https://files.peakd.com/file/peakd-hive/felixxx/23u5SmjF6Ya2X8TzxdWqXjc4Q93v812HtAGqFF29S5VR3EFBFPvRV1apReyuY5XPXx8Vd.png 2x" /> <hr /> <h2>A Better Stream <p dir="auto">In <a href="/@felixxx/a-hacky-guide-to-hive-part-15">another post I explained, how the Hive blockchain is really just a very long list. <h3><code>block_api <p dir="auto">The block_api gives you access to all blocks.<br /> You can access the block_api on all public nodes.<br /> If you want to use <a href="https://@gtg/witness-update-release-candidate-for-eclipse-is-out" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">your own node, having only the block_api should be one of the cheapest options. <h3><code>stream() <p dir="auto">Basically you could build most things around <em>just looking at all blocks as they are written.<br /> That will not include all information for everything (virtual values and such), but a lot.<br /> This might not be the best approach to build everything, but once you've got a stable block stream going, you can build good stuff around it... <h3>Beem <p dir="auto"><a href="https://github.com/holgern/beem/tree/master" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Beem's stream() method still works and you could use it as is. <p dir="auto">The main logic behind Beem's stream is hidden in the <a href="https://github.com/holgern/beem/blob/master/beem/blockchain.py#L394" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">blocks() mehod. That part alone is 278 lines long and does a lot of things.<br /> In the background, Beem can handle: <ul> <li>node switching <li>threading <li>syncing <li>private keys <p dir="auto">... and more.<br /> I could not build it better. I don't have to. <h3>Procedure <p dir="auto">The main procedure to get to a block is still just a querry.<br /> The speed and reliability of that querry depends mostly on the source (the node), not on the Python code. <p dir="auto"><a href="https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/python3-gpp.html" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Python isn't particularily fast to begin with.<br /> But all we need it to do during this procedure: <ul> <li>Querry <em>next block <li>Filter the block for YO <li>Store YO <p dir="auto">That's a job done. <p dir="auto">At the moment, querrying the latest block from api.hive.blog takes about 1 second.<br /> Maximum block size is a <a href="https://github.com/openhive-network/hive/blob/master/doc/witness_parameters.md#maximum_block_size" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">witness parameter: <blockquote> <p dir="auto">The value must not be more than 2MB (2097152). <p dir="auto">...so there are 2 seconds left to handle 2MB at most. (current max: 65536 Bytes)<br /> To just filter and store a block takes only miliseconds even in Python...<br /> Which means, this thing can idle for almost 2 seconds and repeat the procedure. <p dir="auto">Beem actually <a href="https://github.com/holgern/beem/blob/master/beem/blockchain.py#L572" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">does that too 😅: <pre><code># Sleep for one block time.sleep(self.block_interval) <h3>Storage <p dir="auto">It doesn't really matter how I build the stream; without storage, I'll lose all progress when the stream ends or crashes. <p dir="auto">I'll use SQL. I could use Redis, or Mongo... <p dir="auto">There are many different storage solutions and I could never build anything better.<br /> This stuff handles sessions and serialization. It comes with built in backup solutions.<br /> It's fast. It's scalable: I'll use SQLite, but you could plugin in a giant cluster of whatever.<br /> I am trying to move the responsibility of storage handling where it belongs: the database level. <h3>threading and node switching <p dir="auto">Beem can switch through nodes from a list and even manage worker threads.<br /> But why manage that inside Python in the first place? <p dir="auto">I will just build one single procedure and can run it as a background service.<br /> If I need another thread, I can just run another instance of the same procedure.<br /> I could run one thread for every node, or even use separate machines.<br /> Anyhow, the procedure does not need to know which thread it's in.<br /> As long as I funnel the data to the same database in the end, all synchronization and serialization and whatnot is taken care of automatically. <p dir="auto">I am trying to move the responsibility of concurrency where it belongs: the operating system- and database layer. <hr /> <h2>Live Stream <h3><code>block_api.get_block_range <pre><code>import requests def get_block_range(start, count, url): data = '{"jsonrpc":"2.0", "method":"block_api.get_block_range","params":{"starting_block_num":'+str(start)+',"count": '+str(count)+'},"id":1}' response = requests.post(url=url, data=data) return response.json()['result']['blocks'] <p dir="auto">The only function you really need.<br /> I am not even joking. <ul> <li>Usage: <pre><code>url = 'https://api.hive.blog' for block in get_block_range(89040473, 1, url): print(block) <h3>Loop <p dir="auto">For a stream you only need to loop this; you need a <em>start block and then increment.<br /> Repeat every 3 seconds and it's basically Beem's stream(), without all the fluff. <p dir="auto">But that's an infinite loop.<br /> For the final service, that's what I'd want; For a code snippet, I feel like avoiding it. <p dir="auto">In the early days, nodes accepted websockets. I don't know, why that got turned off. Maybe it was too expensive. Maybe you can still do something like that on your own node.<br /> Anyways, if you test this on the public nodes you are stuck with this 3-second-querry loop. It seems crude, but it seems as that's how it's done. <p dir="auto"><a href="https://developers.hive.io/tutorials-python/stream_blockchain_transactions.html" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">The documentation recommends Beem's stream. <p dir="auto"><a href="https://github.com/aaroncox/chainsync/blob/master/chainsync/chainsync.py#L190" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">does it:<span><a href="/@jesta">@jesta's chainsync <pre><code>time.sleep(self.get_approx_sleep_until_block(throttle, config, status['time'])) <p dir="auto">So yeah... I also wait 3 seconds. <h3>Interrupt <p dir="auto">Best case would be, I start the loop once and it runs infinitly (fire&forget).<br /> In reality I have to prepare for what happens should it stop.<br /> Maybe I need to resync the whole service... <p dir="auto">The above is all it takes to rebuild Beem's stream or any other.<br /> Wrap some try excepts around it and it can't really break down. <p dir="auto">But for something useful, storage is necessary.<br /> So that I at least know, where the last tream stopped. And where to begin...<br /> For YO, I could ignore all 89040473 blocks before the first YO. <h3>Traffic <p dir="auto">That 3-second-querry thing may seem like a lot of traffic.<br /> But if it's planned well, and stored well, it only has to be done once for any block.<br /> From that point on, it can feed a whole network of other things, which don't have to make any queries outside of my own database. <p dir="auto">Again: For things like posts and author balance, the standard apis can be enough.<br /> Also: Posts, votes, account balance, can <strong>change, blocks can't. <p dir="auto">Sending one request every 3 seconds, receiving 60KB max data...<br /> I don't know, how annyoing this is for node providers.<br /> I guess it's ok... <p dir="auto">Syncing might be different. In the docs, there's a get_block_range example with count=1000.<br /> The response could be 60MB. But that could also sync 50 minutes in a single call... <h3>Filter YO <pre><code>def get_yos(block): yos = [] for transaction in block['transactions']: for operation in transaction['operations']: if operation['type'] == 'custom_json_operation': if operation['value']['id'] == 'YO': yos.append(operation) return yos <p dir="auto">Returns all YOs in a block, but loses the information, which transaction and which block each YO was in. <p dir="auto">I'll try to avoid data manipulation in this part of the service.<br /> This part is the stream and shouldn't be involved in anything else.<br /> However, I do want to store the <em>block num, which already got lost along the way.<br /> I also want <em>block id and <em>previous. This just demonstrates how to filter data.<br /> It's best to start by building the tables first, though. <h2>Conclusion <p dir="auto">It might not look like much, but the part that <strong>needs to connect to a Hive node is done.<br /> This is the absolute minimum necessary and can only fail at very few points so far.<br /> Most possible problems can be caught outside of this core logic.<br /> All that's missing is persistent storage, which I will conclude next post. <p dir="auto">Anyways, threading, concurrency, data manipulation, whatever... everything else can and should happen later, upstream.<br /> What I keep trying to point out: All extra logic should be avoided.<br /> I am looking at a Hive querry as a single step - a procedure. It should be a single function.<br /> Next post, storage will be wrapped in as few procedures as possible and that will conclude in a YO crawler/watcher that feeds a db, that you could plug <strong>anything into. It will probably be short and include only minimal logic. That's a feature. <h3>Naming <p dir="auto">I think, the hardest question in programming is naming.<br /> 'YO crawler' isn't good. I should give this thing a name, before it's finished.<br /> custom_jacksn, or custom_YOson maybe? Or YOmind...