Part 1: Connecting to STEEM orderbook stream via websockets on different exchanges

<hr /> <h4>Repository <p dir="auto"><span><a href="https://github.com/python" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://github.com/python <h4>What will I learn <ul> <li>Basics of websockets <li>Subscribing to data streams <li>Processing messages <li>Return top bid/ask <h4>Requirements <ul> <li>Python 3.7.2 <li>Pipenv <li>websocket-client <h4>Difficulty <ul> <li>basic <hr /> <h3>Tutorial <h4>Preface <p dir="auto">Websockets allow for real time updates while putting less stress on the servers than API calls would. They are especially useful when data is updated frequently, like trades and the orderbooks on crypto currency exchanges. <h4>Setup <p dir="auto">Download the files from Github and install the virtual environment <pre><code>$ cd ~/ $ git clone https://github.com/Juless89/tutorials-websockets $ cd tutorials-websockets $ pipenv install $ pipenv shell $ cd part_1 <h4>Basics of websockets <p dir="auto">Depending on which websocket one wants to connect to and how the serverside socket has been implemented there are multiple variations. However, in general when connecting to a websocket the following functions are essential. These are <code>on_message, <code>on_error, <code>on_close and <code>on_open. <p dir="auto">When connecting to a websocket it is recommended to read the documentation of the service that is providing the websocket. An object <code>WebSocketApp is created, the functions are passed to overwrite the generic functions and the <code>url is the endpoint of the service. Depending on what type of websocket one is connecting to these functions can be changed accordingly. <pre><code>import websocket def on_message(ws, message): print(message) def on_error(ws, error): print(error) def on_close(ws): print("### closed ###") def on_open(ws): print('Connected to websocket') if __name__ == "__main__": ws = websocket.WebSocketApp( url="wss://stream.binance.com:9443/ws/steembtc@depth20", on_message=on_message, on_error=on_error, on_close=on_close, on_open=on_open ) ws.run_forever() <p dir="auto"><code>on_open gets called immediately after creating the connection, before receiving any messages. When additional parameters are required this is where they get set. <code>on_message deals with all messages received through the websocket, extracting and processing data should be implemented here. <code>on_error gets called when an error occurs, error handling should be implemented here. <code>on_close get called when the connection gets closed, either by the client or the server. <code>run_forever makes sure the connection stays alive. <h4>Subscribing to data streams <p dir="auto">This tutorial will look to connecting to two different exchanges via websockets and get a live data streams of the STEEM orderbook. The exchanges are Binance and Huobi and both use a different technique to subscribe to different data streams. Always read the documentation provided by the websocket service provider. <p dir="auto"><center><img src="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmTaLE7aJWvRcAFuydAA7fYidVkA7vFAqtWGhTwyVg3Fwz/Screenshot%202019-03-27%2013.02.40.png" alt="Screenshot 2019-03-27 13.02.40.png" srcset="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmTaLE7aJWvRcAFuydAA7fYidVkA7vFAqtWGhTwyVg3Fwz/Screenshot%202019-03-27%2013.02.40.png 1x, https://images.hive.blog/1536x0/https://cdn.steemitimages.com/DQmTaLE7aJWvRcAFuydAA7fYidVkA7vFAqtWGhTwyVg3Fwz/Screenshot%202019-03-27%2013.02.40.png 2x" /> <p dir="auto"><a href="https://github.com/binance-exchange/binance-official-api-docs/blob/master/web-socket-streams.md" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Binance documentation <p dir="auto">Binance uses the url to differentiate between data streams. For this tutorial a stream will be set up for the partial book depth. Taken directly from the documentation: <pre><code>The base endpoint is: wss://stream.binance.com:9443 Raw streams are accessed at /ws/<streamName> Top <levels> bids and asks, pushed every second. Valid <levels> are 5, 10, or 20. Stream Name: <symbol>@depth<levels> <p dir="auto">Connecting to the steembtc partial book depth for 20 levels is then achieved by connecting to the url: <code>wss://stream.binance.com:9443/ws/steembtc@depth20. <p dir="auto"><center><img src="https://images.hive.blog/0x0/https://cdn.steemitimages.com/DQmc9zkPw99vE6ML3HzbMprTCzpMRMh13M6mkoqU15XQWfC/binance.gif" alt="binance.gif" /> <hr /> <p dir="auto"><center><img src="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmbFKJdLrzNKAvaoZbhWaaaA6Sr1xt3PxeRVkXKwMyyTDa/Screenshot%202019-03-27%2013.03.01.png" alt="Screenshot 2019-03-27 13.03.01.png" srcset="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmbFKJdLrzNKAvaoZbhWaaaA6Sr1xt3PxeRVkXKwMyyTDa/Screenshot%202019-03-27%2013.03.01.png 1x, https://images.hive.blog/1536x0/https://cdn.steemitimages.com/DQmbFKJdLrzNKAvaoZbhWaaaA6Sr1xt3PxeRVkXKwMyyTDa/Screenshot%202019-03-27%2013.03.01.png 2x" /> <p dir="auto"><a href="https://github.com/huobiapi/API_Docs_en/wiki/Huobi-API" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Huobi documentation <p dir="auto">Huobi uses a subscribe based system. Where the client must subscribe to data streams after the connection has been made. Taken from their documentation: <pre><code>SSL Websocket Connection wss://api.huobi.pro/ws Subscribe To receive data you have to send a "sub" message first. Market Depth market.$symbol.depth.$type $type ：{ step0, step1, step2, step3, step4, step5 } （depths 0-5） //request { "sub": "topic to sub", "id": "id generate by client" } <p dir="auto">Subscribing to the level 0 market depth of steembtc is achieved by connection to <code>wss://api.huobi.pro/ws and then sending the message which contains <code>"sub": "market.eosusdt.depth.step0". Sending the message is done by using <code>send(). The dict has to be converted to a string first, which is done by using <code>dumps() <pre><code>from json import dumps def on_open(ws): print('Connected to websocket') params = {"sub": "market.steembtc.depth.step0", "id": "id1"} ws.send(dumps(params)) <p dir="auto">Will return the following reply on success: <pre><code>{'id': 'id1', 'status': 'ok', 'subbed': 'market.steembtc.depth.step0', 'ts': 1553688645071} <p dir="auto"><center><img src="https://images.hive.blog/0x0/https://cdn.steemitimages.com/DQmdC1aBRzoxsiX4oJq5m1dKo3kf5NskaNWhyuG7axbqpuq/huobi.gif" alt="huobi.gif" /> <h4>Processing messages <p dir="auto">Every service is different and in most cases return data in a different format, noting again the importance of the documentation. <p dir="auto">Binance <pre><code>Payload: { "lastUpdateId": 160, // Last update ID "bids": [ // Bids to be updated [ "0.0024", // Price level to be updated "10" // Quantity ] ], "asks": [ // Asks to be updated [ "0.0026", // Price level to be updated "100" // Quantity ] ] } <p dir="auto">All data gets returned as a string and has to be converted to a dict to access. This is done by using <code>loads() from the JSON library. <pre><code>def on_message(self, message): print(loads(message)) print() <p dir="auto">Huobi returns the data encoded and requires one additional step. <pre><code>import gzip def on_message(self, message): print(loads(gzip.decompress(message).decode('utf-8'))) print() <p dir="auto">Data format: <pre><code>{ 'ch': 'market.steembtc.depth.step0', 'ts': 1553688645047, 'tick': { 'bids': [ [0.00010959, 736.0], [0.00010951, 372.0], [0.0001095, 56.0], [0.00010915, 1750.0], [0.00010903, 371.77], [0.00010891, 1684.65], ], 'asks': [ [0.00011009, 368.0], [0.00011025, 1575.36], [0.00011039, 919.0], [0.00011054, 92.0], [0.00011055, 68.85], [0.00011077, 736.77], ], 'ts': 1553688645004, 'version': 100058128375 } } <p dir="auto">In addition Huobi also sends pings to make sure the client is still online. <pre><code>{'ping': 1553688793701} <p dir="auto">When processing messages from Huobi this has to be filtered out. <h4>Return top bid/ask <p dir="auto">For Binance returning the top bid/ask is straight forward as the messages always have the same format. <pre><code>def on_message(self, message): data = loads(message) bid = data['bids'][0] ask = data['asks'][0] print(f'Top bid: {bid} top ask: {ask}') <p dir="auto">Due to the ping messages doing the same for Huobi requires one additional step. <pre><code>def on_message(self, message): data = loads(gzip.decompress(message).decode('utf-8')) if 'tick' in data: bid = data['tick']['bids'][0] ask = data['tick']['asks'][0] print(f'Top bid: {bid} top ask: {ask}') <h4>Running the code <p dir="auto"><code>python binance.py <p dir="auto"><center><img src="https://images.hive.blog/0x0/https://cdn.steemitimages.com/DQmZjdYqJoarQHSqLGwGU3WJssp1HWV7ps9z2D5ryTUC9QN/binance2.gif" alt="binance2.gif" /> <p dir="auto"><code>python huobi.py <p dir="auto"><center><img src="https://images.hive.blog/0x0/https://cdn.steemitimages.com/DQmRDPoQxN9jCPfDT4jrK5qceHs3FXk3Sut5PDBXCMiS2C5/huobi2.gif" alt="huobi2.gif" /> <hr /> <p dir="auto">The code for this tutorial can be found on <a href="https://github.com/Juless89/tutorials-websockets" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Github! <p dir="auto"><span>This tutorial was written by <a href="/@juliank">@juliank.