Part 2: Manage local 'STEEM' orderbook via websocket stream from exchange

<hr /> <h4>Repository <p dir="auto"><span><a href="https://github.com/python" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://github.com/python <h4>What will I learn <ul> <li>What is a local orderbook <li>Open a diff. depth stream <li>Retrieve orderbook snapshot <li>Process updates <li>Update the local orderbook <h4>Requirements <ul> <li>Python 3.7.2 <li>Pipenv <li>websocket-client <li>requests <h4>Difficulty <ul> <li>basic <hr /> <h3>Tutorial <h4>Preface <p dir="auto">Websockets allow for real time updates while putting less stress on the servers than API calls would. They are especially useful when data is updated frequently, like trades and the orderbooks on crypto currency exchanges. This tutorial will look at managing a local orderbook. For this example the exchange Binance will be used. Depending on the service provider there can be slight modifications. This tutorial is a follow up for <a href="https://steemit.com/utopian-io/@steempytutorials/part-1-connecting-to-steem-orderbook-stream-via-websockets-on-different-exchanges" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">part 1, if anything is unclear, go over the previous tutorial first. <h4>Setup <p dir="auto">Download the files from Github and install the virtual environment <pre><code>$ cd ~/ $ git clone https://github.com/Juless89/tutorials-websockets $ cd tutorials-websockets $ pipenv install $ pipenv shell $ cd part_2 <h4>What is a local orderbook <p dir="auto">In the previous tutorial a partial orderbook stream was created via a websocket. Meaning that only part of the orderbook was send on every update. This reduces the amount of data that has te be send every update. In the case a user wants to have the full orderbook updated via a websocket a different approach is needed. In general this works by first retrieving a full snapshot of the orderbook and then only receiving updates made to the orderbook. This greatly reduces the amount of data that has to be send on every update. However, this does mean the user has to update his version of the orderbook to stay synchronised. In addition, Every update comes with an <code>update id, this allows the user to keep track of any missed updates and ensure the orderbook is synchronised. <p dir="auto">Taken from the <a href="https://github.com/binance-exchange/binance-official-api-docs/blob/master/web-socket-streams.md" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Binance documentation the instructions are as follows. This list will be referred to as the Binance to do list. <p dir="auto"><center><img src="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQma9o83tuupwrSpsKfgug3sTWXqgvqtJy4QT2WtMBRChbW/Screenshot%202019-03-31%2014.31.10.png" alt="Screenshot 2019-03-31 14.31.10.png" srcset="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQma9o83tuupwrSpsKfgug3sTWXqgvqtJy4QT2WtMBRChbW/Screenshot%202019-03-31%2014.31.10.png 1x, https://images.hive.blog/1536x0/https://cdn.steemitimages.com/DQma9o83tuupwrSpsKfgug3sTWXqgvqtJy4QT2WtMBRChbW/Screenshot%202019-03-31%2014.31.10.png 2x" /> <h4>Open a diff. depth stream <p dir="auto">The code from the previous update has been slightly adjusted. A class <code>Client has been made to accommodate local data storage and keeping everything together. Also the websocket <code>url has been set to <code>wss://stream.binance.com:9443/ws/steembtc@depth. This completes step 1 of the Binance to do list. Step 2, the buffering, is done by the python websocket library automatically. <pre><code>import websocket import requests from json import loads class Client(): def __init__(self): # create websocket connection self.ws = websocket.WebSocketApp( url="wss://stream.binance.com:9443/ws/steembtc@depth", on_message=self.on_message, on_error=self.on_error, on_close=self.on_close, on_open=self.on_open ) # local data management self.orderbook = {} self.updates = 0 # keep connection alive def run_forever(self): self.ws.run_forever() # convert message to dict, print def on_message(self, message): data = loads(message) print(data) print() # catch errors def on_error(self, error): print(error) # run when websocket is closed def on_close(self): print("### closed ###") # run when websocket is initialised def on_open(self): print('Connected to Binance\n') if __name__ == "__main__": # create webscocket client client = Client() # run forever client.run_forever() <p dir="auto"><center><img src="https://images.hive.blog/0x0/https://cdn.steemitimages.com/DQmRDfUEg9AFmxt51mRo6jpRYiZBpXgKvYYnhu8rhiE4nbB/websockets_part2.gif" alt="websockets_part2.gif" /> <h4>Retrieve orderbook snapshot <p dir="auto">The depth snapshot can be retrieved by using a get request to <code>https://www.binance.com/api/v1/depth?symbol=STEEMBTC&limit=1000. Data is returned in an encoded string. This has to be converted to a dict with <code>loads() and decoded by <code>decode(). <pre><code>import requests # retrieve orderbook snapshot def get_snapshot(self): r = requests.get('https://www.binance.com/api/v1/depth?symbol=STEEMBTC&limit=1000') return loads(r.content.decode()) <p dir="auto">Data is returned in the following format. The <code>lastUpdateId is important and will be used to synchronise the local orderbook with the updates from the websocket. This completes step 3. <pre><code>{ "lastUpdateId": 160, // Last update ID "bids": [ // Bids [ "0.0024", // Price level "10" // Quantity ] ], "asks": [ // Asks [ "0.0026", // Price level "100" // Quantity ] ] } <h4>Processing updates <p dir="auto">Updates have the following data structure. <pre><code>{ "e": "depthUpdate", // Event type "E": 123456789, // Event time "s": "BNBBTC", // Symbol "U": 157, // First update ID in event "u": 160, // Final update ID in event "b": [ // Bids to be updated [ "0.0024", // Price level to be updated "10" // Quantity ] ], "a": [ // Asks to be updated [ "0.0026", // Price level to be updated "100" // Quantity ] ] } <p dir="auto"><code>U and <code>u are used to synchronise the orderbook. These are the <code>lastUpdateIds. <code>b and <code>a contain the actual new values for the bids and asks. First there is a check done to see if there is already a snapshot in the local orderbook. Initially the <code>lastUpdateId is set to 0 to differentiate between the situation where the snapshot has just been retrieved and may be out of sync. In which case older updates have to be dropped. Then after each update <code>u is set as the <code>lastUpdateId. For each new update <code>U has to be equal to <code>lastUpdateId+1. This completes step 4, 5 and 6 from the Binance to do list. <pre><code># convert message to dict, process update def on_message(self, message): data = loads(message) # check for orderbook, if empty retrieve if len(self.orderbook) == 0: self.orderbook = self.get_snapshot() # get lastUpdateId lastUpdateId = self.orderbook['lastUpdateId'] # drop any updates older than the snapshot if self.updates == 0: if data['U'] <= lastUpdateId+1 and data['u'] >= lastUpdateId+1: print('process this update') self.orderbook['lastUpdateId'] = data['u'] else: print('discard update') # check if update still in sync with orderbook elif data['U'] == lastUpdateId+1: print('process this update') self.orderbook['lastUpdateId'] = data['u'] else: print('Out of sync, abort') <p dir="auto"><center><img src="https://images.hive.blog/0x0/https://cdn.steemitimages.com/DQmWt4fU9UbL3Vt6ERSuQTCUX1jJL8hRX9X4zgvgZXEN6aM/websockets_part2_2.gif" alt="websockets_part2_2.gif" /> <h4>Updating the local orderbook <p dir="auto">The local orderbook is a dict containing the <code>lastUpdateId and two lists <code>bids and <code>asks. The lists are ordered by the price level. There are three different possibilities for adjustments to the list. <ul> <li>Price level already exists, different quantity <li>New price level <li>Price level quality set to 0, remove from list <p dir="auto">Even though the values are strings, python is smart enough to understand when floats are implied. Two functions are used. <code>process_updates is called from <code>on_message: it loops through all the updates and differentiates between the bid and ask side. <code>manage_orderbook then checks which type of adjustment should be made and executes this update into the orderbook. <pre><code># Loop through all bid and ask updates, call manage_orderbook accordingly def process_updates(self, data): for update in data['b']: self.manage_orderbook('bids', update) for update in data['a']: self.manage_orderbook('asks', update) print() # Update orderbook, differentiate between remove, update and new def manage_orderbook(self, side, update): # extract values price, qty = update # loop through orderbook side for x in range(0, len(self.orderbook[side])): if price == self.orderbook[side][x][0]: # when qty is 0 remove from orderbook, else # update values if qty == 0: del self.orderbook[side] print(f'Removed {price} {qty}') break else: self.orderbook[side][x] = update print(f'Updated: {price} {qty}') break # if the price level is not in the orderbook, # insert price level, filter for qty 0 elif price > self.orderbook[side][x][0]: if qty != 0: self.orderbook[side].insert(x, update) print(f'New price: {price} {qty}') break else: break <h4>Running the code <p dir="auto"><code>python binance.py <p dir="auto"><center><img src="https://images.hive.blog/0x0/https://cdn.steemitimages.com/DQmX5EVUTJaN45wgrimvQmZf3WsuQztujCiGnyokEZgMbmF/websockets_part2_3.gif" alt="websockets_part2_3.gif" /> <h3>Curriculum <ul> <li><a href="https://steemit.com/utopian-io/@steempytutorials/part-1-connecting-to-steem-orderbook-stream-via-websockets-on-different-exchanges" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Part 1: Connecting to STEEM orderbook stream via websockets on different exchanges <hr /> <p dir="auto">The code for this tutorial can be found on <a href="https://github.com/Juless89/tutorials-websockets" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Github! <p dir="auto"><span>This tutorial was written by <a href="/@juliank">@juliank.