Part 4: Comparing orderbooks from different exchanges in real time

<hr /> <h4>Repository <p dir="auto"><span><a href="https://github.com/python" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://github.com/python <h4>What will I learn <ul> <li>Connecting to Upbit <li>Change data format <li>Create subsets of exchanges <li>Comparing top bid/ask on each update <h4>Requirements <ul> <li>Python 3.7.2 <li>Pipenv <li>websocket-client <li>requests <h4>Difficulty <ul> <li>intermediate <hr /> <h3>Tutorial <h4>Preface <p dir="auto">Websockets allow for real time updates while putting less stress on the servers than API calls would. They are especially useful when data is updated frequently, like trades and the orderbooks on crypto currency exchanges. This tutorial will look at how to connect to the Upbit exchange, bringing the total connected exchanges to three. In addition the top bid and ask of all exchanges will be compared to each other in pairs. <p dir="auto">This tutorial is part of a series, if anything is unclear refer to the previous tutorials. <h4>Setup <p dir="auto">Download the files from Github and install the virtual environment <pre><code>$ cd ~/ $ git clone https://github.com/Juless89/tutorials-websockets $ cd tutorials-websockets $ pipenv install $ pipenv shell $ cd part_4 <h4>Connecting to Upbit <p dir="auto">Like any other websocket or API interface start of by reading the documentation of the service provider. In this case that would be the <a href="https://api.hitbtc.com/#about-companyname-api" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Upbit API Documentation. <p dir="auto">Upbit's websocket interface works by connecting to the websocket and then subscribing to the data stream. This is done by sending a message with the <code>method, <code>params and <code>id. <code>method refers to which the function, in this case <code>subscribeOrderbook. While params contains the required parameters to do so. <code>id gets returned as is. <pre><code># main.py upbit = Upbit( url="wss://api.hitbtc.com/api/2/ws", exchange="Upbit", orderbook=orderbooks, lock=lock, ) # upbit.py # register to orderbook stream def on_open(self): super().on_open() params = { "method": "subscribeOrderbook", "params": { "symbol": "STEEMBTC" }, "id": 1 } self.ws.send(dumps(params)) <p dir="auto">After subscribing to a specific market the first reply is the full orderbook snapshot. Any reply after contains updates to the orderbook. Similar to the diff. depth stream from Binance is part 2. Replies have the following data structure. <pre><code>{ "jsonrpc": "2.0", "method": "snapshotOrderbook", "params": { "ask": [ { "price": "0.054588", "size": "0.245" }, { "price": "0.054590", "size": "0.000" }, { "price": "0.054591", "size": "2.784" } ], "bid": [ { "price": "0.054558", "size": "0.500" }, { "price": "0.054557", "size": "0.076" }, { "price": "0.054524", "size": "7.725" } ], "symbol": "ETHBTC", "sequence": 8073827, "timestamp": "2018-11-19T05:00:28.193Z" } } <p dir="auto">The method is either <code>snapshotOrderbook or <code>updateOrderbook. <code>sequence is used to stay synchronised with the exchange. The next update should always be <code>sequence+1. If not all data should be dropped and the the stream should be restarted. <pre><code># convert message to dict, process update def on_message(self, message): data = loads(message) # first message is the full snapshot if data['method'] == 'snapshotOrderbook': # process data self.orderbook['sequence'] = params['sequence'] # following messages are updates to the orderbook elif data['method'] == 'updateOrderbook': # track sequence to stay in sync if params['sequence'] == self.orderbook['sequence']+1: self.orderbook['sequence'] = params['sequence'] self.process_updates(params) else: print('Out of sync, abort') <p dir="auto"><br /><br /> The new data structure of the applications is now: <p dir="auto"><center><br /> <img src="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmcc5NSyuk6SPVAQALxRyqRcrxVEQNgEfn7wjnWV7c8vGz/Screenshot%202019-04-04%2015.19.06.png" alt="Screenshot 2019-04-04 15.19.06.png" srcset="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmcc5NSyuk6SPVAQALxRyqRcrxVEQNgEfn7wjnWV7c8vGz/Screenshot%202019-04-04%2015.19.06.png 1x, https://images.hive.blog/1536x0/https://cdn.steemitimages.com/DQmcc5NSyuk6SPVAQALxRyqRcrxVEQNgEfn7wjnWV7c8vGz/Screenshot%202019-04-04%2015.19.06.png 2x" /> <p dir="auto">Each exchange thread writes the exchange data to the shared <code>Orderbooks dict. The main processing thread looks at the <code>last_update variable to determine if there is any new data. If so, the <code>Orderbooks gets locked and the data can be processed. <p dir="auto"><center><br /> <img src="https://images.hive.blog/0x0/https://cdn.steemitimages.com/DQmeC29t8obtqAihcNNLnKSkRmjLRV2H9FGoqTEvsWs3YSs/websockets_part4.gif" alt="websockets_part4.gif" /><br /> <h4>Change data format <p dir="auto">As shown above the orderbook values get returned as a list of dicts. With the keys: <code>price and <code>size. In order to easily process data from all exchanges all data should have the same format. Until now the data structure for the orderbooks has been like: <pre><code>{ "bids": [ // Bids [ "0.0024", // Price level "10" // Quantity ] ], "asks": [ // Asks [ "0.0026", // Price level "100" // Quantity ] ] } <p dir="auto">Python allows for list comprehensions which makes converting the dicts into the preferred lists quite simple. <pre><code># upbit.py # first message is the full snapshot if data['method'] == 'snapshotOrderbook': # extract params params = data['params'] # convert data to right format self.orderbook['bids'] = [[x['price'], x['size']] for x in params['bid']] self.orderbook['asks'] = [[x['price'], x['size']] for x in params['ask']] self.orderbook['sequence'] = params['sequence'] <p dir="auto"><code>[[x['price'], x['size']] for x in params['bid']] is telling python to create a list of lists that look like <code>[[x['price'], x['size']] for each value <code>x inside <code>params['bid']. <p dir="auto">This code is equivalent to: <pre><code>list = [] for x in params['bid']: list.append([x['price'], x['size']]) <h4>Create subsets of exchanges <p dir="auto">To compare the top bid/ask from each exchange with each other in pairs of two all possible unordered subsets have to be determined. Depending on how many exchanges one is connected to the amount of subsets increases drastically. This is known as <code>binomial coefficient. <p dir="auto"><center><br /> <img src="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmXKqh99hhNYKPaYy3BMbM4Yh2RRwU3VABWcovS6G9CDy6/Screenshot%202019-04-04%2014.52.10.png" alt="Screenshot 2019-04-04 14.52.10.png" srcset="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmXKqh99hhNYKPaYy3BMbM4Yh2RRwU3VABWcovS6G9CDy6/Screenshot%202019-04-04%2014.52.10.png 1x, https://images.hive.blog/1536x0/https://cdn.steemitimages.com/DQmXKqh99hhNYKPaYy3BMbM4Yh2RRwU3VABWcovS6G9CDy6/Screenshot%202019-04-04%2014.52.10.png 2x" /> <p dir="auto">Luckily python contains the libraby for just that. <code>n refers to the amount of elements or exchanges in this case. and <code>k is the size of the subset, or 2 in this case. First all exchanges are extracted from the shared orderbooks dicts. Then <code>combinations is used to create all subsets. <pre><code># return subsets of size 2 of all exchanges def exchange_sets(orderbooks): exchanges = [] # extract exchanges for exchange in orderbooks: if exchange != 'last_update': exchanges.append(exchange) # return all subsets return list(combinations(exchanges, 2)) <p dir="auto">Result: <pre><code>[('Binance', 'Huobi'), ('Binance', 'Upbit'), ('Huobi', 'Upbit')] # adding 1 exchange results in quite a lot more subsets [('Binance', 'Huobi'), ('Binance', 'Upbit'), ('Binance', 'Exchange'), ('Huobi', 'Upbit'), ('Huobi', 'Exchange'), ('Upbit', 'Exchange')] <p dir="auto"><br /><br /> Each exchange can be seen as a <code>node that is connected to all other nodes. Each arrow represents a subset. <p dir="auto"><center><br /> <img src="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmSpyJLkMmWGz4nZZSmwTzHDPzYdvVqYZNpLBPGqkW6G6t/Screenshot%202019-04-04%2015.23.19.png" alt="Screenshot 2019-04-04 15.23.19.png" srcset="https://images.hive.blog/768x0/https://cdn.steemitimages.com/DQmSpyJLkMmWGz4nZZSmwTzHDPzYdvVqYZNpLBPGqkW6G6t/Screenshot%202019-04-04%2015.23.19.png 1x, https://images.hive.blog/1536x0/https://cdn.steemitimages.com/DQmSpyJLkMmWGz4nZZSmwTzHDPzYdvVqYZNpLBPGqkW6G6t/Screenshot%202019-04-04%2015.23.19.png 2x" /> <h4>Comparing top bid/ask on each update <p dir="auto">To compare all exchange pairs on each update the exchange names, which are equivalent to the <code>key values for the shared <code>orderbooks dict, are extracted and then passed on to <code>calculate_price_delta() <pre><code># main.py if orderbooks['last_update'] != current_time: with lock: # extract and print data for exchanges in sets: ex1, ex2 = exchanges calculate_price_delta(orderbooks, ex1, ex2) print(f"Last update: {orderbooks['last_update']}\n") # set local last_update to last_update current_time = orderbooks['last_update'] <p dir="auto">The top bid/ask values are extracted from the <code>orderbooks dict and converted to floats. Then the absolute difference is calculated by <code>delta() and converted into a percentage. This is done for each pair of Exchanges. <pre><code># calculate absolute delta in percentage def delta(v1, v2): return abs((v2-v1)/v1)*100 # retrieve top bid/ask for each exchange # calculate deltas def calculate_price_delta(orderbooks, ex1, ex2): bid1 = float(orderbooks[ex1]['bids'][0][0]) bid2 = float(orderbooks[ex2]['bids'][0][0]) ask1 = float(orderbooks[ex1]['asks'][0][0]) ask2 = float(orderbooks[ex2]['asks'][0][0]) bid = delta(bid1, bid2) ask = delta(ask1, ask2) print(f'{ex1}-{ex2}\tBID Δ: {bid:.2f}% ASK Δ: {ask:.2f}%') <p dir="auto">The result is a delta in % from the bid of both exchanges and the ask of both exchanges. The percentage is calculated relative to the first of the two exchanges. <p dir="auto"><center><br /> <img src="https://images.hive.blog/0x0/https://cdn.steemitimages.com/DQmNof3ajrZTwD4ZrdYbdEgzy9sU73ZhbTRBiwsqQsJ42Uw/websockets_part4_2.gif" alt="websockets_part4_2.gif" /><br /> <h3>Curriculum <ul> <li><a href="https://steemit.com/utopian-io/@steempytutorials/part-1-connecting-to-steem-orderbook-stream-via-websockets-on-different-exchanges" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Part 1: Connecting to STEEM orderbook stream via websockets on different exchanges <li><a href="https://steemit.com/utopian-io/@steempytutorials/part-2-manage-local-steem-orderbook-via-websocket-stream-from-exchange" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Part 2: Manage local 'STEEM' orderbook via websocket stream from exchange <li><a href="https://steemit.com/utopian-io/@steempytutorials/part-3-connecting-and-managing-multiple-websocket-streams-in-parallel" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Part 3: Connecting and managing multiple websocket streams in parallel <hr /> <p dir="auto">The code for this tutorial can be found on <a href="https://github.com/Juless89/tutorials-websockets" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Github! <p dir="auto"><span>This tutorial was written by <a href="/@juliank">@juliank.