STEEM Price Feed

STEEM Price Feed: STEEM Currency Price Feed for Witnesses (https://github.com/steemed/steem-price-feed/)

<h1>Introduction <p dir="auto"><a href="https://github.com/steemed/steem-price-feed/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">STEEM Price Feed is yet another<br /> <a href="https://steemit.com/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">STEEM price feed for witnesses.<br /> It has several advantages over simpler feed scripts including: <ul> <li>Extensive configurability through a <a href="http://yaml.org/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">YAML config file. <li>The ability to add exchanges through the config file. <li>Volume-weighted prices that proportionally reduce the influence of low volume markets. <li>Stochastic update based on variance analysis of price history. <h2>Stochastic Updating <p dir="auto">One innovation of STEEM Price Feed is that the decision to update at<br /> any cycle is probabilistic. If the current estimated price satisfies the<br /> "hard" publication criteria (exceeds a minimum price fluctation<br /> and minimum waiting period), then the variance of the feed history<br /> is calculated and the current price is converted to a<br /> <a href="https://en.wikipedia.org/wiki/Standard_score" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">z-score. This score is<br /> then converted to a cumulative probability using the<br /> <a href="https://en.wikipedia.org/wiki/Error_function" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">standard error function<br /> and if a random number in the interval <code>[0,1) is less than the cumulative<br /> probability, then the current price estimate is published. If not,<br /> then STEEM Price Feed tries again after a waiting period. <h3>Rationale for Stochastic Updating <p dir="auto">The stochastic updating utilized by STEEM Price Feed is based on decision<br /> theory, and attempts to answer the question "is the price observed now<br /> significant enough to warrant an update?" The answer to this question<br /> is probabilistic in nature because any significance in a price change has<br /> a component of uncertainty. Updating therefore reflects this uncertainty<br /> in that updating is more likely when there is a higher probability that a price<br /> change is significant (i.e. not due to the random variation observed in<br /> the price history). Because markets are extremely complex<br /> phenomena with far too many variables to model, the most reasonable estimate<br /> for this uncertainty is a<br /> <a href="https://en.wikipedia.org/wiki/Normal_distribution" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">normal distribution. <h1>Dependencies <p dir="auto">STEEM Price Feed has only two dependencies: <a href="http://pyyaml.org/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">PyYaml and<br /> <a href="http://docs.python-requests.org/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Requests.<br /> On <a href="http://www.ubuntu.com/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Ubuntu, these dependencies<br /> can be installed with the following command: <pre><code>sudo apt-get install python-yaml python-requests <h1>Configuration <p dir="auto">An example configuration file is provided in the <code>examples directory as <code>feed-example.yaml.<br /> The configuration file is specified when calling the <code>steem-price-feed.py script: <pre><code>python steem-price-feed.py /home/ima/steem/feed.yaml <p dir="auto">Although the instructions below may appear intimidating,<br /> <strong>only a couple of settings at the top of the configuration file actually need to be changed.<br /> These settings are <ul> <li><code>witness_name <li><code>wallet_password <li><code>rpc_user <li><code>rpc_password <p dir="auto">It is perfectly reasonable to take the rest of the settings as the defaults. <p dir="auto">The configuration file has two main sections, <code>settings and <code>market_data. <h2>Section: <code>settings <p dir="auto">The settings are <ul> <li><strong><code>witness_name: the witness name (e.g. <em>steemed) <li><strong><code>wallet_password: the witness password <li><strong><code>min_publish_interval : do not publish a new feed if it has not been this many <em>hours since the last <li><strong><code>max_publish_interval : pulish a new feed if this many <em>hours have passed since the last publication <li><strong><code>rpc_ip: the ip that the wallet server (<code>cli_wallet) runs on (e.g. <code>127.0.0.1) <li><strong><code>rpc_port: the port that the wallet server runs on (e.g. <code>8091) <li><strong><code>rpc_user: the user set for <code>cli_wallet using the <code>--server-rpc-user flag <li><strong><code>rpc_password: the password set for <code>cli_wallet using the <code>--server-rpc-password flag <li><strong><code>default_base: a base price for 1 STEEM, denominated in SBD; used if all else fails <li><strong><code>log_file: <em>optional; a file to print errors and debugging information;<br /> if not provided, <code>stdout will be used (default: <code>null) <li><strong><code>debug: <em>optional; emit debugging information if set to <code>true (default: <code>false) <h2>Section: <code>market_data <p dir="auto">Market data subsections are market pairs, expressed as <code>query_base, where <code>query is the asset<br /> that is priced in terms of the <code>base asset. For example, the <code>btc_usd subsection means that<br /> BTC pricing information will be reported in terms of USD. Volume will be reported in terms of BTC (<code>query). <p dir="auto">Each <code>market_data subsection has sub-subsections for each exchange. For example, current <code>btc_usd exchanges<br /> are <code>bitfinex, <code>coinbase, <code>okcoin, <code>bitstamp. Each exchange has two subsections, <code>price and <code>volume<br /> that specify how to retrieve these amounts. Key-value pairs for <code>price and <code>volume are: <ul> <li><strong><code>url: the base API url for the query (e.g. <code>https://www.okcoin.com/api/v1/ticker.do) <li><strong><code>query_params: form query parameters used for http GET, express as a JSON object (e.g. <code>{"symbol": "btc_usd"}) <li><strong><code>accessor: specifies how to traverse the JSON object to extract the needed information,<br /> expressed as a list of object members (keys) and/or array indices, intermingled <h3>Example of <code>market_data <p dir="auto">An example of a <code>market_data for the <a href="https://www.okcoin.com/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">OKCoin subsection is: <pre><code>market_data : btc_usd : ... okcoin : price : url : "https://www.okcoin.com/api/v1/ticker.do" query_params : {"symbol": "btc_usd"} accessor : ["ticker", "last"] volume : url : "https://www.okcoin.com/api/v1/ticker.do" query_params : {"symbol": "btc_usd"} accessor : ["ticker", "vol"] <h4>Query URL and <code>query_params <p dir="auto">To query the BTC:USD price for OKCoin, this configuration specifies the URL <p dir="auto"><a href="https://www.okcoin.com/api/v1/ticker.do?symbol:btc_usd" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://www.okcoin.com/api/v1/ticker.do?symbol:btc_usd <p dir="auto">Notice how the <code>query_paramswere appended as form submission data to the URL: <pre><code>query_params : {"symbol": "btc_usd"} <h4>Accessing the JSON: <code>acessor <p dir="auto">The <em>formatted JSON returned by this query as of the time of writing these instructions is: <pre><code>{ "date":"1462307028", "ticker": { "buy":"449.25", "high":"449.62", "last":"449.25", "low":"440.75", "sell":"449.4", "vol":"10025.1961" } } <p dir="auto">Assuming the JSON object representing this result is called <code>OK, accessing the last price<br /> would look like: <pre><code>OK["ticker"]["last"] <p dir="auto">This corresponds to the <code>acessor provided for the query in the YAML configuration file: <pre><code>accessor: ["ticker", "last"] <p dir="auto">Because the syntax of accessing array elements is exactly the same as accessing object members<br /> except that member names are strings and array indices are integers, the <code>accessor allows<br /> simply mixing member names and array indices. For example, as of the time of writing these<br /> instructions, the <em>formatted <a href="https://bittrex.com" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Bittrex market summary for STEEM is: <pre><code>{ "success":true, "message":"", "result": [ { "MarketName":"BTC-STEEM", "High":0.00120025, "Low":0.00090000, "Volume":35316.37017575, "Last":0.00107443, "BaseVolume":34.61316041, "TimeStamp":"2016-05-03T20:35:41.713", "Bid":0.00097042, "Ask":0.00105730, "OpenBuyOrders":105, "OpenSellOrders":341, "PrevDay":0.00109784, "Created":"2016-04-17T01:22:13.35" } ] } <p dir="auto">Notice that this API call has a single element array that contains all of the BTC:STEEM market<br /> information. If this JSON object were called <code>TREX, then the volume information would be accessed<br /> like this: <pre><code>TREX["result"][0]["Volume"] <p dir="auto">The corresponding <code>accessor line would then be <pre><code>accessor : ["result", 0, "Volume"] <h1>Usage <p dir="auto">The STEEM Price Feed service is meant to be run as a daemon, although it can easily<br /> be run manually like a normal python script for debugging. <p dir="auto">The services <code>steem-cli and <code>steem-price-feed will start on boot if properly configured.<br /> Their setup is described below. <p dir="auto">If stopped, these services can be started with the following commands after they are set up: <pre><code>sudo service steem-cli start sudo service steem-price-feed start <p dir="auto">These commands assume that a suitable <code>steemd instance is running and listening at<br /> the default RPC port. <h2>Upstart Service <code>steem-cli <p dir="auto">Because publishing a price feed requires an open wallet, an instance of <code>cli_wallet must<br /> be run as a daemon process, listening on an RPC port. On Ubuntu,<br /> this is best achieved using <a href="http://upstart.ubuntu.com/" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Upstart services. <p dir="auto">To create an Upstart service for <code>cli_wallet, add this simple initialization file at <code>/etc/init/steem-cli.conf<br /> (edited for your system, especially changing the password to something very strong): <pre><code># steem-cli service - steem cli_wallet service for witness description "Steem CLI" author "Ima Witness <ima@example.com>" # Stanzas # # Stanzas control when and how a process is started and stopped # See a list of stanzas here: //upstart.ubuntu.com/wiki/Stanzas # When to start the service start on runlevel [2345] # When to stop the service stop on runlevel [016] # Automatically restart process if crashed respawn # Essentially lets upstart know the process will detach itself to the background # This option does not seem to be of great importance, so it does not need to be set. # expect fork # Specify working directory chdir /home/ima/steem # Specify the process/command to start, e.g. exec ./cli_wallet -u user -p password \ --rpc-endpoint=127.0.0.1:8091 -d 2>cli-debug.log 1>cli-error.log <h2>Upstart Service <code>steem-price-feed <p dir="auto">Similarly, it is desirable for STEEM Price Feed (<code>steem-price-feed.py) to<br /> run as an upstart service so that it starts on reboot and respawns in case<br /> it is terminated undesirably. <p dir="auto">Put the following file at <code>/etc/init/steem-price-feed.conf (edited for your<br /> own system). <pre><code># steem-price-feed service - steem-price-feed service for witness description "STEEM Price Feed" author "Ima Witness <ima@example.com>" # Stanzas # # Stanzas control when and how a process is started and stopped # See a list of stanzas here: //upstart.ubuntu.com/wiki/Stanzas # When to start the service start on runlevel [2345] # When to stop the service stop on runlevel [016] # Automatically restart process if crashed respawn # Essentially lets upstart know the process will detach itself to the background # This option does not seem to be of great importance, so it does not need to be set. # expect fork # Specify working directory chdir /home/ima/steem # Specify the process/command to start, e.g. exec /usr/bin/python steem-price-feed.py feed.yaml 2>feed-debug.log 1>feed-error.log