Automation to the Rescue!
<p dir="auto"> This Python 3.6 script uses the Steem Beem Library and a variety of methods to archive your Steem Blog images as well as markdown files. It used image hash verification to ensure that files are downloaded once saving valuable storage space.
<p dir="auto"><img src="https://images.hive.blog/768x0/https://img.esteem.app/735k3t.png" alt="735k3t.png" srcset="https://images.hive.blog/768x0/https://img.esteem.app/735k3t.png 1x, https://images.hive.blog/1536x0/https://img.esteem.app/735k3t.png 2x" />
<p dir="auto"><img src="https://images.hive.blog/768x0/https://img.esteem.app/dcelof.png" alt="dcelof.png" srcset="https://images.hive.blog/768x0/https://img.esteem.app/dcelof.png 1x, https://images.hive.blog/1536x0/https://img.esteem.app/dcelof.png 2x" />
<h1>Repository
<p dir="auto"><span><a href="https://github.com/anthonyadavisii/SteemYaLater" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">https://github.com/anthonyadavisii/SteemYaLater
<h1>Version 2.0 Change Notes
<ul>
<li>Added PyCurl download method to address issues w steemitboard images
<li>Data Deplication enabled: Prevents redownload of file if already exists in folder structure. Symbolic link with relative path placed instead saving valuable storage space.
<li>Logging and CSV output: Session log file is produced in working directory. Output CSVs are created for each account so users may readily see what failed and may require manual action.
<h1>Version 1.0
<p dir="auto"> Version 1 was the basic framework with wget. We don't talk about version 1 anymore.
<p dir="auto"><span><img src="https://images.hive.blog/768x0/https://i.redd.it/zw17doei2h211.jpg" srcset="https://images.hive.blog/768x0/https://i.redd.it/zw17doei2h211.jpg 1x, https://images.hive.blog/1536x0/https://i.redd.it/zw17doei2h211.jpg 2x" />
<p dir="auto"> I've worked hard and made a ton of progress in order to give my fellow Steemians a way to save their priceless data.
<h1>Roadmap
<ul>
<li>Steem Blog Backup as a Service
<li><span><a href="/@dtube">@dtube thumbnail support
<li>Upload to Skynet web portal
<h1>Known Issues
<p dir="auto"> DTube thumbnails will not download as they are not stored within the Beem Comment json_metadata image property. Logic to be added to accomodate. Also, some links may require escape characters. These will be addressed as time permits.
<h3>Uses Python 3.6
<h1>Install Prerequisites
<pre><code># PyCurl may require the following packages be installed.
sudo apt install libcurl4-openssl-dev libssl-dev
# Python modules installation
Python 3.6 -m pip install beem
Python 3.6 -m pip install wget
Python 3.6 -m pip install urllib3[secure]
Python 3.6 -m pip install pycurl
Python 3.6 -m pip install certifi #may or may not be needed if the [secure] option is used for urllib3
<h1>Execute Script
<pre><code>python3.6 SteemYaLater.py
<h3>Prompts for Steem User. Alternatively, you may populate the accounts list variable with users to backup
<p dir="auto">Account to Backup? anthonyadavisii
<h3>Script will crawl your blog_entries filtering out resteems (reblogs)
<p dir="auto"><img src="https://images.hive.blog/768x0/https://img.esteem.app/st4y66.png" alt="st4y66.png" srcset="https://images.hive.blog/768x0/https://img.esteem.app/st4y66.png 1x, https://images.hive.blog/1536x0/https://img.esteem.app/st4y66.png 2x" />
<h3>Will then cycle through each blog_entry, save the body to a .txt files, and grab any images it can with wget or urllib3
<h1>Feel free to reach out if you need help! If you appreciate the work, consider sending me a tip!
<p dir="auto"><img src="https://images.hive.blog/768x0/https://img.esteem.app/dcelof.png" alt="dcelof.png" srcset="https://images.hive.blog/768x0/https://img.esteem.app/dcelof.png 1x, https://images.hive.blog/1536x0/https://img.esteem.app/dcelof.png 2x" />
<h3>How to put your FREE Downvotes to work in 2 easy steps!
<p dir="auto"><img src="https://images.hive.blog/0x0/https://img.esteem.app/2sxn09.gif" alt="2sxn09.gif" /><br />
<a href="https://steemit.com/esteem/@anthonyadavisii/how-to-put-your-free-downvotes-to-work-in-2-easy-steps" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Learn more!
<p dir="auto"><em>This post was created using the <a href="https://github.com/eSteemApp/esteem-surfer/releases" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">@eSteem Desktop Surfer App.
<p dir="auto"> They also have a <a href="https://esteem.app/hive-125125/@esteemapp/esteem-referrals" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">referral program that promotes users to onboard to our great chain. Sign up using my <a href="https://esteem.app/signup?referral=anthonyadavisii" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">referral link to help support my efforts to improve the Steem blockchain.
<h3>Ditch Partiko and get eSteem today!
<table>
<tr>
<th>PlayStore - Android
<th>Windows, Mac, Linux
<tr>
<td><a href="https://play.google.com/store/apps/details?id=app.esteem.mobile.android" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link"><img src="https://images.hive.blog/768x0/https://img.esteem.ws//twstd2x0xx.png" alt="Get eSteem on Google Play" srcset="https://images.hive.blog/768x0/https://img.esteem.ws//twstd2x0xx.png 1x, https://images.hive.blog/1536x0/https://img.esteem.ws//twstd2x0xx.png 2x" />
<td><a href="https://github.com/eSteemApp/esteem-surfer/releases" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link"><img src="https://images.hive.blog/768x0/https://img.esteem.ws//42dgm1zzo1.png" alt="Get eSteem for Desktop" srcset="https://images.hive.blog/768x0/https://img.esteem.ws//42dgm1zzo1.png 1x, https://images.hive.blog/1536x0/https://img.esteem.ws//42dgm1zzo1.png 2x" />
<tr>
<th>AppStore - iOS
<th>Web
<tr>
<td><a href="https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=1451896376&mt=8" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link"><img src="https://images.hive.blog/768x0/https://img.esteem.ws//ir3o7p26w7.png" alt="Get eSteem on AppStore" srcset="https://images.hive.blog/768x0/https://img.esteem.ws//ir3o7p26w7.png 1x, https://images.hive.blog/1536x0/https://img.esteem.ws//ir3o7p26w7.png 2x" />
<td><a href="https://esteem.app" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link"><img src="https://images.hive.blog/768x0/https://img.esteem.ws/100/bqaxajqbid.png" alt="Get eSteem for Desktop" srcset="https://images.hive.blog/768x0/https://img.esteem.ws/100/bqaxajqbid.png 1x, https://images.hive.blog/1536x0/https://img.esteem.ws/100/bqaxajqbid.png 2x" />
There are errors in your install prerequisites
<p dir="auto">Also, I had a problem installing pycurl, so do this first:<br /> <code>sudo apt install libcurl4-openssl-dev libssl-dev<br /> then<br /> <code>python3.6 -m pip install pycurl <p dir="auto">Another problem when I run it:<br /> <code>FileNotFoundError: [Errno 2] No such file or directory: '/home/drak/SteemYaLater/Backups/drakos' <p dir="auto">So I created that folder manually, then I run the tool again, it fetches the posts but when it starts downloading them, it spits out binary characters on the screen. Did you test this tool properly?Python 3.6should bepython3.6I forgot about that. Yes, that was also encountered during my setup but it slipped my mind. I'll update the readme. Thanks, @drakos!
Terminal is spitting out garbage but it's backing up some of my posts at least after I made the backup folder.
Good catch! Updated repo
https://github.com/anthonyadavisii/SteemYaLater/commit/172a3883a29b40fed8b70d5124e6293e66084439
My man! For those who didn't know, he's been working diligently on this for a while now. He deserves more than the 125 upvotes I'm seeing right now.
Amazing work, thank you
Really appreciate coming up with this! Going to test now.
May want to get the latest as I made a few tweaks and addressed a couple issues.
I ran into some delimiter issues/erros while trying to run. I have raised an issue. Will check whether I can fix (Im stuck with this bloody Bitcoin and ZMQ crash at work!)
Can you set it up to email pdf's?
Or provide a link to download a PDF?
What I'm doing currently is zipping up the backup on my Ubuntu box, transferring that to my cloud provider, and then providing a shareable link.
It's a few too many manual steps at the moment but plan to automate some of the steps such as having the script take care of compression. You're update should be next up btw
Sweet, I would hate to be wiped from the only history likely to be written by me.
I'm not going to try this in current form due to an adversion to the technical nature, but I do like the road map.
Your efforts are greatly appreciated.
I look into putting all the dependencies into a docker container to simplify use and so users on non Ubuntu operating systems can use. Thanks for the sentiment. Glad to help!
I'm running linux minty on a netbook I rarely use...would that manage this?
Unfortunately, the newest version from Github does not work. Any chances to fix that?
<pre><code> File "SteemYaLater.py", line 187 continue ^ SyntaxError: 'continue' not properly in loopThanks for the heads up. Will take a look.
Let me know when it's ready, I would love to contribute by dockerizing it.
Removed those continue statements. Should be good to go now. Putting it in docker would be really helpful. Thanks!
Thanks, I will do a PR tomorrow.
@dexterdev
Thank you @mathowl
@scienceblocks see this. maybe of help to you
Updated again to inject header data to steemitimages.com. Recommend increasing pauseTimeInit due to suspected rate limiting.
Finally!
Thanks Bro!
@tipu curate
NP. Just added another check on image hosts for DNS lookups. Ideally, I'll store unresolved hosts in memory and skip them if they were not resolved previously. Due to the pyCurl settings, the script tends to get hung up for a while on timeouts. It works but definitely more optimization needed.
Appreciate the reblog. Hoping to have a full-fledged service up soon but need to work out my storage solution atm.
Congratulations @anthonyadavisii! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :
<table><tr><td><img src="https://images.hive.blog/768x0/https://steemitimages.com/60x70/http://steemitboard.com/@anthonyadavisii/posts.png?202003080036" srcset="https://images.hive.blog/768x0/https://steemitimages.com/60x70/http://steemitboard.com/@anthonyadavisii/posts.png?202003080036 1x, https://images.hive.blog/1536x0/https://steemitimages.com/60x70/http://steemitboard.com/@anthonyadavisii/posts.png?202003080036 2x" /><td>You published more than 450 posts. Your next target is to reach 500 posts. <p dir="auto"><sub><em>You can view <a href="https://steemitboard.com/@anthonyadavisii" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">your badges on your Steem Board and compare to others on the <a href="https://steemitboard.com/ranking/index.php?name=anthonyadavisii" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Steem Ranking<br /> <sub><em>If you no longer want to receive notifications, reply to this comment with the word <code>STOP <p dir="auto">To support your work, I also upvoted your post! <p dir="auto"><strong><span>Do not miss the last post from <a href="/@steemitboard">@steemitboard: <table><tr><td><a href="https://steemit.com/steemitboard/@steemitboard/downvote-challenge-add-up-to-3-funny-badges-to-your-board" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link"><img src="https://images.hive.blog/768x0/https://steemitimages.com/64x128/https://steemitimages.com/0x0/" srcset="https://images.hive.blog/768x0/https://steemitimages.com/64x128/https://steemitimages.com/0x0/ 1x, https://images.hive.blog/1536x0/https://steemitimages.com/64x128/https://steemitimages.com/0x0/ 2x" /><td><a href="https://steemit.com/steemitboard/@steemitboard/downvote-challenge-add-up-to-3-funny-badges-to-your-board" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Downvote challenge - Add up to 3 funny badges to your board<tr><td><a href="https://steemit.com/steemitboard/@steemitboard/use-your-witness-votes-and-get-the-community-badge" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link"><img src="https://images.hive.blog/768x0/https://steemitimages.com/64x128/https://cdn.steemitimages.com/DQmTugCUsoXX762vg1CuHRrpnPbfnjPogp8iCGv7F2kSVuj/image.png" srcset="https://images.hive.blog/768x0/https://steemitimages.com/64x128/https://cdn.steemitimages.com/DQmTugCUsoXX762vg1CuHRrpnPbfnjPogp8iCGv7F2kSVuj/image.png 1x, https://images.hive.blog/1536x0/https://steemitimages.com/64x128/https://cdn.steemitimages.com/DQmTugCUsoXX762vg1CuHRrpnPbfnjPogp8iCGv7F2kSVuj/image.png 2x" /><td><a href="https://steemit.com/steemitboard/@steemitboard/use-your-witness-votes-and-get-the-community-badge" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Use your witness votes and get the Community Badge <h6><a href="https://v2.steemconnect.com/sign/account-witness-vote?witness=steemitboard&approve=1" target="_blank" rel="noreferrer noopener" title="This link will take you away from hive.blog" class="external_link">Vote for @Steemitboard as a witness to get one more award and increased upvotes!Cool