Pi Life, part two

[chebe]

In Pi Life, part one I only got as far as making my raspberry pi functional again through upgrade. Now on to the script I want to put on it.

After doing a project, and writing it up, I am usually too lazy to tweet my own blog posts, so I wrote a little python script to scrape my RSS feed and post a link to the most recent post to twitter. (For people who don't use RSS.)


Step Two; create twitter app entry and get access tokens

This part seems to change constantly, so here are some posts I used to help me along the way;
https://www.digitalocean.com/community/tutorials/how-to-create-a-twitter-app
https://iag.me/socialmedia/how-to-create-a-twitter-app-in-8-easy-steps/
https://code.tutsplus.com/articles/creating-a-twitter-oauth-application--net-16614


Step Three; create python script

Assuming you have python3 already;
pip3 install tweepy
pip3 install feedparser
pip3 install bs4
pip3 install lxml

And because BeautifulSoup can't seem to find lxml this way;
sudo apt-get install python3-lxml

Create a python script something like this (the exact code will depend on the structure of your RSS);

from bs4 import BeautifulSoup
import feedparser
import os
from requests import get
import tweepy
import urllib3

consumer_key = 'your_key'
consumer_secret = 'your_secret'
access_token = 'your_access_token'
access_token_secret = 'your_access_token_secret'

# set up OAuth and integrate with API
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth)

# get RSS feed
blog = feedparser.parse('https://your_site_rss')

blog_title = 'Untitled'

# only really want most recent post
latest_blog = blog['entries'][0]

if 'title' in latest_blog:
    blog_title = latest_blog['title']

tweet = "New blog post: \"" + blog_title + "\" - " + latest_blog['link']

img_link = ''

soup = BeautifulSoup(latest_blog['summary'], "lxml")

# get last img link
if len(soup('p', attrs={'class':'image'})) > 0:
    img_link = soup('p', attrs={'class':'image'})[-1].find('img')['src']

if len(img_link) > 0:
    filename = 'temp.jpg'
    with open(filename, 'wb') as image:
        response = get(img_link)
        image.write(response.content)
        print(tweet)
        api.update_with_media(filename, status=tweet)
        os.remove(filename)
else:
    print(tweet)
    api.update_status(status=tweet)

Step Four; deal with errors

My main error happened when I wanted to get an image used in a post. My images are self-hosted, Let's Encrypt signed. And the python3 libraries were not happy;
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

Now you can just turn off verification;
response = get(img_link, verify=False)
But you still get an annoying warning message.

I tried so many different ways to try and fix this. This is the only thing I found that worked.

First, go to Let's Encrypt Certs and save the Intermediate, Active, cross-signed cert. Remove the .txt suffix. Verify this cert fixes the error for you with;
curl https://your_website --cacert /path/to/lets-encrypt-x3-cross-signed.pem
If no errors, all good.

Now, in python;
import certifi
print(certifi.where())

Mine returns /home/pi/.local/lib/python3.7/site-packages/certifi/cacert.pem
Edit this file, append everything in lets-encrypt-x3-cross-signed.pem to the end. (Leave a comment telling yourself what you did for future.) Save and exit. Rerun the tweeting python script.


Step Five;

Glory in the lack of errors/warnings. And automatically created tweets, I guess.

Comments

[chebe]

I think I have figured out the root cause of this. My hosting provider makes using self-signed certs tricky (and charges a lot for their own). I have to;
- upload priv as normal,
- upload only the single cert, not the fullchain.pem,
- in a separate section upload the cross-signed intermediate Let's Encrypt cert.