johnpfeiffer

CircleCI for a Pelican static Github site

When I realized my previous CI/CD vendor had finally broken something (after 7 years travis.org incompatibly became travis.com) I decided out with the old and in with the new. Also, Python2 was deprecated so onto Python3!

So this article is a non-comprehensive reprise of How to Setup a Static Site with Github Pages...

Luckily I already had dabbled in CircleCI so the fundamentals were an easy copy paste: https://blog.john-pfeiffer.com/using-circleci-as-continuous-integration-and-continuous-deployment/

Roughly we will:

  1. have python
  2. install dependencies (pelican)
  3. (optional) install plugins
  4. get blog source markdown (ideally from version control like github)
  5. run build commands (to convert markdown to html)
  6. push to a second github repository

A new Pelican Dev Environment

unfortunately getting Docker setup is outside the scope of this tutorial

Just a practice run so you get a feel for it:

sudo docker run --rm -it cimg/python:3.8 /bin/bash
python --version
python -m pip install "pelican[markdown]" beautifulsoup4
exit

an ephemeral docker container with interactive bash shell, cimg is the CircleCI optimized docker image, https://hub.docker.com/u/cimg versus https://hub.docker.com/r/circleci/python/

Installing Pelican is documented: https://docs.getpelican.com/en/latest/

Using pelican-quickstart for a skeleton

assuming you are still in your docker container shell

PYVER=$(ls /home/circleci/.pyenv/versions | grep 3.8)
cd "/home/circleci/.pyenv/versions/$PYVER/bin/"
./pelican --version
mkdir yoursite
cd yoursite
../pelican-quickstart
  Welcome to pelican-quickstart v4.6.0.
  This script will help you create a new Pelican-based website.
  ... (accept all defaults) ...
  Done. Your new project is available at /home/circleci/.pyenv/versions/3.8.11/bin/yoursite

A little directory dancing in the container to get to the pip installed pelican binary

You still need an article with a minimum of content...

vim yoursite/content/my-article.md

Title: My Article
Date: 2021-09-09 22:22
Category: Technology

Hello world.

Generate html from markdown

./pelican yoursite/content/

creates an "output" directory with the HTML and all the index and other pages also updated

my-article.html
archives.html
authors.html
categories.html
category
feeds
index.html
tags.html
theme

Docker volume with your pre-existing blog content

sudo docker run --volume /home/ubuntu/blogsource/:/home/circleci/blogsource --rm -it cimg/python:3.8 /bin/bash
mkdir /home/circleci/OUT
PYVER=$(ls /home/circleci/.pyenv/versions | grep 3.8)
cd "/home/circleci/.pyenv/versions/$PYVER/bin/"
./pelican /home/circleci/blogsource/pelican-project/content -o /home/circleci/OUT -s /home/circleci/blogsource/pelican-project/publishconf.py

Docker will mount your local directory with pelican project markdown mapped to "/home/circleci/blogsource" in the docker container

If you do not specify a new "output directory" then pelican may get confused if there is already content in "pelican-project/output"

Linking Github to CircleCI

assuming you have your blog markdown in a Github repository and logged into CircleCI and authorized it for access to your Github repos

Click on the CircleCI button for your source code: "Setup a Project"

  • Use CircleCI's template for a configuration (a python project)
  • customize it: https://circleci.com/docs/2.0/executor-types/
  • trial and error to get various commands in the docker container right (use the Dev environment above)
  • There is a live-config-editor "Edit Config"

save and merge (.circleci/config.yml)

Now your source code (markdown) for your (pelican) blog should have a CircleCI configuration like the following:

.circleci/config.yml

version: 2.1
jobs:
  build:
    resource_class: small
    docker:
      - image: cimg/python:3.8
    steps:
      - checkout
      - run:
          name: Install Pelican and Build Content
          command: |
            python -m pip install "pelican[markdown]" beautifulsoup4
            pip freeze | grep pelican
            cd ..
            pwd
            ls -ahl /home/circleci/project
            find . -type f -iname pelican
            PYVER=$(ls /home/circleci/.pyenv/versions | grep 3.8)
            cd ".pyenv/versions/$PYVER/bin/"
            ./pelican --version
            mkdir -p /home/circleci/OUT
            ./pelican /home/circleci/project/content -o /home/circleci/OUT -s /home/circleci/project/publishconf.py
            ssh-add -D

      - add_ssh_keys:
              fingerprints:
                          - "4e:c1:a6:83:...:cc"
      - run:
          name: Publish to GitHub Static Site
          command: |
            cd ..
            ls -ahl
            git clone [email protected]:johnpfeiffer/johnpfeiffer.github.io.git
            cd johnpfeiffer.github.io
            git config user.email "[email protected]"
            git config user.name "John Pfeiffer CircleCI"
            git checkout master
            cp -a /home/circleci/OUT/* .
            git status
            git add --all .
            git commit --allow-empty -m "CircleCI publishing $CIRCLE_BUILD_NUM from sha $CIRCLE_SHA1"
            ls -ahl ~/.ssh/
            GIT_SSH_COMMAND='ssh -v -i ~/.ssh/id_rsa_4ec1a683...cc' git push origin master

Everything from add_ssh_keys and below should only be added once you have completed the SSH Key steps below

Removing all SSH keys from the CircleCI agent is because we're done checking out this repo and need to not confuse Git later

Pushing to a Github Page repository

The target for all of this has been your Github static page: https://pages.github.com/

Create a new SSH key dedicated to only this purpose

ssh-keygen -t rsa -b 4096 -C "CircleCI Deploy Key with Write Access" -f /tmp/cikey

Note that this key is just for CircleCI for just this one repo

Navigate to the "Deploy keys" section of the Settings of the Repo in github that will receive the pushes e.g https://github.com/johnpfeiffer/johnpfeiffer.github.io/settings/keys

  • click on "Add deploy key"
  • Paste in the cikey.pub file contents
  • Ensure the "Allow write access" checkbox is checked

In CircleCI: Go to your source project (under your username) and choose the "Project Settings" button, then subsection SSH Keys e.g. https://app.circleci.com/settings/project/github/johnpfeiffer/source.johnpfeiffer.github.io/ssh

Copy the private "cikey" (which starts with "-----BEGIN RSA PRIVATE KEY-----") into your buffer.

Do not share or paste your private key anywhere else than CircleCI

Add an Additional SSH Key "Add SSH Key" Hostname: githubstaticpage Private Key: -----BEGIN RSA PRIVATE KEY-----...

Now copy the "Fingerprint" of the key e.g. something like 4e:c1:... This is what goes in your circleCI config

CircleCI config explained

This portion is your CircleCI agent manually checking out the actual "Pages" HTML repository:

            git clone [email protected]:johnpfeiffer/johnpfeiffer.github.io.git
            cd johnpfeiffer.github.io
            git config user.email "[email protected]"
            git config user.name "John Pfeiffer CircleCI"
            git checkout master

It is important to copy the new HTML content in...

cp -a /home/circleci/OUT/* .

technical debt: should use the rsync command to actually reflect removed content too

Here is where the previously created SSH Key and fingerprint are explicitly used to specify to git to use that ssh key to push to github...

        GIT_SSH_COMMAND='ssh -v -i ~/.ssh/id_rsa_4ec1a683...cc' git push origin master

ssh verbose shows you which Identity/Key is being used to deploy/push to Github

References

Pelican Plugins

My site still uses the old plugins which I "vendored" into the actual repo for simplicity:

Instructions on Pelican Plugins in general (and the new way that I have not yet adopted)

Thankfully this person documented how to use Pelican4.6 with MARKDOWN for the built in Table of Contents (TOC):

Published

Category

build-CI-CD-devops

~975 words

Tags