johnpfeiffer - john pfeiffer

AI Natives do not question the gifts from the gods

2026-04-03T23:59:00-07:00

AI Natives don't question the gifts from the gods

At a recent SF hackathon we used ChatGPT to understand Clinical Trials, defaulted to Claude Code to code up everything, and used LangChain plus Minimax LLMs as the core solution.

My teammates are AI natives: just graduating from university and they don't question the answers at their fingertips.

Unfazed by the entirely new-to-them problem space of clinicaltrials.gov being a mess. Valuable information with awkward technology and clunky documentation was unlocked in just a few hours of "vibe coding".

No hesitation to try something, see what happens, and then just prompt again. Any unfamiliar domain term or technical issue is resolved with a quick prompt. No google, no stack overflow.

It was a joy to work with them and see the new possibilities as they take for granted the powerful capabilities and apply them without hesitation.

https://en.wikipedia.org/wiki/Prometheus 🔥

Reflecting on Arbitrage or Persistent Advantage

Extracting and analyzing data with a customized UI/product is now a weekend project, so what makes a business successful?

"Ship it" hackathon mode means we didn't spend any time thinking about the "7 powers". Identifying investment in opportunities based on clinical trials has no network effect, no economies of scale, no entrenched switching costs, no counter-positioning, no cornered resource, no branding, and especially no process power.

Or as Warren Buffett popularized

...moat that protects a valuable and much-sought-after business castle.

Coding agents raise the software floor dramatically, which increases the speed of competition.

Cloning or reverse-engineering from an interface/API reduces the "Switching Costs" of migration, as exemplified by the recent fierce AI-powered cloud competition: https://newsletter.pragmaticengineer.com/p/the-pulse-cloudflare-rewrites-nextjs

Google DeepMind spun out Isomorphic Labs and Anthropic acquired Coefficient Bio because the major AI players recognize the deep moats in biotech; having very private data is more valuable than ever.

What you get from People

When building something is cheaper and faster, how do you differentiate?

More than a business strategy, you need talented hard-working people who take the initiative because "Culture eats strategy".

Thanks to the organizers Hanwha AI Center + AI Valley, the rooms were abuzz with energy. Everyone was participating in the future: fast experimentation and doing more with less.

In-person had creativity, spontaneous connections... and loud talking-over-each-other, not enough conference rooms, caffeine shortages, and lines for lunch and the bathroom.

People vs Agents is a false dichotomy

Anthropic and OpenAI have unimaginably high valuations while profitable SaaS incumbents are feeling pressure from the stock market 2026 "SaaSPocalypse".

One way of viewing the world is replacing all human employees with AI agents = zero sum.

A more ambitious vision is augmenting every human in your org to 10x productivity.

But doubling business productivity doesn't help when your competitor has something magical with an AI-powered product. 🦄

I believe tackling the hardest problems that AI now unlocks creates huge customer demand... which needs AI-augmented humans to keep up.

Too much of a good thing?

We were coding so fast that we quickly hit complexity overload and spent the last hour of the hackathon debugging. A ball of mud monolith, lots of AI-generated code, no tests. =(

Decades of experience suddenly became invaluable:

Keeping the "main" core loop tight and easy to read
Modularized code like separating "external vendor/network calls" into separate clients
Each specific feature into manageable areas of partitioned complexity
Compartmentalization of failures to not affect the rest of the application
Logging, Tests

We need to question these "gifts from the gods", and know that hard-won principles still matter. Even when we're going as fast as possible for a hackathon demo.

Of course I'm disappointed we didn't win an award, but I am energized by seeing a glimpse of the future and connecting with the next generation of builders.

What a Security Hackathon Taught Me About Agents in the Cloud

2026-03-16T20:20:00-07:00

All you need is Agents and Human intent

Took 3rd place (according to the AI judge) and won the "Scariest" award at the NEBULA:FOG, an AI × Security hackathon in San Francisco on March 14, 2026.

My teammate's genius was giving human intent while 5+ (Claude Opus 4.6) agents did the work in the cloud (they'd melt a laptop). They downloaded the code, built and exercised targets, ran exploits, and auto-generated POC findings with evidence. They even created the presentation.

My role was knowing which tools are widely used in genomics, and observing the future be unevenly distributed.

Responsible disclosures in progress!

Security has always mattered

The tag line from our presentation was:

Low priority for scientists, High priority for security

As a teenager reading "The Cuckoo's Egg" by Cliff Stoll, I was inspired by the insights into the earliest days of networked computing and hacking. Systems need to be secure by design, and people have to understand their role for it to work.

Later, Steven Levy's "Crypto: How the Code Rebels Beat the Government, Saving Privacy in the Digital Age" impressed upon me that a single determined person like Whitfield Diffie could democratize privacy through technology. I also learned how the awesome innovation and implementation of asymmetric cipher keys underpins all secure communication and business today.

Add Schneier's principles and systematic thinking on security, Krebs's reporting on the actual threat landscape, and Troy Hunt's work (https://haveibeenpwned.com/) making breach data accessible to the public, and you understand the worldview that security is not just a feature, it's a transparent and deliberate foundation.

How it happened

We focused on key open source projects for genomics. We found a set of vulnerabilities in the tools and dependencies that are hiding in plain sight. They're not a crypto wallet or a bank, but people's genomic information is incredibly private and sensitive.

My teammate's years of experience were distilled into prompts in English, then the AI agents automated everything.

No Burp Suite. No Metasploit. No hand-crafted exploit scripts. Instead we had tea and pizza.

I'm used to running AI on my laptop, often tethered to foundation model vendors.

But I witnessed the future: that many agents working in parallel - they have to run in the cloud.

Agents in the Cloud

The abstraction pattern: hardware -> hosted infrastructure -> service -> intent-level runtime

evolution of running code:

physical servers in an office (that you maybe sometimes have to kick into rebooting)
data centers ("Colo's") handling the physical stuff (but you still start with bare metal and remotely install the OS, hopefully with a KVM-over-IP)
virtual servers and the rise of VMware (ok, managing dozens of VMs from templates, with snapshots, isn't so bad... until the over-provisioned host crashes)
AWS leading the way in 2006 with "cloud instances" (API for a server, cattle not pets! new scale, new problems like noisy neighbors and thousands of "machines")
PaaS (RIP Heroku) and then FaaS (AWS Lambda): "function as a service" = pure focus on the code

evolution of running generative AI:

2 physical GPUs proved deep learning scales on Alexnet (Alex Krizhevsky and Ilya Sutskever)
Nvidia DGX: GPUs as an appliance aka purpose built hardware for AI, in your datacenter
Renting GPUs from vendors like AWS (P2), Azure (N-series), Google Cloud (P100)
"intelligence on demand" from foundation models like OpenAI and Anthropic, albeit a clunky interface
Agents running "somewhere", taking human intent and just returning results

now for the fun problems with queuing, orchestration, observability, failure modes...

The Results

Those vulnerabilities were real, with evidence for each exploit. We (the humans) had to read the findings to ensure the most critical ones were real.

Note the hackathon judges' reaction — awarding us the "scariest" prize — tells you the severity.

Now the barrier to discovering serious vulnerabilities in critical open source infrastructure just dropped by an order of magnitude.

What previously required deep domain expertise in specific codebases, custom tooling, and hours of manual analysis, can now be directed by someone who understands vulnerability classes and can write clear prompts.

The economics favor the attacker: they only have to find one exploitable path, defenders have to patch all of them.

Hackathon Notes

I love hackathons: the ideas and energy, teams coming together, the action of building. "ShipIt" days were highpoints for me at Atlassian.

At Helix I evolved from participant to organizer of a company-wide 100+ person hackathon, including working closely with external guests (doctors/professors/professionals from the Medical University of South Carolina).

Hats off to the NebulaFog organizers/hosts, they created an environment I rarely see at tech events: genuinely high-trust, positive, and kind. Lots of different tracks and challenges. A 13-year-old demo'd and he was amazing.

https://nebulafog.ai/

No pitches. No spectators. A room full of people who ship, learning from each other, forming alliances, stress-testing what's possible.

A perfect situation to connect and learn, I'll keep in touch with folks and be back next year!

Pragmatic Summit - Nobody has the Answers on AI

2026-03-09T21:21:00-07:00

Nobody Has the Answers

I got my copy of "Refactoring" signed by both Martin Fowler and Kent Beck. That alone would have been worth the conference. What they shared in person at the Pragmatic Summit in SF was pure gold, Kent Beck put it bluntly:

"At this moment, nobody knows the answers to anything."

Martin Fowler and Kent Beck walk into a room full of AI startups

A decade ago I saw Bjarne Stroustrup in person, and he signed my C++ book.

25 years after the Agile Manifesto, two people who shaped the foundations of modern software engineering walked into a room full of AI-native startups and engineering leaders.

If you haven't heard of Martin Fowler then you're in for a huge treat, so many incredible thoughts about software development (shared at https://martinfowler.com/)

And Kent Beck's decades of wisdom (Extreme Programming, TDD, JUnit, etc.) are in books, substack posts, articles, podcasts, etc.

Paradigm shift

There was a huge historical shift into the era of personal computing and the internet. And for 25 years, experienced engineers have had a playbook:

Too many bugs? Here's how you write tests.
Can't write tests? Here's how you design and architect your software.
Go look up a (coding/architecture) pattern in the book

Now with LLMs it has become a completely different software development lifecycle and new tech stack; a whole different set of skills.

Kent Beck's poignant appeal (for geeks who love code): It doesn't feel safe right now.

His antidote:

"You're just as smart as everybody else because you're just as ignorant as everybody else."

In the age of AI hype fatigue and AI agent psychosis, I find this oddly reassuring.

The Craft Still Matters

The most energizing takeaway from the session was that good engineering practices aren't becoming obsolete, they're becoming more important.

Fowler described a repeatedly heard pattern: well-modularized code makes it easier for AI agents to work effectively. Good test suites help agents just as much as they help humans.

"The Venn diagram of developer experience and agent experience is a circle."

One colleague told Beck that 20 years of pushing TDD was paying off now because...

when you have "a big powerful genie," you need to know how to verify it's doing the right thing.

When you step away from hype around AGI and autonomous agents and use first principles thinking, this really makes sense.

Software engineering is not just programming: regardless of human or AI agent, it's based on foundations like complexity/modularity, design/documentation, tests, security, observability, etc.

Learn the Tools

Martin Fowler contended there's been snake oil in software for a long time, and "agile certification factories" (coaches, titles, etc).

His early experience with AI code completion in Emacs was bad enough that he nearly wrote it off entirely. Most completions were garbage.

But he kept probing, found Simon Willison's blog, and saw someone continuously experimenting, with curiosity, learning the new tools; willing to show both the good and bad, to say "I don't know."

Be skeptical about your skepticism

Kent Beck's version: "What's the least I can do to validate, to my own satisfaction, whether this claim is true or not?"

That experimental mindset has become, in his words, "a thousand times more valuable" in the last year. The answers change week to week. A tool that fails on Monday might work on Thursday. You can't have the answer. You can only have a practice of finding out.

SDLC + AI

Previously a generation (or two) of developers used to:

read requirements (PRD)
systems design (hopefully?)
write code (usually in an IDE)
run tests (right?)
submit pull request
continuous integration
continuous deployment
smoketests
observability and monitoring

given the rate of change I hope this doesn't become obsolete in a month ;)

Emerging SDLC With AI

deep research agent creates a report on the domain
LLMs "architect persona" brainstorms and rubber duck options
Using agents with Skills to plan and decompose the work
A carefully crafted prompt kicks off a series of agents in parallel
Human reviews and refines Test Cases and Guardrails, defines verifiability
Human creates Evals for the critical or non-deterministic portions of the workflow/service/product
A series of specialized agents (security, design, etc) review the agent produced artifacts
CI/CD and automated tests run; agents autonomously attempt fixes for anything failing
Human reviews the proposed output, any test failure escalations, eval scores
Production deployment triggers extra LLM traces and privacy protecting prompt logging

Shift Focus to the System

Kent Beck's personal reflection resonated: the deep satisfaction of refactoring code and taking a messy file, making tiny safe steps, watching clarity emerge. Then he said plainly: "I can't do that anymore."

Previously software engineers who were great at managing the code line-by-line and the attendant complexity, like using Tidy First or TDD, they had huge leverage.

There was a time when hand-writing optimized assembly was the high-leverage skill, then compilers made that less valuable.

The leverage in the craft is now understanding the domain and its connection to the software, the system. Not optimizing the code in a specific function or file.

This makes sense to me: so many complaints about microservices are due to an absence of Domain Driven Design, and missing the "value stream" — what are we building, for whom, and how does this piece connect to the whole?

Human intent: what was the system designed to do?

Measuring Success

Kent Beck's right that companies have very high hopes:

Expecting better, faster, cheaper

and his experience has been that inside a company, the incentives are so misaligned with actually achieving that.

AI can generate code fast, but human intent and system understanding are what drive it toward quality outputs and remarkable outcomes.

Kent Beck pointed out: carpentry didn't end because of circular saws (or nail guns). People just got more powerful tools.

In my opinion, measuring Lines of Code or Pull Requests is a poor proxy that's not correlated with the full outcome.

Burnout

And we've always known about software and burnout; clearly people coding need breaks because overwork creates bad ideas and negative value.

I know that when I'm in the zone and stay up well past midnight coding... then the morning comes. After sleeping, reviewing with fresh eyes (and brain) reveals so many obvious bugs and mistakes in the logic.

I also know from more than a decade of managing engineers that product release "death marches" and working nights and weekends degrades a team beyond repair, people are cranky and can no longer function together.

Once I even had to explain to the C-Suite: fine, we'll work weekends, but what days are left when that's no longer enough?

Conclusion

Seeing Kent Beck and Martin Fowler in person reinforced that really wise thought leaders in the industry also like corny jokes and find conference chairs uncomfortable.

Clearly they are beacons for quality craftsmanship, and their message was "experiment and learn to use this new tool well".

I also took away that we need to measure the impacts of the "new way" on outcomes we care about: users/customers, revenue... and the people building the software.

Resources

The conference: https://www.pragmaticsummit.com/

The specific session, and the amazing Pragmatic Summit 2026 playlist:

https://www.youtube.com/watch?v=CZs8J1ZD0CE&list=PLzwJJv8h-icjtYA5oHmc7g6qU1t4OqDqb&index=8

Addendum

I missed seeing Simon Willison (co-creator of Django) in person but did get to watch the video of his session, some important takeways:

His personal evolution from Human writes all the code -> LLM writes some code -> LLM writes all the code and Human reviews the code -> Human does not read the code at all, focus on creating a testable goal

Big organizations had a team create a service and API... and you just trust them until something goes wrong

write prompts on your phone and the LLM writes the code
prompt the LLM "use red/green TDD"; this focuses to minimal code writing, and improves quality and LLMs make writing tests almost free
full time code reviewer is exhausting - humans can't keep up with agents code output
think really hard how the agent proves what's been created really works
be creative to ask for tests how it really works in the real world (unit test of code < curl the webserver)
detailed specs and tests allow for very easy reverse-engineering and building by AI agents

Avoid the "Lethal Trifecta" https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

Have lots of fun, experiment with everything =)

Intro to MCP - give your LLM tools with model context protocol

2025-07-27T19:20:00-07:00

Why is weather so hard?

Let's start with a problem...

(rather than technology looking for a problem ;)

Getting the weather can be frustrating

which of the 6+ websites would you choose?
how many annoying ads?
why does one show the city first, another shows daily temperature, and a third jumps to the week ahead?

It can take many clicks to view a specific city that you're interested in...

And many prompts to "sign up as a user" or "subscribe aka pay money" to have your view of the weather customized

So pay for a weather app - that still doesn't do what you want?

Or find a weather API, write your own app/website, just the way you want it:

"5 day forecast of days and nights - temperatures in both F and C

How useful is your toilet without the plumbing?

LLMs (Large Language Models) excel at natural language processing (NLP) - like asking the computer about the weather (typos and all!)

But the LLM doesn't have access to up to date info (weather, prices) or private data (emails, databases).

Model Context Protocol is a way to solve multiple things:

A user would like the LLM to utilize newer or specific info
Each LLM vendor has a "custom way to configure"
With multiple LLM applications running locally... and you have a lot of tools
Organizations want a centralized way to control access to tools and (potentially private) information

Therefore this is an "N x M" problem: N (LLM) applications for M tools

The grand vision:

LLM vendors have a standard way to give users more leverage/capabilities
Developers can build these "MCP Servers" once - and support every LLM product
Organizations can centralize and manage all the LLMs accessing a given capability or dataset
Users have far more powerful tools with very little setup

Pre-Requisites

Install the python dependency manager

curl -LsSf https://astral.sh/uv/install.sh | sh

Or for MacOS brew install uv

background: https://docs.astral.sh/uv/

Create a default project from a built in template with uv:

uv init mcp-server-demo

This creates a very minimal package (dependency) management configuration file - https://packaging.python.org/en/latest/guides/writing-pyproject-toml/

[project]
name = "mcp-server-demo"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = []

you can ignore or remove the extraneous .gitignore, .git, README.md files

Remove (delete) main.py as it is extraneous for this project.

Next create a python virtual environment and activate it, and install dependencies: - https://docs.python.org/3/library/venv.html

cd mcp-server-demo

uv venv

source .venv/bin/activate

uv add "mcp[cli]" httpx

The code

Use a coding editor to create weather.py that queries the national weather service and provides the MCP hook:

from typing import Any
import httpx
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("weather")

NWS_API_BASE = "https://api.weather.gov"
USER_AGENT = "weather-app/1.0"

async def make_nws_request(url: str) -> dict[str, Any] | None:
headers = {
    "User-Agent": USER_AGENT,
    "Accept": "application/geo+json"
}
async with httpx.AsyncClient() as client:
    try:
        response = await client.get(url, headers=headers, timeout=30.0)
        response.raise_for_status()
        return response.json()
    except Exception:
        return None

@mcp.tool()
async def get_forecast(latitude: float, longitude: float) -> str:
    # First get the forecast grid endpoint
    points_url = f"{NWS_API_BASE}/points/{latitude},{longitude}"
    points_data = await make_nws_request(points_url)

    if not points_data:
        return "Unable to fetch forecast data for this location."

    # Get the forecast URL from the points response
    forecast_url = points_data["properties"]["forecast"]
    forecast_data = await make_nws_request(forecast_url)

    if not forecast_data:
        return "Unable to fetch detailed forecast."

    # Format the periods into a readable forecast
    periods = forecast_data["properties"]["periods"]
    forecasts = []
    for period in periods[:5]:  # Only show next 5 periods
        forecast = f"""
{period['name']}:
Temperature: {period['temperature']}°{period['temperatureUnit']}
Wind: {period['windSpeed']} {period['windDirection']}
Forecast: {period['detailedForecast']}
"""
        forecasts.append(forecast)

    return "\n---\n".join(forecasts)


def main():
    # Initialize and run the server
    mcp.run(transport='stdio')

if __name__ == "__main__":
    main()

The second function in the file specifically creates the "MCP Tool"

The end of the file is the classic python main to have this run as a server

one last config pyproject.toml

Also, modify pyproject.toml to ensure it connects the weather.py and main:

[project.scripts]

weather = "weather:main"

example of the final pyproject.toml file

[project]
name = "mcp-server-demo"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
    "httpx>=0.28.1",
    "mcp[cli]>=1.17.0",
]

[project.scripts]
weather = "weather:main"

Configure Claude to use your new Weather MCP

The following tells your LLM application (i.e. Claude for Desktop)

There is an MCP server named “weather”
launch it by running uv --directory /ABSOLUTE/PATH/TO/PARENT/FOLDER/weather run weather.py

Modify to add the JSON to the file ~/Library/Application\ Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "weather": {
      "command": "uv",
      "args": [
        "--directory",
        "/ABSOLUTE/PATH/TO/PARENT/FOLDER/weather",
        "run",
        "weather.py"
      ]
    }
  }
}

Restart your LLM application (claude for desktkop)

You should be able to see it in the UI: Settings -> Developer

"your new MCP server"

A better response about the weather

(When the LLM application "aka MCP host" starts, it also launches your MCP server as a subprocess)

You (the user) have your LLM running locally and ask:

What's the weather in San Francisco?

The LLM decides it can use the "weather" tool to help answer the question

It actually prompts at first to to approve the permissions:

Claude wants to use Get Forecast from weather

The LLM has geographic data in its training and can use it's "super distilled storage" to convert a city name to coordinates

Request
{
    `latitude`: 37.7749,
    `longitude`: -122.4194
}

The LLM (MCP Host) is the client which communicates via JSON-RPC over stdin/stdout with the "MCP weather server"
The (local) MCP weather server makes HTTP requests to the National Weather Service API
The forecast data (JSON) is returned from the MCP Server to the "MCP Host"

The LLM (MCP Host) incorporates and reformats the JSON into a natural language response

https://modelcontextprotocol.io/docs/develop/build-server

https://github.com/modelcontextprotocol

Last thoughts

Adding tools to an LLM further extends the leverage already available with amazing default NLP that has fact sets and token generation.

Capabilities like "current weather" can be added in a straightforward manner.

MCP unlocks current data, private data, specialized tools, or even running a machine in the physical world - bridging the many fragmented and distinct APIs and interfaces

Bonus Content

You can directly see what the National Weather Service API returns

curl https://api.weather.gov/points/37.780,-122.420

interestingly lat,lon are converted into "grids" which each local forecast office uses, so the JSON returns "Daly City" or "Sausalito"

https://forecast.weather.gov/MapClick.php?lon=-122.4192&lat=37.7793

Send JSON RPC in bash

uv run weather.py << 'EOF'

heredoc>

{"jsonrpc": "2.0", "method": "initialize", "params": {"protocolVersion": "2024-11-05", "capabilities": {}, "clientInfo": {"name": "test", "version": "1.0"}}, "id": 1}

EOF

A response

{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2024-11-05","capabilities":{"experimental":{},"prompts":{"listChanged":false},"resources":{"subscribe":false,"listChanged":false},"tools":{"listChanged":false}},"serverInfo":{"name":"weather","version":"1.17.0"}}}

Maximum leverage and Minimum Ops with Google Cloud Run and the Jules Coding Agent

2025-06-07T06:07:00-07:00

Why Software Agents are Eating the World

Software scales to infinity and prints money the whole way.

Once the cost of distribution went to zero the dominant constraint on growth stopped being infrastructure and became people: how fast could humans design, write, deploy (and maintain!) software.

Over 30+ years, to get increasing leverage (less humans!) and take advantage of economic specialization...

Self-hosted physical servers became leasing space and energy/cooling with colocation
"Colos" physical servers became leasing virtual machines
Overprovisioned VMs became renting "cloud servers"
Cloud services became pay by the millisecond "serverless"

A remaining bottleneck: humans to write and maintain the code.

LLM Coding Agents is "software that writes software"

Personal Goal of Efficiency

Physical wires and failing components, maintaining operating systems (security!), copying files and manually checking if the software even works at all... and even yak shaving down the rabbit hole of code syntax, dependency updates, edge cases, and content typos

Over the years I have leveraged increasing levels of abstractions for provisioning the Environment and Infrastructure (IaaC), and for publishing Content/Code (git to production), all to increase cognitive focus and impact.

see my previous articles on Vagrant, Packer, Terraform, Docker, Static websites, Heroku, AWS, Google App Engine, and AWS Lambdas

Try it for yourself: tell an LLM what you want and asynchronously it comes back (like any remote virtual colleague) with the changes - approve and voila it's live in seconds.

Another day another cloud

The goal is to give the LLM agent Jules instructions in english and go from git to production in seconds.

Previously I have used Google App Engine but it seems to have been superseded by Google Cloud Run (and hopefully less byzantine IAM permissions and "Artifact Registry" costs)

Google Cloud Run "free tier" is still confusing to estimate and calculate (charged by the millisecond) but should "just run".

So here's how to setup a golang web server automatically built and deployed from each GitHub commit.

Note: apparently Cloud Run also uses Artifact Registry to automatically leverage a Docker image that presumably if deployed enough times could go beyond the free tier :(

Different day same Code

If the web server starts automatically on port 8080 then it is "Cloud Run Compatible"

https://github.com/johnpfeiffer/aws-go-lambda

Their official example https://github.com/GoogleCloudPlatform/cloud-run-microservice-template-go

Google is project-centric

Then navigate to a project (or make a new one)

https://console.cloud.google.com/run/overview?project=go-cache

"Deploy a web service" -> Connect repository

Configuration

GitHub: "Continuously deploy from a repository (source or function)" (this works for languages supported by Google)

Cloud Build: Can deploy from GitHub repositories.

"Set up with Cloud Build" (button)

Source repository: GitHub

Currently not authenticated, Authenticate

You will be asked to authorize the Google Cloud Build GitHub App to access your GitHub Account to proceed.
- creates a pop up (check your blockers!): "Authorize Google Cloud Build"

Repository: Manage connected repositories

creates a pop up to GitHub (simplified integration and authentication/authorization)
- Only select repositories (choose from the dropdown)
- - legalese checkbox

Build Configuration

Branch: ^main$

Build Type: Go (or Node.js, Python, Java, .NET, Ruby, PHP)

nothing special for Build context, Entrypoint, or Function target (The name of the exported function to be invoked, leave blank if the source repository is a web server.)

Service name: go-cache

Region: us-west2 (tier2 is cheaper than tier1)

Authentication: Public (unless you have something specific in mind)

Billing: Request-based, Charged only when processing requests. CPU is limited outside of requests.

Service scaling: Minimum number of instances = 0 , Maximum number of instances = 1

very important to set these values to 0 and 1 to stay in the free tier

"Edit Container" has the opportunity to change things or add a variable/secret - ignore for now

Create (button)

https://docs.cloud.google.com/run/docs/continuous-deployment#existing-service

Checking on Builds

When builds are occurring (or failing)

https://console.cloud.google.com/cloud-build/

Cleanup Artifact Registry

You can search and find amid the 100s of services the one that sometimes invisibly goes over a free tier...

https://console.cloud.google.com/artifacts/

Click on "cloud-run-source-deploy" and click in until you get to something like "Digests for aws-go-lambda/go-cache"

Choose an old deploy as a checkbox and then at the top click Delete

Are you sure you want to delete...

hopefully this keeps me under the free tier limit

Agentic Coding with Google Jules

Google is making their LLM offerings like Gemini and Jules free up to a certain limit - an excellent opportunity to experiment. caveat emptor that this may change over time

Google "Gemini" (https://gemini.google.com) is the LLM chat application, useful for questions about technology terms, architecture options, code snippets, and troubleshooting, etc.

...you have encountered the very real friction of Google Cloud's IAM system firsthand. What should feel like a single, simple action often requires a complex combination of permissions... the single App Engine Deployer role is not sufficient for a complete developer workflow, and it does not cover administrative tasks or broader access needs

Getting clarity in your own thinking is important before handing work off to Jules.

"Agentic Coding" is where the LLM agent takes instructions in english and modifies the code.

https://jules.google.com/

Google account as before, Sign in, accept all the terms and conditions of this experimental AI software

Has a purple octopus logo

Choose a Repo from the dropdown (top center) -> "Configure repo access"

opens a new tab (window?) in GitHub for "Google Labs Jules"
Repository access: Only select repositories (choose the ones the agent can work on)

"Save" (button)

Successfully authenticated with GitHub!

https://github.com/settings/installations is where GitHub lists integrated Applications

Choose the repo (i.e. johnpfeiffer/aws-go-lambda)

Give Jules the instructions in the text box (not on the bottom right you can choose a mode like "interactive plan" or "review" - or just "Start")

These tasks run in an isolated sandbox hosted by Google

Watching an agentic task in progress

On the left (expand/collapse) side panel you can see how many "daily sessions" you have left.

Clicking on a session gives you all the details:

how the agent broke the instructions down into a plan
it runs go test -v on its own
sends a (draft) pull request to the code repository

there is a "View PR" button

https://github.com/johnpfeiffer/aws-go-lambda/pull/1

Iterating on the Pull Request

Leaving a comment on the pull request will feedback to Jules (yay for all the permissions) who then automatically resumes the session.

I have received the PR comments and am processing them.

Ready for review

the usual GitHub pull request merge process

Gotchas with Agents

All of this leverage shifts the bottleneck from "programmer" to "good ideas and ability to review changes". Yet this new technology brings new challenges...

the LLM sounds confident even when wrong or has mistakes
often imports random libraries or calls external APIs
deletes tests
modifies code and says "done"... but the code does not work, is not done
mediocre code quality that is either one large ball of mud or a tangle of tiny functions
unnecessarily over-engineered code and abstractions

Mitigations

read and review critically
have specifications, documentation, and anchors
tell it build as simple as possible and not to call external APIs
give it boundary instructions: that it cannot modify or delete tests
give it examples of expected input and output
give it explicit instructions on the level of modularity ("make this 3 functions")

What does it all mean

Even with software writing software there is actually an increasing need for human intent, and managing of complexity (including ensuring generated code is readable, modular, testable, etc.)

Building a desktop app with Golang and Wails

2025-04-06T10:10:00-07:00

tldr: a Golang native desktop app using Wails (Go backend + React frontend = single binary)

Why this tech stack and why a desktop app?

Desktop apps provide privacy for the persistence layer, and bypass the "where is it hosted" challenges.

This is a simple and practical way to leverage some great backend + frontend technologies.

If you need a more basic react intro see my previous post https://blog.john-pfeiffer.com/react-javascript-intro/

The key: Go methods with exported (capitalized) names get auto generated as JavaScript/TypeScript bindings. You call Go functions from React as if they were local async functions.

Why Wails

Wails uses the operating system's native webview (WebKit on macOS, WebView2 on Windows, WebKitGTK on Linux) so the resulting binary is ~10MB.

As a comparison to a very popular technology, electron bundles an entire Chromium browser into your app - which is why a "hello world" Electron app is ~150MB and idles at 100MB+ of RAM.

┌───────────────────────────────────────────────────────────┐
│  WAILS APPLICATION                                        │
│                                                           │
│   Go Backend              React Frontend                  │
│   ┌───────────┐           ┌──────────────────┐            │
│   │ app.go    │◄─────────►│ App.tsx          │            │
│   │ (structs  │  bindings │ (components,     │            │
│   │  & methods│  ───────► │  hooks, UI)      │            │
│   │  & stdlib)│           │                  │            │
│   └───────────┘           └──────────────────┘            │
│         │                        │                        │
│         └──────────┬──────────────┘                         │
│                    ▼                                        │
│         ┌────────────────────┐                             │
│         │  Native WebView    │                             │
│         │  (WebKit on macOS) │                              │
│         └────────────────────┘                              │
│                    │                                        │
│                    ▼                                        │
│            Single Binary (~10MB)                            │
└─────────────────────────────────────────────────────────────┘

Why golang

Golang is a remarkably performant language but it still automatically handles memory management (with garbage collection)

Golang builds to a single binary

Pre-requisites

the assumption here is MacOS...

Golang: https://go.dev/doc/install

brew install golang
which go; go version
echo 'export PATH="$PATH:$(go env GOPATH)/bin"' >> ~/.zshrc

because "GOPATH/bin" is the default destination for go install - now making tools runnable

NPM: https://nodejs.org/en/download/ - brew install node - which npm; npm --version; node --version

Installing Wails

Wails framework: https://wails.io/docs/gettingstarted/installation/

go install github.com/wailsapp/wails/v2/cmd/wails@latest

ls $(go env GOPATH)/bin

wails version

Troubleshooting Wails

wails doctor

This checks Go version, node/npm, and platform-specific dependencies (on macOS you need Xcode command line tools)

Simplest start with Wails

wails init -l

list the different types of wails default project layouts

Plain HTML/JS/CSS              plain
React + Vite                   react
React + Vite (Typescript)      react-ts
Svelte + Vite                  svelte
Vanilla + Vite                 vanilla
Vue + Vite                     vue

for security reasons it is simplest to avoid 3rd party templates unless you've vetted them thoroughly

I'll use "React + Vite (Typescript)": wails init -n myapp -t react-ts

Run the interactive developer view of the Application (it's responsive to rebuilding for changes): wails dev

Executing: go mod tidy
  • Generating bindings: Done.
  • Installing frontend dependencies: Done.
  • Compiling frontend: Done.
  ...

In about 10 seconds you have your example native desktop app running locally.

Privacy checks in the configuration files

wails.json contains the application name - and your email address, so do not send that to a git public repo unless you're ready for it

Also, the name "myapp" will show up in a bunch of places so if you change/update the name, look for

grep -r 'chat-explorer' .
    ./go.mod
    ./wails.json
    ./frontend/index.html
    ./app.go
    /main.go

Explaining the File Structure

myapp/
├── build/
│   ├── appicon.png
│   ├── darwin/
│   │   ├── Info.plist
│   │   └── Info.dev.plist
│   └── windows/
├── frontend/
│   ├── src/
│   │   ├── App.tsx
│   │   ├── main.tsx
│   │   ├── App.css
│   │   └── style.css
│   ├── wailsjs/
│   │   ├── go/main/App.ts    <- auto-generated bindings
│   │   └── runtime/
│   ├── index.html
│   ├── package.json
│   └── tsconfig.json
├── app.go                    <- your Go application logic
├── main.go                   <- entrypoint
├── go.mod
├── go.sum
└── wails.json

There are two halves to understand: the Go backend and the React frontend.

The Golang Backend

main.go is the entrypoint for starting the application

package main

import (
    "embed"
    "github.com/wailsapp/wails/v2"
    "github.com/wailsapp/wails/v2/pkg/options"
    "github.com/wailsapp/wails/v2/pkg/options/assetserver"
)

//go:embed all:frontend/dist
var assets embed.FS

func main() {
    app := NewApp()

    err := wails.Run(&options.App{
        Title:  "myapp",
        Width:  1024,
        Height: 768,
        AssetServer: &assetserver.Options{
            Assets: assets,
        },
        BackgroundColour: &options.RGBA{R: 27, G: 38, B: 54, A: 1},
        OnStartup:        app.startup,
        Bind: []interface{}{
            app,
        },
    })
    if err != nil {
        println("Error:", err.Error())
    }
}

//go:embed all:frontend/dist is a directive doing something clever: embedding the entire compiled frontend into the Go binary at build time, no separate files to distribute!

Bind: []interface{}{ app, is the connection between "Go Exported Methods" and the Javascript/React

app.go is the place for application logic (usually just high level, using /models or other places to get all of the details)

package main

import "context"

type App struct {
    ctx context.Context
}

func NewApp() *App {
    return &App{}
}

func (a *App) startup(ctx context.Context) {
    a.ctx = ctx
}

func (a *App) Greet(name string) string {
    return "Hello " + name + ", welcome to Wails!"
}

The magic: every exported method on a bound struct (in this case "App") automatically gets a TypeScript binding generated in frontend/wailsjs/go/main/App.ts

The React Frontend

Your usual frontend/index.html has <script src="./src/main.tsx" type="module"></script>

frontend/src/main.tsx is the entrypoint for starting the application - nothing surprising here:

import React from 'react'
import {createRoot} from 'react-dom/client'
import './style.css'
import App from './App'

const container = document.getElementById('root')

const root = createRoot(container!)

root.render(
    <React.StrictMode>
        <App/>
    </React.StrictMode>
)

frontend/src/App.tsx is the bridge to the backend Golang Greet function:

import { useState } from 'react';
import { Greet } from '../wailsjs/go/main/App';

function App() {
    const [name, setName] = useState('');
    const [result, setResult] = useState('');

    const greet = () => Greet(name).then(setResult);

    return (
        <div>
            <input onChange={(e) => setName(e.target.value)} />
            <button onClick={greet}>Greet</button>
            <p>{result}</p>
        </div>
    );
}

export default App;

Note that the Greet function is a Promise, the call to the backend is async

Now explore adding your own function that returns a map like func (a *App) GetCurrentTime() map[string]string {

In the frontend you'd use it with

import { GetCurrentTime } from '../wailsjs/go/main/App';

// later in a component...
const [info, setInfo] = useState<Record<string, string>>({});
useEffect(() => {
    GetCurrentTime().then(setInfo);
}, []);

Build and Distribute

wails build

There are more instructions you'd need to follow about all the details of Windows, MacOS, etc.

Conclusion

These are well known concepts of frontend (react) and backend (golang) so you can focus on your domain problems and features.

Using these standard technologies also makes it very easy to leverage AI/LLMs to write the code

Stable building blocks and not re-inventing the wheel allows you to ship faster (and deliver value!)

Cars not helicopters, or running a local LLM with MLX on a Macbook Pro

2024-11-23T19:21:00-08:00

Everything looks like a nail

For Large Language Models (LLMs), Frontier aka "State of the Art" (SOTA) Models provided by big vendors like OpenAI and Anthropic and Google continue to add capabilities at a wondrous rate. Yet there's a very strong case for running LLMs locally, much like every smart phone provides a powerful computer your pocket.

As an example: helicopters can cover in 15 minutes what takes 2 hours by car, but the overhead and extra considerations (fuel, pilots, landings, weather sensitivity, etc.) makes them impractical for routine tasks like picking up kids from school.

(Though those new autonomous drones can carry pretty heavy loads ;)

Given English has ~170,000 words but the average working vocabulary is 30,000 words, and specific domains like "email" (Hello! Best Regards,) are even more narrow and formulaic... Do you need a model trained on the entirety of human knowledge with latency from expensive GPU clusters?

Writing code is specialized due to its utilitarian nature; it has even more structure, repetition, and rules - especially if it's type-checked and compiled.

How they work together

In software, experienced engineers propose and design the architecture, whereas less experienced people do the (simpler parts) of implementation - specialization based on skill and scope.

Frontier models can design and orchestrate, handle the unusual, while local models do specific, simple things well.

Software engineers have learned to have a "plan" or "write a spec" phase specifically created with an advanced LLM, and that a local LLM can write out the code and tests for small well defined components.

Inevitable and Resilient

In 2020 an LLM was a research project and by 2024 it's running on your laptop (or smartphone!). Hardware gets better and costs go down which induces "Jevons Paradox" https://en.wikipedia.org/wiki/Jevons_paradox ; as people adapt they'll use LLM inference non-stop.

Moreover, users (and businesses!) dependent to always having AI will balk at "cloud outages" or "AI outages" - thus a real need and demand for local LLMs.

(Have you ever seen people struggling to function without mobile phone signal/reception ;)

And I didn't even bring up the privacy and security arguments...

We don't use Helicoptors for everything; use the right tool for the job.

Hands On with MLX - install

Apple's silicon architecture of "unified memory" is convenient for those trying out "Local LLMs" - and not buying a separate dedicated server with a GPU.

https://www.apple.com/newsroom/2023/10/apple-unveils-m3-m3-pro-and-m3-max-the-most-advanced-chips-for-a-personal-computer/

brew install uv
uv --version
mkdir llm-demo
cd llm-demo

uv venv

## this installs mlx as it's a dependency of mlx-lm
uv pip install mlx-lm

## optional sanity checks - using explicit calls with the local uv
uv pip list
uv run python -c "import mlx; import mlx.core as mx; print('mlx ok', mx.__version__)"
uv run python -c "import mlx_lm; print('mlx_lm ok')"

Run that LLM with MLX

The following command will both download the model and then load it into memory along with sending it the prompt:

uv run mlx_lm.generate --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit --prompt "tell me a joke"

A man walked into a library and asked the librarian, "Do you have any books on Pavlov's dogs and Schrödinger's cat?" The librarian replied, "It rings a bell, but I'm not sure if it's here or not."

Prompt: 39 tokens, 75.536 tokens-per-sec Generation: 54 tokens, 31.247 tokens-per-sec Peak memory: 4.638 GB

For LLMs a "token" is a part of word - and this output rate of generating tokens is plenty fast enough to not wait while each word of the joke is printed slowly.

Under the hood, let's examine where the "open weight" downloaded model is:

du -sh ~/.cache/huggingface/hub/models--mlx-community--Meta-Llama-3.1-8B-Instruct-4bit

4.2G models--mlx-community--Meta-Llama-3.1-8B-Instruct-4bit

Next Level is a Python Script

Create the wrapper script "llm-demo/mychat.py"

import sys
from mlx_lm import load, generate

if len(sys.argv) < 2:
    print("Usage: python mychat.py 'your prompt here'")
    sys.exit(1)

model, tokenizer = load("mlx-community/Meta-Llama-3.1-8B-Instruct-4bit")
prompt = " ".join(sys.argv[1:])
response = generate(model, tokenizer, prompt=prompt, max_tokens=1000)
print(response)

assuming you are re-using the "llm-demo" directory and all the pre-requisite UV and venv setup

To create the pyproject.toml and ensure the mlx-lm dependency is added run the following:

uv init --bare
cat pyproject.toml
uv add mlx-lm
cat pyproject.toml

Now leveraging the python environment to "just run python" rather than calling UV for everything...

source .venv/bin/activate

python mychat.py "tell me a joke"

You may notice it ran on telling multiple jokes and also abruptly terminated at the end...

Improving the output with a formatted prompt

The following code changes formats the prompt the way the instruction-tuned model expects, returning a more natural

import sys
from mlx_lm import load, generate

if len(sys.argv) < 2:
    print("Usage: python mychat.py 'your prompt here'")
    sys.exit(1)

model, tokenizer = load("mlx-community/Meta-Llama-3.1-8B-Instruct-4bit")
user_prompt = " ".join(sys.argv[1:])

# Provide the role and chat template format
messages = [{"role": "user", "content": user_prompt}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

response = generate(model, tokenizer, prompt=prompt, max_tokens=1000, verbose=False)
print(response)

A much better joke =)

https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py

RAM usage during Inference

When the model is loaded into memory, the most dynamic part of what changes is the "context" - everything provided in the Prompt, and the output.

When there's too much context going in or output coming then the Local LLM can consume all of the available RAM.

Open "activity monitor" and choose "Memory"

Run the following to force more context into the "KV cache" and observe the "Python" application memory slowly creep upward

uv run mlx_lm.generate --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
    --prompt "Write a detailed 5000-word essay on the history of computing" \
    --max-tokens 4000

Simulation of memory in a conversation

Each call to the LLM is stateless, so a wrapper application keeps track of every back and forth interaction during a "conversation".

This allows the LLM to reference things "earlier" in the conversation - all of which is passed in as context for the new Prompt.

Ergo longer conversations will take up more RAM.

import sys
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Meta-Llama-3.1-8B-Instruct-4bit")
system_message = ("You are a helpful, concise Assistant")
messages = [{"role": "system", "content": system_message},]
print("MLX Chat. Type 'exit' or Ctrl+C to quit.\n")
while True:
    try:
        user_input = input("You: ").strip()
        if user_input.lower() in {"exit", "quit"}:
            break
        messages.append({"role": "user", "content": user_input})
        prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
        response = generate(model, tokenizer, prompt=prompt, max_tokens=1000)
        print(f"\nAssistant: {response}\n")
        messages.append({"role": "Assistant", "content": response})
    except KeyboardInterrupt:
        print("\nExiting.")
        break

OpenAI compatible HTTP Server

uv run mlx_lm.server --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit --host 127.0.0.1 --port 8080

curl 127.0.0.1:8080/v1/models

Troubleshooting

Where is that installed?

A python uv gotcha - do not move (or copy) the .venv directory since it includes absolute folder paths - instead use the uv commands for each new project.

UV is awesomely fast - one reason is it uses a cache, here's how to audit how many python versions UV installed/knows:

ls -ahl ~/.local/share/uv/python/

And in case you installed uv with both "install.sh" and homebrew...

which -a uv

/opt/homebrew/bin/uv
~/.local/bin/uv

Cleanup

To find previously downloaded locl models, which are large files

find ~ -type f -size +1G 2>/dev/null
ls -ahl ~/.cache/huggingface/
ls -ahl ~/.cache/huggingface/hub
rm -rf ~/.cache/huggingface/hub/models--mlx-community--Meta-Llama-3...

Or maybe you want to clean up a global pip install of MLX

pip3 list
brew uninstall mlx

What Next?

The right tool for the job: Consider how you leverage and architect this new technology.

Cars may not be as exciting as vertical take off and landing - but maybe solutions don't always have to be exciting.

There is a lot of value in a sub 10ms answer that's practically free.

References

https://github.com/ml-explore/mlx

Upcoming post

Llama 3.1 is a good general model, but it was not created to focus on coding uv run mlx_lm.generate --model mlx-community/Qwen2.5.1-Coder-7B-Instruct-8bit --prompt ""

Addendum:

https://machinelearning.apple.com/research/exploring-llms-mlx-m5

React with Material-UI and Google SSO

2024-09-07T12:34:00-07:00

tldr: React + Vite + MUI + simple Google sign-in for rapid building

React is almost a defacto standard for responsive frontend applications, so it can be helpful to layer on a few more "standard choices" to simplify building apps for production.

if you need a more basic react intro see my previous post https://blog.john-pfeiffer.com/react-javascript-intro/

History of Style

As HTML took over the world, the need became obvious for separating out the "styling" of the presentation and content. Thus CSS was created, though it took awhile to become a fully adopted and usable standard (due to interests that would have preferred a private single browser definition and control).

https://en.wikipedia.org/wiki/CSS

ascii diagram:

┌─────────────────────────────────────────────────────────────┐
│  SOURCE CODE FILES (what you write)                         │
│                                                             │
│   index.html          style.css                             │
│   ┌───────────┐       ┌──────────────────┐                  │
│   │ <h1>      │       │ h1 {             │                  │
│   │ <p>       │       │   color: blue;   │                  │
│   │ <button>  │       │ }                │                  │
│   └───────────┘       └──────────────────┘                  │
│         │                     │                             │
│         └──────────┬──────────┘                             │
│                    ▼                                        │
│              ┌───────────┐                                  │
│              │  Browser  │                                  │
│              └─────┬─────┘                                  │
│                    ▼                                        │
│   ┌─────────────────────────────────────────────────────┐   │
│   │  DOM (Document Object Model) - what the browser     │   │
│   │  builds in memory, combining structure + style      │   │
│   │                                                     │   │
│   │  document                                           │   │
│   │    └── html                                         │   │
│   │          └── body                                   │   │
│   │                ├── h1 (color: blue)                 │   │
│   │                ├── p                                │   │
│   │                └── button                           │   │
│   └─────────────────────────────────────────────────────┘   │
│                    │                                        │
│                    ▼                                        │
│            What you see on screen                           │
└─────────────────────────────────────────────────────────────┘

As an application grows, or if you are not keen on customizing every detail of the style, it can become overwhelming managing a lot of CSS files (and which setting overrides which thing).

Material UI for Styling

Material UI (MUI) provides pre-built components that look decent out of the box and handle responsive design, accessibility, and theming.

It is a framework that generates both HTML and CSS for you - so it is simple, but opinionated.

To add Material UI dependencies to an existing React project:

npm install @mui/material @emotion/react @emotion/styled

https://mui.com/material-ui/getting-started/

example package.json

    {
      "name": "material-ui-example",
      "private": true,
      "version": "1.0.0",
      "type": "module",
      "scripts": {
        "dev": "vite",
        "build": "vite build",
        "preview": "vite preview"
      },
      "dependencies": {
        "@emotion/react": "^11.14.0",
        "@emotion/styled": "^11.14.1",
        "@mui/material": "^7.3.6",
        "react": "^18.2.0",
        "react-dom": "^18.2.0",
        "react-router-dom": "^6.20.0"
      },
      "devDependencies": {
        "@types/react": "^18.2.0",
        "@types/react-dom": "^18.2.0",
        "@vitejs/plugin-react": "^4.2.0",
        "vite": "^7.1.3"
      }
    }

A simple React and MUI code example

App.jsx

import { useState } from 'react';
import { Button, Box, Typography } from '@mui/material';

const content = [
  ["hello world", "more content on line 2"],
  ["github.com/mui has more info"]
];

function App() {
    const [activeContentIndex, setActiveContentIndex] = useState(0);

    return (
        <Box sx={{ p: 2 }}>
            <Box sx={{ mb: 2 }}>
                <Button onClick={() => setActiveContentIndex(0)}>View 1</Button>
                <Button onClick={() => setActiveContentIndex(1)}>View 2</Button>
            </Box>
            <div>
                {content[activeContentIndex].map((line, i) => (
                    <Typography key={i} paragraph>{line}</Typography>
                ))}
            </div>
        </Box>
    );
}

export default App;

Box is a Material-UI generic wrapper: a <div> that accepts the sx property
sx is a Material-UI shorthand for a property
- p = padding, all sides, 2 x 8px = 16px
- m = margin-bottom, 2 x 8px = 16px
Button has Material Design ripple effect, hover states, and theming
Typography applies Material Design font family, sizes, and spacing and renders as <p> by default

There is no longer App.css and index.css since MUI is handling all the styling.

Thus your React framework code is now React + Material-UI, effectively a "Domain Specific Language" that quickly builds and renders frontend apps.

Google Single Sign On

There are quite a few options for security, and identity (followed by authentication and authorization), and if you attempt to write it all yourself you can inadvertantly create a security issue.

One of the most popular Identity Providers to leverage is Google (due to gmail and youtube and large consumer reach).

For simple apps where your users will already have a Google Account, here is an easy way to have them verify their identity with Google and then share their email address with your application.

This allows for a "frontend only" app to have personalization, and is an important pre-requisite when added to a backend.

Create a Google OAuth Client

For an existing (or new) Google Cloud Project, start the flow with the OAuth Consent

Requests user consent so your app can access the user's data

Create https://console.cloud.google.com/auth/overview/create
fill out the questionnaire about the Application and intended use (starting with Internal or Test users is simplest)
fill in contact email address, and "terms and conditions"
"OAuth configuration created"
Create OAuth client (button)
Application type: web application
Add http://localhost:5173 to Authorized JavaScript origins (for local dev)
if you know it already you can add the domain where your production React App runs (i.e. https://yourusername.github.io/)
Create (button)
Copy the Client ID - this is what you put in your environment variables or code

i.e. really-long-random-numbers-and-characters.apps.googleusercontent.com

Note that this client ID is a public facing configuration value (ends up in JS code visible in the browser dev tools)

Environment Variable

Using an Environment variable allows setting a different value for Development, Staging, and Production.

Create the file .env in your project root with the following in it:

VITE_GOOGLE_CLIENT_ID=your-client-id-here.apps.googleusercontent.com

(Also make sure you add .env to your .gitignore ;)

Make sure you also set the value in your build process (or via integration with a hosting provider like AWS Amplify or Cloudflare ) that creates the final bundle

Load the Google Identity Services Script

index.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>My SSO App</title>
  <script src="https://accounts.google.com/gsi/client" async defer></script>
</head>
<body>
  <noscript>You need to enable JavaScript to run this app.</noscript>
  <div id="root"></div>
  <script type="module" src="/src/main.jsx"></script>
</body>
</html>

GoogleSignIn.jsx

import { useEffect, useRef, useState } from "react";

export default function GoogleSignIn({ onAuth }) {
  const [gsiReady, setGsiReady] = useState(false);
  const [authStatus, setAuthStatus] = useState(null);
  const googleButtonRef = useRef(null);
  const renderedRef = useRef(false);

  useEffect(() => {
    const clientId = import.meta.env.VITE_GOOGLE_CLIENT_ID;
    if (!clientId) {
      setAuthStatus({ ok: false, message: "Missing VITE_GOOGLE_CLIENT_ID" });
      return;
    }

    const handleCredentialResponse = (response) => {
      setAuthStatus({ ok: true, token: response.credential });
      if (onAuth) onAuth(response.credential);
    };

    const tryRender = () => {
      const g = window.google;
      if (!g?.accounts?.id || !googleButtonRef.current) {
        requestAnimationFrame(tryRender);
        return;
      }

      setGsiReady(true);

      // Prevent double-render (React 18 StrictMode calls useEffect twice)
      if (renderedRef.current) return;
      renderedRef.current = true;

      g.accounts.id.initialize({
        client_id: clientId,
        callback: handleCredentialResponse,
      });

      googleButtonRef.current.innerHTML = "";

      g.accounts.id.renderButton(googleButtonRef.current, {
        theme: "outline",
        size: "large",
        text: "signin_with",
      });
    };

    tryRender();
  }, [onAuth]);

  return (
    <div>
      <div ref={googleButtonRef} />
      {!gsiReady && <div>Loading Google Sign-In...</div>}
      {authStatus?.ok === false && <div style={{color: 'red'}}>{authStatus.message}</div>}
    </div>
  );
}

App.jsx

import { useState } from 'react';
import { Container, Paper, Typography } from '@mui/material';
import GoogleSignIn from './GoogleSignIn';

function App() {
  const [user, setUser] = useState(null);

  const handleAuth = (credential) => {
    // The credential is a JWT - decode the payload to get user info
    const payload = JSON.parse(atob(credential.split('.')[1]));
    setUser(payload);
  };

  return (
    <Container maxWidth="sm" sx={{ mt: 4 }}>
      <Paper elevation={2} sx={{ p: 3 }}>
        {user ? (
          <div>
            <Typography variant="h6">Welcome, {user.name}</Typography>
            <Typography color="text.secondary">{user.email}</Typography>
          </div>
        ) : (
          <GoogleSignIn onAuth={handleAuth} />
        )}
      </Paper>
    </Container>
  );
}

export default App;

The JWT payload from Google includes: email, name, picture, sub (unique user ID), and expiration info.

Listing all the files and a diagram

Final folder structure:

my-app/
    .env                 (VITE_GOOGLE_CLIENT_ID=...)
    .gitignore           (include .env)
    index.html           (includes gsi script)
    package.json
    src/
        main.jsx
        App.jsx
        GoogleSignIn.jsx

For deployment, remember to add your production domain to the Google OAuth credentials Authorized JavaScript origins.

┌─────────────────────────────────────────────────────────────────────┐
│  GOOGLE SSO (Client-Side Only)                                      │
│                                                                     │
│                                                                     │
│   ┌──────────┐                              ┌──────────────────┐    │
│   │  Browser │                              │  Google Servers  │    │
│   └────┬─────┘                              └────────┬─────────┘    │
│        │                                             │              │
│        │  1. Page loads GSI script                   │              │
│        │────────────────────────────────────────────>│              │
│        │<────────────────────────────────────────────│              │
│        │                                             │              │
│        │  2. User clicks "Sign in with Google"       │              │
│        │────────────────────────────────────────────>│              │
│        │                                             │              │
│        │         ┌─────────────────────┐             │              │
│        │         │  Google login popup │             │              │
│        │         │  - enter email      │             │              │
│        │         │  - enter password   │             │              │
│        │         │  - consent screen   │             │              │
│        │         └─────────────────────┘             │              │
│        │                                             │              │
│        │  3. Google returns JWT credential           │              │
│        │<────────────────────────────────────────────│              │
│        │                                             │              │
│        │                                                            │
│        ▼                                                            │
│   ┌─────────────────────────────────────────┐                       │
│   │  JWT Payload (decoded in browser by javascript)                 │
│   │                                         │                       │
│   │  {                                      │                       │
│   │    "sub": "1234567890",  ← unique ID    │                       │
│   │    "email": "user@gmail.com",           │                       │
│   │    "name": "John Smith",                │                       │
│   │    "picture": "https://...",            │                       │
│   │    "exp": 1234567890                    │                       │
│   │  }                                      │                       │
│   └─────────────────────────────────────────┘                       │
│                                                                     │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Important Caveats

This all depends on trusting the javascript provided by google and their servers - they are responsible for the complexities of passwords, 2-factor-authentication, etc.

This post uses Google Identity Services to get an ID token directly in the browser (no redirect URI required).

If you use "OAuth authorization code flow", you must configure redirect URIs and handle the code exchange on a backend.

Importantly: if you have any backend/API you should send the Google ID token to the backend and have code to truly verify it (signature + issuer + audience + expiry).

Bonus Backend Golang Code

package main

import (
    "context"
    "encoding/json"
    "log"
    "net/http"

    "google.golang.org/api/idtoken"
)

type googleAuthReq struct {
    IDToken string `json:"id_token"`
}

func handleGoogleAuth(w http.ResponseWriter, r *http.Request) {
    var req googleAuthReq
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, "Invalid request", http.StatusBadRequest)
        return
    }
    clientID := "your-client-id-here.apps.googleusercontent.com"
    ctx := context.Background()
    payload, err := idtoken.Validate(ctx, req.IDToken, clientID)
    if err != nil {
        http.Error(w, "Invalid ID token", http.StatusUnauthorized)
        return
    }
    // payload.Claims contains "email", "email_verified", "name", etc.
    email, _ := payload.Claims["email"].(string)
    emailVerified, _ := payload.Claims["email_verified"].(bool)
    if email == "" || !emailVerified {
        http.Error(w, "Invalid email or email not verified", http.StatusUnauthorized)
        return
    }
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(payload.Claims)
}

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/{path...}", handleGoogleAuth)
    err := http.ListenAndServe(":8080", mux)
    if err != nil {
        log.Fatalf("failed to start server: %v", err)
    }
}

curl -X POST http://localhost:8080/ -H "Content-Type: application/json" --data '{"id_token":"foobar"}'

React Javascript Intro

2024-08-25T12:34:00-07:00

The intent of this post is to help people build and understand simple reactive web applications

2025 update: due to the never-ending nature of tech breaking things - this post needed an update (goodbye Create React App)

https://react.dev/blog/2025/02/14/sunsetting-create-react-app

Pre-Requisites

install node.js (which also includes npm) from https://nodejs.org/en

check if they are installed:

node -v

npm -v

Install the default react project

in a directory (probably a code repository)

    npm create vite@latest my-app -- --template react
    cd my-app
    npm install
    npm run dev

(obsolete: npx create-react-app my-app)

note that npm run dev starts a web server at http://localhost:5173

Essential folder structure (that is created by the framework with default files)

my-app/

    index.html  (this is the entrypoint, yes it's all just html and http ;)

    src/
        index.css      (this is for styles for the output - if you want things to look good)
        main.jsx       (connects the App to the starting index.html entrypoint)
        App.css
        App.jsx     (this is where you add code)

/public is a directory with image or binary files that are served directly

(Remove the "vitals" phone home stats because privacy should be a default)

INDEX.HTML

The defacto web standard for the first document (for a web server) to return when no resource is specified is "index.html"

https://en.wikipedia.org/wiki/HTML

    <!DOCTYPE html>
    <html lang="en"><body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
        <div id="root"></div>
        <script type="module" src="/src/main.jsx"></script>
    </body></html>

MAIN.JSX

    // this wrapper is connecting the App to the root
    import React from 'react'
    import ReactDOM from 'react-dom/client'
    import './index.css';
    import App from './App.jsx'

    const rootElement = document.getElementById("root");
    const root = ReactDOM.createRoot(rootElement);
    root.render(
      <React.StrictMode>
        <App />
      </React.StrictMode>
    );

APP.JSX

    // very simple app that switches the content displayed when the button is clicked
    import './App.css';
    import { useState } from 'react';

    const content = [
      ["hello world", "more content on line 2"],
      ["react.dev has tutorials and documentation"]
     ];

     function App() {
      const [activeContentIndex, activeContent] = useState(0);

      return (
        <div className="App">
            <header className="App-header">
            </header>
            <div id="tabs">
            <button onClick={() => activeContent(0)}>Tab 1</button>
            <button onClick={() => activeContent(1)}>Tab 2</button>
            </div>
            <div id="tab-content">
                {content[activeContentIndex].map((line, i) => <p key={i}>{line}</p>)}
            </div>
        </div>
    );
    }

    export default App;

A React component is a JavaScript function that returns JSX (looks like HTML but is not HTML)

Build and Deploy

A critical file created by the default app generation is package.json, it lists all your project dependencies and scripts (including "how to build")

The build framework reads your package.json to understand which tools to run, in this case vite compiles/reformats the code into a production bundle of files that can be uploaded to a server

The command to "build", or convert all of these files into a single bundle that is easily deployed:

npm run build

The new output directory dist contains the index.html and static files along with "minified" javascript and css

https://vite.dev/guide/static-deploy.html

Behind the scenes of the challenges for a Hiring Manager

2024-04-29T12:01:00-07:00

Hiring is more than just a single decision

A lot of interviewing advice is for candidates. I wanted to pull back the curtain on some experiences and tips for hiring managers, especially if it's your first time and you just got a headcount. =)

(Hopefully this is also useful if you're training people managers or ICs on the whole process)

Hiring exists to solve a business need. It can take as little as a few weeks, as long as a year or more. Like many management responsibilities, it is more than just a single decision, often it's creating an effective system.

The Hiring Manager is always accountable for the decision to hire, the process of hiring, and the results of hiring (or not hiring)

Backstory on How a Role Opens Up

There is a lot more to the story for a straightforward job posting like "Senior Engineer".

Here's a short list of reasons that a position may have opened up - and how that influences the hiring process:

rapid growth of users/customers
gaps (think reliability, bugs) due to growth
very lean team reaching a breaking point
someone left
someone was asked to leave
a completely new team (or even whole new initiative)
org shuffle
newly discovered gap in critical skills

Growth headcount is a calculated bet: the organization is investing in a direction.

The approval can hinge on this being aligned with larger leadership strategy and initiatives, and who specifically has budget. Opportunity cost can be subjective, and if this hire comes at the expense of another team?

People in the existing team may wonder why someone's getting hired "above them", or if it means more work and distraction doing interviewing and onboarding, or even worry about this being the beginning of them being replaced.

Backfills are the most straightforward: the organization was paying for a certain amount of work to be done and now needs a replacement. While the approval can be straightforward due to previously allocated budget, it can also mean there's prior expectations.

The existing team/colleagues may be concerned: either that the new person won't be able to do as much as their previous colleague, or that they'll have similar or worse flaws as their departed coworker.

The old saying is "Fool me once, shame on you, fool me twice, shame on the Hiring Manager"

Either way, the team is carrying extra load until the new person starts.

Shuffles and Discoveries are the most at risk. Sometimes the org or plan changed, and it could change again. These headcount may not map cleanly to a business narrative or stable org structure. Sometimes the approval was verbal and never really hardened into a durable commitment. Maybe leadership wants a new capability, but the business case is still fuzzy.

People may ask why someone new is needed for a specific new skill or background/experience instead of uplevelling existing folks.

Ultimately it will be asked:

If this role exists, what other role does not?

Deciding the Level

Here's where the challenge gets harder. Level determines compensation band, which determines budget.

Does your budget really support current market rates for that senior engineer you're sure you need?

What's the distribution of seniority in your team(s), and even the wider org?

Are people needing a mentor - or mentees?

There can be pressure to down-level (which creates room for others in the team to step up) or up-level (which adds skills and bandwidth that may accelerate/improve the team).

Then there's reality of who's applying: if you post for Principal Engineer it's a smaller pool of candidates - which can make finding the right one take a lot longer (or cost more).

Also, candidates at higher levels (more scope, more comp) need to go through more vetting. That may mean navigating cross-org or "all-the-way-to-the-top" approvals.

And candidates may shy away from the increased requirements and extended process. You may have to innovate new ways of assessing these rare candidates, and get approval for that new process too.

One critical tip: if after a couple of months the candidate pipeline isn't quite matching - revisit the job requirements and level.

Something I've personally seen: opening a role at Senior for a wider pool, then having an amazing candidate who will only accept Staff, needing to go back to HR/Finance and renegotiate. In the extra time it took to renegotiate the level, the candidate accepted another offer.

Tools of the Trade

Politics: the art of influencing others.

So first get the facts and strive for first-principles thinking:

What does the company truly need now and for the next 12 months?
- Get feedback from your manager, your manager's manager, and your peers
What element needs to be added into the chemistry of the team?
What can you truly afford - now and during the next compensation and promotion cycle?
What's the opportunity cost for every month the role isn't filled?

Avoid hiring for vanity metrics ("doubled my team") or empire-building ("never dependent on other teams").

The uncomfortable truth: sometimes headcount is assigned because of existing relationships or to keep someone happy.

Long term credibility comes from being seen as objective with decisions that lead to achieving the larger goals, from having a well documented track record of delivering results.

Helping new managers with their hiring process has been some of the most rewarding and impactful work I've done as a people manager:

I helped a peer advocate for and successfully hire 2 great candidates when there was only 1 headcount available by looking at the company's own 12-month forecast
I've pointed out some missing topics in interviewing loops which raised the quality of the candidate debriefs
I've supported a Manager who struggled to say No to everyone on her team, when they kept saying "Yes" to every candidate; via private 1:1s everyone came to understand the real impact of a bad hire
I've even transferred people to another team when they really needed it but didn't have headcount

Writing the Job Requisition

The job description you see on a careers page has been through a journey to get there. It's actually a myriad of compromises about what the manager wants, the team needs, what the company can support and how the company presents itself externally.

It starts with a hiring manager writing briefly what they actually need. This is the important part: documenting what would truly move the organization forward, not just a wish list.

Then the manager of the hiring manager looks it over and considers the context of org priorities and patterns, historical trends for similar roles.

HR reviews and adjusts it for legal/company policy adherence.

Then recruiting edits it for marketing/SEO, company messaging, and consistency with existing job templates.

Since the final polished posting can be different from the original Hiring Manager's version, experienced candidates know to ask, "What does success look like for this role in the first 6 months?"

This is smart because it begins the conversation of alignment on impact - everything on the "required" list may not be needed until month 9+

Working Together with your Recruiter

Maybe you've been wondering the whole time: wait, the Recruiter's last?

If they are last and they just take the req and blindly post it then you may have a long and painful process of starting with poorly-matched candidates =(

The recruiter is not your assistant, they are a skilled professional in a specific domain to achieve company goals.

The collaboration between hiring manager and recruiter is essential for success. Recruiting/Talent have Key Performance Indicators about the number of resumes and candidates screened, interview funnel conversion rates, offers vs accepted, and candidate engagement feedback.

They are also almost always juggling multiple reqs and pipelines at once.

A good recruiter asks questions like:

For candidates, what's the narrative on why this role is open, why the team and project is great?
What's your dealbreaker(s) for passing on a candidate?
What's one requirement or skill you might compromise on?
What are examples of existing employees that'd match the role?
How diverse is your team, and how diverse is the interview panel?
How long would you wait before down-levelling to widen the candidate pool?
How willing are you to up-level or consider a signing bonus for a great candidate?
How will the candidate perform and feel after 6 hours of back to back interviews?

They will tell you historically how long it usually takes to hire this role, if it's a "employer's market" or "candidate's market", and bluntly: that you might be looking for a flying unicorn.

Calibration and Sourcing

Early alignment at the top of the funnel saves a lot of time and effort: what profile is most similar to candidates who will do well during interviewing and get an offer.

An exercise I've been through many times to great effect is having the Recruiter provide a list of randomly selected candidate profiles/resumes and scoring them to 1-5 for how "strong" they seem for the role.

While the first round can be async, what's truly important is to have an actual meeting talking through the Why one candidate really seems better.

Sourcing is finding the channels and methods to reach out to candidates. Your Recruiter/Talent team may know the currently best performing places. Managers and ICs may know the industry niche or geography-specific places that are really high signal.

Finding those proxies for "Who will probably do well" and "Where can I reach them" are the foundation for your pipeline.

Referrals: traditionally the strongest positive signal as someone actually worked with and vouches for this person. These candidates will understandably get special treatment, and it requires diplomacy if things don't work out.
Sourced: they didn't actively come to you, they were found because they fit your target profile. A personal message from a Hiring Manager starts a human connection.
Organic Inbound: high volume and low signal - anyone can click to apply. So time consuming it often needs automation to filter and prioritize, sometimes it even needs (time delayed) auto-rejection emails.

Setting Up the Interview Loop

Candidates, and even Interviewers, rarely see all the machinery underpinning the Interview Loop.

There's usually an existing framework or template in the company, but as a Hiring Manager you may need or want to adapt it.

How many interviews, what type, and in what order:

Recruiter Screen: cross check their resume with reality, quick company/culture fit, scratch test on a few topics
Hiring Manager Screen: is this worth everybody's time?
hands-on technical: core responsibility of the role
architecture: systems design and big picture thinking
cross-functional: input from adjacent disciplines like Product or Quality
presentation: previous project or case study
simulation: a scenario that ramps up
behavioral: experience, judgment, career ambitions
executive: department/company culture fit, adaptability, future promotion potential

Will you have discrete phases and gates where candidates get evaluated like Recruiter/Manager screen, Tech Screen, Onsite? Or a more continuous sequence of sessions?

Since there's an additional cost to your org for every additional session you add, consider the heuristic of increasing investment as you gain confidence and need deeper signal.

And if the candidate fails on a critical area, do you continue with the rest of the sessions?

Choosing interviewers is like casting for a show: you need an experienced expert for each topic, and you need their backups for availability.

I've found being specific on your priority order and max number of interviews one employee should do per a week is critical to keeping things smooth and sustainable.

You may also have to borrow folks from a peer's reporting chain which is a delicate conversation about priorities and "for the good of everyone by increasing the company's overall capacity".

Training Interviewers

Training interviewers before they actually interview an external candidate is incredibly important but always ends up so low a priority that it defaults to "learning on the job" - to the detriment of everyone involved.

Bad outcomes can range from "missed a good candidate" to "waste of time" to "lawsuit against your organization".

The Hiring Manager should write out the minimum acceptable signal for each part of the loop. What's being used to identify good vs weak?

Get clarity on what cannot be legally asked in an interview, and on when and how to discreetly end an interview if it goes off the rails.

Then meet with prospective Interviewers, as a group or individually. Talk through the job req, their specific topic, time management and the expected usage of time (i.e. 5 mins intro, 35 mins on topic, 5 mins candidate questions).

During this process you can and should get feedback from your experienced people about the role's requirements, what should be asked, and what really matters in terms of concrete signal.

For those less experienced overall, or on a specific topic, some useful techniques are mock interviews (you play the role of a candidate) and shadowing (quietly observe in the room) an experienced interviewer's session.

Structured Interviews

Structured interviews (especially for subjective areas) improve the objectivity and effectiveness:

Scheduling Challenges

Scheduling can be a large invisible hole in the hiring pipeline.

It's hard enough to juggle your team's existing responsibilities while prioritizing the candidate's availability, but you may also have to coordinate calendars for cross-functional peers or a whole panel session.

Stuff happens: someone's sick, there's a production or security incident, the candidate last-minute cancels and needs to reschedule.

You'll need debrief meetings too - ideally the first ones are synchronous so you can improve on shared understanding of calibration: what was a good score, a strong candidate, and why? Any yellow or red flags and best questions that uncover those?

If you're lucky there's a dedicated scheduling coordinator, fancy software that matches availability, or your recruiter is a saint and a calendar savant. You should be prepared to be involved: as hiring manager, you know your team’s calendars, constraints, and priorities better than anyone else.

Hiring Manager Interviews

This is a unique piece in the puzzle because no one else can really own:

Can I adequately support their professional needs and career ambitions?
Will I be able to manage them? Would they listen to me and take feedback, will we get along professionally?
What might I need to adapt or prepare the team for during interviewing?
Do they add something special we just don't have, that I didn't even think to ask/require?
Will this person thrive here?

Delegation at its finest: depending on your team and peers to evaluate role specific skills, communication, planning, etc.

This is your opportunity to ask open-ended questions with follow-ups. Get them to open up so you can understand motivation, self-awareness, judgment, and professional style. How do they handle stress and accountability?

The classic question is:

Tell me about a conflict with a peer or your manager, and how it was resolved?

One of my personal favorites:

Where do you see yourself in 2 years time, even regardless of this role?

Tech Screens and the Onsite

There is already a lot of good material on designing technical interviews, so I will not try to recreate all of it here.

Here are a couple that I like:

Debriefs - Data and Humans

Be clear with yourself and everyone on the debrief panel:

This is an attempt at deciding fit of a candidate for a role and organization at a given moment. It is not a personal or professional judgement on the individual

Everyone is "in a room" to work towards a decision. It's not a debate, it's a fact finding mission. How much signal can you extract from the process?

Scorecards appear clinical and objective, but there are many known cognitive biases to watch out for:

anchoring or recency bias (first or most recently seen as the best candidate)
similarity bias (selecting for perceived similarity to the interviewer: background, interests, communication style, etc.)
liking bias (charm is not the only criteria)
prestige bias (brand or fame instead of focusing on evidence)
stereotyping
attribution error (no skill or just nervous?)
groupthink

Interviewer feedback quality will vary, some people write detailed, structured scorecards with specific examples. Others write "Yes, hire".

Some of your interviewers will have a really high bar, while others generally default to yes.

Your role as Hiring Manager is to improve the process: set expectations and get the data.

Having a concrete scoring rubric with examples converts some of the subjectivity into objective numbers. Enumerate ways for candidates to score points, and even consider transparently sharing your scoring rubric with the candidate.

The longer it takes for you and your interviewers to understand what "good enough" looks like, the more likely you are to lose that window of opportunity for an amazing fit candidate.

I've found it goes most smoothly if there's a clear Facilitator and each person provides their feedback with an opportunity for questions. Not a place for the "loudest voice wins", or detours into technology preferences and ideology.

It can feel like the hardest part to sit on your hands and keep your mouth shut, but you really have to listen and stay curious

*I have found that sometimes you can't or shouldn't cleanly tell the interview panel whether you're making an offer - you may need further discussion with your manager and may not even yourself know *

The Decision

The hiring manager makes the decision, informed by the panel. You may want unanimity. You need data. You will deal with the consequences if you hire against consensus or over objections.

If you pass on someone, then maybe you'll see this person again when they're the Interviewer and you're looking for a job.

One of the most memorable debriefs I ever experienced was a group all giving a barely-good-enough score "Yes to hire". Then when I asked each person "Would you sit next to this person and work with them for the next month", an instant "No".

Always Be Closing

Every touch from the recruiting team and interviewer is a signal about your organization. Great candidates have options and other offers.

While the recruiter handles logistics and total compensation offer, you should "close" personally, because after the dust settles this person will become your direct report.

Call the candidate on the phone and tell them why you want them specifically. If it's appropriate, take them out to coffee or lunch, and sell them on the broader vision of the team, the work, and their growth.

The goal isn't perfection or winning, it's to have a strong foundation and keep improving.

The Offer and Everything That Can Go Wrong

Now that you found someone you want to hire, you can find out where things stall, fall apart, or get weird.

The Total Compensation Package is usually a combination of salary and equity, hopefully part of a pre-defined compensation framework that maps to existing employee levels and salary bands.

https://blog.john-pfeiffer.com/compensation-planning-is-sudoku-with-feelings/

Usually a number of people will need to sign off on the Offer:

Your manager (and maybe even someone above them): is this really what the org needs right now?
HR: any discrepancies or irregularities in the interview process, background check, employment conditions, etc?
Finance: this will become an ongoing expense to the business - was it really attributed/budgeted correctly?
Maybe even the Founder or CEO

In a chaotic or high friction system this can cause delays - long enough to frustrate or lose a candidate.

"Ghosting" or never getting back to a candidate, at any stage - especially this stage, is a reputational black eye

Comp negotiation: The candidate wants $X + 20% - is that really doable? In salary band range? How does it look compared to existing team members and employees?

The recruiter (and/or an internal referrer) may be advocating for the candidate.

Are you willing to push your Manager/HR/Finance for this?

Sometimes the first offer feels like a deliberate fiction: of course there's 5% wiggle room, counter-offer back.

Similarly, be careful to not negatively judge a candidate who negotiates: hate the game not the player. Understand they have their own needs and goals - this is one of the few places where someone has leverage with their (prospective) employer.

Sometimes candidates take a long time to consider the offer. Maybe working through a competing interview loop or waiting for a counter-offer or going through a life event. Lean into your Recruiter's people skills to diplomatically reach out. And keep your process going

Counteroffers complicate things. The candidate's current employer, or a competing employer have offered them something "better".

Take a pause before you act. You might not need to counter, nor resign yourself to losing the candidate.

If possible discover what's compelling about the alternative: title? autonomy? equity? "More cash" can mean immediate increase or a future promise, or a larger (future) bonus.

Make your case persuasively, but don't oversell or overpromise. I've hired someone at top of band and then really struggled 10 months later when they wanted a substantial raise but they didn't meet the bar for promotion.

Continuous Improvement

Hiring is always a moving target: needs of the company or team change, candidates appear and disappear from the market.

Having written artifacts for each part of your system means you have a transparent reference for everyone, and a place to update and roll out improvements.

After a few weeks (and a lot of resumes) you should already be learning from the funnel: analyze which things aren't working?

As you experiment with changes monitor using your (ATS aka applicant tracking system) dashboards.

On a single req I've actually experienced changing almost every part from requirements to job posting title to role title and level.

When things aren't working, start from a charitable perspective and use the "blameless post-mortem" and "5 Whys" techniques.

This can feel like a lot of pressure and a big decision. A bad hire impacts the team and maybe even many others in your organization, which is why so often "no hire" is the safe choice.

Also though, economic or political winds may change and your job opening will be closed unfilled. It's a measure of your resilience to learn as much as you can from the experience and improve for next time.

Hiring can be emotionally exhausting. Know that you too will "level up" and get better by doing it.

I definitely remember the naive time I closed a req after a candidate accepted our offer, and then the day they were going to start we learned they actually decided to go with another company

Measuring Success

Of course 100% accepted offers sounds great, unless it's a sample size of 1. Or new hires don't reach the 90 day mark. And having more headcount is also more headache at annual review and compensation planning.

Consider some other ways to measure:

candidate experience becomes marketing for your organization's reputation as an employer
interviewers will remember if things were well organized or chaotic
a well documented system elevates the process for everyone

Great management isn't just intuition and a few good decisions. Instead, like a "sports dynasty" or a finely engineered software system, it reproducibly generates great outcomes.

Hiring is part of the long lifecycle of your relationship with someone, from candidate to direct report to transferred to another team to "alumni".

Resources

After you hire then it's time for onboarding which is a whole different blog post. =]

Here are some other good resources on Hiring:

https://stripe.com/guides/atlas/scaling-eng
https://www.joelonsoftware.com/2006/10/25/the-guerrilla-guide-to-interviewing-version-30/
https://github.com/charlax/engineering-management?tab=readme-ov-file#hiring
https://www.developing.dev/p/meta-hiring-lead-on-behind-the-scenes is Peterman's interview with Austen McDonald, a former Meta hiring committee lead who conducted hundreds of interviews.

Bonus Update - How to update an Interview Loop for AI

I led an update to our engineering interview loop once it was clear that engineers were effectively using AI every day to do the work.

During interviews we were already getting awkward signals: candidates quietly using LLMs during coding screens, long pauses, eyes off-camera, and trouble explaining their own code.

I split the loop into two parts: a non-AI fundamentals exercise first, then an AI-required round where candidates show how they actually work with prompting, evaluating generated code, adding guardrails/tests, debugging, and refactoring.

That made the process far better: transparency removed the awkwardness, candidates relaxed, and some blew us away. The artifacts made debriefs much stronger too: we could evaluate judgment and approach, not just "code".

With agents, the next hard question is what “bring your dev environment” should mean in an interview?

Golang Concurrency Part 1 WaitGroup

2024-04-23T22:22:00-07:00

When I tried to read my previous article on concurrency in Golang I felt like it tried to pack too much in so this is the same topic broken into parts, which may even leave room for more depth. - previous article https://blog.john-pfeiffer.com/golang-concurrency-goroutines-and-channels/

Background on Concurrency and Goroutines

Goroutines are like lightweight threads. This removes some of the overhead of attempting to use concurrency with OperatingSystem threads. Here is someone else's better explanation of threads

https://www.youtube.com/watch?v=oV9rvDllKEg Rob Pike describing Concurrency
https://en.wikipedia.org/wiki/Thread_(computing)
https://www.cs.uic.edu/~jbell/CourseNotes/OperatingSystems/4_Threads.html

Goroutines

Normally in programs main executes sequentially from top to bottom. Go routines can operate concurrently to main (and any other go routines). It only takes the simple syntax of prepending the keyword "go".

package main

import (
    "fmt"
    "time"
)

func main() {
    fmt.Println("start")
    go count()
    go foo()
    time.Sleep(1)
    fmt.Println("done")
}

func count() {
    fmt.Println("sleep then count")
    time.Sleep(2)
    for i := 0; i < 100; i++ {
        fmt.Printf("%d", i)
    }
}

func foo() {
    fmt.Println("foo")
}

https://go.dev/play/p/KX-bMG0KBBu

This code example highlighted that when main exits all goroutines also exit, even if they have not completed.

Golang WaitGroup

In order to add control over the goroutines there are many tools, the simplest is WaitGroup.

package main

import (
    "fmt"
    "sync"
    "time"
)

func example(s string) {
    time.Sleep(1 * time.Second)
    fmt.Println(s)
}

func exampleAsync(s string, wg *sync.WaitGroup) {
    defer wg.Done()
    time.Sleep(1 * time.Second)
    fmt.Println(s)
}

func main() {
    fmt.Println(time.Now())
    example("hello")
    example("world")
    fmt.Println(time.Now())

    var wg sync.WaitGroup
    wg.Add(2)
    go func(s string) {
        example(s)
        wg.Done()
    }("foo")

    go exampleAsync("bar", &wg)

    wg.Wait()
    fmt.Println(time.Now())
    fmt.Println("done")
}

https://go.dev/play/p/YYJVC36uB5r

The whole program executes in 3 seconds: even as sequentially things take 2 seconds, the next 2 sleep statements occur concurrently.

A WaitGroup must in advance be passed a count that matches every execution of "Done()" (usually by goroutines).

Even though the anonymous function that wraps "example()" and "exampleAsync()" both have a 1 second sleep statement, the output shows they run concurrently.

The anonymous function in the middle shows how to pass a string parameter, and also that the waitgroup variable is available through "closure".

For readability, maintainability, and re-use most people write a separate function rather than using anonymous functions.

Passing the waitgroup by reference is safe as it is designed for coordinating goroutines, and the "defer" keyword just ensures that just as the function exits that statement will immediately execute.

package main

import (
    "fmt"
    "sync"
)

    func main() {
            var wg sync.WaitGroup
            wg.Add(2)
            go func(s string) {
                    example(s)
                    wg.Done()
            }("foo")

            wg.Wait()
            fmt.Println("done")
    }

This is a common gotcha where the program will deadlock as the WaitGroup forever expects one more "Done()" than the code provides.

One of the Hardest Parts of the Job - Leading Through Layoffs

2024-02-29T12:01:00-08:00

Disclaimer: this piece reflects my experiences and is not legal or professional advice.

It happens

I've been through multiple layoffs: some I delivered, some I narrowly avoided being on the receiving end. It is never easy, never painless.

This can happen to any kind of company: a startup that runs out of runway or has to pivot, or a large company making a "strategic change." Sometimes it's a broader shock like the 2008 financial crisis or the COVID-19 pandemic.

Even if it hasn't happened directly to you, it has probably happened to someone close to you: a partner, a family member, a close friend, or a former colleague.

Stabilize yourself first

During layoffs, start by acknowledging it is understandable that you're going to have feelings about this.

A useful parallel is the Kubler-Ross model of stages of grief. The point is your emotional state affects your judgment, your patience, and your capacity for the hard decisions and difficult conversations ahead.

Shock and denial: "Maybe we don't really need to do this. Maybe Q4 will turn around."

Then frustration and anger: at the situation, at the leadership decisions that led to this, at yourself for not seeing it sooner. Feeling like some performance issue could have been addressed earlier.

Bargaining: maybe people can work extra on weekends, or temporarily take pay cuts, and the problem will go away.

Then a low period where everything feels heavy, normal work feels impossible.

As these emotions play out, don't make decisions yet, as your judgment isn't reliable. Instead gather facts and prepare to think clearly.

At a high level, consider 3 phases:

Planning: understand the problem, gather data, plan with options
Decision: commit and follow through
Integration: help the new organization stabilize and become healthy

This is one of the ultimate tests of your capacity for confidentiality: leaking your thoughts and emotions early will create anxiety and chaos.

Managing Up

This is an advanced career move, so not for everyone, (and be mindful of your emotional state if you decide to attempt this) but if you have built up enough trust, double check with your manager:

Is this really necessary? (These kinds of changes can have unintended side effects, can lead to irrevocable loss of morale)
Will this be multiple rounds? (longer timeframes means more paralyzing and demoralizing)
Who else will be involved in the planning, and can I discuss it with them?
Do you already have a list in mind?

Be very careful to not sound like you're questioning leadership; seek to understand so you can achieve the best outcome.

Going through these questions may help you have more complete answers and be anchored as you begin the difficult work.

Planning: Work Backwards from the Goal

If the problem is bad enough that it requires a reduction in force, waiting months or even weeks is unlikely to improve things.

Urgency doesn't mean reckless. These are some of the most critical, life-impacting decisions you will make.

Prioritize thinking and working on this before almost anything else - the people affected deserve the best from you. Do it in a quiet, private space.

To be clear: you need a healthy organization of teams at the end to get good results.

Start with the outcome: what does the successful organization look like after?

Work backwards from there...

What does the Future Org look like?

An understandable misstep is to focus on who to remove rather than what you're tasked with building.

Don't start from your existing org, it may have grown organically over time or hastily during a hiring boom. Think outside the box. Start with answering key questions:

What are the most critical projects that have to continue?
Are there natural domains or boundaries around the key initiatives?
What skills/support are most needed?

Conway's Law points out that the product (or software) will mirror the organization that builds it. https://en.wikipedia.org/wiki/Conway%27s_law

Similarly, when you understand the shape of the future work, it should inform how you shape your new organization.

A RIF is an unpleasant forcing function: consider the really painful and radical ideas first, sometimes they're right and dismissing them limits your options.

Look to create teams that own clear domains with minimal external coordination
- prefer clear end-to-end autonomy and value delivery
Consider retaining deep expertise, but balance it against future potential
Prioritize high morale and high productivity: the road ahead will be bumpy

Then layer in:

Are there clearly areas that are lower priority/impact?
Places the business can no longer justify?

Consider your ideal team size: often less than four is fragile (PTO, sickness, on-call coverage), and greater than ten struggles to focus (mega standups, orthogonal sub-projects).

Use a confidential whiteboard; brainstorm and be open to a lot of paths, options, and variations.

I remember talking with my counterparts in other disciplines and departments. Having so many hard conversations about what they were seeing and thinking about. Drawing and redrawing so many variations to not miss a possible improvement.

And some more resources on org planning:

Gathering Data

Gather data, both objective and subjective, into one confidential place.

You will need enough data to show yourself that you were thorough. To show your leadership that you were rigorous. And to help you be confident and thoughtful in the difficult conversations ahead.

What is the driver of the RIF: financial, strategic, or something else?
Every person's current tenure and salary
Performance Reviews and OKRs
Previously created succession plans
Past, present, and future Roadmaps. Plus agile sprint data or charts
An inventory of all responsibilities, special skills, critical external relationships
Emotional data can look like "Which 1:1s do you dread? Which do you look forward to?"
Who's engaged in meetings, taking initiative, amenable to coaching?
Who's been actively cross-training, and is adaptable enough to do more?

Building Careful Systems of Evaluation

Consider starting with a simple table of every name.

Add columns of attributes: Productivity, Cost, etc.

Customize to the attributes that make the most sense to you and your org

Stack-rank the attributes by priority (or add "weightings") - not all data is equal.

Then for each person, add a score of 1 to 5 for each cell: this pseudo-numerical system isn't meant to absolve you of the fact you're making the decisions, but to force you to recognize what you're prioritizing.

I found using positive attributes in the scoring was difficult because I wanted to give everyone a top score.

Looking at negative attributes is more uncomfortable, but also more revealing in this context. It forces you to face the difficult tradeoffs when the work ahead is about to get harder.

Example Table (high score is worse)

NAME	DISRUPTIVE,	LOWER PRODUCTIVITY,	NEEDS SKILLS DEVELOPMENT,	REQUIRES MANAGER ENERGY,	COST,	SUM
Alice	1	2	4	3	2	12
Bob	5	3	1	3	4	16
Charlie	3	5	2	2	2	14

Then iterate again on all the values. Sort and re-sort from different angles.

Next try adding a "safe list" approach: start with the nucleus of your future successful org. (Move them to a separate spreadsheet).

Consider second-order effects: who might be demoralized or leave due to the layoffs.

Like when interviewing a great candidate who doesn't quite fit: this is more about the future organization, its priorities, and possible financial constraints than the individual's qualifications.

Also future team chemistry matters as those remaining need to pull together and push forward.

Will you need as many managers? How will new reporting structures work?

Never delude yourself that the system will give you the answers and absolve you of the decision making responsibility.

Listen to your instincts, not because they're always right, but because maybe they're telling you something you don't fully understand yet.

Spend time reflecting and getting more data when something feels off.

Know the limits of what you know

Having a really complex system and fancy models doesn't mean you can predict the future. Often too much complexity gets in the way.

And here's a really hard thing: do not involve an individual's personal situation.

You can tie yourself in knots thinking about each person's life consequences. Worse, you likely don't know everyone's full situation: family and dependents, expecting children, mortgage and debt, immigration status, etc.

If you start making exceptions where will you stop? Is it fair to the people whose situations you don't know?

The professional role of a people manager is evaluating people's professional circumstances.

Another Evaluation Framework

Another useful framework comes from criteria Jim Collins lays out in "Good to Great":

How would you feel if the person quit today?
Do you risk losing others by keeping this person?
Are there concerns about their values, willpower to get things done, or skills?
When things go wrong, do they look to blame, or in the mirror to improve?
Have they been giving you signals like "this is just a job"?
Do you have them in the right seat on the bus?

Did your bus just get a lot shorter?

Do not be surprised if you are also on a list, even if it's not today

Get Feedback

You are not doing this in a vacuum. All of your research and planning is part of a larger process.

As you reach out to people for feedback, remember to be careful: this is "need to know" only. Only bounce ideas off people you can trust with confidentiality.

Consider having some meetings "off calendar". Be cautious about how you share any lists or plans. Premature sharing can lead to rapid and uninformed decisions.

An open mind for the best outcome

Do be open-minded to different viewpoints. This is about the best possible outcome, not your ego.

Sometimes saying something out loud reveals a very good, or very bad, idea.

Can you be vulnerable and admit if you have favorites? Why might you feel a certain way?

Before you finalize anything: has someone spotted an unconscious bias? Are there unique permissions or access you may have missed?

What's your second-tier succession plan, and how will you cover if others quit?

How will the chemistry change between teams and groups?

Are there responsibilities someone in your group can take over elsewhere in the org? Could there be trades with other managers?

These extra conversations require discretion but can sometimes help retain one more person; this can make all the difference.

Decisions

Once you've made the decision, stop "looking for more signs" to confirm or deny it. You've done the work.

Do stay open to truly last-minute information: someone resigning or otherwise exiting, a material change in the company's plans or needs.

You still need to keep this all confidential, which means trying to avoid a change in behavior or hints. The hardest part is yet to come.

Logistics

Figure out when the decision is final. From there, figure out when is the soonest you can deliver the news well. Note that HR, Legal, Admin, IT, and other departments will need some time to prep too.

Try to keep the gap short: days, not weeks. Long delays create anxiety, speculation, and distraction.

Plan to do it all at once. This may take most of a day, or more. The decision maker should deliver the news privately to each person affected. Be thoughtful of how many people are "in the room" when someone receives news like this.

If it's not possible to do in person, make sure IT is ready for the series of private video meetings.

In many places, leaders prefer to communicate layoffs at the end of the week, with no further work expected that day, so people have private time to process. They should not be expected to absorb life-changing news and then return to a normal workday.

That also means everyone supporting the process should be ready for immediate next steps once the conversations start.

Terminating Access

The goal is not to treat people as untrustworthy. Once someone has been told they are leaving, the organization needs a clean and immediate transition point, especially on a day that may already be emotional and chaotic.

This reduces confusion, avoids accidental mistakes, and makes it clear who is responsible for what from that moment on.

Coordinate with IT, HR, Legal, and others on things like:

Production Systems, Production Data
Confidential or Sensitive info
Public-facing communication channels used on behalf of the organization

On the day of, this may expose gaps in your preparation. In that uncomfortable event, it is better to reduce activity and access, then review it more thoroughly when things are less busy.

If someone needs to keep limited access for a short time, that should be a rare, intentional exception with a clear reason behind it.

Who Knows When

Keep this confidential from everyone inside (and outside) the company who doesn't absolutely need to know.

Sync with your peers and HR who are planning and handling the changes. You're all designing the future successful org. They're your support group during this difficult time.

Rumors may already be swirling. Don't be rushed. You may have to honestly say "I don't have any answers for you right now".

Communications Plan and Delivery

Have a Communications Plan. Each sentence and every word should be carefully reviewed.

During layoffs, this should not be communicated as a personal criticism, or framed like a performance review. There are a lot of factors that go into the decisions. Make sure your language doesn't imply that it's about performance.

For instance, if the company is no longer doing hardware, then excellent people who are experts in hardware will be let go.

Plan down to the minute which people or teams are notified. Prioritize informing those most affected first.

Double-check that the formal parts are ready: this is an extremely emotional one-way door. Not the time for administrative mistakes.

What answers have HR prepared for handling situations like visa issues, parental leave, or other specific circumstances with compassion?

Triple-check your understanding of the new org structure, including connections to other teams, which responsibilities must transition, and especially passwords, credentials, and external contacts.

Prepare a FAQ: what are people most likely to ask or be worried about?

Will those leaving get severance and benefits?
Am I allowed to reach out to folks personally?
Who made the decisions?
Why is this happening?

Rehearse and practice your delivery: both to those who will be departing, and the message you'll be giving to those who remain.

The Day Of

Try and be rested beforehand: this is a really sad day and takes a lot out of you emotionally.

Dedicate a whole day to delivery. Nothing else on your calendar.

There will be no productive work this day. Accept it.

For those departing

If someone is out on vacation or medical leave, delaying communication can create its own harms: they may hear a distorted version secondhand, lose time to process next steps, or be left in limbo while others already know.

In practice, this is usually something to handle carefully with HR and Legal so you balance timely communication with compassion, compliance, and the person's circumstances.

For each person, consider getting approval for a personal note to the official messaging - acknowledging a past success or shared memory.

This is a serious moment: if possible try to avoid the nervous instinct to use humor or forced warmth. It is better to come across as calm than cheerful or casual. And even if it tears you up inside with sadness, you should try to remain composed.

Stay on script: be empathetic but don't argue or explain. This is about them, not you.

Offer to assist in the job search if you mean it. Don't offer if you don't mean it.

HR will take over logistics. That's usually the end of your part of the conversation.

For those remaining

This is an organization-level event.

A common pattern is a leadership meeting, either just before or immediately after, to explain the change at a high level to the whole company.

The individual notices are often followed by a department-level meeting, then even by group and team.

Finally, as a manager, be available for any ICs who may have immediate questions or concerns.

Things you have to clearly explain:

The official reason, and the official messaging to users/customers/partners
The new organizational structure
- What, if any, are the new connections to other teams and departments
Any critical changes in responsibility

Remember to have a consistent message across the many modes: verbal, written, and visual

There will be reactions so consider in advance what's allowable (venting, tears, needing to leave early) versus what's never acceptable (no matter the circumstances).

Shock and some abnormal behavior
Denial, anger, frustration, bargaining, etc
Survivor guilt

Refer back to your comms plan and FAQ.

Tell the truth. Be honest but kind. The truth sometimes is "I can't share that" or "this is not the time to discuss that."

Don't pretend to have answers you don't have. Don't over-explain or re-litigate decisions.

Integrating - The Day After and Moving Forward

The day after: This day too, there will be little meaningful work accomplished.

Understand that trust has been broken. People joined and believed they were all part of a bigger mission. Your role is rebuilding that trust.

The days and weeks after will continue to be challenging while you support those who remain - people can't immediately return to "normal" as everything has changed.

Continue to leverage the communication plan. Patiently explain the mission ahead, the new org chart.

Actively listen.

Did you or the organization make some small mistakes that you can fix?
Is there someone more affected than you anticipated?
- Who else is a flight risk? What are possible mitigations?

The weeks after will bring more hard conversations, often in 1:1s.

The goal is not to have a reorg, and then lose the rest of your org.

Your job is to carefully steer people to look ahead, to integrate and re-engage.

Be careful of the trap of explaining the same thing too many times, especially the hypothetical "what if" scenarios, or arguing with decisions that are now in the past.

The Instinct to Pull Away

You may find yourself shrinking back from 1:1s and skip-levels. Partly because of hard conversations and feedback, partly because it hurts to invest in relationships with people you might have to let go someday.

Resist the instinct to pull away.

But also don't overcorrect by smothering people with attention. Keep the same stable, professional, compassionate cadence you've always had.

People managers represent both change and stability. In moments like these, your steadiness matters more than your words.

Lean into directing thoughts and conversations towards experimenting: trying the new structure, learning to work in the new situation. The goal is eventually integrating the changes into a renewed sense of purpose.

Learnings

Every scar should bring you some wisdom.

Layoffs taught me many lessons: not only about how to try to avoid layoffs, but also about how to proactively build a healthier organization.

As a people leader, strive for transparency around company goals and the company's financial situation. Be accountable, and hold others accountable, for delivering outputs and successful outcomes.

Public OKRs help. So do written records of productivity: roadmaps, demos, features shipped, and concrete work output.

I know how I've felt: I should have spotted performance issues earlier. There was a lot going on, but that's not an excuse. When I saw the business miss its targets, I could have escalated sooner, privately to the CEO, or maybe even to the Board.

I realized I should have pushed harder during an earlier "small reorg" to avoid having multiple rounds of cuts. I wish I had pushed for certain structural changes months earlier.

Did we really need to hire so fast? Were we consistent and thorough in checking on successful onboarding, integration, and the performance of new hires?

Were those hires connected to future revenue, or was I "really busy hiring", and only narrowly seeing the increase in my span of control?

You can't reorg and RIF, over and over, to create success. Happy, paying customers are how you "win".

Be conscious and deliberate with your org design.

Teams with clear identity and purpose make it clearer where work maps to business needs. Teams should refresh their purpose every six months toward the highest-value work. Org resilience comes from overlap and cross-training.

Cost-based modeling of teams mapped to revenue helps. A table of all salaries (and history of raises and promotions) helps.

Promotions need to mean expanded expectations, not just new titles. Do regular succession planning.

Performance review may mean separating earlier with people who aren't working out. Care personally, clarity is kindness.

https://www.radicalcandor.com/blog/how-to-fire-someone

Never get overconfident about cash in the bank and projected revenues... especially without accounting for debt, funding cliffs, and financial risks you don't control.

Consider how to carefully manage up: is the strategy clear and well understood? Are you seeing distraction, dysfunction, performance issues, or accountability gaps in the org?

Leadership

Do not ever think this is someone else's responsibility. Not making a decision... is a decision.

You will need to look every person you let go in the eye and own that you made the choice.

Nobody wants to be in this situation, but someone has to make hard decisions, and make them well.

Be honest with yourself and your leadership that some mistakes likely occurred to get you here. Be humble enough to recognize you will still make mistakes in the future too.

Sometimes this moment is a crucible, a shared bond.

Plan ahead how you'll support folks who are leaving. They carry your company's reputation to the outside world.

Know how you'll support folks who remain. This is one of the moments where leadership happens, or doesn't.

Further Resources

Manager Tools have a number of really good podcasts on this challenging topic:

Compensation Planning is Sudoku with Feelings

2023-11-25T12:34:00-08:00

Compensation planning is like "extra hard Sudoku" except once you've figured it all out... you still have to convince each square that it's the right solution

Always challenging, never a straightforward "satisfying" outcome.

What is compensation planning and why do you care?

Every organization has some fixed budget. Yes there's a little wiggle room, but there's also a breaking point.

Compensation is "buying time with salaries": when you are in charge of an organization you owe it to them, to everyone, to spend responsibly.

It is not sustainable to spend everything today and having nothing for tomorrow. Leadership is providing some measure of predictability.

You must devote at least a couple of hours (or days) to planning and budgeting your largest expense of your most precious resource.

So you have N people, and you have budget X. And if you mess it up really badly you'll either have 0 people or 0 dollars (actually you can go negative but that's very unpleasant).

Fundamentals of compensation planning

Classically you keep salaries (the majority of spending at most successful companies) separate from "everything else" (real estate, vendors, infrastructure). Start with last month's total payroll x 12. That's your baseline spend.

Ask your HR or Board for clarity on any existing or historical policies on how this has been handled previously (or should be handled going forward).

Usually you are not doing this in complete isolation

What your organization needs

Before you start planning, make sure you first learn what your organization needs for the coming year:

Ask your manager
Ask your CEO
Ask your CFO
Ask your Board

There will be conflicting, even contradictory answers. That's okay =]

Everyone's predicting the future, and that's hard.

Ideally there will be a theme, and finding overlaps gives you a direction/pattern.

Is this a growth year with hiring? Maybe it's a "hold tight" year with a slightly shrinking budget.

Even in a shrinking budget there may be projects or opportunities the organization is willing to invest in.

Understand what your company wants to pay for, and where your team members (skills, experience, etc.) may fit in that picture.

Identify which key people (in priority order) would most impact the company if they left. Have you done succession planning?

More importantly: which people have a track record of success and you believe represent the future of your organization?

What your people need

Gather your data about your people and their compensation history

When were they hired?
When were they promoted within your organization?
When have they received raises?
Has there been a trend to their compensation (plateau, spikes, exponentially up)?

Gather your data (hopefully from your regular one-on-ones) about what your people most prioritize.

People want different things (either stated or revealed preferences), so have private notes of their priority ranked list of these things...

cold hard cash
equity
bigger bonus when things go well
flexibility in working hours
flexibility in location/remote
more PTO
more autonomy and/or responsibilities in their work

Some Example Personas

Alice was hired 4 years ago and prefers cash above all else. She has had regular raises every year at 3%. She's still at the low end of the salary band for her level - and you really want to retain her steady troubleshooting skills and institutional knowledge.

Bob was hired 2 years ago and prefers equity and working remote. He's only had one raise during a promotion of 7%. His performance is a roller coaster, and even though he's recently had a promotion he said he thinks he's under-levelled.

Charlie was hired 3 years ago and has never had a raise. Charlie likes PTO. Charlie's well liked by everyone. He's also at the top of his salary band and privately co-workers wonder if he's pulling his weight.

Good Practices

Having distinct levels with responsibilities in a career ladder is a data driven approach to evaluating performance and promotions.

there are many career ladders published on GitHub - consider using even a most rudimentary version of just two categories: "new grad" and "senior"

You should strive to be fair and objective. Not just because it's the right thing to do - but because that's a reproducible framework that creates predictable outcomes.

Salary bands for a given level are a common practice and available through a variety of sources. (see Radford's data or Milkovich et al.'s "Compensation")

If you don't have salary bands you may have trouble recognizing that someone is underpaid (or overpaid) compared to market benchmarks.

Remove your emotions from the process. There is a place for emotions, but not in the current calculus of budgets and planning. The feelings come when you are delivering these decisions to real people.

Framework = rules based, and these rules should be statistically relevant. Rules for one-offs will add a lot unnecessary complexity.

Use a "Parity" principle: keeping people in the same levels or cohorts in a very close (or identical values)

Build your framework

Start with:

What does the company need? (specific skills? leadership? an upcoming key project/feature?)
What are the factors you care most about? (tenure? retention? innovation?)

Create your custom spreadsheet with the header columns.

First pass strategy: ignore the framework for a moment and just write down what feels right for each person.

This gut-check baseline helps you see where your instincts diverge from your rules.

put the basic info in - level, current salary, proposed raise
- have a running sum at the bottom to see what the total is
then remodel and tweak variables

Use a rating system (i.e. performance on a scale of 1 to 5) that maps to percentages:

1: 0% raise
2: 1% raise
3: 3% raise
4: 5% raise
5: 7% raise

of course change those values to whatever you and your organization find best

Raises should be meaningful to the individual for "a very good performer"

Create distinct gaps between the "low performers" and "high performers"

Bonuses are for specific effort in a given time period or year, Salary/Compensation is paying someone for ongoing and future responsibilities

Review

Now that you have your draft...

Ask HR for data

industry comparables (Radford data)
(anonymized?) other departments/groups in your company
Historical trends?
Anomalies or unconscious bias?

Ask for feedback from trusted places:

review with frontline managers (they have to deliver/live with your decisions)
"what did I miss?"
"what did you think about ratings/framework?"
Are you at risk of losing high performers?
examine individual cases

Managing Up

You have done your research and created a draft: this output artifact is essential to get real feedback from the people who need to approve it.

Share it in advance and schedule a meeting. Be open and curious as this is your opportunity to discover more.

Getting alignment early makes it much easier to subsequently get approval.

Nuances

Do look for an opportunity to correct an unconscious/undesired trend - just because someone else made a mistake earlier doesn't let you off the hook

When you have to fix things, small corrections (sometimes over multiple years) are preferred over one large change that looks like a "shadow promotion"

Focus on professional performance, Avoid life/family decisions as input: you don't know every invidual person's situation.

If you skew things and then later learn someone else had "more need" would that be fair? Would you need to ask/pry into every employee's personal and private situations?

Raises as absolute numbers or percentages: most people can understand both so pick one and be consistent.

If you use percentages, at larger salaries this can become surprisingly large absolute-number raises

When someone is getting a raise or promotion, make sure the amount they're receiving "makes a difference".

Giving someone a 1% raise sometimes feels like the worst of both worlds: you are using up budget and the person may get the sense "we don't value you".

Either give a larger raise like 3% or explain "HR/company default for middle performers is 1%", here are the things you can do to be a high performer.

International and Company Patterns

If you have employees spread internationally it may be very difficult to compare without regard to the local economy. Many innovative organizations have attempted to have a "single global salary" and eventually capitulated to economic realities.

Your organization may deliberately choose to not be competitive to the market. You will have to indicate that losing people will affect capacity/timelines, and extra costs for recruiting and hiring a replacement.

Is there an implicit pattern sought or enforced by your company?

The expectation over a large enough distribution of a bell curve or "bottom 10% get 0?"

it's confronting these types of practices that remind you how important your role is in understanding what the organization says vs does - and how your team are impacted

Adjusting to fit your budget

When your compensation proposal doesn't fit your budget, you've got to tweak some things :|

People who are at the top of a salary band are less likely to feel the a small absolute reduction.

If you have excess (lucky you!), consider distributing that to the "bottom" earners first as that may have the most relative impact

Use round numbers: we're human! - This also means, if you need to round down, do so on the higher end first

What if I'm handed a finished compensation plan?

Do your homework. If you spot a serious error you can raise it with your manager and potentially prevent an embarrassing or even consequential mistake.

Showing interest and doing this exercise shows you are preparing to take on more responsibility.

When is it done?

Sleep on your decisions. Review them again with fresh eyes.

These numbers can affect people's life decisions: a progression in their career, where they live, if they get a new car, where their kids go to school or university, etc. They deserve your best when you're making such critical decisions.

If I got this raise from my manager how would I feel?

If I was a manager who had to deliver this compensation change would I know what to say and how explain it?

Can the organization deal with the consequences if someone is unhappy and leaves?

Another round of (hopefully small) tweaks and reviews should go much faster.

Reality bites: sometimes you do this perfectly and then the organization comes back and asks you to reduce your proposal by 10%.

No one ever said the work was easy, but someone has to do it, and do it well.

Delivering the changes

Once it is all approved and you have printed compensation change (or promotion!) letters, practice sharing the news.

Each person deserves an individual conversation, even if you feel like the change was routine or that it was "a large raise and that should make them happy".

(Research confirms that pay systems affect different people in different ways "Personnel Psychology" from Fulmer et. al.)

Do not start with assumptions, every person has unique circumstances and reacts differently
Do give them the letter with the number and at least a minute to read and process
Do listen and observe their reaction, and ask them if they have any questions or feedback
Do not offer any promises to change things, but do take notes and look into concrete concerns

sometimes people need a day or week to fully process, be open and prepared for them to bring those questions about a week later

Part of the reason you put in all the effort to a framework and process is to be prepared to persuade each part of the "Sudoku Puzzle" why the solution you've chosen works.

And sometimes, when you've done this work with care and delivered it with empathy, even if they're not happy with the result, they will understand that you worked hard on their behalf.

The answer is not the solution

2023-01-16T10:00:00-08:00

Look at the number 5: it can have many of representations: IIIII , 10/2 , 05.00 , 5^1

Those representations are not "wrong", they are all still "5".

When people argue about a solution, they can both be "right", and yet there is a reason to pick one solution over another.

If you are adding incrementally, 5 becoming 6 and then 7 is a complete rewrite, whereas IIIII I becomes IIIII II is much easier to append. (Though it also has a downside of readability at scale ;)

If you are already starting a problem knowing "10", then "10/2" is a more natural fit.

When you intend to progress to 05.13 and then 05.99 , clearly decimal has its place.

One should be aware that assigning the number 5 to a group of stones has no impact on those stones. Our form does not change their content.

So when you hear of a different solution, rather than assuming someone else is wrong, ask yourself: does this other representation have a different benefit?

Furthermore, when you have cleverly come up with a solution "5", do not mistake that with necessarily changing the world.

Front Line Manager or Director - Managing People Managers

2022-12-03T12:34:00-08:00

Titles mean nothing

People get hung up on titles and status: "Manager"... of what? Does this mean you get to tell people what to do? So a Director directs people?

The job is always to solve (hard) problems and create value.

The kernel of truth: Manager and Director are distinct in expected outcomes and focus areas. Director is not as simple as Manager++ or VPE--

A front-line manager's focus is the immediate results and to care about people who do the work.

A director's focus is often upward and outward: outcomes that matter to the organization and business, teams and systems that accomplish those things.

Managers are necessary overhead

Why aren't top sports teams "self-managed"? These top athletes are the best at what they do, so why do they have a coach?

People want consistently winning teams.

If the CEO could just directly tell everyone "Do X" and it worked, they would.

Likewise every Individual Contributor would prefer autonomy and to focus on their work rather than having to "report to someone" and "performance evaluations".

A manager represents stability for an organization: a continuing business function that has clear ownership. A dependable and reliable part of the larger structure.

A manager is not "the next rung up the ladder" for an IC but a completely different kind of work, requiring unique skills and responsibilities.

It is a lateral move into a different discipline like Design or Science where prior skills and experience don't transfer automatically.

Good managers are worth their weight in gold - or even their weight in bitcoin ;)

Creating leverage is valuable

What does leverage look like as a People Manager?

An engineering IC creates a frequently used software library or widely adopted architectural pattern: "leverage" to multiply their impact on the organization (and beyond).

A people manager can uniquely scale their effectiveness and influence by:

hiring and retaining great people
converting chaos into an efficient process
building a great culture

Every org has a lot of noise, someone needs to filter for signal and help provide emphasis.

As a great manager - manage yourself

As a Great People Manager, you first manage yourself =)

Are you self aware of your emotional state and mindset - how are you showing up and connecting to others?
- Do you bring positive energy to your team and colleagues?
Do you know what the highest priority most impactful things that you can do are? Effectively manage your own time and calendar?
- Have you spent time trying to understand how your manager and company's leadership team communicate? What is their mindset and favored mechanisms of communication?
Professionalism: on time, prepared for meetings and conversations? Take notes and have action items? Own your action items and follow through?
Do you communicate clearly in writing, verbally, and with images and diagrams?
Do you understand your own strength and areas for improvement, what kind of personalities complement your working style?
- Are you deliberately investing in improving your skills and opportunities for more experience?

What does a Manager even do?

The job is often reminding people of things they already know ;)

The Manager role is indirect and requires patience: long periods of boring (necessary) work punctuated by busy extremes (hiring, performance reviews, promotions, layoffs, receiving resignations)

System/code deep dives are intellectually draining - but often emotionally "neutral". People are not logical software. A people manager's role is integrating humans into a system, and humans are emotionally taxing.

And the job involves common sense: request PTO in advance (so people can cover the work), don't show up 20 minutes late for every meeting.

A manager cares about Results and Retention

A people manager has to "show up", especially in critical times, but overall it is a role requiring reliability.

accountable for success and failure
clarify priorities (because situational awareness and context!) and unblock
aligning the right resources to problems
ensure a good working environment, that people have tools, processes, etc to do their best work
how people's work advances the organization, and also furthers careers
Intensity in 1:1s, Presence for group meetings "crowd control"
https://cate.blog/2015/12/23/the-hardest-shortest-lesson-becoming-a-manager/
https://www.manager-tools.com/2015/06/results-and-retention-part-1-hall-fame-guidance
https://www.manager-tools.com/2015/06/results-and-retention-part-2-hall-fame-guidance

Quality Control

Increased responsibility means more accountability. Every time you hire or promote someone you are putting your credibility (as a decision maker) on the line.

Having a background in a given domain makes it easier to identify when things are going wrong, or performance issues. (For instance, modifying code directly in live production systems is usually a bad idea).

As an engineering manager, have you ever seen an engineer:

write unit test coverage for logging but not tests if the feature actually works?
merge code but always forget to deploy?
have lots of opinions, arguments, and criticisms in meetings but never write code?

A manager observes, actively listens: Is there a pattern? An unvoiced concern or problem?

Then a manager shares with the right people a distilled summary of what's going on, and what change is needed.

Trust and credibility is built by turning pieces of information into action and results.

Mindset of a "legacy manager"

How many people are in my organization, and how do I get more?
Hard work is measured by who sits the longest at their desk
Counting the output of widgets (instead of the outcomes)
Uses biases (especially pre-conceived) and argumentativeness (or "winning")
jumping to a conclusion about who is the decisionmaker; the blame game
One on ones and alignment are overhead - they get paid, don't they?
- Compliments and praise are over-rated, it's their job!
- Why should I develop someone's career when they'll just use it to leave? (or "that's HR's job")
I should send them my random thoughts at any time, day or night - it's their job to be available and figure it out
Unnecessarily going fast and taking risks - activity theater
(over) committing when there's any doubt of followthrough
Stay in my lane, keep my head down, and mouth shut: enforce that for others too
premature public announcements!
- spreading gossip, misinformation, or disinformation

If reading these things makes you uncomfortable, consider reflecting on the ever evolving nature of the knowledge worker

A Modern Manager of Knowledge Workers

A modern manager understands the symbiotic nature of talent and an organization, genuinely finds alignment and valuable opportunities (including when someone's career leads them to another role - even outside of the company), and communicates clearly.

Solving the problems of your "1st Team"

Managing people and resources is leverage to solve business problems. Your peers - your first team - are telling you what they (and the business) need. If you only do what your manager tells you, just an extension of only your manager's asks...

then you're not proactive, not aligned to the business priorities, and not a team player with others genuinely trying to solve company problems.

https://www.youtube.com/watch?v=BjE_mPoZPSg "Team #1" by Patrick Lencioni

A helpful checklist:

do your direct reports know what your quarterly or monthly goal is? How they can contribute to success?
do you communicate clearly what you expect?
when did you last give each direct report a compliment in private? in public?
when have you invested in your direct reports: coached them through something? they learned something new? received a raise or a promotion?
expansive view of everything they do to make their organization and group successful

Litmus test questions about trust:

do your direct reports feel safe enough and are you curious enough, to learn the names of their loved ones, including pets?
do your direct reports feel safe enough to bring you problems and give you feedback?

Recognize people for who they want to be, build trust every day

Prioritizing career development over business needs

If you are prioritizing your own career development or more charitably your direct report's career progressions at the expense of the business outcomes, then you are clearly in a "local maxima".

Aligning what your direct reports or you need to develop along with business priorities is good management.

Don't mistake balance for equality...

Thought experiment: if every manager focused primarily on their direct reports' career development - giving them opportunities to fail and learn and explore - while neglecting what the business needs... and the business fails.

Then nobody has a career to develop!

Challenging Moments and Hard Decisions

If the business is failing then saving the business - which is everybody's job, may be more important than whether your direct report gets a promotion or retained.

Avoiding needless attrition is common sense, but in a crisis, priorities compress and trade-offs get explicit.

Is a Director just Manager++ or VPE--?

Director as Manager++: you're a "super manager" who happens to have managers reporting to you. This works when the business is stable and the primary value is execution consistency. It fails spectacularly when the organization needs strategic pivots or cross-functional coordination.

Director as VPE--: treated as a junior executive, expected to think about business problems, tradeoffs, and org design. This works when the company is scaling or transforming. It fails when you neglect the humans in your organization because you're too busy in leadership meetings.

As a director, the extra layer of indirection between "leadership" and "front line managers" changes everything.

Director reality: all the problems are now your problems. Bubbling up from the front lines, and trickling (or crashing ;) down from above.

Understand the value stream from initial ideas and stakeholders through design and requirements build development, testing deployment, and monitoring.

Have metrics (or a dashboard) doesn't have to be thorough, just enough to give you a warning signal to investigate.

Managing a Manager

Everybody has a first time they became a manager (and the role is already hard!), a great leader helps people nagivate the change.

Hopefully you've learned to become a great people manager before becoming a "manager of managers" - it's never too late!

And just like a Manager is not effective trying to do every one of their direct report's jobs, someone Managing a Manager has to focus on teaching about people management (not taking over or micro-managing).

As a concrete example, when hearing one of my managers bring up not feeling connected to a new direct report, we discussed that there actually many ways to have a one-on-one:

the direct report has their own agenda
John offers a "choose your own adventure" of topics (e.g. how can I help unblock something? larger context in the org or business? career conversation?)
the direct report drives with just a stream of consciousness - your job is to actively listen and spot trends and anomalies
the manager drives (when very comfortable with their manager, likes someone else exposing them to new topics/ideas)

Seek first to understand

Be curious about their perspective: value and opportunities they see, challenges they experience, approaches they've tried
help them construct their own individualized plan for success

For a first time front-line manager the sub-graph of the organization is small: often focused "downward" to just their direct reports.

Teach and encourage (even to the point of discomfort) people managers to navigate peer relationships and "managing up":

invest time and build relationships before you need them
understand other disciplines and roles, what do they need to succeed?
ensure you listen first, ask and follow through on supporting them

An existential lesson is that Managers create a predictable sub-system.

Even Managers with innate people relationship talent have to learn to become systematic, have frameworks, and regular feedback (even difficult conversations) to keep things on track.

What Directors Must Learn (That Managers Can Ignore)

A manager asks "What should my team work on?"

A director asks "What can only I do, what should I delegate, and what can wait?"

Be explicit in what level of delegation you are actually expecting: https://www.jillwetzler.com/resources/delegation

Know what the org/business needs from your domain. A manager can get away with "my team ships good code." A director cannot stop there. You need to understand revenue, runway, competitive positioning, and which parts of your technical portfolio directly impact those things.

Proactively share your group's status and health. Your manager (VP, CTO, whoever) has limited attention. They need a signal, not a firehose of details. Learn to communicate "we're on track" or "we're at risk because X" in 30 seconds or less.

Improve your group's productivity. Not activity, Outcomes! This requires understanding what outcomes matter (see above) and then relentlessly removing obstacles to those outcomes.

Efficiency and cost. At the manager level, headcount is something that happens to you. At the director level, you're accountable for spend. Every engineer-month is real money invested, are you getting a return?

Engineering Leadership is Systems Thinking About People

Here's a mental model to consider:

A great senior engineer knows which service to modify, which file to open, which lines of code to change to have the most impact on the system.

A director of engineering knows which people to talk to, what to say to them, and how to sequence those conversations to have the most impact on the organization.

What is the goal of the system, the weaknesses and greatest opportunities?

growth and transformation
succession planning
personality alchemy, blending skills and experiences
technological investment: balancing innovation and legacy

Directors think in "teams", not "individuals"...

A manager thinks about "Retention and Results", a Director has to systemize and scale:

reproducible successes
manage risk
determinism with people

Find the right people who understand the high leverage techniques (technology or organizational), find/create the right org structure to put those people in the places where they can do impactful work with the sense of ownership and accountability.

Get the right people on the bus, in the right seats

https://www.jimcollins.com/concepts/first-who-then-what.html

Culture: The Thing Directors Actually Own

As a manager you participate in culture, yet at the director level, you shape it.

Outputs vs Outcomes: are you a metrics and process driven organization, or results and outcome driven?

Engineering best practices: are they documented? Enforced? Evolving?

Learning tools: books, video courses, internal wikis. Do people have access and do they know they exist?

Lunch and learns, hackathons, team and company demos: these don't happen by accident, make space for them and protect that space when delivery pressure mounts.

Fun events: connecting people matters. Remote-first makes this harder and no longer optional.

Interviewer training: being an interviewer is a completely different skill. Do your people know how to do it? Are they calibrated?

Ownership: testing, "you build it you run it," production responsibility. Is this just a slogan or a regular practice?

Blameless retros and continuous learning: do people feel safe raising concerns? Do incidents lead to learning or blame?

Psychological safety: "see it, say it." If people can't raise problems, the problems don't go away - they just become surprises.

Embracing difficult conversations and feedback, navigating how to have them even when it's uncomfortable but you need to have a conversation

https://rework.withgoogle.com/intl/en/guides/understanding-team-effectiveness

The spotlight and microphone: what is your organization celebrating? What is getting rewarded as "high performance" and ensuring the optics match reality - help teams see and participate in the wins

Is something missing?

Of course =)

There's no complete guide because it has to be guided by what your organization needs - even as your organization changes.

When considering becoming a people manager or a Director, the promotion is not simply "more work", these are truly different challenges and roles that deserve careful and open minded learning.

Appendix of Checklists

Manager Activities

Ensuring the following for their team members:

roadmaps, sprints
tracking work and bugs and launch checklists
on-call, runbooks, postmortems
accountability on projects and deliverables
team and project retrospectives
how to identify when team has toil, waste? (not all "work" is equal)
individual and team recognition
ownership, alignment and growth
regularly scheduled 1:1s (including taking notes and documenting action items; sharing those for transparency)
approving expenses and PTO

The Unseen Work of a Director

Things a manager rarely thinks about that become a director's problem:

Ensuring compliance and IC training
Access permissions and SSO/audit improvements
Helping with SOC2 or other compliance and external audits
Vendor contracts and licensing (including open source libraries)
Tools and reports to analyze/audit vendors and usage
Frameworks to gather data across many systems - not just deployments and incidents, but interactions with other teams and departments
Cleanup and deletion of old or extraneous systems
Approving PTOs and building a culture of communication (and coverage) during vacations
Training your managers - and other people's managers too (yes, you're responsible for this now)
Coordinating with other departments on how they achieve their goals and their needs
Compensation analysis with HR
Standardization and analysis/improvement of hiring with Recruiting
Creating and Maintaining the career ladder

None of this is glamorous. All of it matters.

The Middle Manager Playbook

Here's what actually happens when you're the quintessential middle manager:

Receive a Vision or Strategy from above (leadership, maybe a passthrough from the board)
Ask questions to gain understanding: context, scope, expectations, timelines, risk, feasibility
Write it up in your own words (refer back to the authoritative source doc)
Understand dependencies and coordination - go from ideas to a plan
Get feedback from your "right hand person" and trusted peers - this is where being cross-functional is a necessity
If necessary, convey that feedback all the way back up to the top
Write up comms (explain the Why!)
Ensure there are milestones, clarity on who does what (in what order), who the leads/champions are, where/how to ask questions and raise issues
Roll it out to your org, capture questions and feedback
Have scheduled check-ins, keep the project on track, and check in "upward" to understand changes in priority, urgency, or scope

It looks obvious written down but it is not obvious when you're in the middle of three conflicting priorities with incomplete information from above and limited capacity below.

Monkeys Stealing Spoons - Why managers struggle and what to do about it

2021-12-29T12:34:00-08:00

Managers run out of time and emotional energy. Imagine a monkey on your back, stealing your spoons - that's what's happening!

The solution = leading with empathy + coaching + ownership clarity

Enduring, high-performing teams aren't built by heroes carrying loads of monkeys, they're built by consistent, sustainable management practices anyone can learn.

Why do you have a monkey on your back?

A metaphor from the classic 1974 Harvard Business Review article "Management Time: Who's Got the Monkey?"

Time is an irreplaceable resource, and every problem is a monkey that takes your time.

You might be familiar with the experience: someone sends you a message or brings something up in a meeting "I'm blocked, I don't know how to do this" - and the problem/monkey "jumps from their back onto yours".

Each time you accept the monkey, you've agreed to carry it. You now own the next action.

Management is Results and Retention

"Manager Tools" frames the manager's job as "results and retention".

The organization wants deliverables within a timeframe, but people are not robots. Managing people means managing emotions too.

Requests range from "I wanted a seat near the window" to "I'm going to leave this place if I have to work with X again".

Problems may not be inherently stressful. Yet how we feel about something can be entirely different from what it may take to complete it.

Your job is to bridge the gap. And that takes energy aka "Spoons"...

I (just) can't get started

"Spoon theory" originated as a way to describe the capacity available to people. We start the day with a finite number of spoons, and someone with chronic illness has "spoons" taken by their illness before they even start their day.

https://en.wikipedia.org/wiki/Spoon_theory

Every task, conversation, and decision costs spoons. When they're gone, they're gone.

When you're low on spoons, activation energy feels insurmountable.

Manage your spoons first

Before you can help anyone else, check your own reserves. Do you have enough spoons today to deal with what might come up?

This is not selfish. You cannot pour from an empty cup. If you're already depleted, you won't have the patience for the small stuff, let alone the capacity for a crisis.

Sometimes the right answer when asked something is "Let me think about it and get back to you".

(I'll have another blog post about time management and a "calendar energy audit")

Reading other people's spoons

Lead with empathy: do they have any spoons left today?

Have you built enough trust that your 1:1s are places where people can be vulnerable?

Where they can tell you "I'm not really up for this today" - and you actually listen?

One of the worst 1:1s I ever had was where I asked how someone was doing and they told me they weren't up for a meeting, and for some reason we kept talking.

Five minutes later they were over-sharing work anxiety, in tears. We agreed to end the meeting and mutually forget everything that was said.

My mistake: I didn't truly listen. I didn't realize that given my role power, they may have thought we had to keep talking.

The lesson: when someone signals "low on spoons", believe them. Don't push through just because the calendar says you have a meeting.

Now when I hear someone say they're not feeling well I offer to reschedule or focus on 1 urgent thing and end early.

Give a spoon

Sometimes you can give a spoon: with good listening skills and the ability to reframe a situation, you can sometimes actually restore someone's capacity.

"My memory was that you let me know your highest priority was sitting next to Y, and they got the window seat. They're really happy there, and I'm hoping you're going to be a great combo."

Helping someone see a situation in a positive light, or not in a zero sum way, is one of the highest-leverage things a manager can do. It costs you relatively little and can shift their entire day or even career.

Protect the team, not just the individual

When you notice dysfunction, especially if you care deeply about people, you may be tempted to endure a situation "a little while longer".

Look back at your 1:1 notes: how many times, and how many people, have brought up the same issue?

That’s spoons being subtracted from everyone: yourself, them, and the team.

Your job isn't only to support individuals. It's also to protect and support the team.

Sometimes everyone benefits from addressing the dysfunction directly instead of absorbing it privately.

I had a manager in my team who knew a direct report wasn't carrying their weight. It was even more difficult as they'd previously been peer-level colleagues.

Once we'd gotten through reviewing the data objectively, and a coaching discussion, it was the realization his other direct reports had been indirectly bringing the issue up that became the catalyst for change.

Once the situation resolved the manager and team were not surprised by the outcome and became healthier and stronger.

Helping on a task versus Helping Growth

A classic first-time manager mistake: taking on your direct reports' work as "helping." You use your IC skills instead of your people management skills. It feels productive because you're good at the work.

But you're not helping, you're enabling dependency and stealing their growth opportunity.

Worse: you create an incentive structure where being "good at fixing things" means everyone brings you their things to fix. More problems, more monkeys.

https://blog.calm.com/blog/hero-complex

I can distinctly remember the one-on-one where my manager once told me "You're working harder than anyone in your team. As a manager, that may not actually be a good thing."

I also remember doing quite a bit of work to help prepare someone's promotion packet. When the promotion didn't happen, it hit me that I'd starting carrying their agency and career growth for them.

People need to build their own problem-solving capacity and self-advocacy - with support.

And every monkey you carry costs you spoons.

Coaching

Coaching is even more abstract than delegating. You're not doing the work. You're not even telling them how to do it. You're asking questions that help them figure out how they want to do it.

This order helps clarify motivation/importance before "What/How"

"What makes this important to work on?"
"What does a successful outcome look like?"
"What have you already tried?"
"What do you think is blocking you? What options are you considering?"
"What would you do if I weren't here?"
"What will you try next?"

(That last part makes it clear - the monkey stays with them)

Measure a leader by what their team does when their leader's absent

Delegation

Specifically for Delegation, rather than seeing it as "all or nothing", Jill Wetzler synthesized a useful framework from Alan Chapman and Jurgen Appelo's work which I paraphrased...

Level 1 - Tell: I make the decision. I also tell you what to do and how
Level 2 - Prep: You do the research, I make the decision (I should explain my reasoning)
Level 3 - Consult: I ask for your input and recommendation first, then we decide together
Level 4 - Needs Approval: Propose your plan and action, wait for my approval to continue.
Level 5 - Expect Feedback: You make the decision and execute but keep me updated, I may have feedback or objections
Level 6 - Informed: You own this - let me know how it goes
Level 7 - Escalate (if needed): I don't need to know the details but if something goes wrong seek help
Level 8 - Autonomous: You got this!

https://www.jillwetzler.com/resources/delegation

An important nuance: someone who's great at one thing might be new to a domain or team or project, adjust your level conservatively and accordingly

Concretely for software engineers

Higher levels of delegation means more autonomy; more experienced folks should be able to take on complex things with high autonomy...

Tell: Giving an intern a fully laid out (small) task with prescriptive How
Prep: Asking a junior developer to take on a bug fix
Consult: Having an engineer work on their first architecture/design proposal
Needs Approval: A remediation for a production incident (note this is "pairing as an ops best practice", and it's similar to "yes we do Pull Requests")
Expect Feedback: An engineer leading a small project
Informed: A senior engineer leading a project
Escalate (if needed): A senior engineer working on a feature (in a domain they know)
Autonomous: An engineer fixing a small bug (routine, low risk)

Because AI and Agents

2025 addendum

Delegating work to AI Agents is becoming common so here are some examples of how I apply this framework to successfully work with agents...

Tell: how to update an unusual core data schema - high risk, large blast radius, difficult-to-recover
Prep: Picking the architecture and tech stack - high risk, large blast radius, not-verifiable
Consult: Working on a large feature - medium risk, large blast radius, hard-to-verify result
Needs Approval: Working on a small feature - some risk, small blast radius, verifiable result
Expect Feedback: Working on a small bug fix - some risk, small blast radius, easily verifiable result
Informed: Writing unit tests - low risk
Escalate (if needed): Updating dependencies - low risk, trivial verification
Autonomous: Updating documentation (have this as a daily cron job) - very low risk/impact

Summary

A manager is great when inspiring and elevating people. Ensure you're in the right frame of mind to be that leader.

Don't be a hero who accepts all the monkeys; they steal all of your spoons!

Give people spoons. Coach them to tame their monkeys.

Over-communicated and mutually understood delegation makes ownership even clearer.

CircleCI for a Pelican static Github site

2021-09-03T21:00:00-07:00

When I realized my previous CI/CD vendor had finally broken something (after 7 years travis.org incompatibly became travis.com) I decided out with the old and in with the new. Also, Python2 was deprecated so onto Python3!

So this article is a non-comprehensive reprise of How to Setup a Static Site with Github Pages...

Luckily I already had dabbled in CircleCI so the fundamentals were an easy copy paste: https://blog.john-pfeiffer.com/using-circleci-as-continuous-integration-and-continuous-deployment/

Roughly we will:

have python
install dependencies (pelican)
(optional) install plugins
get blog source markdown (ideally from version control like github)
run build commands (to convert markdown to html)
push to a second github repository

A new Pelican Dev Environment

unfortunately getting Docker setup is outside the scope of this tutorial

Just a practice run so you get a feel for it:

sudo docker run --rm -it cimg/python:3.8 /bin/bash
python --version
python -m pip install "pelican[markdown]" beautifulsoup4
exit

an ephemeral docker container with interactive bash shell, cimg is the CircleCI optimized docker image, https://hub.docker.com/u/cimg versus https://hub.docker.com/r/circleci/python/

Installing Pelican is documented: https://docs.getpelican.com/en/latest/

Using pelican-quickstart for a skeleton

assuming you are still in your docker container shell

PYVER=$(ls /home/circleci/.pyenv/versions | grep 3.8)
cd "/home/circleci/.pyenv/versions/$PYVER/bin/"
./pelican --version
mkdir yoursite
cd yoursite
../pelican-quickstart
  Welcome to pelican-quickstart v4.6.0.
  This script will help you create a new Pelican-based website.
  ... (accept all defaults) ...
  Done. Your new project is available at /home/circleci/.pyenv/versions/3.8.11/bin/yoursite

A little directory dancing in the container to get to the pip installed pelican binary

You still need an article with a minimum of content...

vim yoursite/content/my-article.md

Title: My Article
Date: 2021-09-09 22:22
Category: Technology

Hello world.

Generate html from markdown

./pelican yoursite/content/

creates an "output" directory with the HTML and all the index and other pages also updated

my-article.html
archives.html
authors.html
categories.html
category
feeds
index.html
tags.html
theme

Docker volume with your pre-existing blog content

sudo docker run --volume /home/ubuntu/blogsource/:/home/circleci/blogsource --rm -it cimg/python:3.8 /bin/bash
mkdir /home/circleci/OUT
PYVER=$(ls /home/circleci/.pyenv/versions | grep 3.8)
cd "/home/circleci/.pyenv/versions/$PYVER/bin/"
./pelican /home/circleci/blogsource/pelican-project/content -o /home/circleci/OUT -s /home/circleci/blogsource/pelican-project/publishconf.py

Docker will mount your local directory with pelican project markdown mapped to "/home/circleci/blogsource" in the docker container

If you do not specify a new "output directory" then pelican may get confused if there is already content in "pelican-project/output"

Linking Github to CircleCI

assuming you have your blog markdown in a Github repository and logged into CircleCI and authorized it for access to your Github repos

Click on the CircleCI button for your source code: "Setup a Project"

Use CircleCI's template for a configuration (a python project)
customize it: https://circleci.com/docs/2.0/executor-types/
trial and error to get various commands in the docker container right (use the Dev environment above)
There is a live-config-editor "Edit Config"

save and merge (.circleci/config.yml)

Now your source code (markdown) for your (pelican) blog should have a CircleCI configuration like the following:

.circleci/config.yml

version: 2.1
jobs:
  build:
    resource_class: small
    docker:
      - image: cimg/python:3.8
    steps:
      - checkout
      - run:
          name: Install Pelican and Build Content
          command: |
            python -m pip install "pelican[markdown]" beautifulsoup4
            pip freeze | grep pelican
            cd ..
            pwd
            ls -ahl /home/circleci/project
            find . -type f -iname pelican
            PYVER=$(ls /home/circleci/.pyenv/versions | grep 3.8)
            cd ".pyenv/versions/$PYVER/bin/"
            ./pelican --version
            mkdir -p /home/circleci/OUT
            ./pelican /home/circleci/project/content -o /home/circleci/OUT -s /home/circleci/project/publishconf.py
            ssh-add -D

      - add_ssh_keys:
              fingerprints:
                          - "4e:c1:a6:83:...:cc"
      - run:
          name: Publish to GitHub Static Site
          command: |
            cd ..
            ls -ahl
            git clone git@github.com:johnpfeiffer/johnpfeiffer.github.io.git
            cd johnpfeiffer.github.io
            git config user.email "me@john-pfeiffer.com"
            git config user.name "John Pfeiffer CircleCI"
            git checkout master
            cp -a /home/circleci/OUT/* .
            git status
            git add --all .
            git commit --allow-empty -m "CircleCI publishing $CIRCLE_BUILD_NUM from sha $CIRCLE_SHA1"
            ls -ahl ~/.ssh/
            GIT_SSH_COMMAND='ssh -v -i ~/.ssh/id_rsa_4ec1a683...cc' git push origin master

Everything from add_ssh_keys and below should only be added once you have completed the SSH Key steps below

Removing all SSH keys from the CircleCI agent is because we're done checking out this repo and need to not confuse Git later

Pushing to a Github Page repository

The target for all of this has been your Github static page: https://pages.github.com/

Create a new SSH key dedicated to only this purpose

ssh-keygen -t rsa -b 4096 -C "CircleCI Deploy Key with Write Access" -f /tmp/cikey

Note that this key is just for CircleCI for just this one repo

Navigate to the "Deploy keys" section of the Settings of the Repo in github that will receive the pushes e.g https://github.com/johnpfeiffer/johnpfeiffer.github.io/settings/keys

click on "Add deploy key"
Paste in the cikey.pub file contents
Ensure the "Allow write access" checkbox is checked

In CircleCI: Go to your source project (under your username) and choose the "Project Settings" button, then subsection SSH Keys e.g. https://app.circleci.com/settings/project/github/johnpfeiffer/source.johnpfeiffer.github.io/ssh

Copy the private "cikey" (which starts with "-----BEGIN RSA PRIVATE KEY-----") into your buffer.

Do not share or paste your private key anywhere else than CircleCI

Add an Additional SSH Key "Add SSH Key" Hostname: githubstaticpage Private Key: -----BEGIN RSA PRIVATE KEY-----...

Now copy the "Fingerprint" of the key e.g. something like 4e:c1:... This is what goes in your circleCI config

CircleCI config explained

This portion is your CircleCI agent manually checking out the actual "Pages" HTML repository:

            git clone git@github.com:johnpfeiffer/johnpfeiffer.github.io.git
            cd johnpfeiffer.github.io
            git config user.email "me@john-pfeiffer.com"
            git config user.name "John Pfeiffer CircleCI"
            git checkout master

It is important to copy the new HTML content in...

cp -a /home/circleci/OUT/* .

technical debt: should use the rsync command to actually reflect removed content too

Here is where the previously created SSH Key and fingerprint are explicitly used to specify to git to use that ssh key to push to github...

        GIT_SSH_COMMAND='ssh -v -i ~/.ssh/id_rsa_4ec1a683...cc' git push origin master

ssh verbose shows you which Identity/Key is being used to deploy/push to Github

References

Pelican Plugins

My site still uses the old plugins which I "vendored" into the actual repo for simplicity:

https://github.com/johnpfeiffer/source.johnpfeiffer.github.io/tree/master/plugins
https://github.com/getpelican/pelican-plugins

10 MB of legacy plugins at "getpelican" because they still work

Instructions on Pelican Plugins in general (and the new way that I have not yet adopted)

https://docs.getpelican.com/en/latest/plugins.html

Thankfully this person documented how to use Pelican4.6 with MARKDOWN for the built in Table of Contents (TOC):

https://cloudbytes.dev/articles/add-a-table-of-contents-using-markdown-in-pelican

Using CircleCI as continuous integration and continuous deployment

2021-04-14T20:43:00-07:00

I avoid providing free advertising for products but often end up writing about how I have leveraged free products. =]

I have many posts over the years using vmware, openshift, heroku, google app engine, aws elastic beanstalk, ec2, aws lambdas, github, bitbucket, bitbucket pipelines, docker, digital ocean, and many more things. (all lowercase ;)

I wanted to document how I successfully migrated some of my other hobby projects to CircleCI so future me (and anyone else) can more easily replicate the steps because CircleCI has a free tier for CI/CD (continuous integration and continuous deployment).

I make no guarantee that CircleCI will continue to be free in the future (but if stops being free I will likely write another blog post about how to use a different service ;)

Why I chose CircleCI

Free (even more free for open source projects)
Straightforward documentation with good examples
Integrated with github (and bitbucket)
It just works (relatively quick execution and feedback loop)

CircleCI Build Terminology

A Project maps to a code repository
You configure each Project to do certain things when a new commit occurs.
A pipeline are all the things that happen when your Project is triggered (i.e. new code or manual re-run).
A workflow is the definition (and execution) of all the jobs in a pipeline; note that jobs can run in parallel.
A job is a collection of steps that are going to happen (i.e. checkout code and run a command)
A job must have an execution environment (i.e. a docker container)
A step is doing a single thing (i.e. checkout code)
https://circleci.com/docs/2.0/concepts/?section=getting-started

Getting Started by Authorizing CircleCI

Start with the source code: setup a repository (i.e. in github) , e.g. https://github.com/johnpfeiffer/stringsmoar

Go to CircleCI's login page and choose to "Sign Up" https://circleci.com/signup/

For the paranoid like me you can choose to only share your public github repos

Once you use the oauth-like permissions screen that provides your username and password to Github so it can authorize CircleCI to access all of your bits.

This is the one time you "authorize all the access", everything afterwards are config files that are fine to be in public code repos.

In Github you can review what applications have access to your github account's source code with https://github.com/settings/applications "Authorized Oauth Apps"

In CircleCI you can now see what projects you can setup builds, apparently segregated by "organization" https://app.circleci.com/projects/

Setup a Project

Not too surprisingly the UI then displays a list of all of the repos https://app.circleci.com/projects/project-dashboard/github/johnpfeiffer/

"Set Up Project" makes sense but for some odd reason the terminology is to "follow a project" when something has been configured by someone else in your organization

The UI will attempt to helpfully suggest a configuration yaml based on auto-detecting the repository's programming language.

For Golang CircleCI presumes you are using go mod so I guess I ought to upgrade my old code repos now that there's an official standard

If you choose the pre-generated configuration file CircleCI will attempt to commit and push that new .circleci/config.yml into your repo and then start a Build.

Instead of the auto-generated configuration you can select Use Existing Config (in which case you should have already uploaded into github remote your preferred CircleCI reference)...

your-repo/.circleci/config.yml

version: 2.1 # https://circleci.com/docs/2.0/configuration-reference
jobs:
  resource_class: small
  build:
    working_directory: ~/repo # the circleCI default for where code is checked out to in the docker build container
  docker:
    - image: cimg/go:1.21.4
  steps:
    - checkout
    - run:
        name: Run unit tests
        command: |
          go test -v ./...

Double checking your Golang Version

The wonderful thing about Docker is the extra transparency. In this case we might want to double check the version of Golang that is being used by the build agent.

You can download and execute the same environment locally:

docker run --rm -it circleci/golang:1.21.4
go version

go version go1.21.4 linux/amd64

https://hub.docker.com/r/circleci/golang/

Cannot find main module is a common error for older golang code repos

go: cannot find main module

This means you have created a golang repo awhile back (Golang 1.15 and older) but are now using a newer/later Golang binary...

To resolve the issue run this golang command in the top directory of your source code:

go mod init

This will create a go.mod file in your repository that allows dependencies to be properly resolved (and go test which implicitly uses "go mod" to execute successfully)

Once that go.mod is committed and sent up to the Main branch in your github repo then CircleCI build will detect it and your "go test" step during the build/test steps will stop failing

https://golang.org/ref/mod#mod-commands

Tweaks to your CircleCI Config

After you have successfully run a build then the UI will show you:

how long the build took
what git sha commit kicked off the build
commit message
all the steps executed and output, etc.

https://app.circleci.com/pipelines/github/johnpfeiffer/stringsmoar/7/workflows/39b3f880-2473-49d7-804d-d1364f08853e/jobs/9

Rerun a build in CircleCI

Sometimes it can take a bit to get used to the CircleCI UI, to drill down to a specific build your "breadcrumbs" will look like:

All Pipelines > your-projectname > branch (main) > workflow > build (4)

In that detailed output UI, to Rerun a build, choose the "Rerun" button from the beginning (or from a failed step)

Flaky tests aka intermittent failures is not resolved by re-running your build/tests all the time ;p

Specific Project Settings in CircleCI

Deploy Keys (specific to a repo) are a better security practice - read below on how to configure your poject to use them =)

In the CircleCI UI, for a given Project, the three little dots will allow you to choose how to configure the project

https://app.circleci.com/settings/project/github/johnpfeiffer/stringsmoar

The one annoying thing is that if you remove your 3rd party access creds in GitHub it's a pain to reconnect CircleCI

In the CircleCI configuration for a Project you should see a listing of SSH keys you have to remove (delete) the old "deployment key" there (which means CircleCI can no longer access github) then choose to re-add a deployment key (if you are signed into CircleCI with GitHub this will automatically generate it)

Or unfollow the project, and then re-follow the project (which will then have CircleCI use your initial OAuth authorization to generate a new SSH deployment key in GitHub).

Afterward you should see a new SSH key that CircleCI created in Github for this Project (the UI's both show the sha of the key but one is sha256 and the other is not)

https://circleci.com/docs/github-integration/#deploy-keys-and-user-keys
https://discuss.circleci.com/t/solved-permission-denied-publickey/19562 (someone else had the same problem and documented their solution)
https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys#deploy-keys
https://github.com/johnpfeiffer/stringsmoar/settings/keys

Picking the size of your build executor

By default CircleCI will choose executor size of "medium", if you want to save (free) credits then for smaller projects use "small" (or conversely if you need more cpu/ram choose a larger size)

jobs:
  build:
    resource_class: small
    docker:
      - image: circleci/golang:1.16

Good news: builds usually trigger almost instantly so all the little config tweaks have a super fast feedback loop

Outputting Test Coverage and Artifacts

CircleCI has an extra space in the UI to display specific test output or artifacts which makes it easy to see the most common pain points rather than digging through all of the build stages output.

your-repo/.circleci/config.yml

version: 2.1 # https://circleci.com/docs/2.0/configuration-reference
jobs:
  resource_class: small
  build:
    working_directory: ~/repo # this is a circleCI default for where code is checked out to in the docker build container
  docker:
    - image: circleci/golang:1.16 # https://hub.docker.com/r/circleci/golang/ , https://hub.docker.com/_/golang?tab=description

  environment:
    TEST_RESULTS: /tmp/test-results

  steps:
    - checkout
    - run:
        name: Run unit tests
        command: |
          go test -v ./...
    - run:
        name: Run code coverage
        command: |
          mkdir -p $TEST_RESULTS
          go test -coverprofile=c.out
          go tool cover -html=c.out -o coverage.html
          mv coverage.html $TEST_RESULTS
          go test -v ./... | go tool test2json > $TEST_RESULTS/test2json-output.json
          gotestsum --junitfile $TEST_RESULTS/gotestsum-report.xml

    - store_artifacts: # Upload files like code coverage html for later viewing https://circleci.com/docs/2.0/artifacts/
        path: /tmp/test-results
        destination: raw-test-output
    - store_test_results: # Upload test results for display https://circleci.com/docs/2.0/collect-test-data/
        path: /tmp/test-results

Artifacts will be deleted after 30 days but would be output like this: https://12-123862890-gh.circle-artifacts.com/0/raw-test-output/coverage.html#file0

The golang coverage.html as an artifact can be opened by your web browser and highlight in color specifically which code paths are covered by unit tests

the CircleCI golang docker container has the opensource helper "gotestsum" to generate junit style XML output from tests

JUnit XML or Cucumber JSON test metadata files

From here on out you should hopefully have only Green Builds!

TODO: an article about how to do continuous deployment (maybe CDK and AWS?)

Bitcoin is a bad business model and when to invest in your Cost Center

2021-01-23T08:44:00-08:00

Bitcoin is a bad business model

A good business model creates value. Eventually it will create good cashflow (i.e. more money in your bank account each month).

While I understand that some may find the concept of an "unowned" virtual currency appealing, creating bitcoin (aka "mining") in and of itself is not differentiated and does not create value.

Buy (or rent) a server, run software with an algorithm, hope that your 1s and 0s are special enough to be "worth something"

Note that the control of the "coins or transactions accepted" is defined by an external entity, so anyone can use the protocol https://en.wikipedia.org/wiki/Satoshi_Nakamoto

Coinbase (https://www.bloomberg.com/profile/company/0776164D:US) has a business model of creating a platform for speculators to gamble (on many crypto-currencies), but just like selling shovels during a gold rush, they are not in the virtual hills digging, instead they are comfortably collecting a huge pile of gold dust.

With Bitcoin, there is no "moat" (https://www.investopedia.com/terms/e/economicmoat.asp); no way to grow your business other than increasing efficiency and reducing costs. It is not inherently more valuable to have more virtual 1s and 0s when there are an infinite number of possible random numbers, nor is it particularly valuable to write your own IOUs that are guaranteed by random other people writing IOUs.

You cannot add so much efficiency to a Cost Center that it suddenly becomes a business model of its own.

Creating Value

Growing food is a way of creating value: people need food and someone creates it.

Do people need cryptocurrency? Creating things that people want can be valuable (see Art).

So Bitcoin is valuable as a way to digitally exchange value, and because other people think it might be more valuable later.

A good business model should create more value, ideally in a novel and innovative way. Intrinsically then Bitcoin and mining Bitcoin are not creating value but "enabling value".

The more you create Bitcoin the more you are helping someone else capture value ;)

Without bitcoin how would crime in the digital world have been able to innovate? So maybe create Bitcoin as a good "loss leader" or a "cost center"?

Cost Center and Profit Center

Like many ideas, it is not the thing, it is just a way to think about things

The "Profit Center" concept was coined by Peter Drucker in 1945 and it refers to thinking about the role a department or group (or project) plays in a company.

One part of the company is making something of value, the core of the business that you are selling.

As an example, "creating and selling food" is the profit center of some example business.

So the management of the people who are "creating and selling food" is a Cost Center. (Yup, your manager is costing the business money!).

Transporting the food is another cost center; a constant question is then "How to get transportation costs down to zero?"

One might even posit that having people create and sell food is a cost center: why not robots? (of course then whomever maintains the robots is the cost center)

Since Profit Centers are what everyone seeks, they are harder to find. They depend on Customers, on satisfying demand, on product market fit, etc... including many things that may be beyond your control.

Cost Centers tend to provide the illusion of control (things you are paying for) and so are eminently more enjoyable to tweak and revise, presumably with less risk.

You cannot save your way into riches

If you save $1 a day from some habit or vice, you will have saved $7,300 in 20 years.

Despite the sage advice to save money, one should be aware that compound interest is not magic.

If you start with $10,000 and compound annually at 10% interest for 20 years you will have $67,275. To further debunk: starting with $10k is not entirely trivial, and I suspect nobody will be famous for having $67k 20 years from now. In fact, with inflation, I suspect very many people will have at least $67k which is a far cry from $1000k (aka $1m), and even a millionaire isn't as glamarous a title as it used to be.

https://www.investor.gov/financial-tools-calculators/calculators/compound-interest-calculator

Therefore to be rich and profitable you must make money. (Perhaps an unsurprising conclusion but for the penny pinchers)

So a Cost Center improvement of 100%, on a company with $0 revenue, only reduces how fast your plane plummets to the earth.

When to invest in a Cost Center

There is an opportunity cost to being alive

Time is not infinite. If you are focusing on your cost center you may be neglecting your profit center, yet there are definitely moments when the investment is worth it.

When the Cost Center deficiencies become a distraction: if your business does not have electricity or your employees are lost in confusion and chaos then it seems much more challenging to expect your Profit Center to succeed.
When the Cost Center is a strategic differentiator: it helps create the moat that others cannot cross. As an example: Amazon began offering free shipping (and escalating into 2-day, then 1-day, then hours etc.), which certainly costs a lot to do. Yet it reduced friction to purchasing (and enabled the very profitable Amazon Prime subscription service) which increased sales, strategically raising the bar for competitors to "have a digital platform for buying things". https://www.vox.com/recode/2019/5/3/18511544/amazon-prime-oral-history-jeff-bezos-one-day-shipping
When the Cost Center will become your Profit Center, see Slack's business pivot away from creating an online game (presumably for profit) into collaborative realtime messaging (an internal tool that cost them money to build and operate). https://mastersofscale.com/stewart-butterfield-the-big-pivot/
When you need to explain to an Investor (present or future) that you are not wasteful, i.e. efficiency as a proxy for smarts/effectiveness.
When you have revenue but your cost center is creating a risk to your cashflow, e.g. you are spending so much it seems unlikely you will be able to last long enough for your profit center to profit.

TODO: add more good reasons, but do it later to avoid over-investing in a cost center ;p

Bad Cashflow versus Bad Business

A bad business model will lose money until it is all gone. Interestingly it is possible to have a good business model and still go out of business.

Bad cashflow can kill a good business: you owe more money in a given month than you can pay your creditors and they want their payment now. Over a longer time horizon a good business model would have made everybody more money.

Infinite time horizons are often similar to ponzi schemes where someone has to keep being the "new sucker born every minute".

Thus the stereotype of a business person being conservative and frugal; they're controlling their cashflow, earning through profit over time.

Contrast with a stereotypical "great entrepeneur" who takes risks and creates profit.

Both approaches have the potential for being a bad business.

How can you discern when "losing money is fine"? There's the usual routine of examining timelines (months and up to a few years - further than that and your crystal ball is the "Profit Center"), then considering risks, growth projections, competition, etc.

One simple technical approach is to see how much improvement can occur to the Cost Center and the impact on the Profit Center.

If you can invest and immediately improve your cashflow then you have a straightforward plan for improvement.

As mentioned earlier, a bad business model does not have a Profit Center. Having no positive cashflow may be something that can be fixed.

Should you work in a Cost Center?

Sensibly a business needs to create revenue and reduce costs, so someone needs to work in the "Cost Center". If you are irresponsible and careless it seems unlikely you will be successful in a Cost Center.

You may genuinely enjoy innovating on efficiency; the larger the scale or harder the problem then potentially the greater the impact.
The company may have a cashflow issue and working in the Cost Center may save the company.
It may be that the opportunity to join a growing/successful company is in the Cost Center; don't look a gift horse in the mouth. (A rising tide lifts all ships).
Sometimes working in a Cost Center is a way to learn the foundations of a business - where are all the dependencies that created the situation for profit?
In the "DevOps" (and TDD) culture there's a deliberate attempt to remove the artificial silos that would have created a "cost center" (like QA or Operations) in the first place by distributing the work (and automating the toil); everybody should work in a Cost center!

Sorting in Golang

2020-09-30T00:00:00-07:00

We maybe take for granted that humans sort things: having a list of favorites e.g. "top ten songs".

Computers (and software) are exceptional for sorting since it is both boring and requires precision; especially for large data sets that are beyond the human brain.

Sorting things (not just numbers!) is foundational in Computer Science, https://en.wikipedia.org/wiki/Sorting_algorithm

Sorting is often considered a "free first step" (since sorting is "Big O n logn") for many problems that already would take a least "n logn" to solve.

Golang has made Sorting really easy and efficient. Both in terms of compute and memory

Verbose implementation of Sorting with Go Primitives like a Map

First, if you want to just stick with primitives, you can sort a particular item like this...

package main

import (
    "fmt"
    "sort"
)

// Person is defined by name and age
type Person struct {
    Name string
    Age  int
}

func main() {
    People := []Person{
        {Name: "Charles", Age: 25},
        {Name: "Alice", Age: 25},
        {Name: "Bob", Age: 24},
    }
    fmt.Printf("Original Listing: %v \n", People)

    // Example of manually sorting using extra primitive data structures
    AgeToPeople := make(map[int][]Person)
    for _, p := range People {
        // handle the edge case of not yet having a list entry in the map
        _, ok := AgeToPeople[p.Age]
        if !ok {
            AgeToPeople[p.Age] = []Person{p}
        } else {
            temp := AgeToPeople[p.Age]
            AgeToPeople[p.Age] = append(temp, p) // this maintains the pre-existing order on duplicates
        }
    }

    // Sort by a property and then dereference the lookup map
    // get the unique list of ages (each map key is unique)
    var Ages []int
    for age := range AgeToPeople {
        Ages = append(Ages, age)
    }

    sort.Ints(Ages)

    var PeopleSortedByAge []Person
    for _, age := range Ages {
        temp := AgeToPeople[age]
        for _, p := range temp {
            PeopleSortedByAge = append(PeopleSortedByAge, p)
        }
    }
    fmt.Printf("Sorted by age Listing: %v \n", PeopleSortedByAge)

    // Optionally create a subset of the "lowest N members"
    // this could be more efficient if it was applied during the PeopleSortedByAge loop
    PeopleSortedByAgeSubset := PeopleSortedByAge[:2]
    fmt.Printf("Stable Subset of Sorted by age: %v \n", PeopleSortedByAgeSubset)

This code is verbose but does provide visibility and control: Not modifying the original slice, the exact property used for sorting, explicit handling of duplicates aka the "stable sort" property

https://en.wikipedia.org/wiki/Sorting_algorithm#Stability

ok, I am "hand-waving" when we use the built in sorting of Integers but I'm not looking to re-invent the wheel...

To execute (or modify) the code yourself https://play.golang.org/p/A9IYxubb8yK

Very easy and fast way to Sort in Go

package main

import (
    "fmt"
    "sort"
)

// Person is defined by name and age
type Person struct {
    Name string
    Age  int
}

func main() {
    People := []Person{
        {Name: "Charles", Age: 25},
        {Name: "Alice", Age: 25},
        {Name: "Bob", Age: 24},
    }
    fmt.Printf("Original Listing: %v \n", People)

    sort.SliceStable(People, func(i, j int) bool { return People[i].Age < People[j].Age })
    fmt.Printf("Original Sorted using BuiltIn SliceStable: %v \n", People)

The anonymous function is passed in as a parameter to define how to compare elements in the slice and be aware that the Sort function modifies the original slice

This sorting capability comes built in with the Go Standard Library https://golang.org/pkg/sort/#SliceStable

Why sorting with Go is efficient

Leveraging a well established implementation of sort saves a lot of developer time (and provides some guarantee of correctness).

As a compiled language Go is performant in terms of compute (which often translates to reasonably fast wall clock time too).

The use of Slices means under the hood there can be pointers and references rather than full copies which reduces memory consumption.

https://blog.golang.org/slices-intro
https://blog.golang.org/slices
https://golang.org/src/sort/sort.go (quicksort but it could be any highly performant sort)

Sorting objects in Go with Customization

It is very powerful is to leverage Golang's interface capabilities and the Sort package.

Since Go uses composition instead of inheritance any arbitrary data structure can support Sorting.

Thus with a struct definition, then writing a few method definitions, one can sort a collection of objects.

package main

import (
    "fmt"
    "sort"
)

// Person is defined by name and age
type Person struct {
    Name string
    Age  int
}

func main() {
    People := []Person{
        {Name: "Charles", Age: 25},
        {Name: "Alice", Age: 25},
        {Name: "Bob", Age: 24},
    }
    fmt.Printf("Original Listing: %v \n", People)

    sort.Sort(ByAge(People))
    fmt.Printf("Original Sorted using a Customized Less: %v \n", People)
}

// ByAge is a list of Persons that can be sorted by age
type ByAge []Person

// Len implements the interface for Sort
func (p ByAge) Len() int {
    return len(p)
}

// Swap implements the interface for Sort
func (p ByAge) Swap(i, j int) {
    p[i], p[j] = p[j], p[i]
}

// Less implements the interface for Sort
func (p ByAge) Less(i, j int) bool {
    // customize sorting on age equivalence to use Name too
    if p[i].Age == p[j].Age {
        if p[i].Name <= p[j].Name {
            return true
        }
        return false
    }
    // otherwise ages are not equal, simply use age
    return p[i].Age < p[j].Age
}

This "strategy pattern" allows Golang to provide Sorting as an abstraction

https://en.wikipedia.org/wiki/Strategy_pattern#Strategy_and_open/closed_principle

The ability to customize logic in Less() is very powerful, yet callers need only pass an extra ByAge() to benefit.

To execute (or modify) the code yourself use https://play.golang.org/p/A9IYxubb8yK

If I ever put more time into various ideas or helper functions it will end up here https://github.com/johnpfeiffer/go-sort-example

Sorting in Reverse

There is a helper function that just reverses the sorting order, so once you can Sort (because you fulfill the Interface), then you can "reverse sort" =) alternatively you could use a for loop and iterate over a sorted slice, starting from index length-1 down to 0

package main

import (
    "fmt"
    "sort"
)

func main() {
    a := []int{4, 2, 1, 3}
    // convert a to an IntSlice so it can be Reverse Sorted
    sort.Sort(sort.Reverse(sort.IntSlice(a)))
    fmt.Println(a)
}

Typically this is how to control "ascending" or "descending" sorted order

Sorting a Map in Go

Sorting a map by Keys

To sort by keys is relatively trivial, for a given map "m" creating a slice and sorting it

m := make(map[string]string)
m["z"] = "zebra"
m["aa"] = "Aardvark"
m["B"] = "Bear"
m["a"] = "antelope"

for k := range m {
    sortedKeys = append(sortedKeys, k)
}
sort.Strings(sortedKeys)

for _, key := range sortedKeys {
    fmt.Printf("%s %s ", key, m[key])
}

Sorting a map by Values

Sorting a map by values is a little harder which is likely why a cursory internet search brings up far fewer code snippet examples ;)

// SortByValue returns the Keys in the sorted order of the Values (does not guarantee Stability)
func SortByValue(m map[string]string) (sortedKeys []string) {

// create a reverse lookup mapping that can handle collisions
// (duplicates) when the same value has multiple occurrences
ValueToKey := make(map[string][]string)
for k, v := range m {
    _, ok := ValueToKey[v]
    if !ok {
        ValueToKey[v] = []string{k}
    } else {
        temp := ValueToKey[v]
        ValueToKey[v] = append(temp, k)
    }
}

// sort the Values
var uniqueValues []string
for v := range ValueToKey {
    uniqueValues = append(uniqueValues, v)
}
sort.Strings(uniqueValues)

// get the corresponding Keys
for _, v := range uniqueValues {
    keys := ValueToKey[v]
    sortedKeys = append(sortedKeys, keys...)
}

return
}

The gotcha is that different Keys could have the same Value

Full source code examples: https://github.com/johnpfeiffer/go-sort-example

Using AWS CDK to configure deploy a Golang Lambda with APIGateway

2020-09-27T14:48:00-07:00

Infrastructure as Code helps guarantee the elusive determinism in infrastructure that we all seek in building applications and services. Often you can defer that complexity by using Heroku, Google App Engine, or other PaaS providers. But when you need to build something really complex (or with specific controls required including managing costs) then using IaaC to tame AWS Serverless reduces some of that pain.

AWS have created their own specific product (domain specific language) "CDK" which competes with the venerable Terraform and nicely focused Serverless.

All of these products use the json syntax of CloudFormation which is the foundational AWS syntax for describing resources

It is straightforward to iteratively setup and use AWS CDK (with the native Typescript syntax).

CDK vocabulary

It all begins with stacks.

From an object oriented perspective, a stack is a definition of components that can be instantiated. An "app" is a collection of related stacks.

Let's say you wanted your persistence (DynamoDB) in one stack, and your more execution oriented components (APIGateway and Lambda) in another stack, and your App might have environment specific parameters it needs to define and pass through to each stack.

This ability to use templates and customize how you structure things, reference other CDK files, etc. makes CDK very modular and re-usable (and yes Typescript is a programming language so you can have linters and tests).

Install the AWS CDK Tool

Have to use npm to install the cdk tool...

npm install -g aws-cdk

https://docs.aws.amazon.com/cdk/latest/guide/cli.html

Setup the project directories

mkdir cdk-example

cd cdk-example

Create a subdirectory to compartmentalize all the infrastructure code (from the rest of the application, i.e. don't mix the business logic with the infrastructure plumbing)

mkdir infra cd infra

cdk init app --language typescript

creates the default files in this directory for an empty CDK app

cdk ls

   InfraStack
Names are hard but whatever you pick, like "Infra", will then show up in the AWS resources everywhere related to this project

Install the dependencies for AWS resources that CDK will manage

Option 1: manually install dependencies (it will automatically insert this into package.json)

npm install @aws-cdk/aws-s3 @aws-cdk/aws-lambda

Option 2: write the lines into packages.json and run at the command line in the infra directory: npm install

Writing files first (and ensuring they are in version control) is a more IaaC pattern

"dependencies": {
  "@aws-cdk/aws-s3": "*",
  "@aws-cdk/core": "*",
  "source-map-support": "^0.5.16"
}

Write the actual Infrastructure Code

The initial application template creates an empty "class" that represents the "stack", we customize and replace that code with the following...

lib/infra-stack.ts has only 2 changes to the default file, the import of S3 and the new Bucket resource "MyExampleBucket"

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';

export class InfraStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new s3.Bucket(this, 'MyExampleBucket', {
      versioned: true
    });
  }
}

The import renames are important to be consistent in subsequent code, so if something is "cdk" then it is cdk.Construct

More complex applications will use a "main entry point" that can refer to specific files for various stacks. - https://docs.aws.amazon.com/cdk/latest/guide/serverless_example.html

Output CloudFormation from a CDK Typescript file

cdk synth

this command will output to the display the CloudFormation that will be sent/used by AWS

Resources:
  MyExampleBucket8D68EFCA:
    Type: AWS::S3::Bucket
    Properties:
      VersioningConfiguration:
        Status: Enabled
    UpdateReplacePolicy: Retain
    DeletionPolicy: Retain
    Metadata:
      aws:cdk:path: InfraStack/MyExampleBucket/Resource
  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:

To preview what will occur between changes to the .ts file...

cdk diff

Deploy resources that CDK has defined

cdk deploy

warning, really making a change in AWS (based on your creds)

InfraStack

InfraStack: deploying...
InfraStack: creating CloudFormation changeset...
(3/3)

Stack ARN:
arn:aws:cloudformation:us-east-1:409670809604:stack/InfraStack/b5167030-00fb-11eb-9f36-12f8925a37c4

Verify the new bucket was created

aws s3 ls | grep MyExampleBucket

this assumes you have installed the AWS CLI like sudo apt install awscli

aws cloudformation list-stacks | grep InfraStack

this listing will also show deleted Stacks

Making updates with CDK

A tiny snippet change to allow bucket deletion...

new s3.Bucket(this, 'MyExampleBucket', {
  versioned: true,
  removalPolicy: cdk.RemovalPolicy.DESTROY
});

Preview changes with diff

cdk diff

Stack InfraStack
Resources
[~] AWS::S3::Bucket JohnPBucket JohnPBucket8D68EFCA
 ├─ [~] DeletionPolicy
 │   ├─ [-] Retain
 │   └─ [+] Delete
 └─ [~] UpdateReplacePolicy
     ├─ [-] Retain
     └─ [+] Delete

Apply the new changes

cdk deploy

InfraStack: deploying...
InfraStack: creating CloudFormation changeset...

cdk destroy

aws s3 ls | grep MyExampleBucket

aws cloudformation list-stacks | grep InfraStack

CDK enables "InfrastructureAsCode" and command line (or scripted) resource management, yet you must still understand the intricacies of the domain (i.e. that s3 buckets have policies and do not get destroyed by default)

s3 buckets are designed by default to not delete with a Stack, you must change the removal policy to do so

APIGateway plus Lambda plus Go

The trick to using Golang with AWS Lambdas is that it is a compiled language. Many tutorials for lambdas (even for CDK) use javascript or python since those dynamic languages can just be put in "one more file" without a build step.

A tiny Go Web Request Handler

Put a simple placeholder Golang Lambda in place using the Gin web framework for convenience

mkdir cdk-example/examplefunction/ ; cd cdk-example/examplefunction/

vim main.go

package main

import (
        "context"
        "log"

        "github.com/aws/aws-lambda-go/events"
        "github.com/aws/aws-lambda-go/lambda"
        "github.com/awslabs/aws-lambda-go-api-proxy/gin"
        "github.com/gin-gonic/gin"
)

var ginLambda *ginadapter.GinLambda

// for convenience leverage the Go init startup concept to define a global web server object
func init() {
        // stdout and stderr are sent to AWS CloudWatch Logs
        log.Printf("Gin starting")
        r := gin.Default()
        r.GET("/ping", func(c *gin.Context) {
                c.JSON(200, gin.H{
                        "message": "pong",
                })
        })
        ginLambda = ginadapter.New(r)
}


// Handler is the function that executes for every Request passed into the Lambda
func Handler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
        return ginLambda.ProxyWithContext(ctx, req)
}

func main() {
        lambda.Start(Handler)
}

go mod init

Ensure dependencies (like the AWS SDK) are recognized by the Go package manager

go test

This is the simplest way to trigger downloading the dependencies (imported packages), you may have to create a tiny main_test.go in order to force this to work

https://blog.golang.org/using-go-modules

Manually build your go binary for AWS Lambda

This will create an output file named "main"

GOOS=linux GOARCH=amd64 go build -o main main.go

compile for the target arch , AWS Lambda ("firecracker") is Linux =]

https://www.usenix.org/system/files/nsdi20-paper-agache.pdf

zip examplefunction.zip main

AWS requires these precompiled binaries to in .zip format

CDK with a Golang Lambda

Focusing on just the "infra" subdirectory in our project:

cdk-example/infra/package.json
cdk-example/infra/lib/infra-stack.ts

cd cdk-example/infra/

First update package.json

"dependencies": {
  "@aws-cdk/core": "*",
  "@aws-cdk/aws-s3": "*",
  "@aws-cdk/aws-s3-assets": "*",
  "@aws-cdk/aws-lambda": "*",
  "source-map-support": "^0.5.16"
}

Do not forget to npm install

Next update the lib/infra-stack.ts

Layout the resources from the deepest dependency first, so in this case a place for the golang function to be zipped

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';
import * as lambda from '@aws-cdk/aws-lambda';
import assets = require("@aws-cdk/aws-s3-assets")
import path = require("path")

export class InfraStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Golang binaries must have a place where they are uploaded to s3 as a .zip
    const asset = new assets.Asset(this, 'ExampleFunctionZip', {
      path: path.join(__dirname, '../../examplefunction.zip'),
    });

    const handler = new lambda.Function(this, "ExampleFunction", {
      runtime: lambda.Runtime.GO_1_X,
      handler: "main",
      code: lambda.Code.fromBucket(
        asset.bucket,
        asset.s3ObjectKey
      ),
    });
  }
}

This crucial glue code indicates that the Lambda will be named "ExampleFunction", that it will get the binary (zipped) from S3, and that the handler expects to have a binary "main"

Note that CDK will handle actually uploading the .zip to an s3 bucket

cdk synth

outputting the CloudFormation is a quick way to valied the syntax and see any warnings

cdk diff

outputting and previewing the changes that will appear in AWS

cdk deploy

InfraStack: deploying...
[0%] start: Publishing 05e95f6b38c932a779e68a7a685e9950eca688e775c77f84787f6fa3e2ade474:current
[100%] success: Published 05e95f6b38c932a779e68a7a685e9950eca688e775c77f84787f6fa3e2ade474:current
InfraStack: creating CloudFormation changeset...
 (4/4)
 InfraStack

Stack ARN:
arn:aws:cloudformation:us-east-1:409670809604:stack/InfraStack/248d2960-0104-11eb-8cc5-0ac853a0932f

Use the AWS Console UI and visually look at the cloud formation stacks

e.g. https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/

When you filter and click on your new stack (e.g. InfraStack) you will be able to see the Resources associated with that stack

The default policy "AWSLambdaBasicExecutionRole" will actually allow the Lambda to output logs to Cloudwatch

CDK with an APIGateway integrated with a Go Lambda

Create the APIG and attach it to the Lambda:

First update infra/package.json

  "dependencies": {
    "@aws-cdk/aws-apigateway": "*",
    "@aws-cdk/aws-lambda": "*",
    "@aws-cdk/aws-s3": "*",
    "@aws-cdk/aws-s3-assets": "*",
    "@aws-cdk/core": "*",
    "source-map-support": "^0.5.16"
  }

npm install

Define the API Gateway in CDK (only adding a few more lines)

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-apigateway-readme.html#aws-lambda-backed-apis

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';
import * as lambda from '@aws-cdk/aws-lambda';
import assets = require("@aws-cdk/aws-s3-assets")
import apigw = require("@aws-cdk/aws-apigateway")
import path = require("path")


export class InfraStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Golang binaries must have a place where they are uploaded to s3 as a .zip
    const asset = new assets.Asset(this, 'ExampleFunctionZip', {
      path: path.join(__dirname, '../../examplefunction'),
    });

    const myhandler = new lambda.Function(this, "ExampleFunction", {
      runtime: lambda.Runtime.GO_1_X,
      handler: "main",
      code: lambda.Code.fromBucket(
        asset.bucket,
        asset.s3ObjectKey
      ),
    });

    // all routes (and REST verbs) will pass through to the lambda
    const api = new apigw.LambdaRestApi(this, 'examplefunction', {handler: myhandler});
  }
}

one gotcha is the ordering of imports, have the "path" one last

cdk synth

now the CloudFormation is very verbose =|

upon inspection you will see the type of integration Lambda is as desired, AWS_PROXY

cdk deploy

There is a warning about security because AssumeRole

InfraStack: deploying...
[0%] start: Publishing 05e95f6b38c932a779e68a7a685e9950eca688e775c77f84787f6fa3e2ade474:current
[100%] success: Published 05e95f6b38c932a779e68a7a685e9950eca688e775c77f84787f6fa3e2ade474:current
InfraStack: creating CloudFormation changeset...
  (14/14)
InfraStack

Outputs:
InfraStack.examplefunctionEndpoint65D9943D = https://abc123.execute-api.us-east-1.amazonaws.com/prod/

Stack ARN:
arn:aws:cloudformation:us-east-1:409670809604:stack/InfraStack/248d2960-0104-11eb-8cc5-0ac853a0932f

Verify your new API Gateway and Golang Lambda with CURL

The very helpful output means you can use CURL directly (as you get to ignore the whole "this apigateway needs to be deployed to a stage")

curl https://abc123.execute-api.us-east-1.amazonaws.com/prod/ping

you should get back either 404 or "pong" =)

To see how much more work it is to do (and understand) the API Gateway concepts...

https://blog.john-pfeiffer.com/localstack-apigateway-lambda-and-s3-integration-testing/

Localstack APIGateway Lambda and S3 integration testing

2020-03-05T21:10:00-08:00

One of the challenges with "serverless" is how to develop locally, especially things like running integration tests in your dev environment.

Imagine writing code for a lambda that reads and writes from S3, but without any AWS.

The AWS tool SAM does have a local mode but does not cover S3 nor Dynamo, etc.

A great tool to fill this need is Localstack. Since its interfaces are compatible with AWS it is an excellent proxy.

Mirroring environments of Development, Staging, and Production, along with Integration or Acceptance tests, are best practices that allow you to write code with confidence and catch issues much earlier (and therefore more cheaply) than "Using your Users as QA in Production"

Prerequisites

Background: my previous article about writing a lambda with Golang and deploying it to AWS.

https://blog.john-pfeiffer.com/go-faas-with-aws-lambda/

Install and configure the AWS CLI

In order to interact with Localstack we will use the AWS CLI, welcome to the beauty of Interfaces (API driven development ;)

sudo apt install awscli or for all the alternate installation options https://aws.amazon.com/cli/

Setup fake credentials...

this will overwrite any existing ~/.aws/ config or credentials

echo -e "[default]\n\
region=us-east-1\n\
output=json" > ~/.aws/config

echo -e "[default]\n\
aws_access_key_id=AKIAFAKE\n\
aws_secret_access_key=FAKE" > ~/.aws/credentials

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

Simplest Localstack Lambda Configuration

version: "3"

services:
  localstack:
    image: "localstack/localstack"
    ports:
      - "4574:4574" # lambda
    environment:
      - SERVICES=lambda
      - LAMBDA_EXECUTOR=docker
      - DOCKER_HOST=unix:///var/run/docker.sock
      - DEBUG=1
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

sudo docker-compose up

Creating task_localstack_1 ... done Attaching to task_localstack_1 localstack_1 | Waiting for all LocalStack services to be ready localstack_1 | Starting mock Lambda service (http port 4574)...

https://github.com/localstack/localstack

A very simple Go Lambda

mkdir task; cd task; vim task.go

creates a directory/package and file with the same name to be the lambda code

package main

import (
    "fmt"
    "github.com/aws/aws-lambda-go/lambda"
)

// MyEvent is a thing
type MyEvent struct {
    Name string `json:"name"`
}

// HandleRequest for an event
func HandleRequest(name MyEvent) (string, error) {
    return fmt.Sprintf("hi %s", name.Name), nil
}

func main() {
    lambda.Start(HandleRequest)
}

go build

outputs a binary file "task", if on MacOS you may want to cross compile with: GOOS=linux go build

zip task.zip task

lambda code uploads must be zipped in advance

Create-Function aka Upload the golang code to the localstack lambda

aws --endpoint-url=http://localhost:4574 lambda create-function --function-name=task --runtime="go1.x" --role=fakerole --handler=task --zip-file fileb://task.zip

localstack_1 | 2020-03-06T05:09:38:DEBUG:localstack.utils.common: Starting download from http://a46a9ed6f485:4574/2015-03-31/functions/task/code to /tmp/tmpfile.6f1d50ce9ccf62f3094d3c7f9eb82573/archive.zip (5060419 bytes)

localstack_1 | 2020-03-06T05:09:38:DEBUG:localstack.utils.common: Writing 1048576 bytes (total 1048576) to /tmp/tmpfile.6f1d50ce9ccf62f3094d3c7f9eb82573/archive.zip

localstack_1 | 2020-03-06T05:09:38:DEBUG:localstack.utils.common: Done downloading http://a46a9ed6f485:4574/2015-03-31/functions/task/code, response code 200, total bytes 5060419

https://docs.aws.amazon.com/cli/latest/reference/lambda/create-function.html

Note that we used a fake role name "fakerole", localstack does not enforce IAM roles or permissions

Get the dependency docker container that actually executes Golang

docker pull lambci/lambda:go1.x

Now you can invoke the Lambda with an input...

aws lambda --endpoint-url=http://localhost:4574 invoke --function-name task --payload='{"Name": "world"}' --region=us-east-1 myout.log

{ "StatusCode": 200 }

localstack_1 | 2020-03-06T05:45:22:DEBUG:localstack.services.awslambda.lambda_executors: Lambda arn:aws:lambda:us-east-1:000000000000:function:task result / log output: "hi world"

Other useful commands for updating or deleting your localstack lambda

aws --endpoint-url=http://localhost:4574 lambda list-functions

aws --endpoint-url=http://localhost:4574 lambda update-function-code --function-name=task --zip-file fileb://task.zip aws --endpoint-url=http://localhost:4574 lambda delete-function --function-name task

API Gateway and Lambda with Localstack

A very simple bit of "handler" code to exemplify the AWS Lambda Proxy Integration

package main

import (
    "context"
    "fmt"
    "net/http"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
)

// HandleRequest for an event https://docs.aws.amazon.com/lambda/latest/dg/golang-handler.html
// input object defined here https://godoc.org/github.com/aws/aws-lambda-go/events#APIGatewayProxyRequest
func HandleRequest(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
    fmt.Printf("DEBUG: %#v", req)

    // https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-output-format
    return events.APIGatewayProxyResponse{
        StatusCode: http.StatusOK,
        Body: req.Body,
    }, nil
}

func main() {
    lambda.Start(HandleRequest)
}

https://docs.aws.amazon.com/lambda/latest/dg/golang-handler.html

How to parse the inputs provided to the Lambda by the AWS API Gateway https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-input-format

Output for a (Proxy Integration) Lambda with APIGateway needs a specific JSON format https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-output-format The custom reponse headers are where CORS can be configured

GOOS=linux go build && zip task.zip task

aws --endpoint-url=http://localhost:4574 lambda update-function-code --function-name=task --zip-file fileb://task.zip aws --endpoint-url=http://localhost:4574 lambda list-functions

arn:aws:lambda:us-east-1:000000000000:function:task

Again invoke the Lambda, with an input... aws lambda --endpoint-url=http://localhost:4574 invoke --function-name task --payload='{"body": "foobar"}' --region=us-east-1 myout.log

{"StatusCode": 200}

aws --endpoint-url=http://localhost:4567 apigateway create-rest-api --name myapi

{
    "createdDate": 1583558847,
    "apiKeySource": "HEADER",
    "tags": {},
    "name": "myapi",
    "endpointConfiguration": {
        "types": [
            "EDGE"
        ]
    },
    "id": "29a3p9encp"
}

That "id" of this REST API is important throughout the rest of the commands

aws --endpoint-url=http://localhost:4567 apigateway get-resources --rest-api-id 29a3p9encp

{
    "items": [
        {
            "id": "62wy7bzofu",
            "resourceMethods": {
                "GET": {}
            },
            "path": "/"
        }
    ]
}

The "id" for the "/" resource is used as the "parent" for adding a "child" resource

aws --endpoint-url=http://localhost:4567 apigateway create-resource --rest-api-id 29a3p9encp --parent-id 62wy7bzofu --path-part mywidget

{
    "pathPart": "mywidget",
    "resourceMethods": {
        "GET": {}
    },
    "id": "jylycd8v4u",
    "parentId": "62wy7bzofu",
    "path": "/mywidget"
}

We have created a REST resource /mywidget

aws --endpoint-url=http://localhost:4567 apigateway put-method --rest-api-id 29a3p9encp --resource-id jylycd8v4u --http-method GET --authorization-type NONE

{
    "authorizationType": "NONE",
    "httpMethod": "GET"
}

Now /mywidget does not require any authentication for GET requests

aws --endpoint-url=http://localhost:4567 apigateway put-integration --rest-api-id 29a3p9encp --resource-id jylycd8v4u --http-method GET --integration-http-method POST --type AWS_PROXY --uri arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:000000000000:function:task/invocations --passthrough-behavior WHEN_NO_MATCH

{
    "type": "AWS_PROXY",
    "httpMethod": "POST",
    "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:000000000000:function:task/invocations",
    "integrationResponses": {
        "200": {
            "statusCode": 200,
            "responseTemplates": {
                "application/json": null
            }
        }
    }
}

This connects the AWS API Gateway GET /mywidget calls as a "proxy" (passing through the request) to the specified Lambda POST is required https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html

aws --endpoint-url=http://localhost:4567 apigateway create-deployment --rest-api-id 29a3p9encp --stage-name foobar

{
    "createdDate": 1583565386,
    "description": "",
    "id": "mbe3fwe0pw"
}

This actually activates the endpoint for traffic

List everything that we have created...

aws --endpoint-url=http://localhost:4567 apigateway get-resources --rest-api-id 29a3p9encp

{
    "items": [
        {
            "resourceMethods": {
                "GET": {}
            },
            "path": "/",
            "id": "62wy7bzofu"
        },
        {
            "resourceMethods": {
                "GET": {
                    "methodIntegration": {
                        "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:000000000000:function:task/invocations",
                        "type": "AWS_PROXY",
                        "httpMethod": "POST",
                        "integrationResponses": {
                            "200": {
                                "responseTemplates": {
                                    "application/json": null
                                },
                                "statusCode": 200
                            }
                        }
                    },
                    "httpMethod": "GET",
                    "authorizationType": "NONE"
                }
            },
            "parentId": "62wy7bzofu",
            "id": "jylycd8v4u",
            "path": "/mywidget",
            "pathPart": "mywidget"
        }
    ]
}

The fully configured API Gateway and /mywidget resource linked to the Lambda code

https://docs.aws.amazon.com/cli/latest/reference/apigateway/test-invoke-method.html aws apigateway test-invoke-method --endpoint-url=http://localhost:4567 --rest-api-id 29a3p9encp --resource-id jylycd8v4u --http-method GET

curl http://localhost:4567/restapis/29a3p9encp/

{"id": "29a3p9encp", "name": "myapi", "description": null, "createdDate": 1583565481, 
    "apiKeySource": "HEADER", "endpointConfiguration": {"types": ["EDGE"]}, "tags": {}
}

curl -i http://localhost:4567/restapis/29a3p9encp/foobar/

curl -i http://localhost:4567/restapis/29a3p9encp/foobar/_user_request_/mywidget

Alternatively, HTTP_PROXY means you have to point it to a URI like https://example.com/my-existing-server

TODO: YAML with APIGateway + Lambda + S3 - Go code for Request Event and write to S3 - curl example integration test

Thanks to:

Career Development and Software Engineering Roles

2018-12-31T12:34:00-08:00

Career Development is Vague but Heavy

It can be vague where someone's career was before they met you and how they view their long term ambitions.

And when the work is getting done now it is easy to understand why future Career Development is one of the overlooked aspects of management.

Knowledge workers have a lot of job choices: if their current trajectory won't help them meet their life goals they will make changes that may come as an unwelcome surprise to their manager or organization.

Aligned interests, beyond salary-for-output (it would be even better if it was compensation-for-outcome), is one of the simpler ways to improve a product and reduce the costs of turnover. (i.e. https://en.wikipedia.org/wiki/Peopleware:_Productive_Projects_and_Teams)

When to discuss Career Development

One of the best tools for building relationships are 1:1's. ("Manager Tools" https://files.manager-tools.com/files/public/product-samples/One_on_One_Shownotes_0.pdf)

At least once a quarter (aka 3 months) a 1:1 session should cover Career Development. (i.e. a regular opportunity to align Company/Team Objectives or Goals with the individual's career)

One of the first questions I generally ask is "Where do you want to be in 2 years?"

2 years is just long enough to be aspirational but near enough we should be planning together how to get there

"Where" should not be reductively just about salary, instead it should encompass talent + experience + direction.

As a caveat, even the largest companies will not have every type of role possible. A good manager will provide guidance about what is immediately available (and research future possibilities).

Great managers grow careers even when it means the outcome will be an individual changing teams or organizations

An incomplete list of technology roles

There are too many roles related to technology to list them all but here are some important ones...

Software Engineer

Front End Engineer: works mostly with implementing UserInterface features and bug fixes (frequently using Javascript and including HTML and CSS)
Mobile Engineer: sort of a specialist "front end engineer" (assuming you agree that the focus is on Users), ideally experience with both iOS and Android but usually specializes in one
Fullstack Engineer: someone who can write/fix "front end" and "back end" code, often a generalist given the necessity of context switching between so many frameworks
Backend Engineer: specializing in APIs, Services, and often data storage (i.e. databases, files, etc.) - by contrast less focus directly on Users

Web Engineer is a less common title as there is less need to distinguish from a Desktop Engineer =p

Software Engineering gradations by years of experience

These levels/titles are really fuzzy because the years may more directly relate to time in an organization or specifically to a framework/techstack.

Also, with talent and extreme focus/specialization someone within 2 years can be producing at the level of a "principal engineer"

Junior (<2 years): focused on implementation under supervision
Senior (~between 2 and 5 years): able to independently gather requirements and run a project from start to finish
Principal (>5 years): influence at the organizational level

Generally it gets exponentially harder to "level up" (it would be exceedingly rare to have an organization that is all principal engineers). Interestingly, given the investment in time, training, and supervision required for junior engineers, it is also rare to have a "pyramid" shaped engineering organization.

One caveat: a Software Engineer will apply best practices in creating solutions and systems (i.e. bigger picture complexity and scale), whereas a Programmer title hints at a more "limited to code" role.

Example Diagram of Engineering Levels

Operations

Technology roles supporting the actual running of software and services...

SiteReliabilityEngineers (SRE): automating monitoring and stability as applied expertise via code (https://en.wikipedia.org/wiki/Site_Reliability_Engineering)
DevOps: automating the operations of systems and services (including monitoring, scripts and code writing, etc.) - leverages Dev skills with Ops experience (https://en.wikipedia.org/wiki/DevOps)
Operations: maintaining services/systems (hopefully still using automation and ideally transitioning to DevOps due to increased scale)
System Administrator: manual job of maintaining a small number of (virtual or physical) machines/systems

Data

Information has to be stored somewhere, this specialization has become even more prevalent with the exponential growth of "big data"...

Data Scientist: someone who uses mathematical tools like statistics, in conjunction with software and "big data", to answer questions (or discover insights)
Data Engineer: someone who builds infrastructure and tools that enables "Data Science", i.e. pipelines, warehouses
DBA (Database Administrator): someone who manages the data for an organization, often an expert in the tooling and optimization

Lateral Moves

Sometimes, besides all the levels and specializations, there are changes in career track (significantly different responsibilities and focus) that are big enough to be called a "lateral move".

Architect: larger systems are inherently complex and designing and communicating the interfaces, especially across multiple teams/services, is an essential "big picture" role
- Having eng experiences leads to designs with fast and effective implementations and prevents "ivory tower"
Engineering Manager: people are non-deterministic, they cannot be debugged, and yet they are a part of every successful organization. Someone who takes care of people, helps build and keep a team running smoothly, and achieves company outcomes. This is usually the role before Director and VP
- Having eng experience vastly increases credibility and the ability to estimate and deliver
Quality: someone who methodically thinks outside the box and regularly breaks boxes, ideally the most valuable boxes first
- Having eng experience means awareness of common shortcomings in frameworks/code and certain boundary conditions or real world scenarios (i.e. load)
Security: someone who thinks way outside of the box and finds ways to get inside of locked boxes
- Having eng experience allows for familiarity with architecture/framework/code flaws and automation of exploits
Product Manager: someone with passion and organizational skills who drives a product forward into the world
- Having eng experience allows for clearer and faster scope/timing discussions, and the ability to help the team with design or debugging
Designer: someone who loves the User and UI/UX and delivers highly desirable features and products
- Having eng experience allows for more effective collaboration and reduced time to market

Other Important Aspects

Career Development is not just getting the next position, it should also be about a lot of qualitative skills that lead to success:

Listening: hearing Users, Stakeholders, Teammates, etc. is one of the most important parts of solving the right problems
Communication: proposing good solutions, asking hard questions, raising concerns, writing skills
Time Management: individual deadlines, team success
Interviewing: being an interviewer is a completely different specialization - yet hiring cannot succeed without good technical interviewers
Business: understanding impact and the consequences of outcomes
Teamwork: multiplying the efforts of those around you

For each skill it is worth considering:

What is the individual's current level?
How important is it to them to improve?
What resources can be provided?
How will they be encouraged and held accountable for progress?

Did I miss something? Are there missing nuances or outright incorrect statements? Of course! =)

There is no one-size-fits-all definition to this NP-Hard problem of "people and organizations over time"

Iterative Improvement on this Article

My brash/blunt style is apparent in this article, but for a peek at how much better things get with collaboration and an editor check out the polished version:

https://web.archive.org/web/20200415071712/https://blog.helix.com/helping-your-engineers-grow/

Also, I have later stumbled across this similar idea of "the range of software careers" done as a podcast back in 2008:

Go FaaS with AWS Lambda

2018-06-11T19:48:00-07:00

The promise of creating functions that do not require server administration is amazing, the reality though includes a huge maze of vendor specific commands and frameworks (including permissions).

Why Function as a Service

In many ways most of the work in software engineering is "accidental complexity". Deployment. Input/Output Parsing. Monitoring. Logging. etc.

The "web request" model conquered (much like the historical domination of SQL) in the 90's as networks and the "inter-network" became popular (overwhelming the fragmented and isolated vendor specific applications approaches).

Yet what if your problem/data is not an input/output web request? (Yes of course there are GPUs and dedicated co-location centers...)

Or what about loads of traffic that spike in very extreme bursts (and diminish to almost nothing)?

The allure then is to have a function that runs on demand: truly elastic compute that does not require provisioning a server (not a virtual one and even without a web server).

Without mangaing an OS (and all that security headache!) and especially not paying for idle resources but instead only getting/paying for lots of compute when needed.

Of course there's no free lunch so as that complexity balloon gets squeezed it is the infrastructure/framework vendor that must "magically" provide the input, execute the function, and return the output.

The irony of "serverless" is that there's still a physical server, drivers, an operating system, and even a web framework, it's just someone else's (problem/revenue).

For "straightforward" web applications it may make more sense to directly offload the hosting/framework but still be in the same comfortable web server model (like Heroku)

Source Code

package main

import (
    "context"
    "fmt"
    "time"

    "github.com/aws/aws-lambda-go/lambda"
)

// MyRequest demonstrates an input value
type MyRequest struct {
    Value string `json:"value"`
}

// MyResponse helps illustrate how AWS Lambda auto
type MyResponse struct {
    Message string `json:"message"`
    Created string `json:"created"`
}

// HandleRequest https://docs.aws.amazon.com/lambda/latest/dg/go-programming-model-handler-types.html
func HandleRequest(ctx context.Context, req MyRequest) (MyResponse, error) {
    t := time.Now().UTC()
    return MyResponse{
        Message: fmt.Sprintf("hi %s", req.Value),
        Created: fmt.Sprintf("%s", t.Format(time.RFC3339))}, nil
}

func main() {
    lambda.Start(HandleRequest)
}

AWS CLI Deployment

To avoid some of the complexity with the CLI you may want to first dip your toes in with the Web UI creation of a Go Lambda: https://us-west-1.console.aws.amazon.com/lambda/home?region=us-west-1#/create?firstrun=true

Packaging for Upload

The easy part is creating a binary...

go build -o examplebinary

build into a single binary

zip deployment.zip examplebinary

wrap it up in a zip

Prerequisites

Installing the awscli may be a chore (if pip is broken), but sudo apt install awscli (from 16.04 should be advanced enough)

Unable to locate credentials. You can configure credentials by running "aws configure".

So in order to have credentials you must create a User in your AWS Account (with programmatic access only) which will generate an API key for you. (I suggest using the WebUI)

Do not use your "Root" Admin account in AWS for the API credentials, security best practice means creating at least one separate User

AWS Permissions and Roles

You may have to use the Web UI or some other mechanism to create the IAM role with the correct permissions

Gotcha: the minimum permission would be "lambda:CreateFunction"

This means the laborious process of creating a Policy that includes all the Lambda permissions (not very secure but it works)

iam:PassRole

Hilariously the pre-created roles listed in the documentation do not have that specific IAM extra permission, somehow by default an Admin of the account should deploy lambdas?

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "iam:PassRole",
                "lambda:*"
            ],
            "Resource": "*"
        }
    ]
}

Create an AWS user with API Credentials

So in order to have API credentials (~/.aws/credentials) you have already created:

A Policy
A Role that can use that Policy
A Group that has the Policy attached
A User that has credentials (and is in the Group)

The Actual Create Lambda Command

aws lambda create-function \
--region us-west-1 \
--function-name ExampleFunction \
--zip-file fileb://./deployment.zip \
--runtime go1.x \
--role arn:aws:iam::1234YOURACCOUNT:role/lambda-all \
--handler examplebinary

fileb format = file binary

The JSON response from creating the new lambda function (and uploading the zipped Go binary)...

{
    "LastModified": "2018-06-12T04:55:36.327+0000",
    "Version": "$LATEST",
    "FunctionArn": "arn:aws:lambda:us-west-1:1234YOURACCOUNT:function:ExampleFunction",
    "MemorySize": 128,
    "Runtime": "go1.x",
    "Role": "arn:aws:iam::1234YOURACCOUNT:role/lambda-all",
    "Description": "",
    "CodeSha256": "45R3BZKesxMM3AuZ96lS9UoiOEGX964oHD/J8QQfLfQ=",
    "Timeout": 3,
    "FunctionName": "ExampleFunction",
    "Handler": "examplebinary",
    "CodeSize": 2793060
}

https://docs.aws.amazon.com/cli/latest/reference/lambda/create-function.html

Trigger a Lambda Manually

Using the WebUI (AWS Console) you can select the function created with the CLI https://us-west-1.console.aws.amazon.com/lambda/home?region=us-west-1#/functions

By using the "Configure test events" (sometimes a dropdown on the right next to the Test button)

You can create a new "test event" , in this case a MyRequest (though any arbitrary extraneous JSON will be ignored)

{
  "value": "john",
  "key2": "value2",
  "key1": "value1"
}

this test event sends one value and some extraneous JSON keys as input to the lambda function

{
  "message": "hi john",
  "created": "2018-06-12T04:34:22Z"
}

the lambda function output is automatically converted from an object to JSON

Note that the WebUI also provides a way to upload a newer definition of the lambda function (zipped in a file)

Lambda Cron aka CloudWatch Scheduled Events

The real power of AWS Lambdas involves how it becomes glue for connecting lots of other AWS services (and transformations).

For example CloudWatch can be used to schedule/trigger a lambda.

In the WebUI you can add a trigger via the Designer (on the left)

Select "CloudWatch Events" -> Configure triggers (scroll down) -> Create a new rule

Configure a trigger, "Scheduled" (as opposed to Rate which is similar but different ;)

Name: "examplecron"

cron(16 * ? * MON-FRI *)

This would be on the 16th minute every hour every day every month on Monday through Friday (every year) cron(Minutes Hours Day-of-month Month Day-of-week Year)

Disabling the "example" Cloud Watch Event is as easy as toggling a radio button (this is an interesting way of controlling how/when your function executes)

https://docs.aws.amazon.com/lambda/latest/dg/tutorial-scheduled-events-schedule-expressions.html

Lambda Monitoring

In the same WebUI where you view the Lambda function (by name) you can click on Monitoring and click around to see Metrics (i.e. Lambda Cron aka CloudWatch Scheduled Events) actually "invoked" your function.

Of course if you haven't enabled CloudWatch Logging or more importantly created a "Log Group" then you get nothing.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html

More Info on Lambdas

Here's a full source code example: https://github.com/johnpfeiffer/aws-go-lambda Compare the unit tests to those from a full web server handler: https://github.com/johnpfeiffer/go-web-example/blob/master/controller_test.go

Apparently in order to test the tight integration with using AWS Lambda to consume events from other AWS Services there's a tool: https://aws.amazon.com/about-aws/whats-new/2017/08/introducing-aws-sam-local-a-cli-tool-to-test-aws-lambda-functions-locally

API Gateway

As a pre-requisite (and to leverage the fully integrated nature of the AWS ecosystem) we will create some data first...

And of course create a new Role that is linked to a Policy =(

https://console.aws.amazon.com/iam/home?region=us-west-1#/roles

Ensure you have a new Role (confusingly in the UI click through as a Lambda Service needing access) and choose the predefined Policy "AmazonDynamoDBFullAccess"

arn:aws:iam::123476797434:role/DynamoDBFull

Of course you'll have your own unique ARN and Security best practice would involve defining a more precise role (i.e. not giving permission for Auto Scaling)

Modify whom can assume the Role (using the AWS Web UI Console) https://console.aws.amazon.com/iam/home?region=us-west-1#/roles/DynamoDBFull?section=trust

If you receive an error later on it will be because of the Edit Trust Relationship, "API Gateway does not have permission to assume the provided role"

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": ["apigateway.amazonaws.com", "lambda.amazonaws.com"]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Click on "Update Trust Policy"

Readonly via DynamoDB

Use the WebUI (console) to create a new table https://us-west-1.console.aws.amazon.com/dynamodb/home?region=us-west-1

Create a table named: stocks
Primary Key: symbol (string)
Sort Key: timestamp (number)

Uncheck Auto Scaling "Read capacity" and "Write capacity" checkboxes

Do not use the default settings, we will disable autoscaling in order to save money and somewhat arrive at fixed cost

(Note that types are quite specific in DynamoDB, e.g. the binary "scalar" value can only contain up to 400KB of base64 encoded data)

After a few minutes of initialization select the "Items" tab then the "Create item" button to manually create an entry:

https://us-west-1.console.aws.amazon.com/dynamodb/home?region=us-west-1#tables:selected=stocks;tab=items

In the Web UI you will have to choose the dropdown (upper left) and change it from "Tree" to "Text" in order to enter a price

{
  "symbol": "VFINX",
  "timestamp": 1529697600,
  "price": 255.44
}

Using a historical stock price (in dollars) at the close of business

(Adding a second entry for 1529611200 and 254.95 can be helpful too)

After clicking Save and possibly a "Refresh data from server" (arrow lines in a circle on the right) you will see the new entries.

Setup a new API Gateway connected to DynamoDB

https://console.aws.amazon.com/apigateway/home?region=us-west-1#/apis/create

"New API"

API Name: stocks Endpoint Type: regional
Actions (Resource Actions) -> Create Resource

Resource Name: stock Resource Path: /{symbol}
Click "Create Resource" to save.
Actions (Resource Actions) -> Create Method (a dropdown appears in the WebUI below /{symbol}) , choose "GET"

Integration type: AWS Service Region: us-west-1 AWS Service: DynamoDB HTTP Method: POST (for interactions with DynamoDB) Action Type: Use action name Action: GetItem Execution role: arn:aws:iam::123476797434:role/DynamoDBFull Content Handling: Passthrough

The permissions/role ARN was created earlier and GET using POST is actually for interacting with DynamoDB

Click on the "Integration Request" for the /{symbol} - Get
Scroll all the way to the bottom and expand "Body Mapping Templates" so that you can click on "Add mapping template"

Content-Type: application/json

Confusingly after you create your type you must click on it to get a UI to define the JSON transformation, use the following:

{
  "TableName": "stocks",
  "Key": {
    "symbol": {"S": "$input.params('symbol')"},
    "timestamp": {"N": "1529697600"}
    }
  }
}

Emulating a direct low level request https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.LowLevelAPI.html

Back at the Web UI for "/{symbol} - GET - Method Execution" there is a button "TEST" that allows for validation of the API Gateway.

This is helpful for detecting early the inevitable bugs that crop up.

(Note: it does not seem to me that debugging an API this is way is actually faster...)

In the Test UI enter the stock symbol "VFINX" underneath {symbol} to emulate a GET request, then click on the "Test" button.

The Response Body on the right shows the response (that the client would see - though often revealing an error directly from DynamoDB)

Careful readers will notice that the timestamp variable is hardcoded in the transformation, todo: make the timestamp part of the query parameter"

Where is the Code?

One thing that is missing from this trivial example is how version control is applied to ensure deterministic change (and best practice reviews).

Rather than just using the WebUI we should leverage "Infrastructure as Code" like CloudFormation but I would suggest more complete tools like:

One thing you might notice as you convert API Gateway + Lambda into code is that it begins to look a lot like code you would see in any web application, except that it is in a DomainSpecificLanguage for a specific vendor framework.

Comparisons

Two things that are practically relevant are:

Latency
Cost

Lambdas will inherently have "warm up time" from a "cold start", while this can be mitigated (with a canary warmer etc.), if your architecture is deliberately tying together lots of services over the network it may feel slow (and indeterministics).

The contrast is to keep more things in memory or co-located in the same server. (Less network calls)

Cost is fickle: Lambdas may be much cheaper but if configured incorrectly a DOS attack could create a lot of API Gateway events or CloudWatch logs (so indirect cost escalation).

So there are workarounds but investing in them once again means the benefits (reduced overhead) come with a new complexity (and skill set).

Google and Azure both provide functions as a service so while it may be possible to get price competition given the vendor lock-in nature switching costs may be non-trivial.

Amazon cleverly have a "free tier" that covers enough to get developer hobby projects (aka time invested learning = biased professional purchasing choices) which covers everything in this article. https://aws.amazon.com/free/ I have no affiliation with Amazon, Google, etc.

From a philosophical perspective the choice is somewhat whether building with predefined vendor blocks (so WebUI or IaaC) is more valuable (time wise)/preferrable than code.

Trust

Something that underlies all of the AWS Lambda thinking and work is trust in the vendor.

Trust in their security practices
Trust in their uptime and operations team
Trust in their business (both longevity and pricing)

Infrastructure as Code with Terraform and AWS

2018-05-12T07:35:00-07:00

Overview

Just like building and maintaining hardware and physical servers by hand is manual drudge work (aka "toil"), so is the manual deployment of servers and networking etc. even on "Infrastructure as a Service".

The "Infrastructure as Code" movement emphasizes the value of human time and leverages source code and version control to manage the increasing complexity. (Think of the exponential growth of logical systems with virtualization, cloud services, containers, and micro-services etc.)

Imagine a tool to automate deploying and managing virtual infrastructure...

While AmazonWebServices CloudFormation works for setting up infrastructure via a config file, the user experience is non-optimal and there is definitely "vendor-lock-in".

Hashicorp are a trusted brand in DevOps and having produced Vagrant, Packer, Consul, etc. it is easy to pick their Terraform product.

I have no affiliation with Hashicorp besides as a user of their software =)

While packer can build a single immutable image (an application or component) of a (virtual) server (or container), Terraform is the higher level tool where the full setup including Load Balancer, Database, etc. are managed. Terraform can also maintain "state" like replacing servers with (deploying) newer images that were built by packer.

The configuration files can be applied to various Cloud and Infrastructure vendors providing some measure of portability.

References:

https://en.wikipedia.org/wiki/Infrastructure_as_Code
https://blog.john-pfeiffer.com/build-automation-using-packer-to-build-an-ami-use-immutable-not-chef (packer runs commands to create a machine image - aka a server frozen as a file)
https://aws.amazon.com/documentation/cloudformation
https://www.digitalocean.com/community/tutorials/how-to-use-terraform-with-digitalocean

Prerequisites

Basically you need access to the remote infrastructure, permissions to make changes (resources cost money!), and of course the Terraform tool.

Download from https://www.terraform.io/downloads.html (and probably unzip to \opt) the terraform binary, you may also want to echo "alias terraform='/opt/terraform'" >> ~/.bashrc
Verify that the terraform binary has been installed and can execute: terraform version ; terraform help
Create an SSH keypair: ssh-keygen -t rsa -C "myemail@example.com" -f $HOME/.ssh/aws.id_rsa
Upload the keypair to AWS https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=us-west-1#KeyPairs:sort=keyName
Create a dedicated User with limited permissions: https://console.aws.amazon.com/iam/home?region=us-west-1#/users
Ensure the new User (i.e. terraform-demo) only has "Programmatic Access" (aka API only, not WebUI)
Ensure the new User has permissions, i.e. is part of a Group (named ec2-only) that leverages the pre-generated policy name of EC2FullAccess)
The last step of the create user wizard should show the ACCESS_KEY and SECRET_KEY , make sure you save these in a password manager.

Using Environment variables is a fairly standard and portable/secure way of providing credentials to Terraform, the region will determine where resources are created

export AWS_ACCESS_KEY_ID="AKIAyourACCESSkey"
export AWS_SECRET_ACCESS_KEY="yourSECRETkeyABC123"
export AWS_DEFAULT_REGION="us-west-1"

Alternatively save the credentials in the default amazon credentials file (~/.aws/credentials) by using aws configure

References:

A simple build

The simple and common example of deploying a new server instance from an AmazonMachineImage (with a new VPC for network control)

Example Terraform File

example.tf

provider "aws" {
  region = "us-west-1"
}
resource "aws_instance" "example" {
  ami = "ami-dc2739bc"
  instance_type = "t2.nano"
  subnet_id = "${aws_subnet.us-west-1a-public.id}"
}
resource "aws_vpc" "example" {
  cidr_block = "192.168.1.0/24"
  enable_dns_hostnames = true
  enable_dns_support = true
}
resource "aws_subnet" "us-east-1a-public" {
  vpc_id = "${aws_vpc.example.id}"
  cidr_block = "192.168.1.0/28"
  availability_zone = "us-west-1a"
}

The provider is the target (with a specific sub region), and each resource has a type and a name (which can be referenced in later variables)

While Amazon Linux images are hardened CentOS with security updates it can be convenient to use Ubuntu 16.04 (and most likely real world use cases will be from an AMI you have built yourself)

The AMI ID is region specific, this example comes from us-west-1

References:

Terraform Plan and Apply

The single configuration file and the following commands will deploy the prescribed infrastructure to AWS.

terraform init

install the AWS plugin (that is detected via the example.tf configuration file), you will need to git ignore .terraform and plugin binaries

terraform plan

This will preview the steps that will occur

Terraform will perform the following actions:
    + aws_instance.example
        id:             <computed>
        ami:            "ami-dc2739bc"
        instance_type:    "t2.nano"
        ...
    + aws_subnet.us-west-1a-public
        id:                               <computed>
        assign_ipv6_address_on_creation:  "false"
        availability_zone:                "us-west-1a"
        ...
    + aws_vpc.example
        id:                               <computed>
        assign_generated_ipv6_cidr_block: "false"
        cidr_block:                       "192.168.1.0/24"

means created , - means removed , the VPC and subnet ids may even be populated

To avoid the issue of VPC or no VPC (and for better default security) this will explicitly create a new VPC.

terraform apply

Once again the expected results are displayed and a prompt requires confirmation

aws_instance.example: Creating...
  ami:                          "" => "ami-44273924"
  ...
aws_instance.example: Still creating... (10s elapsed)
aws_instance.example: Creation complete after 16s (ID: i-0492ba9707e624a66)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

the .terraform.tfstate file contains the current state of the current infrastructure

terraform show

A full listing of the details of the infrastructure is returned

To see the instance in the WebUI: https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=us-west-1#Instances:sort=instanceId
Verify the new VPC https://us-west-1.console.aws.amazon.com/vpc/home?region=us-west-1#vpcs:

Updates to Existing Infrastructure and Instances

If you modify and save example.tf file to add a tag Name:

resource "aws_instance" "example" {
  tags {
    Name = "terraform-example"
  }
...

terraform plan ; terraform apply

The remote infrastructure (server) has been updated with a tag/Name "example" , terraform show

If you change the resource AMI ID (i.e. the base immutable image from which the server was created) then Terraform will destroy the old server and deploy a new one

Destroying

terraform destroy

This is a destructive command, as always you must type "yes" ... "There is no undo"

Terraform will perform the following actions:
  - aws_instance.example
  - aws_subnet.us-west-1a-public
  - aws_vpc.example

terraform show will return nothing and inspection in the AWS Console (web ui) will show all the resources have been cleaned up =)

A Load Balanced Example

Tools to Manage State vs Platform as a Service

Infrastructure as Code focuses on tools to manage complexity. An alternative is to outsource the infrastructure entirely by using something like a PlatformAsAService.

Heroku (or Google AppEngine or Openshift) can simplify application deployment by reducing the input to simply the application code.

The PaaS vendor manages the infrastructure (including load balancers) and can provide a WebUI and APIs for adding dependencies (i.e. databases).

Though it may again require a vendor specific configuration file to specify which application is connected/has permissions to which database...

Go Web Development and Templates with Heroku

2018-05-01T20:37:00-07:00

Prerequisites

Please create a new version controlled source code repository using Github or Bitbucket or ?.
Have Golang installed
Download and install the official Go dependency tool "dep", https://golang.github.io/dep/docs/installation.html

A simple http server

Your source repository currently only needs a single file: main.go

package main

import (
        "fmt"
        "log"
        "net/http"
        "os"
)

func myHandler(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("hi"))
}

func main() {
        port := getEnvOrDefault("PORT", "8080")
        log.Println("Listening on port", port)
        http.HandleFunc("/", myHandler)
        http.ListenAndServe(":"+port, nil)
}

func getEnvOrDefault(key, defaultValue string) string {
        result := defaultValue
        val, ok := os.LookupEnv(key)
        if ok {
                result = val
        }
        return result
}

another example of a trivial web server using only the standard library

Verify it runs locally with: go run main.go

your browser should see "hi" in http://localhost:8080 and use "control + c" to cancel and quit the command line web server application

While previous examples used Google App Engine https://blog.john-pfeiffer.com/go-programming-intro-with-vs-code-and-arrays-slices-functions-and-testing/#a-simple-web-server, here we will leverage Heroku for deploying our web application.

Go Dependency Management

First download and install the Go dependency tool: https://github.com/golang/dep

cd DIRECTORY/OF/WEBAPP
dep init
go list -e .
echo "[metadata.heroku]" >> Gopkg.toml
echo "  root-package = \"`go list -e .`\"" >> Gopkg.toml
git status ; git diff
git add --all .
git commit -m "initial web app and using go dep"
git push

An example simple Go dep configuration file https://github.com/johnpfeiffer/web-go/blob/master/Gopkg.toml
Further heroku metadata config options: https://devcenter.heroku.com/articles/go-apps-with-dep#build-configuration
Similar heroku instructions on dep: https://devcenter.heroku.com/articles/go-apps-with-dep#getting-started

Deploying to Heroku from Github via Travis-CI

Heroku Prerequisites

Download and install the heroku CLI https://devcenter.heroku.com/articles/getting-started-with-go#set-up
cd /opt ; wget https://cli-assets.heroku.com/branches/stable/heroku-linux-amd64.tar.gz
tar xf heroku-linux-amd64.tar.gz
/opt/heroku/bin/heroku --version ; /opt/heroku/bin/heroku --help
/opt/heroku/bin/heroku login this will prompt for your email and password and store the credentials in ~/.netrc

For extra security consider using heroku CLI inside of an ephemeral docker container or a script that removes the credentials after each usage

These instructions will now assume heroku "just works", i.e. alias heroku='/opt/heroku/bin/heroku'

Configuring the Application with Heroku

cd DIRECTORY/OF/WEBAPP
heroku help
heroku create APPNAME --buildpack heroku/go
heroku status

This has created an empty Go application in Heroku

Later on when deploying the application to heroku if you receive the following error: No default language could be detected for this app, it means you did not set the buildpack language yet...

If you missed setting the buildpack language via CLI you can also use the WebUI after the app was already created:

Another gotcha is if you have not configured a dependency manager (even without dependencies!) you will see this error App not compatible with buildpack.

Travis-CI

https://docs.travis-ci.com/user/deployment/heroku/

First you will need to have retrieved your auth token: heroku auth:token

You can connect the repository that was already created in Github to Travis via https://travis-ci.org , more specifically at https://travis-ci.org/profile/USERNAME

you may need to "sync account" if you very recently created a repository , otherwise you will see this error: repository not known to https://api.travis-ci.org/: USERNAME/REPONAME

sudo docker run -it --rm --volume /absolute/path/repo:/opt/repo --volume /opt/heroku:/opt/heroku ruby:alpine /bin/sh
apk update
apk add --no-cache build-base git
gem install travis travis-lint
travis help

This docker example avoids installing ruby (or travis) locally ;p

cd /opt/repo ; touch .travis.yml ; travis setup heroku

/opt/web-go # travis setup heroku
Heroku API token: ************************************
Heroku application name: |web-go|
Deploy only from GITHUBUSERNAME/web-go? |yes| yes
Encrypt API key? |yes| yes

This simple CLI wizard prompts for the Heroku auth token and populates the .travis.yml with the encrypted value

If you want to do it manually you will need (using "vi" ;) to create a dummy .travis.yml file with the content:

deploy:
  provider: heroku
  api_key:
    secure: "YOUR ENCRYPTED API KEY"

You can then run the following command which will update the .travis.yml file with the real encrypted auth token. travis encrypt YOUR-HEROKU-TOKEN --add deploy.api_key -r GITHUBUSERNAME/REPONAME

You may get prompted to login if you use the incorrect USERNAME

In your source code repository your more complete .travis.yml file will be:

language: go
script: go get && go test -v
notifications:
  email: false

deploy:
  provider: heroku
  api_key:
    secure: ABCD1234XYZLONGENCRYPTEDSTRING

An interesting alternative to github and travis is bitbucket: https://confluence.atlassian.com/bitbucket/deploy-to-heroku-872013667.html

https://confluence.atlassian.com/bitbucket/variables-in-pipelines-794502608.html (where you enter your heroku API key as a variable)
e.g. https://bitbucket.org/YOURUSERNAME/YOURREPONAME/addon/pipelines/deployments (on the far right there will be a gear symbol to configure variables)

Note that Using the Deployments UI (via adding the deployment: production line) will change the UI layout, and may require re-entering any Variables

Also, use the Secured checkbox for any passwords or API keys when entered as a Variable

https://bitbucket.org/atlassian/heroku-deploy/src/master/README.md

bitbucket-pipelines.yml

image: golang
clone:
  depth: full
pipelines:
  default:
    - step:
    name: Build and test
    script:
      - source bitbucket-pipelines-go.sh
      - cd ${IMPORT_PATH}/
      - go get
      - go build
      - go test -v
      - git push https://heroku:$HEROKU_API_KEY@git.heroku.com/$HEROKU_APP_NAME.git HEAD

definitions:
  services:
    postgres:
      image: postgres

Try creating new credentials if you get an unauthorized error: Failed to create sources. Expected HTTP Status 200, got 401. Error message: {"id":"unauthorized","message":"Invalid credentials provided."}

heroku authorizations:create

You can see (and rename the description) of the Heroku tokens/api-keys listed at https://dashboard.heroku.com/account/applications

You can input/update the token here: https://bitbucket.org/USERNAME/REPONAME/admin/addon/admin/pipelines/deployment-settings

Test It

Now a browser that hits the Heroku URL will see "hi" , https://web-go.herokuapp.com/

Templates for Content

Separating out the static html content from dynamic and business logic parts of the application is a key way to remain modular. Templates built into the Go standard library can provide output that is safe from code injection.

var indexTemplate = GetIndexTemplate()

func myHandler(w http.ResponseWriter, r *http.Request) {
    indexTemplate.Execute(w, NoData{})
}

These small changes (var indexTemplate and indexTemplate.Execute) to the previous main.go allows the default web handler (aka controller) to return html

indextemplate.go

package main

import "html/template"

// NoData is an empty struct as I do not pass anything into the template
type NoData struct{}

// GetIndexTemplate returns the index.html template https://golang.org/pkg/html/template/#Template
func GetIndexTemplate() *template.Template {
    var indexTemplate = template.Must(template.New("index").Parse(`<html><head><style type="text/css">
body{
  font-size: 1.9em;
}
</style></head><body>hi</body></html>
`))
    return indexTemplate
}

The function just returns the rendered template; since it is only called once in main it is not inefficient, and Must will panic if the template has an error

go run main.go indextemplate.go allows you to test it locally

https://golang.org/pkg/html/template/

Passing Variables to a Template

Dynamic, data driven web sites emphasize the power of software to create tables that nobody wants to write by hand.

Building on the previous two examples (main.go, indextemplate.go) we now have a variation that passes data to the template.

hexcolors.go

package main

import (
    "fmt"
    "html/template"
    "net/http"
)

// HexColors wraps a list of colors as hexadecimal strings
type HexColors struct {
    Colors []string
}

// GetHexTemplate returns the parsed file as a template object
func GetHexTemplate() *template.Template {
    return template.Must(template.ParseFiles("hexcolorstemplate.html"))
}

func hexController(w http.ResponseWriter, r *http.Request) {
    colors := []string{}
    for i := 255; i <= 16711680; i = i * 256 {
        s := fmt.Sprintf("%06X", i)
        colors = append(colors, s)
    }
    data := HexColors{colors}
    hexTemplate.Execute(w, data)
}

A handler/controller that generates hex color data

hexcolorstemplate.html

<table>
  {{range .Colors}}
  <tr><td>{{.}}</td><td style="background-color: {{.}}; ">__</td></tr>
  {{end}}
</table>

range iterates and creates a table row for each color , (the html and body is elided, see the previous example)

For the full implementation see the web-go code example: https://github.com/johnpfeiffer/web-go/commit/c8636ee4f54ff95d4a804a152954874f5c23b682

go run main.go indextemplate.go hexcolors.go allows you to test it locally, note that you do not have to pass the html template as a parameter

More info:

HTML Template Blocks as Reusable Components

Besides dynamic data and tables for "do not repeat yourself" there are also structural components that can be deduplicated. Changes in a base html or css template (or common component definition) can therefore reliably be applied to a large number of files.

Jinja2 is famous in Python for making it easier to work with websites, here are two different helpful mechanisms in Go:

Use the keyword "define" to create a fragment that can be explicitly included
Use the keyword "block" to create a default value that can be overridden

the keyword "template" loads a template that has been created by a "define"

indextemplate.go

...
func GetIndexTemplate() *template.Template {
    indexTemplate := template.Must(template.ParseFiles("base.tmpl", "index.html"))
    return indexTemplate
}

This replaces the previous examples hardcoded html with a default base.tmpl that is then overridden by the "define" block in the index.html file, order matters!

base.tmpl

<html><head>
<style type="text/css">
{{block "style" .}}
body{
  font-family: "Georgia";
  font-size: 1.9em;
}
{{end}}
</style>
</head>
<body>
{{block "content" .}}
{{end}}
</body></html>

The HTML in base.tmpl has a "block" that provides a default style, content is an empty "block", (clearly you can define variations of bases templates)

index.html

{{define "content"}}
hi , try <a href="/hexcolors">hexcolors</a>
{{end}}

The tiny index file heavily leverages the default base template and uses "define" to only override the content block

go run main.go indextemplate.go hexcolors.go allows you to test it locally, you do not pass the .tmpl nor .html template files as parameters

Further illustration by extending the templates usage a little further:

hexcolors.go

func GetHexTemplate() *template.Template {
    return template.Must(template.ParseFiles("base.tmpl", "components.tmpl", "hexcolorstemplate.html"))
}

the template files must exist in the relative path and build in order

components.tmpl

{{define "tablestyle"}}
  table, th, td {
    border: 2px solid black;
    font-size: 2.5em;
  }
{{end}}

defining a specific fragment that can be used anywhere

hexcolorstemplate.html

{{define "style"}}
{{template "tablestyle" .}}
{{end}}

{{define "content"}}
  <table>
    {{range .Colors}}
    <tr><td>{{.}}</td><td style="background-color: {{.}}; ">__</td></tr>
    {{end}}
  </table>
{{end}}

the most complex example: the base template content is overridden with a table that gets data at runtime

the style definition overrides the base template default style ; it actually gets populated by loading the tablestyle fragment definition

go run main.go indextemplate.go hexcolors.go allows you to test it locally, you do not pass the .tmpl nor .html template files as parameters

2 .tmpl files, 2 .html files, 3 .go files

Full source code for this example: https://github.com/johnpfeiffer/web-go/commit/b6da397f89c8a6955a30e665ff1aa99be989e5cb

For further info and advanced features like cloning:

TestDrivenDesign and Gorilla Mux

Tests communicate. Use libraries and composition, not frameworks and magic

Rather than re-invent the wheel it is useful to leverage well designed libraries to reduce bugs (and repetitive boilerplate code).

Gorilla "mux" is a great idiomatic Go library for multi-plexing and routing. Designed to be modular to prefer composition we can also choose to leverage the Logging handler.

Finally adding a bit of JSON and very concisely you have built a high performing API server.

For the full featured source code with tests: https://github.com/johnpfeiffer/go-web-example

References:

https://github.com/gorilla/mux in action https://github.com/johnpfeiffer/go-web-example/blob/master/router.go (with dependency injection for database testability)
https://github.com/johnpfeiffer/time-go/blob/master/main.go leverages https://github.com/gorilla/handlers#example
https://github.com/johnpfeiffer/go-web-example/blob/master/controller_test.go has both unit and integration test examples
TODO: integrating heroku postgres , https://devcenter.heroku.com/articles/getting-started-with-go#use-a-database

Miscellaneous

How to return specific HTTP Response types

Note these code snippets are the Handler function only, you should provide your own HTTP listener and multi-plexer

func headonly(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Allow", "HEAD, GET")
    w.WriteHeader(200)
}

https://play.golang.org/p/GCfxTLdLGYn example returning the HTTP Header where clearly the default status code is 200
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.7

curl --silent --write-out "%{http_code}" localhost:8080

func httperror(w http.ResponseWriter, r *http.Request) {
    http.Error(w, "Internal Server Error", http.StatusInternalServerError)
    return
    // without the return statement execution would continue
}

The http library understands that unexpected errors do occur but make sure to return so as to not continue executing code

https://play.golang.org/p/FyB7FwVp-ZB example showing why the return statement is so important
https://en.wikipedia.org/wiki/List_of_HTTP_status_codes
https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
https://curl.haxx.se/docs/manpage.html

Serving binary files...

func getfile(w http.ResponseWriter, r *http.Request) {
    fp := path.Join("images", "example.png")
    http.ServeFile(w, r, fp)
}

https://play.golang.org/p/G9zQ0SmzjP_K example writing, reading, and serving a file
https://golang.org/pkg/net/http/#ServeFile
https://golang.org/pkg/path/filepath/#Clean for sanitizing user input for loading file paths

An example http server inside the Go playground: https://play.golang.org/p/B-aZuQOdFtB

Logs from Heroku

To see the logs from the web application running in heroku: heroku logs --app APPNAME --tail

Environment Variables for Configuration

https://devcenter.heroku.com/articles/getting-started-with-go#define-config-vars

Custom Domains with Heroku

To use a custom domain name for your traffic https://devcenter.heroku.com/articles/custom-domains

Downloading the source code from Heroku

To download the source code from a running Heroku application: heroku git:clone --app APPNAME https://devcenter.heroku.com/articles/git-clone-heroku-app

Using HTTPS for a Go Web Server with a Self Signed SSL Certficate

One Liner to generate a self signed certificate

openssl req -subj '/CN=example.com/O=My Company Name LTD./C=US' -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout server.key -out server.crt

SSL Web Server in Golang

package main

import (
        "log"
        "net/http"
        "os"
)

func myHandler(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("hi"))
}

func main() {
        certname := "server.crt"
        keyname := "server.key"
        port := getEnvOrDefault("PORT", "8080")
        log.Println("Listening on port", port)
        http.HandleFunc("/", myHandler)
        log.Fatal(http.ListenAndServeTLS(":"+port, certname, keyname, nil))
}

func getEnvOrDefault(key, defaultValue string) string {
        result := defaultValue
        val, ok := os.LookupEnv(key)
        if ok {
                result = val
        }
        return result
}

Note that self signed certificates will prompt a browser warning curl --insecure https://localhost:8080

More Go Data Structures with Heap and Tree and Trie

2018-02-10T07:07:00-08:00

Idiomatic Go depends heavily on the standard library. Given that performance is often implementation dependent and directly related to the data being stored, it makes sense that one must write some of the more common data structures from scratch.

While slices (a.k.a. dynamic arrays) are included (as is a double linked list), a Set must be implemented using a map (for the specific type(s) of interest) https://blog.john-pfeiffer.com/golang-interfaces-stack-linked-list-queue-map-set/

Some of my favorite data structures are based upon trees...

Heap

A heap (https://en.wikipedia.org/wiki/Heap_%28data_structure%29) is a tree that maintains a specific property, i.e. the parent is always larger than its children A binary heap (aka "priority queue" https://en.wikipedia.org/wiki/Binary_heap) has some very helpful and efficient qualities, especially when implemented by using an array (using computations leveraging the inherent relationship with powers of 2). For instance, it is possible to use something like heapsort to sort an array in-place.

A common use for a min-heap or max-heap is to be able to answer a query in O(1). (The top/first item retrieved is the answer!)

And after you get that answer, the next root item you "pop" is the next best answer!

https://golang.org/pkg/container/heap/ is already included in the standard library =]

Binary Search Tree

A binary search tree maintains a specific property between all nodes in the tree. This is most often used to keep a collection of items sorted in order.

If the tree is balanced (https://en.wikipedia.org/wiki/Self-balancing_binary_search_tree) we can mathematically guarantee the time for various operations.

Because each branch splits into 2 it follows intuitively that time bounds are logn (the fancy way of saying the inverse of 2^x) https://en.wikipedia.org/wiki/Binary_logarithm

https://github.com/johnpfeiffer/gotree

Trie

An interesting variation of a tree is a trie that allows for very fast retrieval of information (if it exists)

https://en.wikipedia.org/wiki/Trie

A common example is quickly identifying if a string is available (i.e. autocompletion suggestions of words or names, or "boggle") (Hence the obvious application of a trie also-known-as a "prefix tree") , video of Zelenski at Stanford explaining: https://youtu.be/TJ8SkcUSdbU?t=13m21s

https://github.com/johnpfeiffer/gotree

Benefits

A trie is fast, like O(m) where m = the length of the search string (assuming that m is far smaller than all n keys in the tree)
Unlike hashtables no hashing algorithm is required
"intermediate" or "subset" solutions/queries are possible by returning a subtree
Iterating through the trie can return the keys in a specific (e.g. alphabetical) order

Working successfully with remote contractors

2018-01-20T21:14:00-08:00

Some Background

It can be challenging to scale up an organization. One choice that can be made is "full time employee" versus "contractor". Another choice is "Co-Located" versus "Remote".

There are some important considerations with any combination but I'll share my experiences with "Remote Contractors".

I've worked with remote contractors from Israel, Italy, England, Pakistan, and Ukraine. I've also personally worked as both an on-site contractor and a remote contractor.

A contractor is free to take on other client work in parallel or to say no to a project. They may also decide to stop taking new contracts.
A long term contractor is incentivized to deliver sustainable value for repeat business.
A remote contractor might never be met in-person (instead only via video, chat, email, etc.) - they can be based anywhere in the world.
Contractors typically cost less for a short duration (considering the full cost of HR and the interviewing process, employee benefits, cost of office space, etc.)
Contractors tend to cost more over a longer duration (i.e. you are paying a premium for the overhead of their company administration, marketing, profit margin, etc. , though it still might be absolutely less than an employee)
Contracts and payments can be tricky, especially across international borders, though technology and digital payments have made this a lot simpler.

Myths

The following are some of the many negative things I've heard:

Outsourcing is evil and consultants and contractors are the most expensive way to ruin your business =(
Foreigners are hard to understand and work with
You should only hire contractors for a specific task
Remote workers are virtual and so they don't matter as much
Contractors, especially remote ones, will never be as passionate and invested as your employees

A story with a happy ending

Of my many experiences I'll share a recent one where I've had fabulous success with some contractors in Ukraine.

We started with three open positions and had a happy coincidence that a potential team lead contractor was in the Bay Area finishing a different assignment. The in-person interview went well (communicated well, strong technical skills, showed ambition and experience with leading a team). I interviewed him just like I would any new employee/team member.

In the next phase I immediately encouraged this first contractor, our remote team lead, to be engaged in the interviewing of the remaining two members. (After all, they'd be on his team!) I then received resumes from each prospective candidate he'd already reviewed/interviewed. Next I had a conference call with each candidate where again I focused on communication, experience, and ability.

Once we got the team assembled we codified on a Monday weekly meeting that fit in the narrow window of time that accomodated the 10 hour timezone difference. All other coordination would be through Confluence, Pull Requests, HipChat (chat and, and of course, if needed, video call).

I gave them an example of the work I had done already for the task and asked about their experiences with different solutions/technology choices (test automation and Selenium). I also worked with them to create an inventory of the test coverage. Each week we would discuss the current progress and if there was a change in priority (i.e. a critical security release requiring extra help with manual validation).

After a year it became a clear success: productive part of the team, doubled their head count, and allowing us to ship with more confidence faster than ever before. Some of the trust has been having them own their portion of the system (CI and Automated Testing), very often when they bring up a bug or the tests fail we have to listen: delegation means trusting.

The timezone difference is still hard and they maintain their own Kanban board even though they participate in the common sprint planning meeting. As they've learned our problem domain and culture they have grown to proactively solve problems (like a hackathon where they wrote an API library for testing, or their extension into performance testing, and many other additions). We could not have hired as many full time employees so quickly nor with their specific expertise: our success is due to their success.

Things I've learned to succeed

Especially: Ensure there is clear Communication

Have a regular meeting (I find weekly is the best cadence) to make sure progress is being made on the correct things and identify any "blockers"
Prepare written agendas, repeat any decisions made in writing
Verbally (Video Conference is preferred for visual cues) for "high bandwidth" back and forth, give them an opportunity to ask questions!
Provide asynchronous tools to overcome timezone differences (HipChat/JIRA/Confluence)
Have a clear idea of exactly what you want them to achieve (talk through your "definition of done")

Specifically: People and Organization

Be involved and engaged in the interview process - your people and team are the largest factor for success or failure
Trust is earned over time and is the most valuable component (especially when remote)
You must give trust to receive trust
For hiring a remote team lead: prefer communication and then organization skills and finally technical skills - remember this person must overcome language/cultural barriers
Setup a remote hierarchy: they need to be able to resolve questions and become unblocked quickly via a local escalation process (i.e. a single decision maker in their office/timezone)
Don't be afraid to "lose a week" for them to learn something, research an approach, or just because of a misunderstanding (this is normal and it only feels more expensive than an "internal employee" because you actually get to see a bill for it)
Provide open ended goals (don't micromanage and over-prescribe) and people will often exceed expectations
Being nice is a universal truth

Things to watch out for

Clearly not everything leads to success, here are some important considerations:

Things that materially affect your user count or bottom line may need a stronger compliance/auditing chain than contracting firms can provide
Misunderstandings can turn into personal conflicts if not managed correctly
Beware slipping into an "us and them" mentality or "throwing it over the wall" or "finger pointing"
Skills may not always match up and it's rare remote contractors will directly interact with your customers - don't create unwinnable situations

Be careful when using contractors for critical path projects or key competitive intellectual property: - You are betting your project/company success on a group that has more external risk and less alignment - They may be pressured into taking shortcuts and are less likely to have to "pay" the long term costs (e.g. technical debt by skipping tests or large volumes of support tickets) - Slippage and mistakes happen, by keeping it inside your own organization these can be managed quickly and appropriately

Things that you can takeaway

A foreign contractor can be just a productive as a domestic contractor or employee despite language, geography, cultural, calendar, and timezone differences. A good contractor is like finding a needle in a very large haystack so put in the effort to keep them happy.

Contractors are people:

Personalize the relationship because you should get to know them: their interests, strengths, weaknesses, etc.
Find how they complement your team and give them a path to grow as professionals
Respect that they have national/religious holidays and personal lives
Look to align their interests with yours, there has got to be more than "it's just about the money"

Golang Buffalo Tutorial To Create A Web Site With Authentication

2017-12-02T19:44:00-08:00

Go's considerable tooling makes it a very productive and performant static language. Besides being ideal for developing APIs (goroutines!) it can then be convenient to also write the Web UI in Go.

Although the standard library allows for writing a Go based website with templates and maximum flexibility, what if you want a quick start with "batteries included"?

Code generation makes it easy to get started but all future customization depends on the developer, this is not a CMS =p

Buffalo Installation and Prerequisites

Assuming the Go dev environment is already setup...

go get -u -v github.com/gobuffalo/buffalo/buffalo

buffalo new example

sudo docker run --rm -it --publish 0.0.0.0:5432:5432 --name pg -e POSTGRES_PASSWORD=postgres postgres:alpine

this will make the username and password match the default database.yml that buffalo generates...

cat database.yml

development:
  dialect: postgres
  database: example_development
  user: postgres
  password: postgres
  host: 127.0.0.1
  pool: 5

test:
  url: {{envOr "TEST_DATABASE_URL" "postgres://postgres:postgres@127.0.0.1:5432/example_test?sslmode=disable"}}

production:
  url: {{envOr "DATABASE_URL" "postgres://postgres:postgres@127.0.0.1:5432/example_production?sslmode=disable"}}

buffalo db create --all

the buffalo framework will create all the necessary databases and tables

(you can also buffalo db drop --all or buffalo db drop --env test to remove all or just one db)

references:

It is worth reading about the convention of how Buffalo lays out the directories and code https://gobuffalo.io/docs/directory-structure

Start Developing with a Resource Generator

buffalo dev

(Or use the following syntax for a non default port: PORT=3001 buffalo dev)

curl localhost:3000

Yup a default web page is routed and served

buffalo generate resource users name email title:nulls.Text
buffalo db migrate

Resulting output...

--> actions/users.go
--> actions/users_test.go
--> locales/users.en-us.yaml
--> templates/users/_form.html
--> templates/users/edit.html
--> templates/users/index.html
--> templates/users/new.html
--> templates/users/show.html
--> buffalo db g model user name email title:nulls.Text
v3.41.1

--> models/user.go
--> models/user_test.go
--> goimports -w actions/actions_test.go actions/app.go actions/home.go actions/home_test.go actions/render.go actions/users.go actions/users_test.go grifts/db.go grifts/init.go main.go models/models.go models/models_test.go models/user.go models/user_test.go
> migrations/20171203042126_create_users.up.fizz
> migrations/20171203042126_create_users.down.fizz
--> goimports -w actions/actions_test.go actions/app.go actions/home.go actions/home_test.go actions/render.go actions/users.go actions/users_test.go grifts/db.go grifts/init.go main.go models/models.go models/models_test.go models/user.go models/user_test.go

The code generator is very helpful, especially after the database tables are updated with the migrate command

curl localhost:3000

The list of REST resources now includes all the usual HTTP methods

curl localhost:3000/users

The GET endpoint lists all of the existing users (in the database) and has a button to create a new user

Now you can use a web browser to play with the Web UI

You can also query the database

sudo docker run -it --rm --link pg:postgres postgres:alpine psql --help
sudo docker run -it --rm --link pg:postgres postgres:alpine psql --host postgres --username postgres
    \list
    \connect example_development
    \dt+
    \d+ users
    select * from users;

Avoiding the Frontend

Because javascript moves so quickly (and breaks things) it is easier to skip these steps and focus on the backend.

# maybe skip these (and all the accompanying dependencies) to avoid suckiness
sudo apt-get install -y npm
sudo npm cache clean -f
sudo npm install -g n
npm --version
node --version
npm install

npm installs the authrecipe dependencies but first use npm in order to install nodejs via the "n" helper

Without these the javascript assets or other things that were supposed to make the forms look pretty are not available but everything still works.

Authentication

One of the first building blocks of any decent site is authentication. Luckily there are a couple of packages that make basic and SSO authentication easier.

Basic Authentication with the Database

To just use an example (leverage the Buffalo author's in-progress tutorial ;)

sudo docker run --rm -it --publish 0.0.0.0:5432:5432 --name pg -e POSTGRES_PASSWORD=postgres postgres:alpine
git clone https://github.com/gobuffalo/authrecipe
cd authrecipe
go get
buffalo db create --all
buffalo db migrate
buffalo dev

that is the minimum to get up and runnning...

With a browser you can "sign in" or "register" (create a new user with a password)

TODO: more work to port over how to create secure endpoints with a "CheckAuth" middleware...

Authentication using an External Identity Provider

For instructions on how to have users authenticate and secure resources with identity providers like Github, Facebook, Google, etc.

sudo docker run --rm -it --publish 0.0.0.0:5432:5432 --name pg -e POSTGRES_PASSWORD=postgres postgres:alpine
go get -u -v github.com/gobuffalo/buffalo/buffalo
go get buffalo-goth
buffalo new example
buffalo generate goth-auth bitbucket

In order to set up the Bitbucket OAuth credentials: Log into bitbucket -> from your user profile (avatar) dropdown choose "bitbucket settings" -> click on OAuth (on the left) -> Add consumer

CallbackURL (for development): http://127.0.0.1:3000/auth/bitbucket Permissions: account email

Now you have a KEY and SECRET

BITBUCKET_KEY=foobar BITBUCKET_SECRET=barfoo buffalo dev

Dynamic Programming and Memoization and the Compute versus Storage Tradeoff

2017-11-06T20:45:00-08:00

Tradeoffs

There is an inevitable tradeoff of storage versus computation speed, or space used versus time to run, in every program.

Caching is often viewed as a performance optimization but sometimes it is the only way to achieve a result in a finite amount of time.

https://en.wikipedia.org/wiki/Dynamic_programming is about breaking down a problem into sub-problems where each solution can be stored so that it can be looked up rather than recomputed.

https://en.wikipedia.org/wiki/Memoization is related in that it is about storing the output of a function call when specific parameters are provided. Thus it is often conceptualized as a special kind of caching.

Naive Fibonacci Recursion

https://en.wikipedia.org/wiki/Fibonacci_number is a classic numerical series where each subsequent number is the sum of the previous two numbers: 1 1 2 3 5 8 13...

An interesting problem is to calculate the series up to a certain point. The straightforward solution uses recursion:

package main

import (
    "fmt"
)

func main() {
    fibSeries(37)
}

func fibSeries(n int) []int {
    a := make([]int, n)
    for i := 1; i <= n; i++ {
        a[i-1] = fib(i)
    }
    return a
}

func fib(x int) int {
    if x < 2 {
        return x
    }
    return fib(x-1) + fib(x-2)
}

go run main.go

Each iteration in the series discards the previous results and then re-calculates the intermediate steps for each subsequent iteration

Fibonacci timings and golang playground time is frozen

https://play.golang.org/p/esejwsN0lQ is an example with timings but... ...sorry, the Go Playground does not really use time (and caches all output) so to really see the difference you must run the program locally (Andrew Gerrand explains)...

https://groups.google.com/forum/#!topic/golang-nuts/Dh0P1VzXmq8
https://groups.google.com/forum/#!topic/golang-nuts/NLZJahiMk58
https://github.com/golang/playground/blob/master/sandbox/play.go
http://www.gophercon.in/blog/2015/02/17/andrew/
https://talks.golang.org/2014/go4gophers.slide#3

Memoization aka Caching with Fibonacci

It almost seems common sense that we should not be re-calculating answers that we already know for every step...

func fibSeriesMemoization(n int) []int {
    a := make([]int, n)
    m := make(map[int]int)
    for i := 1; i <= n; i++ {
        a[i-1] = fibMemo(i, m)
    }
    return a
}

func fibMemo(x int, m map[int]int) int {
    if x < 2 {
        m[x] = x
        return x
    }
    _, ok := m[x-1]
    if !ok {
        m[x-1] = fibMemo(x-1, m)
    }
    _, ok = m[x-2]
    if !ok {
        m[x-2] = fibMemo(x-2, m)
    }
    return m[x-1] + m[x-2]
}

Using a map as a lookup table caches the result of each function call during each iteration

https://play.golang.org/p/Wxgl_OwkTY again provides the full code (despite the Go playground not really providing time elapsed)

That's it: identifying the extra work and storing it somewhere that can be referenced, the trade-off is now that more memory is required. =]

Dynamic Programming with Fibonacci Numbers

This alternative implementation removes the recursion (and helper function) and instead uses a simple for loop and a slice of ints. It highlights the nuance of how Dynamic Programming is not necessarily just storing the result of a function call but genuinely understanding the nature of the problem.

func fibDynamic(n int) []int {
    a := []int{1}
    if n == 1 {
        return a
    }
    a = append(a, 1)
    if n == 2 {
        return a
    }
    for i := 2; i < n; i++ {
        a = append(a, a[i-2]+a[i-1])
    }
    return a
}

this trivial example also benefits from the slice index corresponding well to a key for each fibonacci value in the "lookup table"

The example Dynamic Programming solution avoids the map lookup and so should be the most performant https://play.golang.org/p/VY9ul6ievC, but since the Go Playground time elapsed does not work...

Comparing with Benchmarks

Besides the "manual performance testing" with time and print statements you can use Go's more sophisticated tooling with bench.

Create main_test.go and run go test -v -run=NOMATCH -bench=BenchmarkFibonacciSeries

func BenchmarkFibonacciSeriesRecursive(b *testing.B) {
    for n := 0; n < b.N; n++ {
        fibSeriesRecursive(20)
    }
}

func BenchmarkFibonacciSeriesMemoization(b *testing.B) {
    for n := 0; n < b.N; n++ {
        fibSeriesMemoization(20)
    }
}

func BenchmarkFibonacciSeriesDynamicProgramming(b *testing.B) {
    for n := 0; n < b.N; n++ {
        fibDynamic(20)
    }
}

The nanoseconds per operation are dramatically less in the side-by-side comparison

goos: linux
goarch: amd64
pkg: bitbucket.org/johnpfeiffer/gosrc/benchmarking
BenchmarkFibonacciSeriesRecursive-4                10000            191632 ns/op
BenchmarkFibonacciSeriesMemoization-4             200000             13675 ns/op
BenchmarkFibonacciSeriesDynamicProgramming-4     2000000               814 ns/op
PASS
ok      bitbucket.org/johnpfeiffer/gosrc/benchmarking   7.157s

https://github.com/johnpfeiffer/go-fibonacci for full source code
Reference for running go benchmarking https://blog.john-pfeiffer.com/golang-testing-benchmark-profiling-subtests-fuzz-testing/#running-specific-benchmarks

Insights

The hardest part of applying caching is understanding the problem well enough to see where the extra work can be avoided. Thus I recommend pen and paper (or whiteboard) for diagramming the tree of (usually recursive) calls in order to see any patterns.

The tradeoff of memory for computation (time!) is usually worth it given modern large amounts of cheap memory available (assuming we do not have to worry about cache invalidation ;).

https://blog.john-pfeiffer.com/caching-data-and-common-gotchas-and-an-intro-to-redis-memcached-and-varnish/

Further exercises:

the coin problem (do not need to re-calculate the sub problems) , https://en.wikipedia.org/wiki/Coin_problem
towers of hanoi instructions series (rotate previous instructions rather than full recursion) , https://en.wikipedia.org/wiki/Tower_of_Hanoi

Productivity is a Myth

2017-06-29T21:00:00-07:00

Productivity is dependent on and interwoven with so many things that anytime anyone pushes "more productive" I wonder...

If you're "productive" and you get 3 things done in a day, or in a week even...

The Many Measures of Productivity

Were the 3 things done actually the important ones? (i.e. do they create value? reduce drag or improve velocity?)
How are you able to actually measure the effect/impact of the 3 things? Against what baseline and over what time period?
Do those 3 things even matter if the market suddenly shifted away from your product/segment/industry?
Do they contradict previous features/changes? (i.e. did they create confusion rather than clarity?)
Were 3 things being compared to 1, or were they being compared to 30? (If expectations aren't correctly set and met you cannot ever be "productive")
Does anybody even know that you did those 3 things? ("if a tree falls..." and "success has many fathers")

Hidden Costs

When getting 3 things "done", did you ignore the cost of quality?

Does it actually solve the intended problem?
Are there unacceptable side effects or regressions?
Do you even want to know the answers to the previous two questions?

Without test automation you probably have a feature or code that cannot be validated:

How will this value persist when the next bug fix or refactor occurs?

With code that is unreadable and therefore unmaintainable:

Will the next developer be unable to fix a bug or maybe just removes the code entirely?

Will you be unable to grow your organization as you cannot find anyone to work on your codebase anymore?

Did these changes dramatically increase the complexity of the architecture or the product?

spaghetti-ball-of-mud-rube-goldberg-birth-of-a-blackhole?

Was this "done" at the expense of burnout of an individual or the team?

Who will continue development of the feature or product?

How do these changes affect operational complexity or cost? Does crashing often or going bankrupt faster count as productivity?

Is this just squeezing the complexity bubble or shifting the burden?

In "getting things done" did you burn bridges within your organization? What about partners or existing customers?

Did you make a small subset happier at the expense of a much larger group?

What about the future, did you just trade loyal customers for a new ephemeral cohort?

John's Phases of Value

Least Valuable

an unshared idea
a proposal
a plan
some code
a pushed commit
in testing in a branch
reviewed and merged to master
testing the deployment (alpha)
released and distributed
experimental usage with feedback (beta)
stable usage

Most Valuable

Things become more valuable as they become more real

(becoming widely distributed/adopted for more people is a kind of proof that it actually solves a problem)

Traditional Methods of Productivity

There are amazing methods and techniques to "sustainably" and reliably create the "right thing" that is the "tipping point feature" of a "killer app" where you can measure and watch your key metrics and indicators go "up and to the right".

"Everyone loves a winner" and survivor bias certainly helps but...

The simplest way, in our capitalism-with-money-as-a-fungible-intermediary, is to receive dollars. :$

Yup, very crass, and a "lagging indicator" that is not nearly as exciting as "clicks" or "eyeballs" or "MAU/DAU/HAU", but... Monthly Active Users, Daily Active Users, Hourly Active Users, etc.

When you get people to part with their money you have definitely convinced them of some value

Organizations that have lasted longer, with the same people in them, are more likely the ones that have figured out a process for delivering value.

But maybe, instead of focusing on being productive, you should figure out how to get old and rich. =p

Doing Something Different

While imitation is the sincerest form of attempting to copy someone else's financial success, competition in a "race to the bottom" can turn a "win-win" into a "zero-sum" and then a "lose-lose".

Productivity should therefore also be compared against not only "what-is" but the "not-yet-imagined".

Strategy is creating a rising tide that lifts all boats. Tactics is poking holes in other people's stuff

So go on then, be productive! =)

Continuous Delivery with Bitbucket Pipelines and Google App Engine Deployment and the storage.objects.list error

2017-05-23T22:24:00-07:00

One thing that critical to writing software is actually delivering value. That means shipping the bits, luckily the internet makes the cost of moving 1's and 0's near zero.

Like any other chore or drudge work the release process should be automated.

https://en.wikipedia.org/wiki/Continuous_delivery

I attempted to follow Google App Engine's documentation on how to setup Bitbucket Pipelines (yeah it's crazy how much free compute there is in the world such that I can tie together a free Continuous Integration service with a free Application Hosting service) but ran into a few snags I thought I'd document.

Leveraging SaaS source control, CI, and CD means inherently placing trust in those vendors (and their ops and security teams)

Google Cloud Setup

https://cloud.google.com/solutions/continuous-delivery-bitbucket-app-engine

Besides creating an App Engine Project (python) also create the credentials to allow the automated deployment.

Create a new Cloud project in Google: https://console.cloud.google.com/projectcreate
Also create a new App Engine Project: https://console.cloud.google.com/projectselector/appengine/create?lang=python&st=true (you will need to choose your region such as us-central, or use the Web UI https://console.cloud.google.com/iam-admin/iam/project?project=example-john (Use the "Select a project dropdown" in the middle and in the Select menu use the + symbol on the right to "Create project")
Note the ID: https://console.cloud.google.com/home/dashboard?project=example-john
BILLING on the left, enable that (and link your credit card I guess, Google are moving everything away from free - and the credit card is another way to track and correlate everything you do)
Enable the App Engine Admin API https://console.cloud.google.com/apis/library?project=example-john
Navigate to the Google Cloud Platform Console credentials page https://console.cloud.google.com/apis/credentials?project=example-john
Click Create credentials -> Service account key
Select "New service account" from the Service account dropdown
Input a name like "Bitbucket authorization" in the Service account name field
ENSURE the ROLES contain at least: App Engine -> App Engine Admin and Storage -> Storage Admin
Click the Create button. A copy of the JSON file will automatically download to your computer. (if necessary on the right three dots choose "create key" -> create private key for "..." JSON
Click Create credentials > API key

Bitbucket Setup

Create a new repository in bitbucket that will contain the code deployed to Google App Engine (in my example I'll use the Python native one but I know soon everything will Docker based)

The minimum version of the two app engine python app you will need are:

app.yaml

runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: .*
  script: main.app

libraries:
- name: webapp2
  version: "2.5.2"

For more info https://developers.google.com/appengine/docs/python/config/appconfig

main.py

#!/usr/bin/env python
import webapp2

class MainHandler(webapp2.RequestHandler):
    def get(self):
        self.response.write('hello')

app = webapp2.WSGIApplication([('/', MainHandler)], debug=True)

A previous example of writing a Python web app with Google App Engine https://blog.john-pfeiffer.com/google-app-engine-python/

Bitbucket Pipelines Configuration

Enable the Pipelines feature: https://bitbucket.org/johnpfeiffer/continuous-deployment-bitbucket/addon/pipelines/home#!/getting-started

Configuring the secure environment variables in Bitbucket Pipelines

Use the Bitbucket WebUI in order to securely add the three variables (project id, api key, secrets json).

i.e. https://bitbucket.org/johnpfeiffer/continuous-deployment-bitbucket/admin/addon/admin/pipelines/repository-variables

CLOUDSDK_CORE_PROJECT (the app engine project id)
GOOGLE_API_KEY
GOOGLE_CLIENT_SECRET (all of the contents of the json file pasted in)

Ensure the Pipelines Environment variable of GOOGLE_CLIENT_SECRET is "Secured" so it is encrypted (i.e. all asterisks in the forms or in the log output)

Full instructions: https://cloud.google.com/solutions/continuous-delivery-bitbucket-app-engine#setting_up_environment_variables

bitbucket-pipelines.yml file

Before using a secret (or really any environment variables) with bitbucket pipelines you will need to have a placeholder bitbucket-pipelines.yml so that you can officially "enable" it for the repository - after that the WebUI for configuration for pipelines will become accessible =(

image: python:2.7
pipelines:
  default:
    - step:
        script:
          - python --version

Use this very trivial bitbucket-pipelines.yml in order to get at least one successful validation so that you can press the "Enable" button in the Bitbucket UI (I had to zoom my browser out to workaround some broken css/javascript)

A yaml configuration file describes the work that you instruct Bitbucket Pipelines to do, in this case we are doing the extra work of grabbing the remote google cloud SDK and installing it so that we use it with the credentials to deploy the app.

image: python:2.7

pipelines:
  default:
    - step:
        script:
          - export CLOUDSDK_CORE_DISABLE_PROMPTS=1
          - SDK_VERSION=127.0.0
          - SDK_FILENAME=google-cloud-sdk-${SDK_VERSION}-linux-x86_64.tar.gz
          - curl -O -J https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/${SDK_FILENAME}
          - tar -zxvf ${SDK_FILENAME} --directory ${HOME}
          - export PATH=${PATH}:${HOME}/google-cloud-sdk/bin
          - GAE_PYTHONPATH=${HOME}/google_appengine
          - export PYTHONPATH=${PYTHONPATH}:${GAE_PYTHONPATH}
          - python scripts/fetch_gae_sdk.py $(dirname "${GAE_PYTHONPATH}")
          - echo "${PYTHONPATH}" && ls ${GAE_PYTHONPATH}
          - echo "key = '${GOOGLE_API_KEY}'" > api_key.py
          - echo ${GOOGLE_CLIENT_SECRET} > client-secret.json
          - gcloud auth activate-service-account --key-file client-secret.json
          - gcloud --verbosity=error app deploy app.yaml --promote

A best practice is to pull the dependency from storage you have control over (or simply vendor the SDK in the source code) rather than downloading it every time and risking the upstream pinned version being removed

https://bitbucket-pipelines.prod.public.atl-paas.net/validator did not work for me (crazy javascript) but in theory this would be a very helpful tool

Full bitbucket pipelines documentation: https://confluence.atlassian.com/bitbucket/build-test-and-deploy-with-pipelines-792496469.html

scripts/fetch_gae_sdk.py

#!/usr/bin/env python
# Copyright 2015 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Retrieved from https://github.com/Google/oauth2client
"""Fetch the most recent GAE SDK and decompress it in the current directory.
Usage:
    fetch_gae_sdk.py [<dest_dir>]
Current releases are listed here:
    https://www.googleapis.com/storage/v1/b/appengine-sdks/o?prefix=featured
"""

import json
import os
import StringIO
import sys
import urllib2
import zipfile

_SDK_URL = (
    'https://www.googleapis.com/storage/v1/b/appengine-sdks/o?prefix=featured')


def get_gae_versions():
    try:
        version_info_json = urllib2.urlopen(_SDK_URL).read()
    except:
        return {}
    try:
        version_info = json.loads(version_info_json)
    except:
        return {}
    return version_info.get('items', {})

def _version_tuple(v):
    version_string = os.path.splitext(v['name'])[0].rpartition('_')[2]
    return tuple(int(x) for x in version_string.split('.'))


def get_sdk_urls(sdk_versions):
    python_releases = [
        v for v in sdk_versions
        if v['name'].startswith('featured/google_appengine')]
    current_releases = sorted(
        python_releases, key=_version_tuple, reverse=True)
    return [release['mediaLink'] for release in current_releases]


def main(argv):
    if len(argv) > 2:
        print('Usage: {} [<destination_dir>]'.format(argv[0]))
        return 1
    dest_dir = argv[1] if len(argv) > 1 else '.'
    if not os.path.exists(dest_dir):
        os.makedirs(dest_dir)

    if os.path.exists(os.path.join(dest_dir, 'google_appengine')):
        print('GAE SDK already installed at {}, exiting.'.format(dest_dir))
        return 0

    sdk_versions = get_gae_versions()
    if not sdk_versions:
        print('Error fetching GAE SDK version info')
        return 1
    sdk_urls = get_sdk_urls(sdk_versions)
    for sdk_url in sdk_urls:
        try:
            sdk_contents = StringIO.StringIO(urllib2.urlopen(sdk_url).read())
            break
        except:
            pass
    else:
        print('Could not read SDK from any of {}'.format(sdk_urls))
        return 1
    sdk_contents.seek(0)
    try:
        zip_contents = zipfile.ZipFile(sdk_contents)
        zip_contents.extractall(dest_dir)
        print('GAE SDK Installed to {}.'.format(dest_dir))
    except:
        print('Error extracting SDK contents')
        return 1

if __name__ == '__main__':
    sys.exit(main(sys.argv[:]))

This is google's script to download and extract their App Engine SDK , included here only for completeness

Push a commit to trigger a deployment

The whole point of all of that work is to make life easier for every commit that follows.

git add --all .
git commit -m 'example google app engine app deployed via bitbucket pipelines'
git push

You can monitor the results of every change in the Addon's output logs:

https://bitbucket.org/johnpfeiffer/continuous-deployment-bitbucket/addon/pipelines/home#!/

You can monitor the deployment history in Google App Engine's console:

https://console.cloud.google.com/appengine/versions?project=bitbucket-pipelines&serviceId=default

And of course see the currently deployed application in action:

https://bitbucket-pipelines.appspot.com/

Actually using automated testing

CI (continuous integration) testing is the ideal next thing to add as this allows you to automatically prevent critical errors from being deployed to production.

This means adding unit tests (test_main.py) and then running them by updating bitbucket-pipelines.yml

- python test_main.py

It is also possible to keep extending the work required by installing any requirements.txt dependencies (i.e. only required for testing) or running post deployment end-to-end smoke tests.

Double checking security

If your repository is public the pipelines log outputs will be public. Double check that you are not "leaking" your API key, secrets, or hardcoded passwords. =|

Troubleshooting

The current Google Cloud project [go-lanscan] does not contain an App Engine application. Use gcloud app create to initialize an App Engine application within the project.

You need to create the initial app engine project, either with the WebUI https://console.cloud.google.com/appengine/start?project=bitbucket-pipelines or using the GCloud CLI

You will need to be prompted for your region, i.e. us-central

"Does not contain a valid app engine project"

Most likely the app.yaml has deprecated fields or invalid characters

"does not contain an App Engine application."

Most likely the app.yaml has deprecated fields or invalid characters, or maybe missing a main.py altogether

"ERROR: The [application] field is specified in file [/opt/atlassian/pipelines/agent/build/app.yaml]"

This is because previously app.yaml included "application" and "version" but those lines are now deprecated, delete them

"You do not have permission to access app "

Most likely the Role of "App Engine Admin" still needs to be added, use the IAM for the project to update the Permissions

e.g. https://console.cloud.google.com/iam-admin/iam/project?project=bitbucket-pipelines

Or ensure you have an API key generated and added to the Bitbucket Pipelines environment ... or delete all API keys and service accounts and do them again (because it seems to get stuck if you have encrypted API key or something)

"Caller does not have storage.objects.list access to bucket staging.bitbucket-pipelines.appspot.com."

Ensure the Role "Storage Object Admin" was added to the Roles during creation, see above

This guy got really close and helped me find the hint about Storage Object Admin: http://www.deadunicornz.org/blog/2017/01/31/travis-ci-and-deploying-golang-apps-to-gae/

Using Docker with Bitbucket Pipelines

I could reduce the time required to build (and decouple the build stages) by using a Docker image that already contains the GCloud SDK.

This also extends the flexibility if part of your build flow is to build and tag a Docker image as an artifact that can be used for multiple tests (i.e. parallelization) and especially if you are already using Docker in Production.

Create an automated docker image build in Docker Hub

Link a source code repository to Docker (sadly the bitbucket one requires overly broad permissions whereas the github one only requires public read)

Follow the steps in their tutorial to create a repository with a Dockerfile and then have it auto-build an image and upload it to Docker Hub

Unfortunately sometimes you have to just log in to Docker Hub and press the "Trigger" button to start the build since "automatically" does not seem to trigger correctly, e.g. https://hub.docker.com/r/foupfeiffer/gcloud-sdk/~/settings/automated-builds/

Using a public image from Docker Hub

To verify App Engine publishing from the Docker container manually: docker run --rm -it --volume /opt/mysecrets:/opt/mysecrets --volume /opt/myrepo:/opt/myrepo foupfeiffer/gcloud-sdk /bin/bash

Run these commands in the Docker container...

cp -a /opt/mysecrets/api_key.py /opt/myrepo
export CLOUDSDK_CORE_DISABLE_PROMPTS=1
export CLOUDSDK_CORE_PROJECT=example-gae-id
cd /opt/myrepo
gcloud auth activate-service-account --key-file /opt/mysecrets/client-secret.json
gcloud --verbosity=error app deploy app.yaml --promote

You are about to deploy the following services: Uploading 7 files to Google Cloud Storage

This assumes you have already correctly setup your Google Cloud Project, Google App Engine Project, Google Permissions, and that your repository has the correct app.yaml and main.py

Updating your bitbucket-pipelines.yml to use a public docker image

With your open sourced Dockerfile and docker image built in docker hub we can simplify some of the steps in our previous bitbucket-pipelines.yml file:

image: foupfeiffer/gcloud-sdk

pipelines:
  default:
    - step:
        script:
          - export CLOUDSDK_CORE_DISABLE_PROMPTS=1
          - echo "key = '${GOOGLE_API_KEY}'" > api_key.py
          - echo ${GOOGLE_CLIENT_SECRET} > client-secret.json
          - gcloud auth activate-service-account --key-file client-secret.json
          - gcloud --verbosity=error app deploy app.yaml --promote

One you have pushed this change every future build will have the capabilities of your docker image (i.e. go build)

Bitbucket Pipelines and Go

This article was about deploying a python application (which in my use case happened to distribute a Go binary).

For more information about a Golang application with Go Build and Tests...

Bitbucket Pipelines and Testing with a Database

There is an example of how to leverage two of the most common databases, MySQL and Postgres, in your testing pipeline:

https://confluence.atlassian.com/bitbucket/test-with-databases-in-bitbucket-pipelines-856697462.html

Obviously since you can execute the pipeline with your own customer Docker image you can extend and build your required test/environment dependencies.

Mid 2017 technology and business prediction for 2018 and beyond

2017-05-07T20:00:00-07:00

The future already happened

The internet boom has arrived silently and the "everything is online" happened while we were all looking down at our smart phones.

New giants have emerged to rule the economic landscape that have eclipsed, at least in virtual specie, their physical-and-brick-and-mortar competitors: Apple, Alphabet (Google), Microsoft, Amazon.com... Facebook...

https://en.wikipedia.org/wiki/List_of_public_corporations_by_market_capitalization#2017

The market can stay irrational longer than you can stay solvent

-John Maynard Keynes

We have entered into an uncharted area in terms of economics; much financial data and many models indicate that 2017 is very near the peak of a trend.

The Case-Shiller Home Price Index is topping the maximums of 2006 and almost double the "normal" of 100 and the CyclicallyAdjustedPriceEarnings ratio is at 29, also soon to be double the median of 16. Neither of these tools are intended for "timing the market" or identifying the end of a trend, but they certainly indicate a longer wait for expected return on investment and higher level of risk of a "return to the mean".

The CAPE ratio hit 44 right before the "dot-com" bust (pokerface)

The most optimistic perspective is that globalization and technology have dramatically increasing efficiencies, though that same combination drove the "Roaring 20's" and (along with political and economic factors) created the "Great Depression".

Prediction of a Crash

I'll make a prediction (not that it will change the future in any way). Given the cyclical nature of economies, it has to come to pass sometime ;]

The United States is due for an economic crash soon

At a minimum a recession, most likely another "Great Recession", hopefully not another "Great Depression".

There will most likely be a catalyst like unexpected political collapse (Presidential impeachment), armed conflict (i.e. North Korea), or environmental disaster (i.e. global pandemic or solar flare).

Technology is not a root cause but creates the sense of instability in society and economics as efficiencies disrupt existing balances.

A "perfect storm" of global economic weakness with globalization that interconnects all economies

Populist moves to nationalism and tariffs

Deregulation and "pro-business" policies

Increased Migration

In the wake of global instability people will seek, in ever larger numbers, societies and infrastructure that are stable.

Those places that adopt policies and mechanisms to successfully integrate and take advantage of the influx of newcomers will do quite well.

Those places that attempt to remain isolated will incur ever growing costs and simultaneously be outcompeted by places that have successfully leveraged new ideas, new labor, and new capital.

Technology Predictions

We are rapidly moving into a new era of technology and information where energy is becoming cheaper than ever and computing has become more powerful and ubiquitous than ever.

Scarcity of Privacy

The proliferation of smart phones with cameras everywhere, drones, actively listening devices like Siri/Alexa/Google means that it is far harder than ever to be alone, unwatched, and unheard.

Big data means an unprecedented amoutn of correlation can be created.

While this benefits society in emergency situations (i.e. earthquakes, fire, crime, etc.) it also means there will be a lucrative business opportunity in technology that can actually restore privacy.

http://timelines.latimes.com/snapchat-stanford-ipo/ (a $30B+ social network that makes data ephemeral)
https://en.wikipedia.org/wiki/Faraday_cage

Pendulum swings back to decentralization

"The Cloud" is very popular now, the availability of energy and local compute (including smartphones) means it will become cheaper and cheaper to compute things locally.

Even as the internet has allowed incredible flows of data and exchange of information the amount of locally generated information ("internet of things") is growing very large very quickly as well.

Yet given the increasing requirement for security (and privacy) it makes a lot more sense to keep data and compute inside a protected space. One of the simplest ways to prevent hacking, keep data secure, and increase privacy is to never have the data leave its place of origin.

Homes, and more importantly businesses, will not want every camera, microphone, and sensor sending data into "The Cloud" to centralized (and often vulnerable) warehouses. Instead there is an opportunity for businesses and technology that leverage cheap compute to create "mini private clouds".

The slow deprecation of SMTP and HTTP (both unencrypted plaintext) being the backbone of the internet will be increasingly replaced with secure alternatives (encrypted, dedicated networks, extra authentication layers like 2FA and MFA) to create virtual tunnels.

The companies and tools that allow redundancy and accessibility of data in a secure fashion will profit from the abundance of local compute/storage and scarcity of security and privacy.

Golang Concurrency Goroutines and Channels

2017-04-12T22:54:00-07:00

If there is a killer feature to Go it is the focus on concurrency. This article captures some of the basics and I hope to someday write a follow-up article on more advanced topics.

"Go is a compiled, concurrent, garbage-collected, statically typed language"

https://talks.golang.org/2012/splash.article

Distributed systems and large data sizes mean developers are forced to think in parallelism, or are they?

Using Go channels (based upon Communicating Sequential Processes https://en.wikipedia.org/wiki/Communicating_sequential_processes, http://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf) developers can write code that feels very imperative and sequential, but designed in such a way that parallelism comes easily.

Go and Goroutines

A goroutine is a "lightweight thread" that allows for a far higher amount of concurrency than just depending on OS processes or even traditional threading (and much simpler than attempting to explicitly organize around a defined number of processors or threads).

In a very meta sense every Go program uses concurrency because the main function itself is an implicit goroutine (and will not wait or might block forever ;)

package main

import (
    "fmt"
    "time"
)

func example() {
    time.Sleep(1 * time.Second)
    fmt.Println("example")
}

func main() {
    go example()
    fmt.Println("done")
}

This will exit without printing "example" because using the "go" keyword runs the example function in a new goroutine

There is no explicit external management of a goroutine once it has started, terminating a goroutine is implemented via an exception or exit in the code that the goroutine is running, usually signalled via a channel

Since Goroutines are cooperative they are not pre-empt-able...

Sync with a WaitGroup

The most straightforward way to fix the previous trivial example is to specify in advance that the implicit main goroutine should wait before continuing...

package main

import (
    "fmt"
    "sync"
    "time"
)

func example() {
    time.Sleep(1 * time.Second)
    fmt.Println("example")
}

func main() {
    var wg sync.WaitGroup
    wg.Add(1)
    go func() {
        example()
        wg.Done()
    }()
    wg.Wait()
    fmt.Println("done")
}

The example function remains unchanged

The waitgroup will expect one call of "Done"

The go keyword now calls an anonymous function that calls wg.Done() (accessed using closure) after example()

The waitgroup.Wait() blocks until the correct number of Done() calls have been made

The example function sleep and print finally finish

The waitgroup unblocks and the main goroutine can finally print "done" and exit

Channels

Channels are the recommended way of communicating when using goroutines (and sharing resources).

<- the arrow always points to the left

package main

import (
    "fmt"
    "time"
)

func main() {
    c := make(chan int)
    go func() {
        fmt.Println("sleeping...")
        time.Sleep(1 * time.Second)
        c <- 42
        close(c)  // IF NOT CLOSED THEN DEADLOCK
        // c <- 2  DO NOT SEND TO A CLOSED CHANNEL, IT WILL PANIC
    }()
    fmt.Println("anonymous function, a channel passed a value via closure", <-c)
    v, ok := <-c
    fmt.Println(v, ok)
}

since a channel is a reference type "make" is used and the <- sends a value to the channel, later <- is used to receive a value

a closed channel cannot be written to further and will PANIC "send on a closed channel"

if you forget to close a channel, later reading from an open unbuffered channel which does not have data will exit "fatal error: all goroutines are asleep - deadlock!"

an "unbuffered" channel "blocks" until both the sender and receiver are ready

the Println function reads from the channel, in this case the channel acts as a synchronization tool that blocks at the Print statement and prevents the main goroutine from exiting

The final printed output will be: "0 false", since the channel is closed and empty subsequent receives will return a the empty value "zero" and the state of the channel (in this case "false")

Buffered channels and returning a value

A common problem is one part of the application running faster than another part and one way to "unblock" the fast part is to use a buffer to create a queue for the slower part to catch up. This kind of issue occurs in a "pipeline" of producers/consumers (also known as sources/sinks)...

package main

import (
    "fmt"
    "log"
    "sync"
    "time"
)

func slowReceiver(c <-chan int, wg *sync.WaitGroup) {
    for {
        n := <-c
        fmt.Println("received", n)
        time.Sleep(time.Second)
        // c <- 42 // THIS WOULD CAUSE AN ERROR "(send to receive-only type <-chan int)"
        wg.Done()
    }
}

func fastSender(c chan<- int, wg *sync.WaitGroup) {
    for i := 0; i < 5; i++ {
        c <- i
        wg.Add(1)
        fmt.Println(i)
        // fmt.Println(<-c) // THIS WOULD CAUSE AN ERROR "(receive from send-only type chan<- int)"
    }
}

func main() {
    start := time.Now()
    var wg sync.WaitGroup
    c := make(chan int, 3)
    go slowReceiver(c, &wg)
    fastSender(c, &wg)
    wg.Wait()
    log.Println(time.Since(start))
}

This example shows how to specify a channel of type that either only sends or only receives

log works the same as fmt and Since() a very convenient way to output elapsed time

Without the waitgroup Wait() main would exit after 1 second with only "received 1" and never reach "received 4" (5 seconds)

Using Select to not block a channel

Channels are most useful when they can block asynchronously until an event occurs.

package main

import "time"
import "fmt"

func main() {
    c := make(chan string)
    q := make(chan bool)
    go myDelayedQuit(q)
    go mySleep(c, 1)

    fmt.Println("begin non blocking wait...")
    for {
        select {
        case msg := <-c:
            fmt.Println("received:", msg)
        case <-q:
            fmt.Println("done")
            return
        }
    }
}

func mySleep(a chan string, n int) {
    time.Sleep(time.Second * time.Duration(n))
    a <- "woke up"
}

func myDelayedQuit(b chan bool) {
    time.Sleep(time.Second * 2)
    b <- true
}

for loops forever until the return statement

select will wait and whenever a case can be filled it will unblock

after 1 second the sleep function is done and sends the "woke up" message

after 2 seconds the true boolean is sent and main finishes

https://play.golang.org/p/JBrhHZVq6a

Real Example of Concurrency in a LAN Scanner

A real world example is discovering all of the hosts listening on a given port in local area network (subnet).

In a serial example waiting 2 seconds for each host to respond would mean waiting 512 seconds in a "normal" /24 subnet of ~256 hosts (ignoring the .255 broadcast and .0)

found := make(chan IPCheckResult, max)
var wg sync.WaitGroup
wg.Add(max)
for _, a := range addresses {
    fmt.Println("checking", a)
    // https://golang.org/doc/faq#closures_and_goroutines
    go func(ip string) {
        checkIP(ip, *port, found)
        wg.Done()
    }(a)
}
wg.Wait()
close(found)

Here the channel is simply used as a "lock free" place to aggregate all of the results of the goroutines, sync occurs via the waitgroup which will wait until every pre-added item is decremented by a Done(), there is definitely a possibility for an off-by-one gotcha that will hang your program!

https://bitbucket.org/johnpfeiffer/go-lanscan/src

Troubleshooting Race Conditions

A common gotcha is that in Go maps are not safe for concurrent use:

In this example of a simple in memory cache the expiration was implemented...

func (memory MemoryCache) Set(key string, value string, expiresSeconds int) {
    memory.m[key] = value
    timer := time.NewTimer(time.Duration(expiresSeconds) * time.Second)
    go func() {
        <-timer.C
        memory.Delete(key)
        // log.Println("Timer triggered cache expiration for", key)
    }()
}

The error here is that the goroutine that wakes up to "expire" and remove a key/value pair from the map may contend with any other later operation (i.e. Get, Set, Delete) that is also modifying the map

https://bitbucket.org/johnpfeiffer/go-cache/src

A really helpful tool is to run go test -race , it may take a little bit but "WARNING: DATA RACE" is pretty clear.

https://blog.golang.org/race-detector

More random thoughts on concurrency and control

Controlling goroutines is like controlling threads or even any other control flow.

Iterative vs Recursive calculation of factorial means either predetermined count of iterations or an indeterminate count (recursion) with a (base case) signal for termination (often called a sentinel value).

So either the External Controller knows when to stop or each actor checks if it is time to stop.

An important consideration to termination is cleanup

If the actor is responsible for self cleanup (as it knows what resources it is using) this can lead to resource leaks if the actor terminates unexpectedly without cleaning up.

If using "dependency injection" then the Controller has knowledge of what resources were shared with the agents and can do cleanup, even if actors terminate unexpectedly.

An increasingly common approach is for the Framework to facilitate cleanup so that the complexity is removed from both the Controller and the actors (e.g. Garbage Collection or Go deferred)

Poison Jobs cons and pros

One challenge with workers and a queue is the "poison job" which may create inefficiency or halt the system entirely as each worker who takes the job blocks/loops forever or terminates unexpectedly.

One possible solution is to have a retry count so that any job which has timed out or failed and retried repeatedly is moved to a FailedJob queue (for future manual inspection and debugging) or logged and dropped entirely.

Interestingly something like a "poison job" is actually a useful way to signal to concurrent actors to have an orderly termination even if they have not completed their jobs (i.e. a full system shutdown has been initiated and we want to trigger self cleanup).

Patterns for Channels and Flow Control

Go Channels represent a way to map out the dependencies and then allow the compiler to optimize for parallelization.

Waiting indefinitely for all goroutines to return is naive, and any termination signal must have the ability to truly interrupt work in progress, which it does NOT for goroutines, so any call that a goroutine is making MUST have a timebound where it can check for the termination signal.

Therefore write your goroutines carefully knowing that you cannot cancel/aka force them to return from an infinite loop/long call, unless you exit main entirely.

This means that for architecture decisions it is important to consider small separate services/applications that can provide resource usage transparency and termination control. While this is becomes a tradeoff with coordination/orchestration complexity it is worth having modularity and clear boundaries in any application of decent complexity.

Simple and deterministic: fan out with predefined count, close the channel
Provide timeouts and retries: a failure can occur anywhere and graceful degradation means setting limits and dealing with ephemeral errors
Use the select statement for a non-blocking way to check for early application termination events
Use buffers to even out spikes in work from sources/production

https://www.ardanlabs.com/blog/2017/10/the-behavior-of-channels.html

Code is for Humans

2017-02-25T20:34:00-08:00

Code is the automation of a solution

I often feel like the purpose of Programming is lost in the many discussions and debates about Programming.

Sometimes the code is conflated as being the solution. Whatever problem is being worked on has to be thought through and different possible solutions evaluated. Then one approach has to be implemented and tested. That implementation can take many forms: https://en.wikipedia.org/wiki/Turing_completeness
Alternatively a solution is seen as perfected because it is written in code. Yet all it takes is one more edge case, one more nuance on a variable, and the "solution" will fail. (Bugs!) Perhaps this is because the trivial examples of coding are often related to math, like adding two numbers, that we are somewhat misled to believe that the correct code will always solve the problem (and amplified by Computer Science which uses proofs to mathematically prove algorithmic solutions). https://en.wikipedia.org/wiki/P_versus_NP_problem
Selecting the correct algorithm assumes understanding of the problem, the constraints, and of course the end goal. Much like picking an off-the-shelf tool or open source library or framework, there has to be some comprehension of whether their solution is your solution.

The purpose of the automation is to repeatedly solve the problem (assuming the same environment and same inputs).

So first you really have to sit down and think through the solution. Usually this involves pen (or pencil) and paper though the whiteboard is certainly one of my favorites.

I certainly understand people who want to work towards solutions by coding (i.e. the IntegratedDevelopmentEnvironment and programming paradigm as just another tool for our brains) but I suspect it can also lead to distractions of accidental complexity (limitations or unfamiliarities of the programming language syntax/framework) and sometimes worse yet unnoticed biases of the tools preclude some of the best solutions. (The infamous "imperative vs functional" debates ;)

Non trivial problems will require solutions that involve tradeoffs and compromises (i.e. the classic "execution time" vs "resources required").

https://en.wikipedia.org/wiki/Software_engineering

Code is for humans, not computers

Human readable code must be transformed into instructions for a machine which is what executes all of the computations. The machine has no understanding of whether the instructions will solve the problem.

This fundamental impedance mismatch is one of the major challenges to programming. Humans do not always know all of the correct instructions to provide. Machines will faithfully execute whatever is given to them, including conflicting commands or erroneous data.

So the history of the abstraction of computer programming very often reads like the evolution away from the physical hardware towards humans expressiveness because the better able we are to describe something the more likely we are to document a correct automation of a solution.

https://en.wikipedia.org/wiki/History_of_programming_languages

Solving Yesterdays Problems of Performance

While the earliest hardwired machines filled large rooms and had different kinds of accidental complexity and bugs https://americanhistory.si.edu/collections/search/object/nmah_334663 , later generations had to deal with getting the most performance out of the machines https://en.wikipedia.org/wiki/Assembly_language#Historical_perspective.

These powerful low level languages can also generate some of the most persistent and pernicious bugs via manual memory management, pointers, and buffer overflows.

The amount of developer time required to create correct code has dramatically reduced as the speed of computation has increased and the tools (including the programming languages) are better able to "get out of the way" and avoid the accidental complexity of optimizing for performance.

It is rare that the purpose of a program is to add numbers as quickly as possible

Programming languages help with explicitness by removing ambiguity in natural languages

A sentence in English, "We saw her duck", can have multiple meanings https://en.wikipedia.org/wiki/Ambiguity#Linguistic_forms)

Programming languages force human expressiveness to be less ambiguous (i.e. the responsibility of the mismatch impedance of incorrect instructions falls squarely on the humans).

Names are important and should be well thought out https://en.wikipedia.org/wiki/Naming_convention_(programming)
Short variables and acronyms can confuse, mislead, or misdirect other humans who will modify or extend code based on that misunderstanding
Performance specific changes can become digressions and noise that distract or make brittle the tracing of the required solution logic

Explicit communication because magic is incomprehensible

The problem with short meaningless variable names in unreadable code littered with performance optimizations is they prevent solving The Problem.

Some considerations and anti-patterns:

Straight to coding (no research)
- Real time systems that keep an aircraft in the air must pay attention to runtime constraints
- Overlooked requirements and misunderstanding the problem domain
- Not knowing what the correct answer will look like (i.e. not having test/control inputs and outputs)
Absence of acceptance tests to prove that it really is solved and the infamous "it works on my machine"
The fallacy of "Perfection"; for a solution to provide value there has to be a mechanism to empirically prove it works
"Done" is not "code complete" (even with tests ;) , it is Integration Tests, Acceptance Tests, Performance Tests, Soak Tests, actually shipped to the "wild" where it survives real environments and edge cases
Intermittent behavior = automating of a solution should provide consistent results
Ideologue = a technology looking for a problem
Premature Optimization https://en.wikipedia.org/wiki/Program_optimization#When_to_optimize
Premature Generalization http://wiki.c2.com/?PrematureGeneralization
https://en.wikipedia.org/wiki/Spaghetti_code
https://en.wikipedia.org/wiki/Big_ball_of_mud
Now is the only time that matters, when actually the more successful you are the more likely the code will continue and need maintenance https://en.wikipedia.org/wiki/Software_maintenance

Process Improvements to actually Solve a Problem

Considering the complexity required to actually solve a problem it would be fair to say many iterations are required.

Some of the "tools" that have helped the iterative process:

Software https://en.wikipedia.org/wiki/Stored-program_computer#History
Testing http://web.archive.org/web/20161024015955/http://www.testingreferences.com/testinghistory.php
Logging https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
Monitoring
Automated deployments
Version Control http://web.archive.org/web/20170104162946/http://layervault.tumblr.com/post/102541175774/the-history-of-version-control

Additionally any implementation must be bound by a "good enough" state as perfection cannot coexist with the ever changing real world.

Data is required to understand progress (or regression).

Humans must keep modifying until "done" (even in the advanced example of https://en.wikipedia.org/wiki/Genetic_programming)

The steady increase in computing power (https://en.wikipedia.org/wiki/Moore%27s_law) means Maintainability trumps Performance. Being able to get reproducible results was codified long ago in the https://en.wikipedia.org/wiki/Scientific_method. =]

Collaboration and Crowdsourcing without Group Think

There seems to be a strong reaction to the term "pair programming" and it is no uncommon for code reviews and pull requests to be a source of emotional angst and team friction.

These techniques are well established ways of improving quality:

While the myth of the individual genius over-emphasizes the outlier it is far more common for a group to achieve projects of any large size (even with software and technology acting as a multiplier).

I am not advocating pure democracy (or just accepting status quo) as the only way of building things but it is clearly preferrable to have a variety of skills (i.e. architecture, mathematics, verification, design, etc.)

It may be that working together on intellectual endeavours is currently less intuitive for humans than working together on physical challenges and that a fledgling industry that is chronically short of trained and experienced workers is not selecting and creating environments that are conducive to group working.

One of the most challenging aspects is disambiguating where something has been successful in the field (mature) versus "it's always been that way" complacence. Different isn't always better but should always be honestly evaluated.

Configuration Patterns

Configuration is just another part of "Do Not Repeat Yourself" https://en.wikipedia.org/wiki/Don't_repeat_yourself .

Without configuration each program would have to be rewritten with the a new "configuration" portion hardcoded each time. (Though software is an improvement over having to build new hardware for each new configuration...)

So it is a pragmatic way to extend the utility of an automated solution.

"When should configurations be applied"?

Configuration is passed as a parameter when the program first starts
Configuration is loaded from a configuration file when the program first starts
Configuration is loaded from Environment variables when the program first starts
Configuration is loaded from a configuration file whenever a change to that file is detected by the program
Configuration is loaded from a configuration file whenever a module is loaded (i.e. "lazy loading")
Configuration is loaded from Environment variables whenever a module is loaded (i.e. "lazy loading")
https://12factor.net/config
https://en.wikipedia.org/wiki/Lazy_loading

For a short execution time there is little difference between loading at startup versus runtime

What are the impacts of restarting the service (in order to reload the new configuration changes)?

Are there parts that have to be loaded first and then wait for a slower dependency to be available?

Prefer Determinism for Reasonability

Loading and initializing all configuration when the program first starts is one way of attempting to create determinism in the code paths that are running in memory.

The benefit of "hot swapping" is applying changes to existing code while it is still running.

The computer is not going to get confused. It does not care if the data being passed to the module in memory is correct or incorrect, but the new dynamic result may ruin that beautifully automated solution, sometimes in almost undetectable ways.

As we humans struggle with ever increasing complexity (both in the software and hardware) we should focus on how to reduce variability (that includes during coding, during compilation, and especially at run time).

Some alternatives tend to take advantage of the cheaper cost of computing and increasingly distributed/networked systems:

Send a reconnect signal to clients to use a new endpoint
Start up a second process and have the operating system pass the network connection from the old to the new process
A Load Balancer or other connection holding component that can direct traffic to the new service

https://www.martinfowler.com/bliki/BlueGreenDeployment.html

Modularity

Smaller pieces can be easier to understand and assert for validity. The "Do One Thing" principle helps the human who is composing a solution (ideally from re-usable components) to understand which tool is right for the job.

This also allows for leveraging "seams" to investigate or decouple code (which drastically helps with maintenance).

Updating, upgrading, or replacing a well defined component will be easier than something that is tightly interwoven with all of the other pieces.

There is a natural tension with re-usability since something with a slight modification that can be re-used reduces the overall code footprint. The key here is a clear understanding of whether the actual test footprint and complexity have been reduced.

Also it is possible to decompose into such small parts that they have no logical coherence =[

Immutability

One last technique I would like to highlight is immutability. The idea is by preventing change it can be easier to trace and determine the expected outcome. (Or discover the exact point at which there is an unexpected deviation).

It comes at a trade-off of increased resource consumption (i.e. memory) and creating an unmanageablely large number of entities (with possible corresponding orchestration or scale issues)

Functional programming definitely tends towards immutability ;)

Security Encryption HTTPS OpenSSL SSH Keygen VPN Letsencrypt Certbot

2017-02-16T23:34:00-08:00

As our lives become increasingly monitored and digital the privacy of being unobserved or having a private conversation that we used to be able to take for granted now requires extra effort. The more people who choose to use these easy and readily available tools the more privacy will become the standard rather than the exception.

Symmetric and Asymmetric Encryption

Using a shared secret key is generally the simplest way to encrypt, both parties use the same key to encrypt and decrypt.

Asymmetric encryption (aka "public and private key") allows for a message to be encrypted without the parties having to meet or exchange a secret.

Both are often used together in a complementary fashion.

Codes and the ability to communicate in secret have a very long history which many others have documented and thanks to the amazing efforts of many many people we have the ability to communicate our billions of messages with relative privacy.

It is important to understand that good encryption depends on randomness to make it as hard as possible for an "attacker" to reverse engineer or guess the key, i.e. https://en.wikipedia.org/wiki//dev/random

The following is my summary of some of the most common and useful tools...

Pretty Good Encryption for Pretty Good Privacy

While there is a lot of value in leveraging the GPG public and private keys for authenticity checking this is just about encrypted data...

GPG Encryption

gpg -c example.tar.gz

prompts for a password to access the keyring and leverages existing (or automatically generates) public and private keys, and outputs example.tar.gz.gpg

Note: encrypting before compressing is meaningless since encrypted data is random and compression depends on repetition/patterns

gpg --yes --passphrase=password -c example.txt

non interactive encryption (if the private key is password protected), outputs example.txt.gpg

echo "password" | gpg --yes --no-tty --batch --passphrase-fd 0 --output encrypted.txt.gpg  --symmetric --cipher-algo AES256 plain.txt

non interactive AES 256 symmetric cipher rather than public/private keypairs

GPG Decryption

gpg example.tar.gz.gpg

enter the passphrase to access the private key to decrypt the file

gpg --yes --passphrase=password example.txt.gpg

non-interactively access the private key to decrypt the file

echo "password" | gpg --yes --no-tty --batch --passphrase-fd 0 --output plain.txt --decrypt encrypted.txt.gpg

non interactively decrypt with symmetric encryption

Do not pick password as your password ;p

https://www.gnupg.org/gph/en/manual/x110.html

AES 256 Encryption

AdvancedEncryptionStandard is one of the US and world standards as an encryption algorithm.

Security benefits from transparency in that if you provide the algorithm and source code in plain sight and attackers are still unable to decrypt/crack/manipulate the data then you are probably in good shape.

Perhaps one of the most well known projects (open source and free!) to advance the practice of encryption is https://www.openssl.org/

Here we encrypt and decrypt a text file:

openssl aes-256-cbc -in plain.txt -out message.encrypted

prompted for a password (symmetric key) to encrypt the file with AES 256

openssl aes-256-cbc -d -in message.encrypted -out plain.txt

prompted for a password to decrypt the message

openssl aes-256-cbc -in plain.txt -out message.encrypted -pass pass:YOURPASSWORD

non-interactively provide the password to encrypt the file with AES 256

openssl aes-256-cbc -d -in message.encrypted -out plain.txt -pass pass:YOURPASSWORD

non-interactively provide the password to decrypt the file with AES 256

an incorrect password will create a zero byte file

openssl aes-256-cbc -a -d -in message.encrypted -out plain.txt

if it's base64 encoded do not forget the -a

Generating SSL Certificates and Private and Public Keys

Perhaps the most common use of TLS/SSL are the keys used to encrypt communication with a web server, these examples use a "self signed certificate" which most libraries and browsers will not trust.

"Root Certificate Authorities" are the ones you send a "Certificate Signing Request" to generate a certificate that can be mathemetically trusted by the existing software libraries and browsers

More Private and Public Key basics for SSL Certificates

The private key contains a series of numbers. Two of those numbers form the "public key", the others are part of your "private key".

The "public key" bits are also embedded in your Certificate (we get them from your CSR - certificate signing request).

Verify the SSL Certificate and Private Key match

(openssl x509 -noout -modulus -in server.pem | openssl md5 ; openssl rsa -noout -modulus -in server.key | openssl md5) | uniq

outputs a single hash if matching, if two hashes are output then it is not unique and the key and cert do not match

To check that the public key in your cert matches the public portion of your private key:

openssl x509 -noout -text -in server.crt
openssl rsa -noout -text -in server.key

But since the public exponent is usually 65537 and it's challening to compare a long modulus you can use the following approach:

openssl x509 -noout -modulus -in server.crt | openssl md5
openssl rsa -noout -modulus -in server.key | openssl md5

the "modulus" and the "public exponent" portions of the cert and key ... aka the two md5sum hashes should match

Verify the Cert and Intermediate match and then Verify

openssl verify -purpose sslserver -CAfile intermediate.pem -verbose server.pem

ensure the intermediate certificate matches the SSL certificate (if it does not SSL trust will not work correctly)

Verify a CSR matches the Key and Certificate

To check to which key or certificate a particular CSR belongs you can compute

openssl req -noout -modulus -in server.csr | openssl md5

Creating keys and certificates and certificate requests

openssl New Key and Cert One Liner

openssl req -subj '/CN=example.com/O=My Company Name LTD./C=US' -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout server.key -out server.crt

one liner to generate a self signed SSL key and certificate, since SSL certs are rotated regularly we can use 2048 instead of 4096 bits

openssl certificate commands

openssl req -out CSR.csr -new -newkey rsa:2048 -nodes -keyout privateKey.key

create 2048 bit nopass key + csr (certificate signing request - usually sent to a Certificate Authority that is in the root chain bundled with major browsers and libraries)

openssl x509 -req -days 365 -in CSR.csr -signkey privateKey.key -out cert.crt

self sign a CSR and generate a self signed SSL certificate

openssl rsa -in mykey.pem -pubout

display the public key generated from the private key

openssl rsa -in mykey.pem -pubout -out mykey.pub

use the private key to generate and save the public key to a file

openssl req -text -noout -verify -in CSR.csr

Verify a CSR certificate request

openssl rsa -check -in cert.key

Verify a key

openssl x509 -text -in cert.crt

View INFO about a cert and see the cert

openssl x509 -noout -issuer -subject -dates -in cert.crt

View specific items in the certificate (and do not print out the full certificate)

openssl req -out CSR.csr -key privateKey.key -new

Generate a certificate signing request (CSR) for an existing private key

openssl x509 -x509toreq -in certificate.crt -out CSR.csr -signkey privateKey.key

Generate a certificate signing request based on an existing certificate and private key

Creating an RSA certificate with a password

openssl genrsa -des3 -out domainname.key 2048

Create a 2048 bit private key with passphrase

openssl req -new -key domainname.key -out domainname.csr

CREATE A CSR, CEERTIFICATE SIGNED REQUEST

Common Name (domain name) = fully qualified domain name

openssl x509 -req -days 365 -in CSR.csr -signkey privateKey.key -out cert.crt

Generate a self signed cert

Remove a passphrase to install a private key on a server

openssl rsa -in domainname-passphrase.key -out domainname-server.key

openssl rsa -in privateKey.pem -out newPrivateKey.pem

Use openssl's built in webserver

openssl s_server -cert server.pem -key server-nopass.key

start a TCP server with the provided certificate and key on the default port of 4433

openssl s_server -cert server.pem -accept 4433 -www

start an HTTPS server with the cert and key combined in a .pem on the specific port (e.g. 4433)

telnet localhost 4433

verify basic network connectivity

wget https://localhost:4433 --no-check-certificate

verify basic network connectivity and download the contents without validating the SSL certificate

openssl s_client -showcerts -connect localhost:4433

show the ciphers and certificates of a server, "18 (self signed certificate)

openssl s_client -connect ldaps.example.com:10636 -CAfile intermediate.crt

show the ciphers and certificates of a server while providing client-side the Root + Intermediate Chain certificates

Get a remote server ssl certificate using openssl and sed

echo | openssl s_client -connect ${REMOTEHOST}:${REMOTEPORT} 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'

Getting and verifying an Intermediate Certificate

Popular web browsers will often have lock symbols or other ways for you to see (and download) a certificate)

startup vm with webserver (tomcat) + current certificate
chrome browse to the https://10.10.10.199
Click on the lock (certificate) symbol => Certificate information "Issued to: example.com"
Click on the Certification Path (tab) ... see the root CA + intermediates + cert.
click on "GeoTrust" (or the top level Certificate listed in the path) -> View Certificate
Details (tab) -> Copy to File -> Save Base-64 encoded x.509 (.cer)
Verify that it is different than your original cert.crt
Copy paste that file into a place where you have openssl installed (i.e. ubuntu linux)
Hostfile (/etc/hosts) so that your openssl command can use the DNS name
openssl s_client -showcerts -connect example.com:443
Without the intermediate you should receive "Verify return code: 21 (unable to verify the first certificate)"
openssl s_client -showcerts -connect example.com:443 -CAfile geotrust-intermediate.crt
With the intermediate in the command above: "Verify return code: 0 (ok)"

Finally, if the DNS name is publicly available, you can verify: https://www.sslshopper.com/ssl-checker.html

Other certificate formats

CONVERT FROM WINDOWS (IIS) PKCS12 OR PFX TO PEM

openssl pkcs12 -info -in keystore.p12

openssl pkcs12 -in original.pfx -out cert.pem -nodes

This creates both the key and the cert

You can copy and paste the certificate portion into cert.crt

Also copy and paste the private key portion into the cert.key

Configure an existing key + certificate into PKCS12

openssl pkcs12 -export -inkey cert.key -in cert.pem -out keystore.p12

KEYTOOL AND JAVA KEY STORES

Java, just like web servers, have public and private encryption keys in order to enable cryptography and encryption from within the applications.

keytool -list -v -keystore /usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/security/jssecacerts

hit enter as it has no password

keytool -list -v -keystore /etc/java-6-sun/security/cacerts

default password is changeit

keytool -list -v -keystore /etc/java-6-sun/security/cacerts -alias alpha.domain.net  -storepass changeit
keytool -list -v -keystore jssecacerts -storepass changeit | grep atmos -n4


Keystore type: JKS
Keystore provider: SUN

Your keystore contains 76 entries

Alias name: digicertassuredidrootca
Creation date: Jan 7, 2008
Entry type: trustedCertEntry

Import a signed primary certificate to an existing Java keystore

One of the most common tasks is adding a certificate to trust

keytool -import -trustcacerts -alias mydomain -file mydomain.crt -keystore /etc/java-6-sun/security/cacerts -storepass changeit
Trust this certificate? [no]:

type "yes"

keytool -list -v -keystore /etc/java-6-sun/security/cacerts -storepass changeit

keytool -printcert -v -file mydomain.crt

keytool -import -trustcacerts -alias root -file geotrust.intermediate.crt -keystore /etc/java-6-sun/security/cacerts -storepass changeit

keytool -delete -alias mydomain.com -keystore cacerts -storepass changeit

EXTRACT A CERTIFICATE FROM A JKS INTO A .DER, THEN CONVERT IT INTO A .PEM AND IMPORT INTO THE JVM cacerts

keytool -export -alias zanzibar -keystore zanzibar.jks > zanzibar.der
openssl x509 -in zanzibar.der -inform DER -out zanzibar.crt -outform PEM

keytool -import -trustcacerts -alias zanzibar -file zanzibar.crt -keystore /etc/java-6-sun/security/cacerts -storepass changeit

SSH Encryption

Secure Shell is for network access over an insecure network https://en.wikipedia.org/wiki/Secure_Shell

Create a public/private key pair for SSH

Backup any existing ~/.ssh/id_rsa (cp -a ~./ssh ~./ssh-bak) Backup any existing ~/.ssh/id_rsa.pub

ssh-keygen -t rsa -C "your_email@example.com"
chmod 400 /path/to/id_rsa*

id_rsa and id_rsa.pub *e.g. in /home/USERNAME/.ssh , https://en.wikipedia.org/wiki/Ssh-keygen chmod to modify permissions (to the "only the owner can read") since if the private key is not restricted in security the ssh client will not run but instead return an error

id_rsa IS YOUR PRIVATE KEY, GUARD IT!

id_rsa.pub IS THE PUBLIC PORTION WHICH YOU ADD TO REMOTE SERVERS

if you add a passphrase to your SSH key (to prevent hackers from simply copying the file)

ssh-keygen -y

prompted for the path to the file, then prompted for the password to outpout a public signature (.pub)

ssh-keygen -t dsa

RSA is generally preferred, protocol 2, I only include the DSA command for completeness http://security.stackexchange.com/questions/5096/rsa-vs-dsa-for-ssh-authentication-keys

ssh-keygen -l -f ~/.ssh/id_rsa.pub

Check the number of bits for key strength, it should at least 2048 bits

ssh-keygen -t rsa -C "myemail@example.com" -f $HOME/.ssh/myserver.id_rsa

Create an RSA based key with a specific email address label as an output in a specific directory and named file

output a .pub from a private key

ssh-keygen -y -f id_rsa

generate a fingerprint for verification of a host

ssh-keygen -lf id_rsa.pub

AMAZON ec2 instances have a different method, ec2-add-keypair

openssl pkcs8 -in myec2key.pem -nocrypt -topk8 -outform DER | openssl sha1 -c

ec2-import-keypair:

openssl pkey -in ~/.ssh/ec2/primary.pem -pubout -outform DER | openssl md5 -c

Adding a Public Key to a remote server

ON THE REMOTE SERVER IT SHOULD ONLY HAVE THE public key from .ssh/authorized_keys

mkdir /home/username/.ssh
sudo vi /home/username/.ssh/authorized_keys
> or  cat .ssh/id_rsa.pub | ssh username@123.45.56.78 "cat >> ~/.ssh/authorized_keys"

chmod 400  /home/username/.ssh/authorized_keys

Don't forget to modify:

sudo nano /etc/ssh/sshd_config
#AuthorizedKeysFile     %h/.ssh/authorized_keys

/etc/init.d/ssh force-reload
/etc/init.d/ssh restart

ON THE SERVER: ssh-agent sh -c 'ssh-add < /dev/null && bash' OPTIONAL? exec ssh-agent sh -c 'ssh-add </dev/null && exec /usr/local/bin/wmaker'

Connecting with SSH client to a remote server

ssh -v -i id_rsa username@123.45.56.78

verbose, use identity from private key

ssh -vvvv -i id_rsa username@example.com

if errors like (nil) verify permissions and ownership (whoami=username)

Don't forget you sometimes have chmod 400

SSH ignore strict hosts checking

(i.e. developing against an FQDN with a dynamic ip, this does expose you to the improbable risk of an imposter "man in the middle" server)

ssh -i /usr/local/bamboo/bamboo_id_rsa -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no

this command line parameter will not store nor verify the remote server's signature

A more permanent configuration change:

vi ~/.ssh/config

IdentityFile ~/.ssh/id_rsa

Host example.com
        StrictHostKeyChecking no
        UserKnownHostsFile=/dev/null

SSH Agent Forwarding

It can be insecure to forward your SSH access through a jumpbox to a machine deeper in your network, though it is generally worse to leave SSH keyson a jumpbox.

ssh-add ~/.ssh/example.pem

add a specific key to the ssh agent ssh-add -L an optional step to list the keys that ssh-agent has loaded in memory

ssh -At user@1.2.3.4

SSH and explicitly forward the key so that the second shell session (ssh-agent) has access to it

    ssh user@5.6.7.8

the extra indentation is to indicate that from the jumpbox you are able to ssh to the internal machine

A more permanent configuration change by using ~/.ssh/config

Host 1.2.3.4
    ForwardAgent yes
    IdentityFile ~/.ssh/example.pem

https://linux.die.net/man/1/ssh-add

SSH TCP tunnel

In this example the SSH tunnel could be used for a SOCKS proxy for the browser...

ssh -ND 9999 user@example.org

connect via ssh to a remote server, after the password prompt the process will stay open for connections locally via port 9999

Use a docker container to isolate the VPN software and instead only allow an SSH tunnel as a web proxy

An alternative to a remote machine as the SSH target for the tunnel you can instead target a docker container running locally. - This allows you to package everything required (i.e. vpn software) into the container - Has a security benefit that only the container (and its filesystem) are directly exposed to the VPN

Note that for an even more limited security profile the container could run as an HTTP proxy as it will only handle HTTP traffic (rather than all TCP)

docker run --detach --rm --privileged --publish 2222:2222 ubuntu-xenial-vpn

the container is running SSHD

ssh -p 2222 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@localhost
openconnect https://vpn.example.com

SSH into the container and start the openconnect vpn connection and authenticate

ssh -ND 9999 -p 2222 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@localhost

Configure Firefox to use the SOCKS Host

Firefox -> Edit -> Preferences -> Advanced -> Network -> Connection -> Settings
Manual Proxy configuration: SOCKS Host: localhost, Port 9999

firefox sends requests to the server over the ssh encrypted connection

Configure Firefox to use the socks5 proxy for DNS

about:config
network.proxy.socks_remote_dns

use the address bar to access the advanced configuration option and set socks_remote_dns true

This ensures the browser will use the SOCKS proxy domain name server rather than the local machines

SSH tunnel for Windows RDP

ssh -L 3389:172.24.32.40:3389 172.24.32.100 -l sshusername -N

L = local port is forwarded to the remote host and port
l = login_name
.40 = the windows rdp server
.100 = the remote ssh server which has access to the windows rdp server
N = do not execute a remote command (port forwarding only)

ssh tunneling in general

local port:host:remote-port

ssh tunnel on port 9090 (cherokee-admin)

sudo vi /etc/sysctl.conf
    net.ipv6.conf.all.disable_ipv6=1

** On the remote server start the admin UI (which only allows access via 127.0.0.1 by default for security reasons)** sudo cherokee-admin

    Cherokee Web Server 1.2.101 (Jan 30 2012): Listening on port 127.0.0.1:9090,
    TLS disabled, IPv6 enabled, using epoll, 4096 fds system limit, max. 2041
    connections, caching I/O, single thread

    Login:
      User:              admin
      One-time Password: xoKmLN0aISztVMFs

    Web Interface:
      URL:               http://127.0.0.1:9090/


ssh -L 9090:localhost:9090 -p 22 user@host.com -N

L = local port is forwarded to the remote host and port, so ssh binding the localhost to port 9090

ssh the remote server on port 22

ssh with the specified user to the hostname (assuming DNS is correct)

N = do not execute a remote command (port forwarding only)

ssh -L 9090:localhost:9090 remote_IP

assumes the ssh port is 22 and the remote user is the same as the local user, not a good assumption

On your browser access http://localhost:9090 to see the Cherokee Admin UI

Future ideas: iptables 9090? forwarding?

Virtual Private Networks and openVPN

Docker OpenVPN

Using Docker is one of the easiest ways to leverage all of the open source tools (assuming for security you inspect the upstream source code, clone the Dockerfile, build your own docker image/container ;)

# https://github.com/kylemanna/docker-openvpn
# https://openvpn.net/index.php/open-source/documentation/howto.html
export FQDN="example.com"
export OVPN_DATA="ovpn-data"
docker volume create --name $OVPN_DATA
# generate the initial configuration in the volume
docker run -v $OVPN_DATA:/etc/openvpn --rm kylemanna/openvpn ovpn_genconfig -u udp://$FQDN
# generate the certificate in the volume (you must choose a passphrase)
docker run -v $OVPN_DATA:/etc/openvpn --rm -it kylemanna/openvpn ovpn_initpki
# start the openvpn service
docker run -v $OVPN_DATA:/etc/openvpn -d -p 1194:1194/udp --cap-add=NET_ADMIN kylemanna/openvpn

# generate the client certificate without the passphrase
docker run -v $OVPN_DATA:/etc/openvpn --rm -it kylemanna/openvpn easyrsa build-client-full $FQDN nopass
# export the client config with embedded certificates
docker run -v $OVPN_DATA:/etc/openvpn --rm kylemanna/openvpn ovpn_getclient $FQDN > $FQDN.ovpn

Digital Ocean OpenVPN

Dedicated SSH User

create a new ssh key (ops sec best practice)
Setup a free Firewall with ports tcp 22 and udp 1194
Create and Deploy a new droplet ($5 in a region near you for lower latency, with your newly created SSH key)
Once it has finished booting use the Digital Ocean web ui: Networking -> Firewalls -> FirewallName -> Droplets to add the Droplet to the firewall
ssh -i ~/.ssh/your-new-ssh-key root@1.2.3.4
reset the root password: passwd
Use the WebUI from Digital Ocean to verify you can access local console with the root user (and the new password)
harden the machine by adding a dedicated ssh user (not root!)
- useradd -s /bin/bash -m NEWUSERNAME
- usermod -a -G admin NEWUSERNAME
- passwd NEWUSERNAME
- verify this with: cat /etc/passwd | grep NEWUSERNAME ; cat /etc/group | grep NEWUSERNAME
- give the new user sudo permissions: visudo
- - UPDATE THE LINE TO: %admin ALL=(ALL) NOPASSWD:ALL
- - MOVE THE LINE BELOW: %sudo ALL=(ALL:ALL) ALL
- - VERIFY: cat /etc/sudoers
- mkdir -p /home/NEWUSERNAME/.ssh
- vim /home/NEWUSERNAME/.ssh/authorized_keys (paste the public key that matches the new ssh key)
- /etc/init.d/ssh force-reload
- /etc/init.d/ssh restart
- BE SURE that /home/NEWUSERNAME/.ssh and authorized_keys is owned by the new user (chown)
- VERIFY WITH VERBOSE the new user can SSH with: ssh -v -i ~/.ssh/your-new-ssh-key NEWUSERNAME@1.2.3.4
- VERIFY PERMISSIONS ONCE LOGGED IN WITH SSH: sudo su

More Hardening

disable root user from using SSH access and harden with a non-standard port
- vim /etc/ssh/sshd_config
- - Port 22222
- - PermitRootLogin no
- - ADD TO THE END OF THE FILE: AllowUsers NEWUSERNAME
- service sshd restart
- VERIFY: netstat -antp # should see 0.0.0.0:22222

Prerequisites

apt update
apt install openvpn easy-rsa
vim /etc/sysctl.conf
    net.ipv4.ip_forward=1
sudo sysctl -p

Setting up the certificate authority (since OpenVPN uses keys)

make-cadir ~/openvpn-ca
cd ~/openvpn-ca

vim vars

update the export KEY_NAME, KEY_ORG, KEY_EMAIL, KEY_OU to something more personalized

source vars
./clean-all
./build-ca

You can just press enter a bunch to use all the defaults that you preconfigured in the vars file previously

./build-key-server KEY_NAME (i.e. server or myvpn)

Just press enter to accept all the defaults again (no challenge password!) and press y at the end to Sign and then Commit the certificates

./build-dh
openvpn --genkey --secret keys/ta.key

./build-key client1

if you want to build another one later you must use: sudo su ; cd /root/openvpn-ca ; source vars ; ./build-key client2

Configure openvpn

cp ~/openvpn-ca/keys
sudo cp ca.crt myvpn.crt myvpn.key ta.key dh2048.pem /etc/openvpn
gunzip -c /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz | sudo tee /etc/openvpn/myvpn.conf

vim /etc/openvpn/myvpn.conf :::bash tls-auth ta.key 0 # Remove the preceding ; to uncomment this line key-direction 0 cipher AES-256-CBC auth SHA256 user nobody # Remove the preceding ; to uncomment this line group nogroup push "redirect-gateway def1 bypass-dhcp" # Remove the preceding ; to uncomment this line push "dhcp-option DNS 208.67.222.222" # Remove the preceding ; to uncomment this line push "dhcp-option DNS 208.67.220.220" # Remove the preceding ; to uncomment this line block-outside-dns cert myvpn.crt key myvpn.key

some tweaks for more security https://community.openvpn.net/openvpn/wiki/Hardening (the ta.key = tls auth) 208.67.222.222 = opendns , https://openvpn.net/index.php/open-source/documentation/howto.html

sudo systemctl start openvpn@myvpn sudo systemctl status openvpn@myvpn sudo systemctl enable openvpn@server

Should end with "ovpn-myvpn[21142]: Initialization Sequence Completed" "enable" means that the service will start at boot

netstat -antup

Should list all the TCP and UDP listening services including port 1194, 0.0.0.0:1194

ip addr show tun0

Should show the tunnel interface is created what the network range is (e.g. 10.8.0.1)

Consider updating myvpn.conf to use port 443 and proto tcp in order to assist clients traverse/reach the openvpn server in restricted environments

TODO: automate using packer and digital ocean apis to create an image

https://www.digitalocean.com/community/tutorials/how-to-set-up-an-openvpn-server-on-ubuntu-16-04
https://wiki.openwrt.org/doc/howto/vpn.openvpn#tab__using_openssl_commands_most_secure (more secure by directly using openssl commands)

Connect a client to the OpenVPN server

OpenVPN client config infrastructure on the remote server

mkdir -p /root/client-configs/files
chmod 700 /root/client-configs/files   
cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf /root/client-configs/base.conf
vim /root/client-configs/base.conf
    remote NEW-VPN-SERVER-IP-ADDRESS 1194
    user nobody
    group nogroup
    # ca ca.crt
    # cert client.crt
    # key client.key
    cipher AES-256-CBC
    auth SHA256
    key-direction 1

if you have modified the port and protocol ensure that 1194 and udp are updated accordingly windows clients will not be able to downgrad the user and group to nobody/nogroup commenting out the files as they will be embedded in the .ovpn config file

vim /root/client-configs/make_config.sh :::bash KEY_DIR=/root/openvpn-ca/keys OUTPUT_DIR=/root/client-configs/files BASE_CONFIG=/root/client-configs/base.conf cat ${BASE_CONFIG} \ <(echo -e '') \ ${KEY_DIR}/ca.crt \ <(echo -e '\n') \ ${KEY_DIR}/${1}.crt \ <(echo -e '\n') \ ${KEY_DIR}/${1}.key \ <(echo -e '\n') \ ${KEY_DIR}/ta.key \ <(echo -e '') \ > ${OUTPUT_DIR}/${1}.ovpn

This automates creating a config file via a script

chmod 700 make_config.sh
./make_config.sh client1
ls -ahl /root/client-configs/files/

Once you have the configuration

sudo apt-get install -y openvpn

For a linux client this installs the openvpn client software

scp -i ~/.ssh/VPNKEY -P 22222 NEWUSERNAME@VPN-IP-ADDRESS:client1.ovpn .

of course no matter what we need the open vpn client configuration

Or... From a client computer SCP to download the $FQDN.ovpn and then connect to the openvpn server

sudo openvpn --config client1.ovpn

# verify with the following in the output: "/sbin/ip addr add dev tun0 local 192.168.255.6 peer 192.168.255.5"
# ifconfig -a  "tun0 ... inet addr:192.168.255.6"  , as you send traffic: RX bytes:4145707 (4.1 MB)  TX bytes:319025 (319.0 KB)
curl checkip.amazonaws.com  # should return the IP address of the VPN server (not your local Wifi/ISP)
curl https://dnsleaktest.com/
# https://whoer.net/#extended

Import a .ovpn config file into ubuntu network manager

sudo nmcli connection import type openvpn file myname.tcp.ovpn

[sudo] password for your-username: Connection 'mynameVPN' (abcd-37c175b2325d) successfully added.

Connect to a VPN from the CLI

nmcli connection up mynameVPN

presuming the prior connection config was named mynameVPN

nmcli connection show mynameVPN

DNS Security

Moreover your DNS server can "leak" or be hijacked, here are some good alternatives to your snooping ISP or "big brother" evil corp.

1.1.1.1 (cloudflare https://developers.cloudflare.com/1.1.1.1/commitment-to-privacy/)
185.121.177.177 or 169.239.202.202 (http://dnsrec.meo.ws/ part of opennic)
84.200.69.80 or 84.200.70.40 (dns.watch free, neutral, privacy)
https://www.opennic.org/

echo 185.121.177.177 >> /etc/resolvconf/resolv.conf.d/tail sudo resolvconf -u

This should append the new resolver and force an update, though you may have to ifdown eth0 ; ifup eth0

dig -trace example.com nslookup -debug example.com

either command will provide copious debugging output (and nslookup has an interactive mode with "d2" for extra debugging)

These are DNS providers that have been assimilated into the borg:

216.146.36.36 (Dyn DNS was acquired by Oracle)
209.244.0.4 (Level3 Communications telco was acquired by CenturyLink)
208.67.220.220 (OpenDNS was acquired by Cisco and they basically store your data)
208.69.38.205 (also opendns)
8.8.8.8 (Google - so basically giving them even more data - especially when you use Chrome too)

A non standard protocol to encrypt DNS traffic: https://en.wikipedia.org/wiki/DNSCrypt

Letsencrypt and certbot for free SSL Certificates

A relatively recent development has been a widespread effort to help secure more of everyone's communications by encouraing web sites to install SSL certificates (and automate renewals) for free.

Here is the tool that allows you to easily automate getting a free SSL certificate (trusted by libraries and browsers no less)...

Once again Docker simplifies things slightly (as long as you trust the container hosted by quay.io CoreOS who was acquired by RedHat ;)

Prerequisite: setup a DNS record for yourdomain.com to point to the server ip address.

Create a free SSL cert with certbot in docker

systemctl stop nginx
sudo docker run -it --rm -p 443:443 --name certbot -v /etc/letsencrypt:/etc/letsencrypt -v /var/log/letsencrypt:/var/log/letsencrypt quay.io/letsencrypt/letsencrypt certonly --standalone -d yourdomain.com
systemctl start nginx

Downloads the container if it is not already in docker images
Starts a container with a web server that binds to port 443
The same web server/tool sends a certificate signing request (from your server running at yourdomain.com) to letsencrypt.org
letsencrypt.org then attempts to contact the provided domain (DNS -> IP -> server -> docker container)
The containerized-certbot-web-server/tool then securely downloads the new SSL certificate
Files created in the process are stored in /etc/letsencrypt
The "evergreen" certs are presented via symlinks in /etc/letsencrypt/live/yourdomain.com/
Specific versioned copies of the certificates are stored in /etc/letsencrypt/archive/yourdomain.com
letsencrypt certbot tool automatically generates the /etc/nginx/conf.d/default.conf (which points to the config and cert files in /etc/letsencrypt/)

Renew your SSL certificate with certbot and cron

You can keep renewing the certificate (which lasts 90 days) for free and there are a number of other open source tools (which leverage the API/process)

systemctl stop nginx
docker run -it --rm -p 443:443 --name certbot -v /etc/letsencrypt:/etc/letsencrypt -v /var/log/letsencrypt:/var/log/letsencrypt quay.io/letsencrypt/letsencrypt certonly --standalone --force-renewal -d yourdomain.com
systemctl start nginx

you must stop the production web server briefly in order to generate the updated certificate

If you want to use a cron job for certificate renewal then put the previous instructions into an executable file named ssl-cert-renewal.sh

echo '59 23 01 * * /home/USERNAME/ssl-cert-renewal.sh' >> /home/USERNAME/mymonthly.cron
crontab /home/USERNAME/mymonthly.cron
crontab -l

be sure to append to mymonthly.cron as the crontab command will override any previous cron jobs

Cryptography Exercises

In order to really understand and enjoy cryptography you can dive deeper via some of these exercises

https://cryptopals.com/

Golang JSON is challenging

2016-12-30T20:34:00-08:00

Parsing JSON can be a relatively simple subject covered early in other programming languages (i.e. JavaScript ;)

JSON is a really popular way to persist or transmit data, especially for APIs. So it is really common to need to use it and yet...

JSON can be surprisingly difficult in Go because even though it is built into the language it depends already understanding a few other somewhat advanced topics. And the challenge can be compounded by the Go philosophy of "We Did Not Put It In the Language Do It Yourself".

A quick overview of JSON

JSON (I never hear humans actually say the full thing: JavaScript Object Notation , though "Jason" does get annoyed) is a way to format data structures as text and it is the modern alternative to XML.

There is some fuzziness about Numbers and other definitions but its beauty is definitely simplicity.

Why does Go in JSON seem (unnecessarily) advanced or challenging?

Go with JSON requires defining/using structs for objects. This upfront cost certainly fits the "statically compiled" model ;) But Javascript or Python magically "just make an object" (or dictionary) which has fields/values that are very accessible.

It is actually pretty common that we might not know or want to define the full (nested?) object structures we've received (as if the JSON format was for portability of data from a service outside of your control), but we're forced to figure something out...
To generically parse an object from JSON (i.e. you do not know the full structure) you must use the "empty interface" (the most generic object), and Interfaces are more advanced than simply defining structs.
Inferring or attempting to lazily load JSON (i.e. a mixed list of different objects - since there isn't a slice of mixed types in Go!) requires Reflection which is a relatively advanced topic for a beginning programmer.
Static typing is great, except for when you're reading from JSON and you're not sure which type you should really use and you probably just want it to work simply. This impedance mismatch is natural when moving from a portable data format to a specific language and application, but it doesn't reduce the cursing.
To really have the Go compiler figure out the translation between JSON to object requires "hinting" which helps with compiling Reflection magic, but starts to complicate your structs (and interfaces!)
Nested Structs (fields) are the answer to Nested JSON, but then you have to really figure out how many (and lists with multiple types of objects!) and how deep you expect any nesting you'll receive. (Or just give in to your recursive desires.)
Pointers. They are efficient. Since JSON parsing can be memory intensive you will end up using them... in Nested Structs... with Interfaces... and Reflection Hints... (and since Pointers are messy and confusing there will be bugs).

associative array < struct/object < interface < pointers < reflection

a completely made up ordering of complexity, arrays being the least hard to grok

To summarize, it is really common to get some json from somewhere from someone else and want to just peek at one field, update another field, add a key and value, and save the json.

And that kind of dynamic behavior isn't inherently easy in Go. =[

Example code of Marshalling and Unmarshalling JSON with Go

package main

import (
        "encoding/json"
        "fmt"
        "io"
        "io/ioutil"
        "log"
        "os"
)

func logIfError(err error) {
        if err != nil {
                log.Fatal(err)
        }
}

// readFile is a convenience function to read a whole file at once, LOL similar to ioutil.ReadFile()
func readFile(f *os.File) {
        var data = make([]byte, 1024)
        totalBytes := 0
        for {
                count, err := f.Read(data)
                // https://golang.org/pkg/io/ , EOF is an expected error condition
                if err != io.EOF {
                        logIfError(err)
                }
                // TODO: 0 bytes could be returned when not an EOF
                if count == 0 {
                        break
                }
                totalBytes += count
                fmt.Printf("Read %d bytes: \n%s\n", count, string(data))
        }
        fmt.Printf("Read %d total bytes from the file\n", totalBytes)
}

// genericParsing is an example of the empty interface https://blog.golang.org/json-and-go
// https://en.wikipedia.org/wiki/JSON
func genericParsing(data []byte) {
        var f interface{}
        err := json.Unmarshal(data, &f)
        logIfError(err)

        // https://golang.org/doc/effective_go.html#interface_conversions
        m := f.(map[string]interface{})
        fmt.Println("\ngeneric json parsing")
        for k, v := range m {
                switch vv := v.(type) {
                case string:
                        fmt.Println("  ", k, "is string:", vv)
                case int:
                        fmt.Println("  ", k, "is int:", vv)
                case bool:
                        fmt.Println("  ", k, "is bool:", vv)
                case []interface{}:
                        fmt.Println("  ", k, "is an array:")
                        for _, u := range vv {
                                fmt.Println("    ", u)
                        }
                default:
                        fmt.Println("  ", k, "is of a type I don't know how to handle")
                        fmt.Printf("  but I could have checked another way and found %v is a %T\n", v, v)
                        // JSONNumber https://golang.org/pkg/encoding/json/#Decoder.UseNumber
                        // http://json.org/ no floats so hinting is appreciated
                }
        }
}

// Assuming top level keys are strings, i.e. NOT [] , https://gobyexample.com/json
func rootStringsOnlyParsing(data []byte) map[string]interface{} {
        // A map of string to any type https://blog.golang.org/laws-of-reflection , http://research.swtch.com/interfaces
        var datmap map[string]interface{}
        e := json.Unmarshal(data, &datmap)
        logIfError(e)
        fmt.Println("\nKeys are Strings in a Map:", datmap)
        return datmap
}

// ExampleSimpleObject must be exported to parse correctly , the fields order here is used by json.Marshal output
type ExampleSimpleObject struct {
        Age  int    `json:"age"`
        Name string `json:"name"`
}

// ExampleComplexObject is the magic of auto parsing, if your data never gets corrupted...
// helpful understanding of Go and JSON nesting https://eager.io/blog/go-and-json/
// hints are very powerful leveraging of Reflection that Go core libraries use for JSON
type ExampleComplexObject struct {
        ArrayOfObjects []ExampleSimpleObject `json:"jsonArrayOfObjects,omitempty"`
        ArrayOfStrings []string              `json:"jsonArrayOfStrings"`
        JSONBoolean    bool                  `json:"jsonBoolean"`
        JSONNumber     int                   `json:"jsonNumber"`
        JSONString     string                `json:"jsonString, omitempty"`
        // jsonArrayOfNumbers is not defined and so is not included in the parsed object
}

// autoUnmarshal shows Go structs making parsing JSON look easy https://golang.org/pkg/encoding/json/#example_Unmarshal
func autoUnmarshal(data []byte) ExampleComplexObject {
        var ex ExampleComplexObject
        err := json.Unmarshal(data, &ex)
        logIfError(err)
        fmt.Printf("Auto Unmarshal: %+v \n", ex)
        return ex
}

// writeJSONFile demonstrates the power of interfaces for shared functionality
func writeJSONFile(name string, thing interface{}) {
        theJSON, err := json.MarshalIndent(thing, "", "  ")
        logIfError(err)
        err = ioutil.WriteFile(name, theJSON, 0644)
        logIfError(err)
        // See the omitted fields with: diff --ignore-all-space types.json output.json
}

func main() {
        // https://golang.org/pkg/os/
        myFile, ferr := os.Open("types.json")
        logIfError(ferr)
        readFile(myFile)

        // hint: read a file and return a slice of bytes: https://golang.org/pkg/io/ioutil/#ReadFile
        data, _ := ioutil.ReadFile("types.json")

        genericParsing(data)
        datamap := rootStringsOnlyParsing(data)
        // modifying or adding to a JSON file can be tricky
        datamap["injectedKey"] = "injected value"
        writeJSONFile("dataMapModified.json", datamap)

        auto := autoUnmarshal(data)
        writeJSONFile("autoUnmarshalOmits.json", auto)
        fmt.Println("done")
}

Common JSON gotchas with Go

type Oops struct {
    Name string `json:"name"`
    i int    `json:"timestamp"`
}

The i int field will not be Marshaled and will therefore not exist in the JSON object

The data structures need to be exported, otherwise you'll only end up with an empty JSON object: https://play.golang.org/p/ukkjLQnSSq , https://golang.org/pkg/encoding/json/#example_Unmarshal
Types are strict in Go. JSON is unclear about "Number". Golang will assume float64 without any hints. Use hints, or reflection and type assertions and a magic wand... https://golang.org/pkg/encoding/json/#Decoder.UseNumber
Marshal() returns a slice of bytes which is not a string. so string() , https://golang.org/pkg/encoding/json/#Marshal
"The argument to Unmarshal must be a non-nil pointer", https://golang.org/pkg/encoding/json/#InvalidUnmarshalError

An example Go JSON helper utility

The internet has many "helper" (usually performance focused) utilities/libraries for JSON with Go, here's mine:

https://bitbucket.org/johnpfeiffer/go-jsondao/src

The idea is to simplify just doing minimal parsing in order to add or update a Key

Hint: RawMessage is the performance trick to not parse all of the fields.

A more "real world" code example of parsing JSON with Go

I wanted to import the bookmarks from Chrome but I hadn't exported them. I wrote this utility to parse the default chrome bookmarks json file that I did have:

https://bitbucket.org/johnpfeiffer/bookmarks/src

Golang Testing Benchmark Profiling Subtests Fuzz Testing

2016-10-22T20:44:00-07:00

Go makes it very easy to unit test with the packagename_test.go file right next to the package source code file(s).

As a pragmatic language designed for developers who ship to production the amount of built in tooling (testing, benchmarks, etc.) is impressive.

Taking an earlier example I gave of MergeSort let's examine how TestDrivenDevelopment (or Design) was used to implement it.

Small functions make for good tests

Small functions are easier to read and code is read 1000 times more often than it is written. (completely made up but believable fact).

Less lines of very-readable-code is usually an ok approximation for complexity, and less complexity means your program is easier to reason about (and easier to validate with tests!).

Some reasons to not use MEGA-OBJECTS

One of the things that TDD helps focus on is modularity and requirements. Two tensions to balance are the needs of the caller versus the needs of the function.

What I mean is that the function caller wants to understand what they have to provide and what they'll get back. If the function asks for a MEGA-OBJECT then somehow the caller has to find or create a MEGA-OBJECT (which sounds very expensive). And if the function didn't really need the MEGA-OBJECT then the function will extract the one value it actually needs and throw all that work away.

If instead the function asks for the integer primitive that is the value of the MEGA-OBJECT's this should be very easy to fulfill. (Which is how tests help discover this MEGA-OBJECT anti-pattern, because even MEGA-OBJECT mocks are difficult).

A second reason to not pass a MEGA-OBJECT is that those are usually "pass by reference" for performance reasons and if modifying/side-effects are allowed then the function may accidentally invalidate other values (or intentionally corrupt data or override permissions).

The less state being passed around the easier it is to quickly write a large base of non brittle unit tests to isolate exactly where the logic goes wrong when doing "the simplest thing" and of course to communicate to others/callers how they might use your function or how it handles failure modes.

Example function with go unit tests

This example function requires two slices of integers.

package main

import (
    "fmt"
)

// SlicesMerge takes two sorted slices of integers and merges them into a single sorted slice of integers
func SlicesMerge(a []int, b []int) []int {
    s := make([]int, len(a)+len(b))
    for ai, bi, si := len(a)-1, len(b)-1, len(a)+len(b)-1; si >= 0; si-- {
        if ai < 0 {
            s[si] = b[bi]
            bi--
        } else if bi < 0 {
            s[si] = a[ai]
            ai--
        } else if a[ai] > b[bi] {
            s[si] = a[ai]
            ai--
        } else {
            s[si] = b[bi]
            bi--
        }
    }
    return s
}

func main() {
    a := []int{1, 3, 5}
    b := []int{2, 4, 6}
    fmt.Println("merged:", SlicesMerge(a, b))
}

a main function with print statements is the tried and true way of manual testing , go run main.go

main_test.go

Automated units tests means another developer can fix a bug and ensure it does not regress, read and learn expected behavior, and of course have automated continuous integration catch problems as early as possible, go test

package main

import (
    "reflect"
    "testing"
)


var nilslice []int

var empty = []int{}

func TestSlicesMergeEmpty(t *testing.T) {
    t.Parallel()
    assertSlicesEqual(t, empty, SlicesMerge(empty, empty))
    assertSlicesEqual(t, empty, SlicesMerge(empty, nil))
    assertSlicesEqual(t, empty, SlicesMerge(nil, empty))
    assertSlicesEqual(t, empty, SlicesMerge(nil, nil))
}

func TestSlicesMergeHalfEmpty(t *testing.T) {
    t.Parallel()
    a := []int{1}
    assertSlicesEqual(t, a, SlicesMerge(a, empty))
    assertSlicesEqual(t, a, SlicesMerge(empty, a))

    a = []int{1, 2}
    assertSlicesEqual(t, a, SlicesMerge(a, empty))
    assertSlicesEqual(t, a, SlicesMerge(empty, a))
}

func TestSlicesMergeNegative(t *testing.T) {
    // if os.Getenv("MY_ENVIRONMENT_VARIABLE") == "" {
    t.Skip("skipping test: $MY_ENVIRONMENT_VARIABLE is not set")

    t.Parallel()
    a := []int{1, 3}
    b := []int{-2870837225030527764, -2}
    assertSlicesEqual(t, []int{-2870837225030527764, -2, 1, 3}, SlicesMerge(a, b))
}


// Helper Functions

func assertSlicesEqual(t *testing.T, expected []int, result []int) {
    if !reflect.DeepEqual(expected, result) {
        t.Error("\nExpected:", expected, "\nReceived: ", result)
    }
}

The idea is to have code that is testable, and just use Go code to write the tests (not another DomainSpecificLanguage to learn)

https://golang.org/pkg/testing/

Skipping tests is fairly important to applying logic to parts of the test suite (or maybe deferring paying some technical debt)

Parallel indicates the tests can run in parallel, not useful here but in larger test suites taking advantage of extra processor power (GOMAXPROCS) to speed up the feedback loop is always appreciated

Even in a statically compiled language making comparisons of lists of potentially nested objects is unguaranteed, but the reflection function DeepEqual does a best effort job

https://golang.org/pkg/reflect/#DeepEqual

Running the tests in sequence and not parallel

t.Parallel() indicates a test can run in parallel, but if you are debugging or really need a specific order then using go test -p 1 will force it to run each test sequentially.

Note that Go Test will execute multiple different package tests in parallel...

-parallel n
   Allow parallel execution of test functions that call t.Parallel.
   The value of this flag is the maximum number of tests to run
   simultaneously; by default, it is set to the value of GOMAXPROCS.
   Note that -parallel only applies within a single test binary.
   The 'go test' command may run tests for different packages
   in parallel as well, according to the setting of the -p flag
   (see 'go help build').

Running a specific test

go test -v -run TestSlicesMergeHalf

Getting verbose output and specifying tests is quite helpful when fixing a piece of code or test. Note the run parameter takes a regular expression

Test Coverage

Sometimes people talk about "test coverage" and while it's clear that 100% coverage is rarely possible (nor entirely desirable from the idea of diminishing returns and exponential growth in integration combinations outside of the simplest function) , it's still a useful metric/tool to discover if there's a chunk of code that's "whistling in the wind".

go test -cover

"coverage: 75.0% of statements"

Generate a "coverage profile" of how many times each statement was run, use the current directory: go test -covermode=count -coverprofile=count.out .

ok command-line-arguments 0.004s coverage: 94.7% of statements

go test -covermode=count -coverprofile=count.out ./stringsmoar.go ./stringsmoar_test.go

Alternatively, pass the name of the package (and test files)

go tool cover -func=count.out

utilize the "coverage profile" to see the coverage breakdown by function

bitbucket.org/johnpfeiffer/stringsmoar/stringsmoar.go:23:       RuneFrequency                   100.0%
...
bitbucket.org/johnpfeiffer/stringsmoar/stringsmoar.go:82:       RemoveNthRune                   100.0%
bitbucket.org/johnpfeiffer/stringsmoar/stringsmoar.go:96:       RemoveNthItem                   0.0%
...
bitbucket.org/johnpfeiffer/stringsmoar/stringsmoar.go:115:      Permutations                    100.0%
total:                                                          (statements)                    94.7%

If you have a default browser configured you can use the following: go tool cover -html=count.out to generate a "heat map" and see exactly how often each line of code is covered. (red means not at all ;)

https://blog.golang.org/cover

Subtests

Using the pattern of table driven tests improves the readability and extensibility of the "merge empty test" by applying "Don't Repeat Yourself" and removing the copy pasting of the driver function call.

package main

import (
    "reflect"
    "testing"
)

var empty = []int{}

func assertSlicesEqual(t *testing.T, expected []int, result []int) {
    if !reflect.DeepEqual(expected, result) {
        t.Error("\nExpected:", expected, "\nReceived: ", result)
    }
}

// defining the test structure separately and clear naming helps readability
type slicesMergeTest struct {
    a        []int
    b        []int
    expected []int
}

func TestSlicesMergeEmpty(t *testing.T) {
// alternative "anonymous struct" example 
/* var testCases = [] struct {
       a        []int
       b        []int
       expected []int
   }{ */

    testCases := []slicesMergeTest {
        {a: empty, b: empty, expected: empty},
        {a: empty, b: nil, expected: empty},
        {a: nil, b: empty, expected: empty},
        {a: nil, b: nil, expected: empty},
    }

    // Without subtests
    // for _, tc := range testCases {
    //  actual := SlicesMerge(tc.a, tc.b)
    //  assertSlicesEqual(t, tc.expected, actual)
    // }

    for _, tc := range testCases {
        t.Run(fmt.Sprintf("%#v merged with %#v", tc.a, tc.b), func(t *testing.T) {
            actual := SlicesMerge(tc.a, tc.b)
            assertSlicesEqual(t, tc.expected, actual)
        })
    }
}

One variation with the "subtest" feature (which may apply more to benchmarks than straightforward unit tests) is not only that a fatal will not skip subsequent tests but that the output is more verbose

=== RUN   TestSlicesMergeEmpty
--- PASS: TestSlicesMergeEmpty (0.00s)
PASS

With subtests...

=== RUN   TestSlicesMergeEmpty
=== RUN   TestSlicesMergeEmpty/[]int{}_merged_with_[]int{}
=== RUN   TestSlicesMergeEmpty/[]int{}_merged_with_[]int(nil)
=== RUN   TestSlicesMergeEmpty/[]int(nil)_merged_with_[]int{}
=== RUN   TestSlicesMergeEmpty/[]int(nil)_merged_with_[]int(nil)
--- PASS: TestSlicesMergeEmpty (0.00s)
    --- PASS: TestSlicesMergeEmpty/[]int{}_merged_with_[]int{} (0.00s)
    --- PASS: TestSlicesMergeEmpty/[]int{}_merged_with_[]int(nil) (0.00s)
    --- PASS: TestSlicesMergeEmpty/[]int(nil)_merged_with_[]int{} (0.00s)
    --- PASS: TestSlicesMergeEmpty/[]int(nil)_merged_with_[]int(nil) (0.00s)
PASS

Making the "table" of inputs and outputs more obvious AND the output verbosity clearer seems like a small refinement but goes a long way to making production quality testing easier

Running a specific subtest

go test -v -run=TestSlicesMergeEmpty/"nil"

--- PASS: TestSlicesMergeEmpty (0.00s)
    --- PASS: TestSlicesMergeEmpty/[]int{}_merged_with_[]int(nil) (0.00s)
    --- PASS: TestSlicesMergeEmpty/[]int(nil)_merged_with_[]int{} (0.00s)
    --- PASS: TestSlicesMergeEmpty/[]int(nil)_merged_with_[]int(nil) (0.00s)

Yes! You can pattern match on the string from the subtest table and only run a subset of subtests (mindblown)

More info on the "table driven test" pattern:

Benchmarking

Benchmarking is most useful if you're attempting to answer a question of two variations on how to implement something.

I suppose if you recorded every result and ran against exactly the same hardware you might be able to detect performance regressions, though I'd be worried about overly inconsistent/flaky results taking up way too much valuable time.

Inside of a _test.go file you can also write benchmark test functions, here is one of the classic questions of "concatenating strings in Go"

Here we compare the simplest concatenation of two strings and also the continued concatenation of many strings with either + or buffer.WriteString()

Create myconcat_test.go and execute the following with go test -v -bench=.

package main

import (
    "bytes"
    "testing"
)

func MyConcatSimple(a string, b string) string {
    return a + b
}

func MyConcatSimpleLooped(a string, b string) string {
    for i := 0; i < 101; i++ {
        a += b
    }
    return a
}

func MyConcatBytesBuffer(a string, b string) string {
    var buffer bytes.Buffer
    buffer.WriteString(a)
    buffer.WriteString(b)
    return buffer.String()
}

func MyConcatBytesBufferLooped(a string, b string) string {
    var buffer bytes.Buffer
    buffer.WriteString(a)
    for i := 0; i < 101; i++ {
        buffer.WriteString(b)
    }
    return buffer.String()
}

func BenchmarkConcatSimple(b *testing.B) {
    for n := 0; n < b.N; n++ {
        MyConcatSimple("foo", "bar")
    }
}

func BenchmarkConcatSimpleLooped(b *testing.B) {
    for n := 0; n < b.N; n++ {
        MyConcatSimpleLooped("foo", "bar")
    }
}

func BenchmarkConcatBytesBuffer(b *testing.B) {
    for n := 0; n < b.N; n++ {
        MyConcatBytesBuffer("foo", "bar")
    }
}

func BenchmarkConcatBytesBufferLooped(b *testing.B) {
    for n := 0; n < b.N; n++ {
        MyConcatBytesBufferLooped("foo", "bar")
    }
}

The b.N is automatically filled in by Go until the benchmark runner is "satisfied with stability" though you can create a wrapper function for the code under test if you wish to attempt more control over iterations

Remember that every benchmark is only valid against a specific set of hardware, operating system, libraries, etc. and with any changes (i.e. upgrade from Go 1.6 to 1.7) you may need to retest... unless you're just proving O(N) is better than O(N^2) ;)

Oh right, the results...

BenchmarkConcatSimple-4                 30000000                40.1 ns/op
BenchmarkConcatSimpleLooped-4             100000             20728 ns/op
BenchmarkConcatBytesBuffer-4             5000000               373 ns/op
BenchmarkConcatBytesBufferLooped-4        300000              7227 ns/op

the iterations show that the simplest naive concatentation with + is very fast for a couple of small arguments (40 nanoseconds)

BUT if appending many (100+) items together buffer.Write() is better

This example ignored all sorts of real world questions around how the strings are provided (i.e. a slice of strings might be better served by Join()) (or examine it's source code to see how they implement string concatenation!) , or memory consumption, etc.

Running specific benchmarks

go test -v -run=NOMATCH -bench=BenchmarkConcatSimple

Since it is using the test runner the -run= regexp not matching allows you to skip any unit tests

-bench= can take a regexp to match only a subset of benchmark tests

Go Benchmark with an expensive setup

If you have some setup (e.g. creating a slice with test data) you probably do not want it inside of the benchmark ;)

go test -v -run=NOMATCH -bench=BenchmarkKey

func BenchmarkKey(b *testing.B) {
    a := getData(100)
    b.ResetTimer()  
    for i := 0; i < b.N; i++ {
        problemKey(100, a)
    }
}

func getData(n int) []int {
    a := make([]int, n)
    for i := 0; i < n; i++ {
        a[i] = i
    }
    return a
 }

the timer of the benchmark testing has been reset before the "real" load , note that the compiler is smart and this may not always be necessary

Profiling

TODO:

Go Fuzz Testing

Fuzz testing is furthering the principle of automation (and that computers are inherently better at some things than humans) to have software discover edge cases for tests.

Basically the idea is to have software run over an extreme range or with randomness and then the edge cases that are discovered can be added into the test suite. It has been used to enormously good effect on the Go standard library by Dmitry Vyukov.

One thing you start to see when attempting to apply it is that it really a tool for helping validate handling of a specific input. It is not a magic wand to discover bugs. ;)

This kind of tool assisted exploratory testing is usually reserved for a more mature phase of a product (or in special use cases where there is high value in attempting to prove correctness).

func TestSlicesMergeRandom(t *testing.T) {
    f := fuzz.New()
    var randomSeed int
    f.Fuzz(&randomSeed)
    fmt.Println("random seed:", randomSeed)
    r := rand.New(rand.NewSource(int64(randomSeed)))

    xLength := r.Int() % 5
    yLength := r.Int() % 5
    fmt.Println(xLength, yLength)
    var x []int
    var y []int

    for i := 0; i < xLength; i++ {
        var r int
        f.Fuzz(&r)
        x = append(x, r)
    }
    for i := 0; i < yLength; i++ {
        var r int
        f.Fuzz(&r)
        y = append(y, r)
    }
    sort.Ints(x)
    sort.Ints(y)
    result := SlicesMerge(x, y)
    expected := append(x, y...)
    sort.Ints(expected)
    if !reflect.DeepEqual(expected, result) {
        t.Error("\nExpected:", expected, "\nReceived: ", result)
    }
}

Seeding randomness is part of how this gofuzz library is used; the Vyukov version actually produces output that must be parsed separately.

Golang Interfaces Stack Linked List Queue Map Set

2016-10-05T20:34:00-07:00

Two of the most useful tools in daily professional programming are hash maps and frameworks for modular abstractions. It's the blend of performance and composability that makes Go an attractive language for getting things done.

Maps aka Hash Tables in Go

For hash tables basic operations like insert, lookup, delete are basically "constant time" in big O notation which makes it the "go to" default data structure for a lot of basic coding. That helps avoid having to write and maintain all the complexity and operations of an in-memory sorted data structure.

It is also often the desired data structure answer in a programming interview problem =]

Here's the basics of using a map since someone else has already done all of the hard work of figuring out the correct math of the hash function to avoid collisions and the engineering behind implementing it in a performant fashion.

Using a map in go

package main

import (
    "fmt"
)

func inSet(s string, m map[string]bool) bool {
    // gotcha: do not depend on just the return value but use a the second value which returns in map or not
    value, ok := m[s]
    fmt.Println(s, "has value", value, "and is in?", ok)
    if ok {
        return true
    } else {
        return false
    }
}

func main() {
    m := make(map[string]bool)
    // never use var m map[string]bool because m is nil and assign causes a panic
    m["foo"] = true
    m["bar"] = false

    fmt.Println(inSet("foo", m))
    fmt.Println(inSet("bar", m))
    fmt.Println(inSet("foobar", m))

    fmt.Println(len(m), " elements in map", m)
    delete(m, "bar")

    m2 := map[rune]int{'a': 1, 'b': 2, 'c': 3}
    for k, v := range m2 {
        fmt.Println(string(k), v)
    }
}

the two main gotchas for maps in go are to not accidentally assume that the value coming back from a map means that the key was in the map since if the key is not in the map it will return the default "empty" value which will be 0 or false, and to not start with a nil pointer since later assignment will cause a panic

https://play.golang.org/p/fqvNrELy1S play with maps yourself
https://en.m.wikipedia.org/wiki/Hash_table
https://blog.golang.org/go-maps-in-action
https://golang.org/src/runtime/hashmap.go#L9

Sets

A common question in coding is whether a certain key or object has already been seen before. The lookup cost in a set is cheaper than a full search through a more complex data structure like a binary tree.

Golang currently does not have a "set" built into the language but given the code example you can see how to quickly achieve the same functionality with a map. This is a common pattern: they keep the language simple and focus on low level components that perform and compose well while we code exactly what we need for a given usage. (i.e. a huge meta "YouAren'tGoingToNeedIt")

Low memory footprint map as a set

When implementing a Set, in some cases you can be very parsimonious with memory by using an empty struct. This means you will rely on the very explicit _, ok pattern of checking for presence in the set.

package main

import (
    "fmt"
    "unsafe"
)

func main() {
    m := make(map[rune]struct{})
    fmt.Println(m)
    m['a'] = struct{}{}
    fmt.Println(m)
    fmt.Printf("%#v \n", m)
    fmt.Println(unsafe.Sizeof(m['a']))

    var b bool
    fmt.Println(unsafe.Sizeof(b))
    var n int8
    fmt.Println(unsafe.Sizeof(n))
    var i interface{}
    fmt.Println(unsafe.Sizeof(i))
}

An empty struct is 0, a boolean and an int8 are size 1 (byte), an interface is size 8

Polymorphism with Go Interfaces

While definitely a distinctly different approach on objects and polymorphism than Object Oriented languages like Java and Python, the critical capabilities of Interfaces allows flexibility and re-usability in functionality enabling things like the Strategy Pattern.

Starting with the fundamentals of Types and Structs, Interfaces are a natural extension to separating "What" behavior is desired versus "How" it is implemented.

Using Interfaces can feel tricky at first but really the main challenge is the distinction between pointer receiver methods vs the default that Go has of pass by value (considering a pointer to be an integer value of an address ;)

Also, testing in Go relies heavily on the developer paying that up front cost of creating Interfaces (which is a much better longer term modular abstraction) rather than using Mocks.

This example uses a trivial Stack data structure but implements it 3 different ways to illustrate how the interface can have a variety of implementations and the caller can decide which one they prefer.

package main

import (
    "fmt"
)

// Stacker is a data structure that has specific data access and storage properties
// also known as a Last In First Out queue, and not fully implemented for this example =]
// This interfaces only requires two methods to implement
type Stacker interface {
    Push(n int)
    Show() []int
}

// DemoStack shows a demo example of different implementations of the interface
func DemoStack(s Stacker, name string) {
    fmt.Println("Pushing 2, 1, 0, onto", name)
    s.Push(2)
    s.Push(1)
    s.Push(0)
    fmt.Println(name, "contains: ", s.Show())
}

// FakeStack implements the interface using only value receivers
type FakeStack struct {
}

var fakeStackCheater []int

// Show returns the global slice
func (s FakeStack) Show() []int {
    return fakeStackCheater
}

// Push uses a global variable (evil) so it can get away with a value method receiver
func (s FakeStack) Push(n int) {
    fakeStackCheater = append(fakeStackCheater, n)
}

// SliceStack is an integer stack data structure built using a slice
type SliceStack struct {
    s []int
}

// Show displays the contents of the stack, pass a copy as no modification needed
func (s SliceStack) Show() []int {
    return s.s
}

// Push an integer onto the stack, pass a reference so the receiver method can directly modify
// https://github.com/golang/go/wiki/CodeReviewComments#receiver-type
func (s *SliceStack) Push(n int) {
    s.s = append(s.s, n)
}

/* A real stack based upon a linked list */

// IntNode is a pointer data structure for holding an integer
type IntNode struct {
    value int
    left  *IntNode
    right *IntNode
}

// LinkedListStack is an integer stack data structure built using a slice
type LinkedListStack struct {
    head *IntNode
}

// Show returns the total number of elements currently stored in the LinkedListStack
func (s LinkedListStack) Show() []int {
    var result []int
    for current := s.head; current != nil; current = current.right {
        result = append(result, current.value)
    }
    return result
}

// Push an integer onto the stack, pass a reference so the receiver method can directly modify
func (s *LinkedListStack) Push(n int) {
    new := IntNode{value: n}
    if s.head == nil {
        s.head = &new
    } else {
        new.right = s.head
        s.head = &new
    }
}

func main() {
    // a fake stack does not use pointer method receivers and must modify the data structure some other way
    var f FakeStack
    DemoStack(f, "FakeStack")

    // stacks that modify their underlying data structure must pass a reference
    var s SliceStack
    DemoStack(&s, "SliceStack")
    var k LinkedListStack
    DemoStack(&k, "LinkedListStack")
}

This example contradicts one of the best practices: pointer receivers should be consistent https://tour.golang.org/methods/4 , so Show() and Push() should have the same receiver type

A Pointer Receiver allows the method to make a modification (i.e. syntactic sugar where (s *SliceStack) Push(n int) could be thought of as Push(s *SliceStack, n int)
A Pointer Receiver prevents an extra copy (in the instance of a very large object)
All of the methods of an interface should be consistent (as a part of the developer user experience)

References:

Doubly Linked List and First In First Out Queue

With some small additions the linked list can be enhanced to provide the functionality of a Queue. The "doubly linked list" (https://en.wikipedia.org/wiki/Doubly_linked_list) means that one can traverse from either the head (using next) or the tail (using previous). It is not too expensive to add the extra previous pointer to each node and a tail pointer and this makes the FIFO capabilities fairly straightforward. Thankfully with a garbage collected language like Go we do not have to worry about manually allocating or deallocating memory, though we should always keep an eye out for memory leaks

package main

import (
    "fmt"
)

// IntNode is a pointer data structure for holding an integer
type IntNode struct {
    value    int
    previous *IntNode
    next     *IntNode
}

// MyList is a linked list of pointers
type MyList struct {
    head *IntNode
    tail *IntNode
}

// Enqueue adds an integer onto the end of the list
func (q *MyList) Enqueue(n int) {
    new := IntNode{value: n}
    if q.head == nil {
        q.head = &new
        q.tail = &new
    } else {
        q.tail.next = &new
        q.tail = &new
    }
}

// Dequeue removes the first integer added to the list (from the front)
func (q *MyList) Dequeue() int {
    // TODO: error handling for dequeuing when the list is empty
    result := q.head
    q.head = q.head.next
    return result.value
}

func main() {
    var q MyList
    q.Enqueue(3)
    q.Enqueue(4)
    fmt.Printf("FirstInFirstOut head: %v at memory address %p \n", q.head.value, q.head)
    fmt.Printf("FirstInFirstOut tail: %v at memory address %p \n", q.tail.value, q.tail)
    fmt.Println("dequeuing a value: ", q.Dequeue())
    fmt.Printf("FirstInFirstOut head: %v at memory address %p \n", q.head.value, q.head)
    fmt.Printf("FirstInFirstOut tail: %v at memory address %p \n", q.tail.value, q.tail)
}

The terminology changed a little bit from Push to Enqueue but now we can have a simple "fair buffer"

Doubly Linked List in the Go Standard Library

Wonderfully there is an implementation of a Doubly Linked List in the Go standard library: https://golang.org/pkg/container/list/

package main

import (
    "container/list"
    "fmt"
)

func main() {
L := list.New()
e1 := L.PushBack(1) // enqueue
L.InsertAfter(3, e1)    
e3 := L.Back()
L.InsertBefore(2, e3)

displayFIFO(L)
fmt.Printf("\nTraverse in reverse with Prev(): ")
displayLIFO(L)

L2 := list.New()
L2.PushBack("D")
displayFIFO(L2)
L.PushBackList(L2)
displayFIFO(L)

L.Remove(e1)
fmt.Printf("\nRemoved %v", e1.Value)
displayFIFO(L)
fmt.Printf("\nFront is now: %v", L.Front().Value)
fmt.Printf("\n%#v", L)
}

func displayFIFO(L *list.List) {
    fmt.Printf("\nList length: %v \n", L.Len())
    for e := L.Front(); e != nil; e = e.Next() {
        fmt.Printf("%v ", e.Value)
    }
    fmt.Println()
}

func displayLIFO(L *list.List) {
    fmt.Printf("\nList length: %v \n", L.Len())
    for e := L.Back(); e != nil; e = e.Prev() {
        fmt.Printf("%d ", e.Value.(int))
    }
    fmt.Println()
}

All of the node and pointer implementation is already handled for you, play with it here: https://play.golang.org/p/FUEtqMNoaP9

Golang Slices Functions Filters Mergesort

2016-10-02T23:45:00-07:00

Tips on learning a new programming language

Learning a new programming language takes a lot of effort. Here are some tips I've found to make it easier:

Layering: read through the material and just accept things without trying to understand every piece, small pieces are easier to digest and absorb
Repetition: it is natural to read and re-read before comprehension occurs, learning right before sleep is scientifically proven to better enter long term memory
Immersion: surround yourself with reminders of the material like books, web sites, on your phone, flash cards, online and real life discussions, videos
Variety: sometimes it sticks better with a book, other times a video or podcast. Different teachers will emphasize or convey certain topics better. Do not give up because it's confusing from one source, there are many many sources to learn from
Practice: apply yourself as programming is far better internalized "in action" rather than just theory
Engage: using provided exercises will get you to the solution the instructor expects, creating your own coding projects will force you to use the tools in ways that make sense to you (and discover why certain things are "best practice")
Go Deep: once comfortable moving around, real fluency and understanding requires digging into exactly how something is architected and implemented, what are the corner cases, and why things are done one way over another

Arrays underlying slices with memory addresses and unsafe pointers

Arrays are a mainstay of programming (mechanical sympathy for memory addresses and our very serial brains). Here is a major digression into Go slices which is a pointer structure that holds an array and manages the dynamic resizing. Hint: the whole point of Go is to not need to use low level unsafe pointer arithmetic

package main

import (
    "fmt"
    "unsafe"
)

func main() {
    capacityDoubles()
}

func capacityDoubles() {
    // slice of zero length and zero capacity, basically just a "slice header"
    // NOTE this is different from a "nil slice" which is a nil pointer
    var nilPointer []int
    fmt.Printf("example nil pointer: %#v \n", nilPointer)
    fmt.Printf("has memory adddress: %p \n", &nilPointer)

    s := make([]int, 0, 0)
    fmt.Printf("example empty slice: %#v \n", s)
    fmt.Printf("has memory adddress: %p \n", &s)
    fmt.Println("both have the same length:", len(nilPointer), len(s), "but does each equal nil? ", nilPointer == nil, s == nil)

    fmt.Println(s)
    for i := 0; i < 3; i++ {
        // capacity doubles so a new underlying array, if we checked with reflect the memory address would change
        s = append(s, i)
        fmt.Printf("len=%d cap=%d %v\n", len(s), cap(s), s)
    }

    // https://golang.org/pkg/reflect/#SliceHeader
    // since a slice is a struct containing an underlying array of memory addresses...
    // https://golang.org/pkg/builtin/#byte
    h := make([]byte, 2, 2)
    h[1] = 5
    fmt.Printf("Slice of byte memory addresses: %p ", &h)
    // 0xc42006e200 example address of the slice
    fmt.Printf("%p ", &h[0])
    fmt.Printf("%p ", &h[1])
    // 0xc420074458 0xc420074459 , the two elements are exactly 1 byte apart in address locations
    // Getting the value from the memory location using a piont
    fmt.Printf("\nThe byte value in the second element is binary five: %b", *(&h[1]))

    // https://golang.org/pkg/unsafe/
    fmt.Printf("\nUnsafe pointer of the first memory location: %p", unsafe.Pointer(&h[0]))
    // unsafe pointer arithmetic
    lastAddress := uintptr(unsafe.Pointer(&h[0])) + unsafe.Sizeof(h[0])
    // convert back into a usable pointer type
    lastAddressPtr := (*byte)(unsafe.Pointer(lastAddress))
    fmt.Printf("\nAfter pointer arithmetic and unsafe re-typed: %p, value: %b", lastAddressPtr, *lastAddressPtr)
    fmt.Printf("\nSafe examination of the second location in the slice: %#v", &h[1])
    fmt.Println()
}



example nil pointer: []int(nil)
has memory adddress: 0xc42000e440
example empty slice: []int{}
has memory adddress: 0xc42000e4a0
both have the same length: 0 0 but does each equal nil?  true false
[]
len=1 cap=1 [0]
len=2 cap=2 [0 1]
len=3 cap=4 [0 1 2]
Slice of byte memory addresses: 0xc42000e580 0xc42000a618 0xc42000a619 
The byte value in the second element is binary five: 101
Unsafe pointer of the first memory location: 0xc42000a618
after pointer arithmetic and unsafe re-type: 0xc42000a619, value: 101
Safe examination of the second location in the slice: (*uint8)(0xc42000a619)

Reminder, just because we can does not mean we should =p

Examples of Go Slice operations and tricks

Some examples of slices in action https://github.com/golang/go/wiki/SliceTricks

package main

import (
    "fmt"
)

func main() {
    a := []int{1, 2, 3}
    fmt.Println(a[1:2]) // 2
    fmt.Println(append(a, a[1:2]...)) // 1, 2, 3, 2

    // pre-allocating might be premature optimization and lead to bugs
    premature := make([]string, 10, 10)
    premature[0] = "foo"
    premature = append(premature, "bar")
    fmt.Println(len(premature), premature) // "11 [foo          bar]"

    var s []string
    s = append(s, "add", "multiple", "items", "at", "once")
    fmt.Println(len(s), s) // "5 [add multiple items at once]"

    // COPY also known as ADD to a slice (in this case add to a nil slice)
    // the triple dots (ellipsis in english) indicates a variadic parameter
    b := append([]string(nil), s...)
    // https://golang.org/ref/spec#Passing_arguments_to_..._parameters
    // https://golang.org/src/builtin/builtin.go?s=4716:4763#L124
    fmt.Println(len(b), b) // "5 [add multiple items at once]"

    c := make([]string, len(s)) // perhaps more readable
    copy(c, s)
    fmt.Println(len(c), c) // "5 [add multiple items at once]"

    // CUT - but warning, this only removes it from the slice... 
    // NOT the underlying array so a possible memory leak!
    s = append(s[:1], s[4:]...) // up to but not including index 1, start at index 4 to the end
    fmt.Println(len(s), s)      // "2 [add once]"

    // DELETE index 2 (same ordering)...
    // so no memory leak by correctly setting it to the zero value (usually nil for objects)
    copy(b[2:], b[2+1:])
    b[len(b)-1] = ""
    b = b[:len(b)-1]
    fmt.Println(len(b), b) // "4 [add multiple at once]"

    // INSERT into index 2 a new value
    // b = append(b[:2], append([]string{"foobar"}, b[2:]...)...)
    // Preferred more readable and avoid the extra slice creation with copy sleight of hand
    b = append(b, "")
    copy(b[3:], b[2:])
    b[2] = "foobar"
    fmt.Println(len(b), b) // "5 [add multiple foobar at once]"

    // SWAP two values
    b[0], b[1] = b[1], b[0]
    fmt.Println(len(b), b) // "5 [multiple add foobar at once]"

    // REVERSE by using the mirror image effect and multiple assignments
    for i := len(b)/2 - 1; i >= 0; i-- {
        opp := len(b) - 1 - i
        b[i], b[opp] = b[opp], b[i]
    }
    fmt.Println(len(b), b) // "5 [once at foobar add multiple]"
}

Slices are fairly fundamental and while most things are easy there are definitely some gotchas

A critical thing to remember when reasoning is that slices are references to underlying arrays, so small subslice from a very large slice will prevent that larger object/array from being garbage collected

https://play.golang.org/p/znwrmQavmn (work with the slice tricks example yourself)
https://golang.org/ref/spec#Passing_arguments_to_..._parameters
https://golang.org/src/builtin/builtin.go?s=4716:4763#L124

Of course, read the docs and test it for your requirements, situation, and circumstances!

Gotcha with Slices and Pointers

Assigning a new slice variable (pointer) to an existing slice will link the two together. This may lead to undesired "side-effects".

package main

import (
    "fmt"
)

func main() {
    s := []string{"a", "b", "c"}
    fmt.Println(s) // [a b c]
    i := 1
    temp := s
    temp = append(temp[:i], temp[i+1:]...)
    fmt.Println(temp) // [a c]
    fmt.Println(s) // [a c c]

    fmt.Printf("%p \n", &s)
    fmt.Printf("%p \n", &temp)
}

removing (cut or delete) a specific element in the slice using the new pointer affects the original slice

https://play.golang.org/p/togSAlnj6J

Functions, Anonymous Functions, Functions as Parameters, and Filters

Go has functions as first class citizens. Just assign a function to a variable or define a type that is a function signature. Since go starts from simple blocks we build, as needed, more complex tools like filtering from a slice of integers.

package main

import( "fmt"
)

// https://golang.org/doc/codewalk/functions/
// filter is a type that allows us to apply a test to the provided integer parameter
type filter func(x int) bool

// applyFilter is a trivial examply of using a filter as a parameter
func applyFilter(fn filter, x int) bool {
    return fn(x)
}

// SliceFilter returns a new slice filtered using the filter function parameter
func SliceFilter(fn filter, n []int) []int {
    var result []int
    for _, x := range n {
        if fn(x) {
            result = append(result, x)
        }
    }
    return result
}

// isEven returns a filter (the anonymous function inside)
func isEven() filter {
    return func(x int) bool {
        if x%2 == 0 {
            return true
        }
        return false
    }
}

func main() {
    a := []int{2, 4, 6}
    b := []int{1, 3, 7, 9}
    c := []int{5, 3, -1, 12}

    var iseven = func(x int) bool {
        if x%2 == 0 {
            return true
        }
        return false
    }
    fmt.Println("2 is even: ", iseven(2))
    fmt.Println("4 is even: ", applyFilter(iseven, 4))
    fmt.Println(a, "is even: ", SliceFilter(iseven, a))  // [2 4 6] is even:  [2 4 6]
    fmt.Println(b, "is not even: ", SliceFilter(iseven, b))  // [1 3 7 9] is not even:  []
    fmt.Println(c, "filtered for evens becomes: ", SliceFilter(isEven(), c))  // [5 3 -1 12] filtered for evens becomes:  [12]
}

With a function that returns a function, and a function that requires a parameter that is of the type "function signature", we can apply the Strategy Pattern

https://play.golang.org/p/Uxm7HZzS-V (yes you can mess with functions too)
https://en.wikipedia.org/wiki/Anonymous_function
https://en.wikipedia.org/wiki/Strategy_pattern

MergeSort

And because I like source code in blogs, a highly imperfect mergesort.

package main

import (
    "fmt"
)

// SliceSplit is a function to split a slice into roughly even partitions
// https://golang.org/doc/effective_go.html#two_dimensional_slices
/* A more specific implementation than SliceSplit with the special case partitionSize = 1 could have been
result := make([][]int, len(n), len(n))
for i := 0; i < len(n); i++ {
    element := make([]int, 1, 1)
    element[0] = n[i]
    result[i] = element
}
return result
*/
func SliceSplit(n []int, count int) ([][]int, error) {
    result := [][]int{}
    // TODO: is there a better way of handling split into 0 pieces?
    if count == 0 || count > len(n) {
        return result, fmt.Errorf("Cannot split length %d into %d pieces", len(n), count)
    }

    partitionSize := len(n) / count
    for i, k := 0, 0; i < count; i, k = i+1, k+partitionSize {
        a := n[k : k+partitionSize]
        // special case to pad the last partition with all elements
        if i == count-1 {
            a = n[k:]
        }
        result = append(result, a)
    }
    return result, nil
}

// SlicesMerge takes two sorted slices of integers and merges them into a single sorted slice of integers
func SlicesMerge(a []int, b []int) []int {
    s := make([]int, len(a)+len(b))
    for ai, bi, si := len(a)-1, len(b)-1, len(a)+len(b)-1; si >= 0; si-- {
        if ai < 0 {
            s[si] = b[bi]
            bi--
        } else if bi < 0 {
            s[si] = a[ai]
            ai--
        } else if a[ai] > b[bi] {
            s[si] = a[ai]
            ai--
        } else {
            s[si] = b[bi]
            bi--
        }
    }
    return s
}

//MergeConsecutiveElements joins two consecutive slice elements together
func MergeConsecutiveElements(a [][]int) [][]int {
    var result [][]int
    for i, k := 0, 0; i < len(a); i++ {
        if i+1 < len(a) {
            result = append(result, SlicesMerge(a[i], a[i+1]))
            k++
            i++
        } else {
            result = append(result, a[i])
        }
    }
    return result
}

// MergeSort uses the merge sort algorithm to return a sorted a slice of integers
func MergeSort(n []int) []int {
    parts, _ := SliceSplit(n, len(n))
    result := MergeConsecutiveElements(parts)
    for len(result) > 1 {
        result = MergeConsecutiveElements(result)
    }

    return result[0]
}

func main() {

    n := []int{2, 1, 0, -1}
    fmt.Println("unsorted start:", n)
    fmt.Println("sorted:", MergeSort(n))

    n = []int{0, 1, 2}
    fmt.Println("unsorted start:", n)
    fmt.Println("sorted:", MergeSort(n))

    n = []int{9, 2, 4, 3}
    fmt.Println("unsorted start:", n)
    fmt.Println("sorted:", MergeSort(n))

    n = []int{9, 8, 7, 3, 2, 5}
    fmt.Println("unsorted start:", n)
    fmt.Println("sorted:", MergeSort(n))

    fmt.Println("done")
}

https://en.wikipedia.org/wiki/Merge_sort
https://play.golang.org/p/nAHF50zk7m (modify the merge online)
https://bitbucket.org/johnpfeiffer/go-slice-mergesort (because version control means the fun never stops)

Consul Service Discovery and Cluster Configuration

2016-08-27T21:01:00-07:00

Overview

Basically consul is an out-of-the-box service discovery system intended for clustered and highly available applications.

This kind of infrastructure simplifies the programming of distributed systems so that it is easier to deliver value quickly on the actual domain problems.

I have certainly done my fair share of hardcoded config files to "discover" dependency services and even used chef for "config management"... But with the evolution of dev-ops, web scale, microservices, containers, etc. it is great to leverage an existing battle tested solution

Consul Cluster using Docker

Following the straightforward work from this Docker Image we can run a cluster on a single machine:

https://hub.docker.com/r/progrium/consul/

sudo su
docker run -d --name node1 -h node1 progrium/consul -server -bootstrap-expect 3
JOIN_IP="$(docker inspect -f '{{.NetworkSettings.IPAddress}}' node1)"
docker run -d --name node2 -h node2 progrium/consul -server -join $JOIN_IP
docker run -d --name node3 -h node3 progrium/consul -server -join $JOIN_IP
docker run -d -p 8400:8400 -p 8500:8500 -p 8600:53/udp --name node4 -h node4 progrium/consul -join $JOIN_IP

The second 2 nodes join the first one in the cluster by using the inspected IP Address, the last container is a consul agent (not in the quorum) but has public ports for interactivity

curl localhost:8500/v1/catalog/nodes
    [{"Node":"node1","Address":"172.17.0.2"},{"Node":"node2","Address":"172.17.0.3"},
    {"Node":"node3","Address":"172.17.0.4"},{"Node":"node4","Address":"172.17.0.5"}]
dig @0.0.0.0 -p 8600 node1.node.consul
    ;; QUESTION SECTION:
    ;node1.node.consul.     IN  ANY

    ;; ANSWER SECTION:
    node1.node.consul.  0   IN  A   172.17.0.2

REST API call to the list of nodes, then DNS client to get the Record for the first node

curl http://localhost:8500/v1/status/leader
    "172.17.0.2:8300"
curl http://localhost:8500/v1/status/peers
    ["172.17.0.2:8300","172.17.0.3:8300","172.17.0.4:8300"]

curl http://localhost:8500/v1/health/node/node1

some more REST calls about the basic nodes, RAFT leadership and peers, and node health

https://www.consul.io/docs/agent/http.html
https://www.consul.io/docs/agent/dns.html

:::bash

curl http://localhost:8500/v1/catalog/services {"consul":[]} curl http://localhost:8500/v1/catalog/service/web []

Listing of the services available, no web service yet =)

Alternative Install from Zip

apt-get install unzip
wget https://releases.hashicorp.com/consul/0.7.0/consul_0.7.0_linux_amd64.zip
unzip consul_0.7.0_linux_amd64.zip
BINDIP=$(ifconfig eth0 | grep "inet addr" | cut -d ':' -f 2 | cut -d ' ' -f 1)
./consul agent bootstrap server bind=$BINDIP datadir /tmp/consul

netstat antp  | grep consul
curl http://localhost:8500/v1/status/peers

Note getting the IP Address on ubuntu 16.04 uses enp3s0 or enp25 which can be changed back via grub workaround: GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"

https://www.consul.io/docs/agent/options.html

Registering a Service

Creating and running a very simplistic golang web server (assuming you have go installed ;) , go run web.go https://blog.john-pfeiffer.com/go-programming-intro-with-vs-code-and-arrays-slices-functions-and-testing/ (though you could also use nginx in docker)

import (
    "fmt"
    "net/http"
)

func myHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "hi\n")
}

func main() {
    http.HandleFunc("/", myHandler)
    http.ListenAndServe(":8080", nil)
}

From the previous steps we should have an Agent where we can register the new web service...

#!/bin/bash

curl --header "content-type: application/json" -X PUT -d'{
  "ID": "web1",
  "Name": "web",
  "Tags": [
    "master",
    "v1"
  ],
  "Address": "127.0.0.1",
  "Port": 8080,
  "EnableTagOverride": false,
  "Check": {
    "HTTP": "http://localhost:8080/health",
    "Interval": "10s",
    "TTL": "15s"
  }
}' http://localhost:8500/v1/agent/service/register

In order to verify the new service is registered (besides the 200 response code)

curl http://localhost:8500/v1/catalog/services
    {"consul":[],"web":["master","v1"]}
curl http://localhost:8500/v1/health/service/web

Our new service is created and doing well

So many more things can be done with https://www.consul.io/docs/agent/http/agent.html#agent_service_register

Stopping the web server (control + C) and checking that Consul has noticed Status is critical |o/

curl http://localhost:8500/v1/health/checks/web
    [{"Node":"node4","CheckID":"service:web1","Name":"Service 'web' check","Status":"critical","Notes":"","Output":"TTL expired","ServiceID":"web1","ServiceName":"web"}]

Starting the web server again and check

curl http://localhost:8500/v1/health/service/web
    [{"Node":{"Node":"node4","Address":"172.17.0.5"},
    "Service":{"ID":"web1","Service":"web","Tags":["master","v1"],"Address":"127.0.0.1","Port":8080},
    "Checks":[{"Node":"node4","CheckID":"service:web1","Name":"Service 'web'check","Status":"critical",
        "Notes":"","Output":"TTL expired","ServiceID":"web1","ServiceName":"web"},
    {"Node":"node4","CheckID":"serfHealth","Name":"Serf Health Status","Status":"passing","Notes":"","Output":"Agent alive and reachable","ServiceID":"","ServiceName":""}]}]

Redis in Containers as another Service

docker run --rm -it -p 0.0.0.0:6379:6379 --name redis redis:alpine

docker run --rm -it --link redis:redis redis:alpine redis-cli -h redis -p 6379 help keys
docker run --rm -it --entrypoint=/bin/sh --link redis:redis redis:alpine

By running a local redis service we can modify our simple Go web service to query consul and dynamically discover how to reach the correct dependency, "look mom, no config files!"

https://hub.docker.com/_/redis/

Distributed Configuration and the Go Client Library

A simple use case is to use the key value store to distribute other information besides services that need to be discovered.

Obviously interacting directly with Consul as a client from inside the application is beneficial to "keeping it all in the code" and not relying on config files or shell scripts.

Documentation on the Key Value store and the official Go client library

Python Client

Using an open source client can help avoid "do not repeat yourself" of writing the REST API wrapper (and benefiting from crowd source at work)

http://consulate.readthedocs.io/en/stable/
https://github.com/gmr/consulate

sudo pip3 install consulate

Like all open source projects this has some bugs and outstanding PRs but it is better than another one I tried which was still in alpha (aka not really fully implemented) , https://www.consul.io/downloads_tools.html

Some Gotchas

Consul has a few edge cases that you may need to address specifically:

If a node reboots and changes ip address it will not go well: https://github.com/hashicorp/consul/issues/457 , the simplest case might be to just remove it's data directory and force it to rejoin without any data
If a new node attempts to join a cluster it needs to know the ip address of an existing node, there is no "auto discovery-join" mechanism except to delegate to Atlas, the paid SaaS product from HashiCorp, or of course to write your own workaround https://www.consul.io/docs/guides/bootstrapping.html
If all of the server nodes in the cluster go down then there is no auto-recovery (which is not surprising I suppose...) https://www.consul.io/docs/guides/outage.html , https://github.com/hashicorp/consul/issues/454 , https://github.com/hashicorp/consul/issues/526, again, if you write your own wrapper to detect this scenario as the nodes reboot (or in an immutable world are re-added assuming you have solved #1 and #2 ;) they "should" be able to recover and reload from raft/peers.json

A micro story about migrating a personal monolith into microservices

2016-07-29T20:59:00-07:00

I'm a fan of best practices (who isn't?) and as complexity increases one of the modern paradigms is to use microservices to more transparently manage complexity, reduce tight coupling, and decrease maintenance overhead.

Taking a different approach to my "less than webscale" personal web services has still made great improvements to the services and my quality of life.

This article is categorized as Build and CI/CD and DevOps because while microservices is transformative for larger organizations to avoid the negative consequences of Conway's Law and provides a better architecture for really complex systems, in my case it was simply about decoupling deployments and not having all of my eggs in one basket. (e.g. the Drupal migration below actually did not affect the User Experience in any visible way).

How to grow a monolith

Value, value, value

I've had a Virtual Private Server for many years and have been always able to find something for free (or as my requirements increased at most $10 a month). This has been at least 1000x worth the investment as it has provided a platform for learning and experimenting with all sorts of technology (Linux, PHP, Python, Twisted, MySQL, Redis, etc.) and of course a useful catch all tool (to check DNS resolution or temporarily store a file).

I'm a firm believer in understanding the whole stack. Maintaining the OS and managing deployments has made me conscious of the many hidden costs and compromises/trade-offs in software and services.

Organic Sprawl

While I started with a personal notes site in PHP, I eventually added a couple of personal blogs using Drupal, and then another Drupal site about physics (http://physicstime.com) for my father in law, and then a few experiments in Python (including uwsgi and Twisted). This was all of course underpinned by the shared underlying Cherokee web server, Drupal, PHP, Python, MySQL, and Ubuntu.

Physicstime alone has served over 500,000 visitors =)

I envisioned that having a common platform would make it easier to add more sites and the shared maintenance meant I would only have to pay it once, especially considering the alternative of paying for and deploying many separate servers.

I got to try a lot of different technologies but it started to be clear that it was not a "common platform" and maintenance (and just the mental energy of worrying about deferred maintenance) started to take up way too much time and effort.

The downsides of a monolith

Anecdotes: a security update meant the infrastructure provider needed to restart all of their host machines which meant "hmmm, will all my services restart correctly on reboot?" - luckily yes. Then I had to patch my virtual server (anything on the Internet is one vulnerability away from becoming taken over and in the least case being used for spam/DDoS and eventually being unable to actually serve my sites, in the worst case rooted or keystroke-logged in an attempt to hack my life or in a serious criminal pursuit).

Oh and then there's the times the logs (web or auth due to anonymous attackers scanning) or backups filled the disk... (facepalm)

Unavailability Due To:

any infrastructure vendor maintenance
any OS maintenance/upgrade
incorrectly configured/rogue application
a security issue in one affects them all
they all contend for the same resources
they all share the same version/dependency requiring upgrading and testing everything at once

In Context: Linode, Ubuntu, Cherokee, and Drupal were good choices at the time

Firstly let me say I picked a great vendor (Linode) who was very solid (they limit bad actor customers which tends to make resources predictable) and that Ubuntu OS and Cherokee webserver are very easy to setup and maintain which is one reason why I've put this off for so long.

Drupal does an ok job of separating the tech stack from the content publishing so it was possible to ignore the tech side for awhile.

Another factor is that iPhones/Android, Dropbox, Bitbucket/GitHub, PaaS, and a whole generation of technologies were not around when I set this up.

Finally, maybe it's a corollary to Moore's law and the prevalence of the cloud but there's quite a bit of free compute around than there used to be =]

Thinking Microservices

Thinking about microservices is like TDD (Test Driven Design): it exposes assumptions, unmanaged organic evolution, and accidental complexity.

Discovering the real problem domain

When you have a hammer everything looks like a nail

It becomes too easy to just use an interesting or popular technology for everything regardless of the true problem.

Analyzing what I actually did with the various services I realized there were actually two distinct phases: creation and distribution.

I did not have a "realtime" or "high volume dynamic data" use case, nor even a large number of content publishers that needed extra tooling. (The plugins/additions I used weren't even that exotic.)

In contrast the overhead was my irreplaceable time spent for the maintenance of Ubuntu patching/upgrades, including the underlying PHP, Drupal upgrades, backups, and of course the inestimable risk of running something on the Internet. =p

How I Converted to Microservices: Big Bang vs Kanban

A common question is "how", and since "it depends" ;) in this case I had limited free hours to accomplish change and a strong desire to not break existing service.

Rather than "Big Bang" I went Kanban https://en.m.wikipedia.org/wiki/Kanban_(development) (a much better analogy than Martin Fowler's "strangler pattern" analogy http://www.martinfowler.com/bliki/StranglerApplication.html). This allowed gradual migration with the least disruption and the most flexibility in when changes would occur. As a not-to-be-underestimated bonus it was also the least stressful.

Since I publish a post maybe once a month my blog was the simplest place to start.

Feature parity requirements:

publish read-only pages to the world
editing
some minimal data structure (category and tags)
search
preserve content and meta data
regular backup of the content

I migrated my content to a Pelican static site (with advanced functionality via a nice JavaScript theme) running on GitHub static pages and Travis CI.

Having text in git provides a built in backup (published as html and stored remotely in markdown in the free "Software as a Service" repository) with of course a local git repo working copy on my laptop.

The result is I only need to do a single git push of markdown text to deploy to production. It's highly available, scalable, and as a bonus there is versioning and I got a fairly snazzy facelift with improved search.

Migration has extra costs of research and tweaks but is an opportunity for new benefits

First I had to research and evaluate static site generators and static site hosts.

Next I had to learn how to setup a workflow with test data to automate the process from writing content to publishing.

The actual data migration was quite lengthy. This was due to the inevitable format change (export from Drupal to .md) and post transformation validation.

Of course I also had to update DNS entries and even setup subdomains and 301 redirects.

Like any rewrite I also ended up adding things (like tags).

One of the most beneficial "while I'm already redoing everything anyways" enhancements was adding CDN and SSL via Cloudflare which actually added another layer of availability and security.

Cloudflare also has very useful feature for blocking noise with Security -> Security Rules -> Create rule , to block "Country is in RU, CN" or "AS Num is in 16276, 14061, 24940" (OVH, DigitalOcean, Hetzner"

https://www.cloudflare.com/learning/network-layer/what-is-an-autonomous-system/

Data Gravity is expensive and Microservices allows polyglot so Go Programming

My personal notes are full of years of research and was the most time consuming, fulfilling the adages of "data gravity" and "unstructured data costs you" =(

A free static site on bitbucket.org (markdown in a free git repository transformed to html for free via Shippable.com) used the same pattern (and Pelican tech) as my blog: version control, offloading the hosting to someone else, and JavaScript for search.

While the file system organically captures metadata like "created date" I had to inject that into the content; I found this to be a data integrity improvement as I had noticed before that FTP and git have a tendency to discard that metadata.

One of the most time consuming transformations was just tweaking the plain text into markdown but this was worth the improvement in readability since the content is far more often read than written =]

While Markdown violates the principle of separating data and presentation I found it to be a pragmatic compromise as it IS a standard and it's machine readable. (I could theoretically use a script to convert it back to plaintext ;)

Oh right, so Go, aka Golang?

I leveraged Google AppEngine and for fun got to use the relatively new programming language Go to write a custom 301 redirector to prevent links on the Internet from breaking and allow search engines link from all of the previous URLs to the new locations.

While Python is very easy to pickup I found Go to be similar enough (especially to C) to be also not so hard to learn (lots of documentation!) and better able to do what I needed simply within the language (though Python pretty much has a library for everything it also has C dependencies that don't always play nice with a PlatformAsAService).

Drupal to Drupal

The more things change the more they stay the same.

Wherever you go there you are.

The most active Drupal site will stay Drupal on DigitalOcean (to leverage their one click example) and cheaper prices.

Edit: one click was for Drupal8, yet another headache migration, so not in the scope of this project.

Now at least as an isolated service (website) on a dedicated server, updates will be specific to it.

I also put in the effort to use automation via Packer and experiment with Docker...

Immutable Packer

I considered Docker Machine and Ansible but both seemed the wrong tools for my purpose.

Docker Machine is still relatively new and is more oriented towards a cluster of nodes. Additionally the post docker image phase (ssh commands to install things) seems overly complex.

Ansible (SSH paradigm) is simpler than chef but encourages a mutable long lived server.

Packer has a simple and straight forward way of building an immutable server image for DigitalOcean yet retains the flexibility to adapt to other cloud vendors later if needed.

Drupal website context and domain problems

Besides the basic components (Docker will simplify this): nginx, php, MySQL

The ongoing issues:

Backups of the MySQL and uploaded files
Upgrades of the OS and components (security)
Upgrades (security) of Drupal and modules

Hint: DigitalOcean offer full disk image backups as a paid service so if I ever stop being cheap this resolves pretty quickly

The full details are in a separate post but I'm pretty happy that setting up the box from scratch again, upgrading the various subcomponents, or even migrating to a different vendor will be a lot easier in the future (and won't affect any of the other projects I have going on).

Why not just automate the monolith

The "easy" answer may have been to automate more of my "ball of mud" to address the effort/efficiency of applying security updates.

Yet the "better monolith" would mean I still owned the maintenance and uptime for a large percent of my services.

The microservices approach of diversity means that with different providers (GitHub, Bitbucket, Google AppEngine, DigitalOcean, etc.) it is nearly impossible for them to all go down simultaneously or be affected by one another.

Leveraging other platforms that better fit my use case means I benefit from their expertise and by reducing the moving parts I have a reduced security risk.

My experiments in other frameworks and programming languages were never a good match for my "production" web services.

"because it's there" or "because I can" are very often the reason things continue to be done in a suboptimal way ;)

Now I will focus more on the "top" of the tech stack and high value content (like this blog post), less on the "how to automate and deploy" portion.

Ongoing and Future Work

I still have to purchase/renew domains, update DNS, and write content.

I still have to eventually find a platform that allows my father-in-law to publish content (and upload files) where I'm not responsible for security patches or backups ;)

My experimenting is now done via a PaaS like Google AppEngine, Heroku, Openshift etc. or using Docker containers. That means more admin and cognitive sprawl but PaaS and CaaS are more predisposed to version control and elastic/disposable architecture so in all a lot less maintenance.

The biggest new cost is managing the increased number of services but this at least makes explicit what I am working on and is mostly mitigated by automation.

Drupal with Docker Compose and nginx and php-fpm and mariadb

2016-07-28T23:59:00-07:00

Replacing: cherokee webserver, php-cgi, MySQL

In my professional life I've seen the webserver front end default choice shift from Apache to NGINX occur for production services.

Nginx is compelling due to simpler configuration and improved performance so I suspect the existing preponderance of Apache deployments are because "if it ain't broke don't fix it" (which makes lots of sense in Operations) along with the large number of Ops/SysAdmins who already know how to configure and integrate Apache (which makes a lot of sense to Management).

https://www.nginx.com/blog/nginx-vs-apache-our-view/

For a personal project I experimented with and chose Cherokee web server (http://cherokee-project.com/) because it offered good performance and a simpler setup (Web UI even!)

Unfortunately as the project Dev and adoption slowed it seemed to make a lot of sense to finally convert my personal project to nginx. https://github.com/cherokee/webserver/commits/master

I also wanted to experiment with a Docker based infrastructure (with docker-compose and a single YAML config file, ideally leveraging as much as possible the many official upstream docker images (https://hub.docker.com/explore/))

Benefits:

this would isolate my app from the Host OS (increased portability)
allow for simpler component swapping or upgrades (with testing and rollback and even local Dev)
leave room for adding other isolated components

Almost able to leverage everything right out of the box

I spent time researching nginx and php-fpm and created a minimal config that worked (two docker commands) based on the slimmer alpine linux docker images. https://blog.john-pfeiffer.com/nginx-with-docker/

This wasn't quite far off from the official Drupal Dockerfile except there are some more OS dependencies like GD and XML so I instead opted to use the official Drupal Docker image. (Non alpine linux so larger size but they've done the work of packaging the complexity and by not customizing it will be easier to update in the future - and avoid a Docker Build entirely!)

docker-compose.yml for nginx and drupal:7-fpm and mariadb:5.5

# https://docs.docker.com/compose/compose-file/
# https://docs.docker.com/compose/environment-variables/
# https://www.nginx.com/resources/wiki/start/topics/recipes/drupal

nginx:
  image: nginx:alpine
  ports:
    - "80:80"
  volumes:
    - ./nginx.conf:/etc/nginx/nginx.conf:ro
    - ./default.conf:/etc/nginx/conf.d/default.conf:ro
    - /var/www/html:/var/www/html/
  links:
    - fpm

# https://hub.docker.com/_/drupal/
fpm:
  image: drupal:7-fpm
  ports:
    - "9000:9000"
  volumes:
    - /var/www/html:/var/www/html/
  links:
    - mysql

# https://hub.docker.com/_/mariadb/
mysql:
  image: mariadb:5.5
  ports:
    - "3306:3306"
  environment:
    - MYSQL_ROOT_PASSWORD
    - MYSQL_USER
    - MYSQL_PASSWORD
    - MYSQL_DATABASE

Adding the MariaDB (compatible with MySQL) image/container was fairly straightforward (just reading the docs on how to override and setup the default user and DB via Environment Variables)

I am ok with the Database files being kept completely inside of a persistent container. I expect to do regular backups (mysqldump and scp) and may also spend the money to just use DigitalOcean's backup service (no poweroff required!)

And the bugs

I discovered a bug where the css/themes didn't render correctly. As part of the troubleshooting I installed nginx, php, and MySQL locally on the host with my override configs (no bug exhibited). I then replaced each "native host installed service" with the Docker one, working from the backend DB forward I was able to identify the issue.

Unfortunately the nginx image uses a default user of "nginx" whereas the drupal:php image uses "www-data" which caused issues when accessing the Drupal source files (which I was sharing via --volume on the host, preferring to eschew a "data container").

My personal Linux maxim still holds true: "If there's a problem with something running in Linux it's probably a Permissions issue". ;)

Do not add complexity

I avoided creating my own docker image build FROM drupal:php with nginx included because:

simplest with as few build steps possible
contrary to the "do one thing well" principle
contrary to the "one container one app" Docker principle
tight coupling of nginx and php would make it harder to update one or the other independently

Cache your upstream dependencies

Ironically I also ended up providing my own drupal.tar.gz downloaded from Drupal.org but cached in Bitbucket as the upstream ftp.drupal.org proved unreliable.

The pragmatic solution

Instead, as I was already using packer on DigitalOcean to automate building the Host and uploading the custom configs, docker-compose.yaml, and Drupal source, I added a few steps to install nginx directly to the Host OS.

packer.json

NEWUSER_NAME=username NEWUSER_PASSWORD=yourpassword DIGITALOCEAN_API_TOKEN=yourapitoken /opt/packer build packer.json

A few files and a single command (packer is just a single binary downloaded from hashicorp) I can have another Docker based Drupal host ready (awyeah)

{
  "_comment": "https://www.packer.io/docs/builders/digitalocean.html",
  "variables": {
    "digitalocean_api_token": "{{env `DIGITALOCEAN_API_TOKEN`}}",
    "newuser_name": "{{env `NEWUSER_NAME`}}",
    "newuser_password": "{{env `NEWUSER_PASSWORD`}}"
  },

  "builders": [{
    "type": "digitalocean",
    "api_token": "{{user `digitalocean_api_token`}}",
    "size": "512mb",
    "region": "lon1",
    "image": "ubuntu-16-04-x64",
    "droplet_name": "drupal-from-packer-{{timestamp}}",
    "snapshot_name": "drupal-from-packer-{{isotime \"2006.01.02.030405\"}}"
  }],

  "provisioners": [
    {
      "type": "shell",
      "inline": [
        "ip a",
        "curl -s http://checkip.amazonaws.com",
        "apt-get update",
        "sudo apt-get install -y vim curl wget byobu ntp tar gzip",
        "timedatectl set-timezone Etc/UTC",
        "cat /etc/timezone",
        "date",
        "useradd -s /bin/bash -m {{user `newuser_name`}}",
        "usermod -a -G admin {{user `newuser_name`}}",
        "echo '{{user `newuser_name`}}:{{user `newuser_password`}}'|chpasswd",
        "cat /etc/passwd",
        "sed -i 's/Port 22/Port 2222/g' /etc/ssh/sshd_config",
        "sed -i 's/PermitRootLogin yes/PermitRootLogin no/g' /etc/ssh/sshd_config",
        "echo 'PasswordAuthentication no' >> /etc/ssh/sshd_config",
        "cat /etc/ssh/sshd_config",
        "mkdir -p /home/{{user `newuser_name`}}/.ssh",
        "fallocate -l 1G /swapfile",
        "chmod 600 /swapfile",
        "mkswap /swapfile",
        "swapon /swapfile",
        "swapon --show",
        "free -h",
        "echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab",
        "echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf",
        "echo 'vm.vfs_cache_pressure=50' | sudo tee -a /etc/sysctl.conf"
      ]
    },
    {
      "type": "file",
      "source": "authorized_keys",
      "destination": "/home/{{user `newuser_name`}}/.ssh/authorized_keys"
    },
    {
      "type": "shell",
      "script": "install-docker.sh"
    },
    {
      "type": "file",
      "source": "docker-compose.yml",
      "destination": "/home/{{user `newuser_name`}}/docker-compose.yml"
    },
    {
      "type": "shell",
      "inline": [
        "mkdir -p /var/www/html",
        "wget https://bitbucket.org/yourusername/yourrepository/raw/97360be2edd53b93149d750db24f749aebc27988/binaries/drupal-7.50.tar.gz",
        "tar xf drupal-7.50.tar.gz --strip-components=1 -C /var/www/html",
        "ls -ahl /var/www/html",
        "mkdir -p /var/www/html/sites/default/files",
        "chmod 777 /var/www/html/sites/default/files",
        "cp -a /var/www/html/sites/default/default.settings.php /var/www/html/sites/default/settings.php",
        "chmod 777 /var/www/html/sites/default/settings.php"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "apt-get install -y nginx"
      ]
    },
    {
      "type": "file",
      "source": "nginx.conf",
      "destination": "/etc/nginx/nginx.conf"
    },
    {
      "type": "file",
      "source": "default.conf",
      "destination": "/etc/nginx/conf.d/default.conf"
    }

  ]

}

HINT: A DigitalOcean 512MB droplet needs swap enabled due to the MySQL/MariaDB memory defaults, otherwise you get strange errors about inode unavailable while starting the MariaDB container

https://www.suse.com/documentation/opensuse114/book_tuning/data/cha_tuning_memory_vm.html
https://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.7/2.6.7-mm1/broken-out/vfs-shrinkage-tuning.patch
One thing that is annoying is as a single partition it is potentially vulnerable to a /var/log DenialOfService
A very nice tool to add is fail2ban (apt-get install fail2ban) but I want to ensure I tweak the configs correctly

install-docker.sh

#!/bin/sh
# ubuntu 16.04 is xenial , https://blog.john-pfeiffer.com/docker-intro-install-run-and-port-forward/
sudo apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
sudo sh -c "echo 'deb https://apt.dockerproject.org/repo ubuntu-xenial main' > /etc/apt/sources.list.d/docker.list"
sudo apt-get update || exit 1
sudo apt-get install -y linux-image-extra-$(uname -r) || exit 1
sudo apt-get install -y docker-engine docker-compose || exit 1
service docker status || exit 1
docker info || exit 1

linux-image-extra is to ensure we have AUFS because docker needs a proper storage driver

An alternative is to download the docker-compose binary directly (to /usr/local/bin): - https://github.com/docker/compose/releases

sudo curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose which docker-compose ; sudo docker-compose --version

docker-compose.yml with drupal:7-fpm and mariadb:5.5

# https://hub.docker.com/_/drupal/
fpm:
  image: drupal:7-fpm
  ports:
    - "127.0.0.1:9000:9000"
#    - "9000:9000"
  volumes:
    - /var/www/html:/var/www/html/
  links:
    - mysql

# https://hub.docker.com/_/mariadb/
mysql:
  image: mariadb:5.5
  ports:
    - "3306:3306"
  environment:
    - MYSQL_ROOT_PASSWORD
    - MYSQL_USER
    - MYSQL_PASSWORD
    - MYSQL_DATABASE

2 out of 3 ain't bad

nginx.conf

# slightly modified /etc/nginx/nginx.conf from "apt-get install nginx"
# https://www.nginx.com/resources/wiki/start/topics/examples/full/

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        gzip on;
        gzip_disable "msie6";

        gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        include /etc/nginx/conf.d/*.conf;

}

default.conf

server {
    listen      80;
    server_name physicstime.com;
    root /var/www/html; ## <-- Your only path reference.

    location = /favicon.ico {
        log_not_found off;
        access_log off;
    }

    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }

    # Very rarely should these ever be accessed outside of your lan
    location ~* \.(txt|log)$ {
        deny all;
    }

    location ~ \..*/.*\.php$ {
        return 403;
    }

    location ~ ^/sites/.*/private/ {
        return 403;
    }

    # Allow "Well-Known URIs" as per RFC 5785
    location ~* ^/.well-known/ {
        allow all;
    }

    # Block access to "hidden" files and directories whose names begin with a
    # period. This includes directories used by version control systems such
    # as Subversion or Git to store control files.
    location ~ (^|/)\. {
        return 403;
    }

    location / {
        # try_files $uri @rewrite; # For Drupal <= 6
        try_files $uri /index.php?$query_string; # For Drupal >= 7
    }

    location @rewrite {
        rewrite ^/(.*)$ /index.php?q=$1;
    }

    # Don't allow direct access to PHP files in the vendor directory.
    location ~ /vendor/.*\.php$ {
        deny all;
        return 404;
    }

    # In Drupal 8, we must also match new paths where the '.php' appears in
    # the middle, such as update.php/selection. The rule we use is strict,
    # and only allows this pattern with the update.php front controller.
    # This allows legacy path aliases in the form of
    # blog/index.php/legacy-path to continue to route to Drupal nodes. If
    # you do not have any paths like that, then you might prefer to use a
    # laxer rule, such as:
    #   location ~ \.php(/|$) {
    # The laxer rule will continue to work if Drupal uses this new URL
    # pattern with front controllers other than update.php in a future
    # release.
    location ~ '\.php$|^/update.php' {
        fastcgi_split_path_info ^(.+?\.php)(|/.*)$;
        # Security note: If you're running a version of PHP older than the
        # latest 5.3, you should have "cgi.fix_pathinfo = 0;" in php.ini.
        # See http://serverfault.com/q/627903/94922 for details.
        include fastcgi_params;
        # Block httpoxy attacks. See https://httpoxy.org/.
        fastcgi_param HTTP_PROXY "";
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_intercept_errors on;
        # PHP 5 socket location.
        #fastcgi_pass unix:/var/run/php5-fpm.sock;
        # PHP 7 socket location.
        # fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;

        fastcgi_pass 127.0.0.1:9000;
        # fastcgi_pass fpm:9000;
    }

    # Fighting with Styles? This little gem is amazing.
    # location ~ ^/sites/.*/files/imagecache/ { # For Drupal <= 6
    location ~ ^/sites/.*/files/styles/ { # For Drupal >= 7
        try_files $uri @rewrite;
    }

    # Handle private files through Drupal.
    location ~ ^/system/files/ { # For Drupal >= 7
        try_files $uri /index.php?$query_string;
    }

    # prevent hotlinking
    location ~ ^/sites/.*/files/ {
        valid_referers none blocked www.physicstime.com physicstime.com;
        if ($invalid_referer) {
          return 403;
        }
    }

    location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
        # prevent hotlinking
        valid_referers none blocked www.physicstime.com physicstime.com;
        if ($invalid_referer) {
          return 403;
        }
        expires max;
        log_not_found off;
    }

}

slightly modified from the wonderful reference provided by nginx, mostly the first 3 lines and later the fastcgi_pass

https://www.nginx.com/resources/wiki/start/topics/recipes/drupal/

Post boot manual configuration (ohgodwhy)

Post boot of a "fresh from built-by-packer snapshot" there were still the basic steps of configuring Drupal (which I did manually with a browser and install.php though I'm pretty sure I could have just overridden the settings.php directly).

And for this migration project a MySQL dump, SCP of the existing extra modules, and of course the already uploaded user images/files.

I'm sure with more tinkering I can overcome the nginx vs www-data user permissions issue but since I time boxed this project I stuck with this compromise which is still much more improved and automated than my previous setup.

For migrating data there is of course the prerequisite: mysqldump -uroot -p physicstime > backup.sql

vi /var/www/html/themes/bartik/css/style.css
    font-size: 2.929em;

cat /var/www/html/themes/bartik/css/style.css | tr '\n' '\r' | sed -e 's/#site-slogan {\r  font-size: 0.929em/#site-slogan {\r  font-size: 2.929em/' | tr '\r' '\n' > /var/www/html/themes/bartik/css/style.css.updated

customize increased visibility of the #site-slogan { , occasionally needs cache cleared in admin/config/development/performance The oneliner is a complicated way of sed replacing two lines at once (tr replaces newlines with \r temporarily)

sed -i 's/font-size: 87.5/font-size: 120/' /var/www/html/themes/bartik/css/style.css

Because the world does not need yet another 10pt font website

hostnamectl set-hostname physicstime.com
docker-compose up &

Hint: /etc/hosts locally and use a browser to do initial install.php config before a DNS cutover

DB connection string requires 172.17.0.1 (the default docker private IP network bridge)

root@physicstime:/home/USER/physicstime.com# mv files/ /var/www/html/sites/physicstime.com/
mv all /var/www/html/sites/

ifconfig | grep Bc
    172.17.0.1

docker run -it --rm --volume /opt:/opt mariadb:5.5 /bin/bash
    mysql -h172.17.0.1 -uYOURUSER -p physicstime
        show tables;
        exit
    mysql -h172.17.0.1 -uroot -p physicstime < backup.sql

chown root:root -R /var/www/html/sites/
chmod 400 /var/www/html/sites/default/settings.php

ensure /var/www/html/sites/default/files has the correct permissions, otherwise no CSS for you!

docker-compose stop
docker ps --all
docker-compose up &
docker-compose logs |& tee /var/log/physicstime.log
tail -f /var/log/physicstime.log /var/log/nginx/access.log

This starts it manually and displays any traffic

cd /etc/init.d; touch physicstime.sh; chmod +x physicstime.sh; vi physicstime.sh

    #!/bin/bash
    echo "starting physicstime after boot" >> /var/log/physicstime.log
    cd /home/YOURUSER; docker-compose up &

update-rc.d physicstime.sh defaults

this starts the docker based services automatically on boot (not including extra logging)

Adhoc Performance and Latency Testing

Data driven decisions is a big deal, but let's be honest, engineers love numbers :)

ab -l -r -n 500 -c 100 -k -H "Accept-Encoding: gzip, deflate" http://physicstime.com/node?page=1

The unscientific "watching top": 80% for about 5 seconds vs 78% for about 5 seconds

https://httpd.apache.org/docs/2.4/programs/ab.html

Linode and Cherokee and php-cgi 5.3 and MySQL 5.5

Concurrency Level:      100
Time taken for tests:   7.255 seconds
Complete requests:      500
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      5321000 bytes
HTML transferred:       5089500 bytes
Requests per second:    68.92 [#/sec] (mean)
Time per request:       1451.029 [ms] (mean)
Time per request:       14.510 [ms] (mean, across all concurrent requests)
Transfer rate:          716.22 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        2    2   0.4      2       6
Processing:   102 1313 320.0   1437    1525
Waiting:       89 1308 320.2   1431    1521
Total:        104 1315 319.8   1439    1526

Percentage of the requests served within a certain time (ms)
  50%   1439
  66%   1456
  75%   1466
  80%   1473
  90%   1488
  95%   1505
  98%   1516
  99%   1519
 100%   1526 (longest request)

DigitalOcean and nginx and Docker php-fpm and MariaDB 5.5

Concurrency Level:      100
Time taken for tests:   7.791 seconds
Complete requests:      500
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      5320068 bytes
HTML transferred:       5082000 bytes
Requests per second:    64.18 [#/sec] (mean)
Time per request:       1558.145 [ms] (mean)
Time per request:       15.581 [ms] (mean, across all concurrent requests)
Transfer rate:          666.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.1      0       4
Processing:   542 1421 201.6   1453    2020
Waiting:      531 1417 201.2   1447    2017
Total:        545 1422 201.0   1453    2022

Percentage of the requests served within a certain time (ms)
  50%   1453
  66%   1470
  75%   1485
  80%   1494
  90%   1510
  95%   1674
  98%   1877
  99%   1930
 100%   2022 (longest request)

So it seems there's a slight edge in either cherokee over nginx OR not running dockerized applications OR linode vs digitalocean BUT...

When I tweaked it for higher concurrency ab -l -r -n 500 -c 200 -k -H "Accept-Encoding: gzip, deflate" http://physicstime.com/node?page=8

Linode + Cherokee + php-cgi

Time taken for tests:   9.933 seconds
Requests per second:    50.34 [#/sec] (mean)
Time per request:       3973.367 [ms] (mean)

DigitalOcean + Nginx + Docker + php-fpm

Time taken for tests:   8.120 seconds
Requests per second:    61.58 [#/sec] (mean)
Time per request:       3248.067 [ms] (mean)

I can see the trend reverse (shrug)

Latency testing

https://tools.pingdom.com/

website speed test (NY) for cherokee and php-cgi had a load time of 1.17s
website speed test (NY) for nginx and Docker had a load time of 1.46s

meh

Conclusions

Adding docker simplifies one kind of complexity (stringing together multiple services that are packaged upstream) but can come at some cost to performance. (Though this might also be due to the cheaper node price)

The other major advantage of using docker is that different components/services can be upgraded independently (and even just "test upgraded") which allows for a faster adoption of upstream project improvements.

I personally don't like relying on the OS of the host for all of the global dependencies to play nice (and especially that different packages won't have conflicting dependencies).

Since all of these processes are running in the same host that I own I expect I'm not worse off security wise.

I'm curious to see if given enough time I run into the infamous /var/lib/docker issues of orphan containers, running out of disk space, or other issues.

A next step might be to run this in a PlatormAsAService like OpenShift or better yet break up each container to run via a ContainerAsAService (if it's free ;)

Obviously if performance and scale of a mostly read-only content distribution system was really an issue adding a cache layer like Varnish or Cloudflare or even just tweaking the various configurations would help =]

Alpine Linux Introduction Tutorial

2016-02-28T21:19:00-08:00

Alpine Linux is a minimalist secure linux distro.

In security terms less "footprint" often means less vectors of attack and less complexity to analyze for vulnerabilities

Alpine Linux is becoming a preferred base OS for many foundational official Docker Images (python, php, ruby, nginx, redis, haproxy, go) since downloading many large Docker Images (aka Deploying Docker Containers) can saturate the network at scale.

Getting started with Alpine Linux in Docker

This will pull the latest alpine image (around 4MB) and run it in an ephemeral container.

sudo docker run -it --rm alpine /bin/sh

https://hub.docker.com/_/alpine/

Basics

Most of the very basic commands are similar to other linux distros like Debian/Ubuntu/Redhat, but of course there are differences ;)

ls -l /bin/sh

/bin/busybox

cat /etc/passwd
less /etc/passwd
vi /etc/passwd
grep root /etc/passwd
ls -l /usr/bin /usr/sbin | more

more busybox

Package Management

apk update
apk --help

update the local index for all remote packages, list the options of the package manager

apk info

list all of the packages installed locally

apk search curl
apk search curl | sort
apk info curl
apk add curl

search the remote packages for a keyword (unsorted results), get info for a specific package, install a specific package

Networking

hostname
cat /etc/resolv.conf
ifconfig
netstat -anp

traceroute
apk add iptables

Compiling C on Alpine Linux

apk add build-base gcc abuild binutils

This should probably be part of a Dockerfile rather than run every time in an ephemeral container

https://wiki.alpinelinux.org/wiki/How_to_get_regular_stuff_working#Compiling_:_a_few_notes_and_a_reminder

vi hi.c

hi.c

#include <stdio.h>

int main()
{
  printf("hi");
}

compiling

apk add file
gcc -static hi.c
chmod +x a.out
file a.out

The default is a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

./a.out

hi

Note that this was just gcc not musl-gcc :(

Incomplete musl compiler on alpine linux

WARNING: below did not work, TODO: https://bitbucket.org/GregorR/musl-cross

I suspect that I am overwriting the existing gcc toolchain and I need to specify a different prefix (/usr/local)

apk update
apk add wget tar gzip gcc make

apk add musl-dev wget tar gzip gcc make

wget --no-check-certificate http://www.musl-libc.org/releases/musl-1.1.15.tar.gz
tar xf musl-1.1.15.tar.gz
cd musl-1.1.15
./configure
make install

Now we've installed the musl compiler?

/usr/local/musl seemed terribly empty of binaries (i.e. no /usr/local/musl/bin/)

https://www.musl-libc.org/faq.html
http://www.musl-libc.org/how.html

apk add alpine-sdk

But maybe because it's already an alpine linux container the gcc already uses musl instead of libc and does not need the musl-gcc wrapper?

http://www.guidesbyeric.com/statically-link-c-programs-with-musl-gcc

Compile Go on Alpine Linux

Just use the golang image based on alpine linux ;)

https://github.com/docker-library/docs/tree/master/golang

docker pull golang:alpine
docker run -it --rm golang:alpine /bin/sh

Now that we've got the container running we can use the shell

vi intro.go

package main

import "fmt"

func main() {
  fmt.Println("hi")
}

To just run the source code go run intro.go

hi

ls -l
go build intro.go
ls -l
./intro

hi

Maybe

http://dominik.honnef.co/posts/2015/06/go-musl/ ?

Git with Alpine

apk update
apk add --no-cache git
git --version

Debian Packages for Deployment and Reprepro for a local apt repository

2016-02-24T20:40:00-08:00

What is a Debian Package?

A debian package is a way to distribute and install a collection of files (aka software) onto a system (i.e. Debian or Ubuntu).

While a piece of software might depend on other debian packages (e.g. libraries) usually a single .deb file represents some sort of module that serves a single purpose.

Once a debian package is built any client (dpkg or apt which also uses dpkg ;) can use it to install the software.

https://www.debian.org/doc/manuals/debian-faq/ch-pkg_basics.en.html

Why use a Debian Package?

When you're developing on your own box you can pretty much get away with anything

A complex and large scale production environment typically has a lot of costs (both operationally and in not becoming a bottleneck to dev velocity). Any opportunity to increase determinism and reduce risk is welcome.

Deploying source code directly from version control does not always scale well (streaming tons of small files, dedicated read only service user direct to production, etc.) nor does it create enough determinism with regards to dependency management.

As deployments become more frequent and Continuous Integration becomes more complex it is really important to embrace the "build once" principle so that a single artifact (hopefully with all of its dependencies) can pass through the gauntlet of integration testing and canary/incremental rollout.

So now that you're convinced "Artifacts" are the way to go lets just skip .exe, .msi, .jar, .etc and go straight to...

The Debian Package is a "battle tested" format with lots of features (dependency requirements, preinst scripts, postinst scripts, etc.) but if there is a bug in a specific .deb file it is not always practical to get the full source code and rebuild the whole thing (especially considering static bindings and specific compilation environment/parameters).

One example people give is an erroneous pre install or post install script that is preventing either installation or removal.

The example below is more on just simply changing the control file "Description:"

How to unpack a debian package, modify the control file, and repack it

To unpack, modify, and repack a debian package:

docker run --rm --it --volume /tmp:/tmp ubuntu:14.04 /bin/bash
sudo apt-get update
apt-get install --yes vim wget
wget https://example.com/example.deb --output-document /tmp/example.deb
cd /tmp
mkdir emptydir
dpkg-deb -R example.deb /tmp/emptydir
ls -ahl /tmp/emptydir
ls -ahl /tmp/emptydir/DEBIAN
vi /tmp/emptydir/DEBIAN/control
dpkg-deb -b emptydir /tmp/example-fixed.deb

On your host /tmp should now contain example.deb and example-fixed.deb

reprepro for a local apt repository

Just as a debian provides more control over packaging and dependency management, you can also have your apt repository where you store debian packages.

By hosting your own apt repository you can:

Create your own distribution server (e.g. in an s3 bucket)
Create your own intermediate mirror or cache of an upstream repository (e.g. in a local area network shared drive)
Create a local apt repository on local disk for a non internet connected device

The tool reprepro creates and manages the apt database and filesystem.

Setup Ubuntu 14.04 to install reprepro

It might be as simple as a single command to install reprepro but here is the full example in the case where you have broken or corrupted your sources.list:

cat << EOF > /etc/apt/sources.list
    deb http://archive.ubuntu.com/ubuntu/ trusty main
    deb http://archive.ubuntu.com/ubuntu/ trusty universe
    deb http://archive.ubuntu.com/ubuntu/ trusty multiverse
    deb http://security.ubuntu.com/ubuntu trusty-security main restricted
EOF

rm -rf /var/lib/apt/lists
apt-get clean; apt-get update
apt-get install reprepro

http://manpages.ubuntu.com/manpages/trusty/en/man1/reprepro.1.html

Setup the GPG key

A gpg key is an important part of apt for providing a digital signature of authenticity

gpg --list-keys
gpg --list-keys --with-fingerprint
gpg --list-secret-keys --with-fingerprint
gpg --allow-secret-key-import --import YOURKEY.gpg

This allows for importing an existing gpg key into the local keyring (otherwise reprepro actions will not persist)

gpg --yes --batch --delete-secret-keys "21E29B5B3F6D550EF4E2C2C9E9991E312341234"
gpg --yes --batch --delete-keys "21E29B5B3F6D550EF4E2C2C9E9991E312341234"

Removing or deleting a key seems to only work when you delete a key exactly by fingerprint

echo ENCPASSWORD | gpg --yes --no-tty --batch --passphrase-fd 0 --output 8F13E123.key --decrypt 8F13E123.key.gpg

Decrypt a password encrypted gpg key (that was encrypted with gpg - so meta!)

https://wiki.debian.org/SettingUpSignedAptRepositoryWithReprepro

Setup a simple reprepro

Here we will setup a local apt mirror that is a filtered subset of the upstream mariadb repository

mkdir -p /home/admin/apt/conf
mkdir -p /home/admin/apt/logs
mkdir -p /home/admin/apt/archive

cat << EOF > /home/admin/apt/conf/distributions
Origin: digitalocean-mariadb-10-trusty
Codename: digitalocean-mariadb-10-trusty
Description: local mirror of mariadb 10 trusty from the digital ocean sf mirror
Architectures: amd64
Components: main
SignWith: 8F13E123
Update: - digitalocean-mariadb-10-trusty
Log: /home/admin/apt/logs/bintray-mirror.log
EOF

cat << EOF > /home/admin/apt/conf/options
    outdir /home/admin/apt/archive
    ask-passphrase
EOF

cat << EOF > /home/admin/apt/conf/updates
Name: digitalocean-mariadb-10-trusty
Suite: trusty
Method: http://sfo1.mirrors.digitalocean.com/mariadb/repo/10.0/ubuntu/
Components: main
Architectures: amd64
FilterList: deinstall /home/admin/apt/conf/mariadb-partial.list
VerifyRelease: blindtrust
EOF

cat << EOF > /home/admin/apt/conf/mariadb-partial.list
mariadb-partial.list
libmariadbclient-dev install
libmariadbclient18 install
libmariadbd-dev install
libmysqlclient18 install
mariadb-client install
mariadb-client-10.0 install
mariadb-client-core-10.0 install
mariadb-common install
mariadb-connect-engine-10.0 install
mariadb-server install
mariadb-server-10.0 install
mariadb-server-core-10.0 install
mysql-common install
EOF

The configuration created - defines the Distribution - the option of output directory - how the Distribution becomes updated (blindtrust (lol)) - explicitly what to download or blacklist from the upstream (exclude everything else)

https://mirrorer.alioth.debian.org/reprepro.1.html#CONFIG FILES

Updating a remote source

Assuming the installation of reprepro and correct configuration of conf/options, conf/distributions, conf/updates, and conf/NAME-partial.list

Update a local repository from an upstream:

reprepro --verbose --basedir /home/admin/apt check digitalocean-mariadb-10-trusty

reprepro --verbose --basedir /home/admin/apt update digitalocean-mariadb-10-trusty
    aptmethod got 'http://sfo1.mirrors.digitalocean.com/mariadb/repo/10.0/ubuntu/dists/trusty/InRelease'
    aptmethod got 'http://sfo1.mirrors.digitalocean.com/mariadb/repo/10.0/ubuntu/dists/trusty/main/binary-amd64/Packages.gz'
    Calculating packages to get...
    Getting packages...
    aptmethod got 'http://sfo1.mirrors.digitalocean.com/mariadb/repo/10.0/ubuntu/pool/main/m/mariadb-10.0/libmariadbclient-dev_10.0.26+maria-1~trusty_amd64.deb'

ls -ahl /home/admin/apt/archive

should contain two directories: dists and pool

reprepro --verbose --basedir /home/admin/apt check

This will check all Distributions that have been configured for upstream changes

Listing or Adding or Removing a debian package with reprepro

Assuming you have setup your gpg signing key and config files correctly you can also just add a single package ad-hoc to your local apt repository

reprepro dumpreferences
reprepro --verbose --basedir . remove digitalocean-mariadb-10-trusty SOMEPACKAGENAME
reprepro --verbose --basedir . includedeb digitalocean-mariadb-10-trusty SOMEFILENAME.deb
reprepro export digitalocean-mariadb-10-trusty

Configuring a client

cat /etc/apt/sources.list.d/mariadb.list
    deb     file:///home/admin/apt/archive trusty main
apt-get clean
apt-get update
apt-get search mariadb

Now you can install mariadb from your local apt repository, or host it on s3 and change mariadb.list to have the bucket FQDN (aws s3 --delete --exact-timestamps sync ./archive s3://mybucket)

Python packaging pip wheels

2016-01-05T20:00:00-08:00

Installing packages with python

Pip is the standard way to install python packages

sudo pip install packagename

searches https://pypi.python.org and finds the latest version of the package, full docs https://pip.pypa.io/en/latest/

If you get the package name wrong you will have installed something completely different.

sudo pip search packagename

find packages similar to the name you provided (from pypi or whatever provider you are using)

https://pip.pypa.io/en/stable/reference/pip_search/

sudo pip freeze

displays what is installed

Virtualenv for sanity and isolation

A common mistake is to use the global pip installation of the Operating System to store all of the installed dependencies. As soon as you have conflicting version requirements this breaks. As soon as you have multiple applications/services installing globally it becomes unmanageable.

Virtualenv creates a virtual environment (basically injecting a PATH into the environment) for python binaries and package installation.

Pinning Versions and Guaranteed Sourcing

One common mistake is to not pin version numbers and depend on https://pypi.python.org

Without pinning version numbers for your dependencies (listed one per line in requirements.txt) you will receive a nasty surprise when the maintainers make a breaking change and you get a newer version unexpectedly. Since python is a dynamic language you may receive the worst kind of surprise in production (hopefully nothing as bad as data corruption or security issues).

When you do pin the version number BUT still depend on https://pypi.python.org to provide the file then the project maintainers may remove the version you are pinned to (causing your builds to fail - though some for some cowboys this will cause production deployments to fail).

Pin the version of your dependencies, provide the dependencies locally or through a system under your control (your own pypi server or s3 bucket) , explicit is better than implicit.

Wheels are better Python Packaging

Wheels are (awesome), it's the beginning of trying to make python installations more deterministic and pip less dynamic at install time. http://wheel.readthedocs.org/en/latest/

An output directory of the wheels of a project are known by convention as a "wheelhouse".

As a side effect the wheel directory, “/tmp/wheelhouse” in the example, contains installable copies of the exact versions of your application’s dependencies. By installing from those cached wheels you can recreate that environment quickly and with no surprises.

When you install using pip it looks for a “wheel file” (*.whl which is the newer zip compressed format, goodbye .egg) of the correct name for your (virtual) environment (e.g. py2 or py3 or x86 linux). This wheel file saves time and bugs from installing a package/.egg from source (usually that time is spent compiling C code for the python library).

sudo pip install wheel
cd projectsource
python setup.py bdist_wheel
ls -l ./dist
pip wheel  --find-links /root/wheelhouse --wheel-dir=/root/wheelhouse -r requirements.txt

installing using a wheel file

pip install somepackage-version-py2.py3.whl
pip freeze

error: invalid command 'bdist_wheel'

https://pypi.python.org/pypi/docutils#downloads only provided a py3 wheel (facepalm)

Downloading the source .tar.gz and running python setup.py bdist_wheel resulted in: error: invalid command 'bdist_wheel'

Reading the internet provided no comprehensible answers (lots of "setuptools does not match your version of pip or wheel or whatever")

The following hacking seems to have provided a solution:

:::bash python --version pip --version cd /tmp wget https://pypi.python.org/packages/source/d/docutils/docutils-0.12.tar.gz#md5=4622263b62c5c771c03502afa3157768 tar xf docutils-0.12.tar.gz cd docutils-0.12 virtualenv venv source venv/bin/activate python --version pip --version pip install wheel pip freeze python setup.py install pip freeze pip wheel . ls /tmp/docutils-0.12/wheelhouse docutils-0.12-py2-none-any.whl

Static site with Bitbucket and Shippable and Pelican

2015-12-21T12:24:00-08:00

Running Software Costs Money

One of the most overlooked costs in running a service is operations. While Research and Development (aka coding) is often cited as the largest expense (software developer salaries! https://www.quora.com/What-are-the-average-operating-costs-of-SaaS-companies), and 80% (or more) of (successful) software's life is maintenance (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3610582/), you have to run the darn thing all the time.

A Free and Efficient Static Web Site

What is one cost effective (free!) and efficient solution to running a static site?

Bitbucket also has a free static site capability (as long as the DNS is USERNAME.bitbucket.org), so no server/hosting required
Bitbucket have free private repositories
The Bitbucket static site repository can be private (only the html exposed will be visible to anonymous users)
Shippable have a free plan with 1 container that will do your builds (fine by me, Docker is fast!)
Pelican converts markdown into .html and you can still use javascript for fancy things http://docs.getpelican.com/

Sadly it looks like the Shippable startup is having trouble maintaining/updating their Docker images so I recommend instead to use Bitbucket Pipelines https://blog.john-pfeiffer.com/continuous-delivery-with-bitbucket-pipelines-and-google-app-engine-deployment-and-the-storageobjectslist-error/#bitbucket-pipelines-configuration

The basic process is to be triggered by a git push to the private repository of new/updated source markdown, use pelican to process it into .html, and then publish (git push) the new/updated .html to the static site repository.

One reason to use two seperate repositories instead of only one repository is that if you make a commit to your markdown source repository that will trigger a CI run which will push the updated .html files to the repository which would be detected and maybe trigger an infinite loop. Or at the least interleave your source code changes with generated output changes in the commit logs. =]

An alternative is using multiple branches but you'd better hope nobody ever deletes your source branch by accident.

Another alternative is to include an IF statement in your shippable code to not push if the diff/md5 of the source files (or maybe check against the output .html?) still match.

I say keep it super simple ;)

Bitbucket setup

Create The Source and Target Repositories

Create a new private repository (for your markdown), consider prefixing the name with source or something (good names makes for good maintenance)

Make sure you have cloned the pelican project and setup a basic static site: https://blog.john-pfeiffer.com/how-to-set-up-a-pelican-static-blog-site/

Inside your .gitignore you will probably want to exclude .pyc and ./output and any other pelican created artifacts.

Inside your repository at the root level you will need a shippable.yaml file:

language: python
python:
    - "2.7"
install:
    - pip install pelican Markdown beautifulsoup4
script:
    - rm -rf ./output
    - rm -rf ./cache
    - rm -rf ./plugins/*
    - mv ./pelican-project/* .
    - ls -ahl ./content
    - pelican ./content -o ./output -s ./publishconf.py
after_script:
    - ls -ahl
    - ls -ahl ./output

This assumes that the pelican-project is a subdirectory in the repository using the best practice of leaving the top level of a repository for build and test artifacts and isolating the source code into a subdirectory.

Create another private repository for your public html. It MUST be named USERNAME.bitbucket.org to make use of the bitbucket static site capabilities.

(yes, the name must include those dots/domain name of the service)

https://confluence.atlassian.com/bitbucket/publishing-a-website-on-bitbucket-cloud-221449776.html

Get OAuth Access

The goal here is to leverage the one-hour-access-token-generation-via-Oauth in order to write to a different repository in Bitbucket.

Generate a Consumer OAuth2 Token with https://confluence.atlassian.com/bitbucket/oauth-on-bitbucket-cloud-238027431.html#OAuthonBitbucketCloud-Createaconsumer

(The UI in https://bitbucket.org/account/user/USERNAME/api is fairly straightforward with "Add consumer")

The least privilege required permissions would be "Repositories: Read, Write" You must define a callback URL, i.e. http://YOURDOMAIN:8888 , even if it is not used by this workaround

Use curl to verify your token (this is how Shippable will get a 1 hour expiring access token to work on the target output repository)

curl https://bitbucket.org/site/oauth2/access_token -d grant_type=client_credentials -u yourkeyhere:yoursecrethere
    {"access_token": "abcd1234...", "scopes": "repository:write", "expires_in": 3600, ...}

The curl command should get a JSON response which includes an access_token

http://stackoverflow.com/questions/24965307/how-to-manipulate-bitbucket-repository-with-token

Shippable setup

Sadly it looks like the Shippable startup is having trouble maintaining/updating their Docker images so I recommend instead to use Bitbucket Pipelines https://blog.john-pfeiffer.com/continuous-delivery-with-bitbucket-pipelines-and-google-app-engine-deployment-and-the-storageobjectslist-error/#bitbucket-pipelines-configuration

Enable the integration with Bitbucket: http://docs.shippable.com/#step-0-prerequisite

Right from the beginning Shippable tries to ask which source repository provider (either GitHub or Bitbucket) you will be using.

Use Shippable's OAuth implementation (Account Integration) to pick which Bitbucket repository

Home -> CI (dropdown) -> Bitbucket (hopefully you have a different avatar between Bitbucket and GitHub)

Press "Sync" if you have a newly created repository that is not listed yet

Add the Bitbucket OAuth2 Key and Secret as a Shippable secure environment variable in the format KEY:SECRET

Go to https://app.shippable.com/projects/1234d2ea1895ca4474661234/settings and look for the Encrypt section Fill it in with something like OAUTH_USER=yourkeyhere:yoursecrethere

http://shippable-docs-20.readthedocs.org/en/latest/config.html#secure-environment-variables

Copy the output to your shippable.yml file

Putting it all together

Update the source repository top level shippable.yaml file:

language: python
python:
    - "2.7"
install:
    - pip install pelican Markdown beautifulsoup4
env:
    - secure: yourencryptedkeyandsecret==
script:
    - rm -f token.json
    - rm -rf ./output
    - rm -rf ./cache
    - rm -rf ./plugins/*
    - mv ./pelican-project/* .
    - ls -ahl
    - ls -ahl ./content
    - pelican ./content -o ./output -s ./publishconf.py
    - ls -ahl ./output
    - curl https://bitbucket.org/site/oauth2/access_token -d grant_type=client_credentials -u $OAUTH_USER >> token.json
    - BBTOKEN=$(cat token.json | python -c 'import sys, json; print json.load(sys.stdin)["access_token"]')
    - git clone "https://x-token-auth:$BBTOKEN==@bitbucket.org/USERNAME/USERNAME.bitbucket.org"
    - ls -ahl USERNAME.bitbucket.org/
    - rm -rf USERNAME.bitbucket.org/*
    - cd USERNAME.bitbucket.org
    - mv ../output/* .
    - ls -ahl .
    - git config user.email "me@example.com"
    - git config user.name "John Pfeiffer"
    - git add -f .
    - git commit -m "build $BUILD_NUMBER commit $COMMIT"
    - git push -fq origin master > /dev/null
    - rm -f ../token.json
after_script:
    - ls -ahl

Adding the bitbucket oauth2 consumer key and secret (separated by a colon) as an encrypted environment variable

using curl to generate a temporary access_token and extracting it into a local environment variable

cloning with the access)token and removing the previous contents and replacing them with the newly generated output

leveraging the CI variables to indicate on the output html repository what markdown source commits triggered this build

Reviewing the output

The full output of the run is available at something like https://app.shippable.com/builds/1234dec1d00e020c0011234

This is really helpful for debugging (especially seeing how many seconds each step took)

Possible improvements:

use a python application best practice of documenting dependencies with a requirements.txt file at the top level
putting all of the commands into a script like publish-in-ci.sh so that it could be run locally in a dev environment
add the Dockerfile used for local development into the source repository to consolidate and simplify development in one place

Misc

One thing that is interesting about this is that using OAuth tokens through a service is merely wrapping all of the manual steps I have in a previous blog post into a nice SaaS wrapper =) https://blog.john-pfeiffer.com/publish-a-pelican-blog-using-a-bitbucket-post-webhook/

Go Programming Intro with VS Code and Arrays Slices Functions and Testing

2015-11-15T21:19:00-08:00

Introducing the Go Programming Language (aka golang) basics, interactive sandbox with https://play.golang.org

Prerequisites

Tooling is often essential to productivity.

Download and install the go language compiler and tools

vim and Docker

Or alternatively just avoid the IDE and Environment and use vim and a docker container ;) https://hub.docker.com/_/golang/

Installing and the Environment

https://golang.org/dl/ and cd /opt; tar xf go.tar.gz

Instead of the ephemeral export PATH=$PATH:/usr/local/go/bin I prefer the persistent ~/.profile (or for all users /etc/profile though clearly /opt indicates a single user system ;)

# Go Programming
export GOROOT=/opt/go
export PATH=$PATH:$GOROOT/bin
export GOPATH=/opt/goprojects
export PATH=$PATH:$GOPATH/bin

source ~/.profile

The last step gets you going without needing to reload your shell, full docs at https://golang.org/doc/install

WARNING be careful how you name your executables as the $GOPATH/bin will contain the names of the projects as binaries (so don't create a project or binary named bash!)

Go Docs locally

If you need the go standard library docs (and access to any of the code docs from repos in the path) you can run: godoc -http=:6060

A local web server with documentation for the go standard library https://godoc.org/golang.org/x/tools/cmd/godoc

Manual Go CLI execution and compilation

The traditional command line method is:

cd $GOPATH/path-to-your-project/PROJECTNAME
go run PROJECTNAME

This will run go in a "dev mode" where it pulls in dependencies and executes immediately (no artifacts are created)

To compile and install the binary into the local path:

cd $GOPATH/path-to-your-project/PROJECTNAME
go install
$GOPATH/bin/PROJECTNAME

This compiles and builds and installs the binary into the $GOPATH

To build a binary in the current directory:

cd $GOPATH/path-to-your-project/PROJECTNAME
go build

This will just build the binary locally (i.e. name.go becomes "name") in the current directory, not in the $GOPATH

Be aware that sometimes you may forget and commit this new binary to version control (badpokerface)

A concrete example:

Compile with: cd /opt/goprojects/src/github.com/johnpfeiffer/intro/ ; go install
Execute with: /opt/goprojects/bin/intro

https://golang.org/doc/code.html is a very complete tutorial, notes for myself:

cat ~/.profile
export GOPATH=$HOME/Desktop/repos/goprojects
export PATH=$PATH:$GOPATH/bin

source ~/.profile

an example of setting up the environment in Mac OSX

If your files are laid out like this...

/Users/johnpfeiffer/Desktop/repos/goprojects
  - bin
  - pkg
  - src

mkdir -p $GOPATH/src/bitbucket.org/johnpfeiffer/myproject/mystrings

a wrinkle on the official go tutorial where both packages are in the same remote git repository

mystrings/mystrings.go

package mystrings

func StringLength(s string) int {
    return len(s)
}

A simple "library" package that can be re-used

cd /tmp; go install bitbucket.org/johnpfeiffer/myproject/mystrings ALTERNATIVELY: cd $GOPATH/src/bitbucket.org/johnpfeiffer/myproject/mystrings; go install

Installing the package builds the .a binary which can be used by other builds/programs

mkdir -p $GOPATH/src/bitbucket.org/johnpfeiffer/myproject/hello hello/hello.go

package main

import (
    "fmt"
    "bitbucket.org/johnpfeiffer/myproject/mystrings"
 )

func main() {
    fmt.Println("hello", mystrings.StringLength("hello"))
}

a simple main package that can be called from the CommandLineInterface

cd $GOPATH/src/bitbucket.org/johnpfeiffer/myproject/hello; go install

Execute your new program with: $GOPATH/bin/hello

(or since $GOPATH/bin is in $PATH, just type: hello)

An ASCII diagram of the file system

$GOPATH/src/bitbucket.org/johnpfeiffer/myproject
|
 - hello
     | - hello.go
 - mystrings
     | - mystrings.go

Cross Compiling with Go

If developing on Mac OSX and wanted to compile/build a 64bit linux binary

cd $GOPATH/src/bitbucket.org/johnpfeiffer/myproject/hello
GOOS=linux GOARCH=amd64 go build -v
    runtime/internal/sys
    runtime/internal/atomic
    runtime
    bitbucket.org/johnpfeiffer/myproject/mystrings
    bitbucket.org/johnpfeiffer/myrpoject/hello

the "go build -v" extra flag outputs verbosely the intermediate steps

file hello
     ./hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

using the "file" command we can inspect it really has been built correctly in the current directory

Dave Cheney's suggestion about "build vs install" is a good one since the cached intermediate .a files in the directory pkg/ may complicate things, especially for a cross compilation.

Download and install an IDE

If you are used to larger projects then an IDE is quite helpful for colorization, auto completion, (right click or f12) goto definition, rename, build on save, auto formatting, etc.

Suprisingly one of the most popular and effective Go IDE combinations is: https://code.visualstudio.com/Docs/?dv=linux64_deb

dpkg -i code_...amd64.deb

To install https://marketplace.visualstudio.com/items?itemName=lukehoban.Go aka https://github.com/Microsoft/vscode-go you actually...

The plugin has been renamed to "ms-vscode.go" =|

open Visual Studio Code
Control + P (Launches VS Code Quick Open)
ext install ms-vscode.Go
restart when prompted
File -> Preferences -> Color Theme (Light Visual Studio ;)

At this point you probably need to reboot to get VSCode to recognize the GOPATH correctly

In your shell where you setup the GOPATH run the following to get all of the analysis tools that "Go for Visual Studio Code" uses, though it's a lot easier to just use the IDE in the bottom right corner "Analysis Tools Missing"

To take advantage of those tools (like gofmt on save), in your workspace (GOPATH) there will be a .vscode directory with settings.json https://code.visualstudio.com/docs/customization/userandworkspace

// Place your settings in this file to overwrite default and user settings.
{
    "go.buildOnSave": true,
    "go.lintOnSave": true,
    "go.vetOnSave": true,
    "go.buildFlags": [],
    "go.lintFlags": [],
    "go.vetFlags": [],
    "go.coverOnSave": false,
    "go.useCodeSnippetsOnFunctionSuggest": false,
    "go.formatOnSave": true,
    "go.formatTool": "goreturns"
}

Just in case restart VSCode to recognize the updated settings

Unfortunately it is not quite simple to execute the code directly in VSCode https://github.com/Microsoft/vscode-go/issues/21

VS Code preferences to disable telemetry and automatic updates

To prevent the software from sending (some) data to Microsoft...

File -> Preferences -> Settings

"telemetry.enableTelemetry": false,
"telemetry.enableCrashReporter": false,
"update.channel": "none"

disabling sending stats and crash reports to vortex.data.microsoft.com and also prevent checking for software updates

https://code.visualstudio.com/Docs/supporting/FAQ#_how-to-disable-telemetry-reporting

Custom VS Code icons and disabling annoying file icons

There is a feature to show file icons at the top of every file (i.e. distinguish visually/graphically between Go source and HTML)

https://code.visualstudio.com/docs/getstarted/themes#_icon-themes

To graphically disable/modify the file icons: File -> Preferences -> File Icon Theme and choose "None"

To disable the theme using the settings file at the User level (or workspace level if you want to have to modify this for every project):

File -> Preferences -> Settings

USER SETTINGS (settings.json)

"workbench.iconTheme": null,

(previously it was vs-seti , apparently Seti is the poor rendition of gophers as ugly brown blobs)

"workbench.colorTheme": "Visual Studio Light",

because simple text with a light background is actually easier on the eyes

https://code.visualstudio.com/docs/getstarted/settings

Disabling the minimap enhanced scroll bar

It is pretty nifty to see the scroll bar have colors/graphics indicating roughly where in the file you are.

If you want to maximize your available editing width then you just want a "normal" scroll bar (or removed entirely because "clean code" your files are small ;)

File -> Preferences -> Setting

In USER SETTINGS add the line to the settings.json that the IDE opened for you (in proper JSON syntax):

"editor.minimap.enabled": false

Pro tip to avoid VS Code melting your system down

If you find your machine suddenly slows down terribly with VS Code open and that CPU, then RAM, then finally swap (kswapd0) then you have probably run into an annoying trap:

/usr/share/code/code /usr/share/code/resources/app/out/bootstrap --type=watcherService

That high cpu utilization is either the project file watcher indexing or one of the language plugins (because "open source") attempting to lint every file in the project.

That's right, if you have a large git repository, a temp directory with some test data, or anything else it can get it reach Visual Studio Code will scan it at the expense of your machine.

(and it may even be scanning all sorts of random locations throughout your file system (stare))

https://github.com/Microsoft/vscode/issues/3998

The following may have helped (but more likely is that I moved the large files to another directory outside of the Project):

settings.json

"files.watcherExclude": {
    "**/.git/objects/**": true,
    "**/.git/subtree-cache/**": true,
    "**/node_modules/**": true,
    "**/*.aes": true,
    "**/TEMP": true
}

Install or Build or Run

Because Go is a static language there is a compilation (and linking) phase where the source code is transformed into a binary.

Using Visual Studio Code Tasks to Build Go

With your main.go file open press Control + Shift + B
"No task runner configured" -> click on "Configure Task Runner"
From the dropdown choose "Others"
fill in tasks.json

{ "version": "0.1.0", "command": "pwd", "isShellCommand": true, "showOutput": "always" }

The command will execute in the folder that is opened in "File -> Open Folder"

{ "version": "0.1.0", "command": "go", "isShellCommand": true, "args": ["build", "-v"], "showOutput": "always" }

This builds verbosely in the current directory, assuming that the IDE has opened the project folder since Go only uses relative path https://golang.org/ref/spec#ImportPath (and the GOPATH is set correctly)

If you prefer to File -> Open Folder a top level folder that has many subfolders with go projects then...

The workaround for not having "change working directory" or "command chaining" is to create a shell script, build.sh

#!/bin/sh
cd $1
pwd
ls -l
go build -v
ls -l

The script could be reduced to "cd $1; go build -v" but having the extra debugging output can help...

The task runner must be defined to take advantage of the new script:

{
    "version": "0.1.0",
    "taskName": "build",
    "command": "bash",
    "args": ["${cwd}/build.sh", "${fileDirname}"],
    "isShellCommand": true
}

The IDE stores the files in PROJECTFOLDER/.vscode/tasks.json

https://code.visualstudio.com/Docs/editor/tasks#_variable-substitution

Using Visual Studio Code Tasks to Run Go

You can continue to create more key bindings (hotkeys) for running Go or running Tests, I will just override the Build (Control + Shift + B) hotkey for now...

#!/bin/sh
cd $1; go run *.go

This does not build and instead just executes directly

{
    "version": "0.1.0",
    "taskName": "build",
    "command": "bash",
    "args": ["${cwd}/run.sh", "${fileDirname}"],
    "isShellCommand": true
}

this extra indirection of a separate run.sh is only necessary if you prefer having a top level "meta" folder

https://code.visualstudio.com/Docs/customization/keybindings#_tasks

Go Test with a Visual Studio Code Test Task

The "test" task is unassigned (at least for my linux installation) so the first step is to inspect and customize the hotkey keybindings.

Control+K Control+S

Or use File -> Preferences -> Keyboard Shortcuts

In the left pane search the "Default Keybindings" file for "build", i.e. on line 502 you should see:

{ "key": "ctrl+shift+b",          "command": "workbench.action.tasks.build" },

Add to the empty "keybindings.json" on the right a new hotkey

[
    { "key": "ctrl+shift+t",          "command": "workbench.action.tasks.test" }
]

Save the file (the IDE will store your customization in ~/.config/Code/User/keybindings.json)

Now when you use Control+Shift+T you should see "No task configured" and "Configure Task Runner"

This example defines multiple tasks, both go build and go test (this will save to .vscode/tasks.json)

{
    "version": "0.1.0",
    "tasks": [
        {
            "taskName": "build",
            "command": "go",
            "args": ["build", "-v"],
            "isShellCommand": true,
            "showOutput": "always"
        },
        {
            "taskName": "test",
            "command": "go",
            "args": ["test", "-v"],
            "isShellCommand": true,
            "showOutput": "always"
        }
    ]
}

Assuming you have opened File -> Open Folder at the top level of your code tree as Go expects relative paths,

Control+Shift+T will now open the Tasks output pane and display the results of any tests run

Debugging with Delve

go get github.com/derekparker/delve/cmd/dlv
cd /opt/goprojects/src
go install /derekparker/delve/cmd/dlv

The first time I attempted to do it manually:

ls -ahl /opt/goprojects/bin/
cd /opt/goprojects/src/github.com/johnpfeiffer/YOURPROJECT
/opt/goprojects/bin/dlv  debug --headless --listen=:2345 --log

Now in the VSCode IDE open the project folder and create your helloworld.go source file and Control + S to save (and auto gofmt) and then press F5 and it will connect to the Delve Debugger and display the output

Delve Debugging and running your application with F5 is automatic once installed correctly

The first time you run "Continue" with F5 on a file it will prompt you to setup your launch.json (and the IDE will open the default template for you)
Use the IDE to go back to your source .go file and press F5 again, this time since the .vscode subdirectory was created and the default delve launch.json file was created, it will just start in debug mode with the Debug Console output at the bottom

Coding and Compiling

For VSCode IDE keyboard shortcuts: https://code.visualstudio.com/docs/customization/keybindings

Comments Types Strings Slices For Loops

The main package it where the execution begins (aka "main" in c https://en.wikipedia.org/wiki/Entry_point)

Comments are either single line with double slashes or block comments https://golang.org/doc/effective_go.html#commentary

intro.go

package main

import "fmt"

/* https://golang.org/doc/effective_go.html#mixed-caps
   https://golang.org/ref/spec#Constants */
const alphabetMax int = 26

func main() {

    // while
    j := 0
    for j < 4 {
        fmt.Println(j)
        j += 2
    }
    for i := 0; i < 4; i += 2 {
        fmt.Println(i)
    }
    for {
        fmt.Println("break or return exits an infinite loop")
        break
    }

    // arrays are contiguous memory, fixed size and type
    // initialized to capacity 5 with values inserted, alternatively just initialized to empty with: var a [5]string
    a := [5]string{"a", "b", "c", "d", "e"}

    // prefer Slices which are Reference Objects that wrap the underlying arrays
    // https://blog.golang.org/go-slices-usage-and-internals
    s := []string

    for i := 0; i < len(a); i++ {
        fmt.Println(i, a[i])
    }

    // cleaner way of iterating over key and value
    for k, v := range a {
        fmt.Println(k, v)
    }
    fmt.Println(s)    // []
    s = a[:]
    fmt.Println(s)    // [a b c d e]
    s = a[2:]
    fmt.Println(s)    // [c d e]
}

arrays are contiguous memory and 4 bytes is normal

// int is usually the 4 byte int32 https://golang.org/ref/spec#Numeric_types
b := [2]int{1, 2}
// dereference the addresses that are holding the values 1 and 2
fmt.Printf("%d %d\n", &b[0], &b[1])

// rune is also int32
c := [2]rune{'a', 'ä'}
fmt.Printf("%d %d\n", &c[0], &c[1])

fizzbuzz and switch

https://golang.org/ref/spec#Switch_statements

for i := 1; i < 16; i++ {
    // usually static case values and "switch i {" , note it will NOT fall through by default
    switch {
    case i%3 == 0 && i%5 == 0:
        fmt.Println("fizzbuzz")
    case i%3 == 0:
        fmt.Println("fizz")
    case i%5 == 0:
        fmt.Println("buzz")
    default:
        fmt.Println(i)
    }
}

time

import "time"
// https://golang.org/pkg/time/#Now
now := time.Now()
fmt.Println("local:", now)
fmt.Println(now.UnixNano()/1000000, "ms")
fmt.Println("in UTC:", now.UTC().Format(time.UnixDate))

Packages and String Reverse

When you modularize your code into packages then multiple programs can make use of DRY https://en.wikipedia.org/wiki/Don%27t_repeat_yourself

main

package main

import (
    "fmt"
    "github.com/johnpfeiffer/mystringutil"
)

func main() {
    fmt.Printf(stringutil.Reverse("!oG ,olleH"))
}

package mystringutil with Reverse

// Package mystringutil contains utility functions for working with strings. "go build"
package mystringutil

// Reverse returns its argument string reversed rune-wise left to right.
func Reverse(s string) string {
        r := []rune(s)
        for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 {
                r[i], r[j] = r[j], r[i]
        }
        return string(r)
}

palindrome and string conversion integer to ascii

Besides the main function for executing you will obviously create re-usable packages which will contain functions.

Here is the source code for a simple "is this string a palindrome" and "is this integer a palindrome" programs:

package main

import (
    "fmt"
    "strconv"
)

func main() {
    fmt.Println(isPalindrome("a"))
    fmt.Println(isPalindrome("ala"))
    fmt.Println(isPalindrome("noon"))
    fmt.Println(isPalindrome("ab"))
    fmt.Println(isPalindrome("racecar"))
    fmt.Println(isPalindrome("abfooba"))

    // slower and uses extra memory
    fmt.Println(isPalindrome(strconv.Itoa(1991)))
    fmt.Println(isPalindrome(strconv.Itoa(1981)))

    // takes advantage of math (mindblown)
    number := 9
    reversed := reverseInteger(number)
    fmt.Println(number, reversed)
    fmt.Println(number, "is a palindrome: ", number == reversed)
    number = 123
    reversed = reverseInteger(number)
    fmt.Println(number, reversed)
    fmt.Println(number, "is a palindrome: ", number == reversed)
    number = 121
    reversed = reverseInteger(number)
    fmt.Println(number, reversed)
    fmt.Println(number, "is a palindrome: ", number == reversed)
}

func isPalindrome(word string) bool {
    for index, value := range word {
        oppositeIndex := len(word) - index - 1
        opposite := rune(word[oppositeIndex])
        fmt.Printf("%d %T %c compared to %d %c \n", index, value, value, oppositeIndex, opposite)
        if index >= oppositeIndex {
            break
        }
        if value != opposite {
            return false
        }

    }
    return true
}

func reverseInteger(x int) int {
    reversed := 0
    for ; x > 0; x /= 10 {
        remainder := x % 10
        reversed = (reversed * 10) + remainder
        fmt.Println(remainder, x, reversed)
    }
    return reversed
}

Random Integers

// https://golang.org/pkg/crypto/rand/
// https://golang.org/pkg/math/big/
var max big.Int
var myRandom *big.Int
var err error
max.SetUint64(10)
myRandom, err = rand.Int(rand.Reader, &max)

fmt.Println(*myRandom)
fmt.Println(err)

Binary Search

Using main and print is the poor man's Unit Testing ;)

package main

import (
    "bytes"
    "fmt"
)

//todo pass by ref?
func binarySearch(a []int, target, low, mid, high int) int {
    fmt.Println("low =", low, "mid =", mid, "high =", high)
    if a[mid] == target {
        return mid
    }
    if mid >= high || mid <= low {
        return -1
    }

    if a[mid] < target {
        low = mid + 1
    }
    if a[mid] > target {
        high = mid
    }
    mid = ((high - low) / 2) + low
    return binarySearch(a, target, low, mid, high)
}

func main() {
    a := []int{1, 4, 6, 8}
    fmt.Println(a)
    targets := []int{1, 4, 6, 8, 0, 3, 5, 7, 9}
    for _, t := range targets {
        result := binarySearch(a, t, 0, len(a)/2, len(a))
        fmt.Println("found", t, "at location", result)
    }

}

Efficient String append and replacement

https://stackoverflow.com/questions/1760757/how-to-efficiently-concatenate-strings-in-go

func myReplace(source string) string {
    var b bytes.Buffer
    for _, c := range source {
        if c == ' ' {
            b.WriteString("%20")
        } else {
            b.WriteString(string(c))
        }
    }
    return b.String()
}

Testing with Go

A simple web server

package main

import (
    "fmt"
    "net/http"
)

func myHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "hi")
}

func main() {
    http.HandleFunc("/", myHandler)
    http.ListenAndServe(":8080", nil)
}

Verify with curl localhost:8080

Deploying a Go Web Application to Google AppEngine

First create an app.yaml file in your package

application: MyApplicationName
version: 1
runtime: go
api_version: go1

handlers:
- url: /.*
  script: _go_app

Second adapt your source code to the Google App Engine entrypoint:

package myapplicationname
// package main

import (
        "fmt"
        "net/http"
)

func myHandler(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "hi")
}

// The App Engine PaaS provides its own main() that handles the Listening and Serving ;)
//func main() {
func init() {
        http.HandleFunc("/", myHandler)
//      http.ListenAndServe(":8080", nil)
}

Assuming you have created the project with https://console.cloud.google.com/ and received a unique application id...

A prerequisite is to use the SDK if you want to test it locally: https://cloud.google.com/appengine/downloads#Google_App_Engine_SDK_for_Go

unzip go_appengine_sdk...zip
/opt/go_appengine/goapp serve /path-to-project/MyProjectFolder/

  INFO     2016-06-02 06:30:27,493 devappserver2.py:769] Skipping SDK update check.
  INFO     2016-06-02 06:30:27,527 api_server.py:205] Starting API server at: http://localhost:38837
  INFO     2016-06-02 06:30:27,530 dispatcher.py:197] Starting module "default" running at: http://localhost:8080
  INFO     2016-06-02 06:30:27,531 admin_server.py:116] Starting admin server at: http://localhost:8000

To deploy to Google App Engine

/opt/go_appengine/appcfg.py -A MyApplicationID update ./MyProjectFolder/

View the version deployed and stats with https://console.cloud.google.com/appengine/versions?project=MyApplicationId

curl http://MyApplicationId.appspot.com/

A gotcha is future updates deployed need the app.yaml version to increment AND either use the Web UI to set the new "default" or...

/opt/go_appengine/appcfg.py -A MyApplicationId set_default_version /MyProjectFolder

HandlerFunc and Anonymous Functions and Closure

The decorator pattern Anonymous functions an

package main

import (
    "fmt"
    "net/http"
)

/* using an anonymous function and closure to wrap the HandlerFunc
https://golang.org/pkg/net/http/#HandlerFunc
https://medium.com/@matryer/the-http-handlerfunc-wrapper-technique-in-golang-c60bf76e6124
*/
func makeHandler(name string) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // https://golang.org/src/net/http/request.go
        fmt.Println("serving: ", r.URL.Path)
        fmt.Fprintf(w, "<h1>%s</h1>", name)
    }
}

func main() {
    fmt.Println("starting...")
    indexHandler := makeHandler("Index")
    myHandler := makeHandler("John")
    // https://golang.org/pkg/net/http/#HandleFunc , string, func(ResponseWriter, *Request)
    http.HandleFunc("/", indexHandler)
    http.HandleFunc("/john", myHandler)
    http.ListenAndServe(":8080", nil)
}

More Info

Build Automation using packer to build an AMI use immutable not chef

2015-07-06T20:40:00-07:00

Why build automation?

Software has always been about automation and leveraging the computer's capacity for precision and repetition.

Somehow though, software is sometimes still deployed using a series of often poorly documented steps (to physical hardware even!). I've been there, it ain't pretty. (badpokerface)

The second time you need to build a server running service(s) you may be under time pressure. (Murphy's law says you might be building it again because the first one which was business critical blew up unexpectedly.)

Building things by hand is possibly the most expensive way to generate impossible to reproduce bugs and job security for the personality challenged.

(Almost everyone agrees that technology employees are expensive and so by extension their time is constantly being wasted by everything they do).

As virtualization (and linux!) took over the world there was an explosion of virtual machines that needed to be deployed and an evolution of a fairly standard virtual harwdare layer. (x86 cpu and Intel NIC anyone?)

Suddenly you couldn't hire enough antisocial people to run around with floppies and scratching cds while shoving them into servers.

Why not chef?

Chef, Puppet, and Ansible are the well known configuration management and build/deployment automation tools.

Automated configuration management which tries to keep a remote server in a specific state seems like a good recipe for things going wrong

I've used chef successfully quite a few times and the main things that make it a specialized tool that I prefer not to use:

It's really easy to do chef wrong: nested roles and recipes that keep exploding exponentially with circular dependencies which make you think software development starts looking easy again.
Community cookbooks are written to allow deployment on every architecture ever created (Debian, Ubuntu, RedHat, Windows, SPARC, etc.) which makes them challenging to read and debug, almost impossible to customize to do what you actually want.
The ruby based DSL isn't bad but it's pretty annoying to constantly make syntax errors (which unless you're all TDD rambo and use Kitchen you'll find during the never ending waiting many minutes for a deployment to fail)
It's difficult to debug the non intuitive "compilation phase" and "execution phase" way chef does its dependency tree magic, and the "shoot yourself in the foot" is compounded with the apparently edge case necessary compile time run executions
The "best practices" have changed 3 or 4 times (write your own custom cookbooks, leverage the community cookbooks, write a custom wrapper for the community cookbook, don't ever use set_unless even though it still exists, etc.) and the 6 layers of variable overrides makes it hard to keep track of what the actual output of a script will be (don't worry, they have pages of documentation explaining it)
The recommended "chef client server architecture" does not scale to really large numbers well and creates administration overhead and a lot of authorization complexity - and my preferred method with "chef solo" still requires an annoying amount of bootstrap setup on the target machines.
Polling not only creates network congestion but worse creates windows of uncertainty about deployment state and the possibility of nodes silently dropping out https://docs.chef.io/chef_client.html
Chef tends to encourage the pattern of long lived mutable servers (with their therefore necessary expensive and obnoxious biological caretakers)
https://docs.chef.io/resource_common.html#lazy-evaluation
https://docs.chef.io/resource_common.html#run-in-compile-phase
http://erik.hollensbe.org/2013/03/16/the-chef-resource-run-queue/

So is there a simpler way to just reliably, reproducibly, build a box?

Packer to the rescue

Packer is from the same people who brought you Vagrant https://www.vagrantup.com/ , that really easy way to set up a virtual machine... https://blog.john-pfeiffer.com/using-vagrant-to-deploy-instances-on-aws

It is very straightforward to read and actually you can still leverage chef (unless you realize that a series of shell commands is all you wanted anyways...)

This leads to the better path of "immutable servers" http://martinfowler.com/bliki/ImmutableServer.html

packer --version

my_example_box.json

{
  "variables": {
    "aws_access_key": "",
    "aws_secret_key": ""
  },
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
    "region": "us-east-1",
    "source_ami": "ami-de0d9eb7",
    "instance_type": "t1.micro",
    "ssh_username": "ubuntu",
    "ami_name": "packer-example {{timestamp}}"
  }]
}

packer validate mybox.json

packer build mybox.json

packer build -debug mybox.json

This will prompt for the enter key to continue at each step

Once it's done it will terminate the EC2 instance for you (it only runs as long as it takes to build the machine and then burn the Amazon Machine Image).

us-east-1: ami-19601234

Unfortunately it is not machine readable json output so you have to do some bash-fu to extract just the id

Also unfortunately there is no way to tell packer to not terminate so you can troubleshoot, the workarounds are the -debug which is essentially "interactive" or adding sleep commands

my_advanced_box.json

{
  "_comment": "This is a comment",
  "variables": {
      "my_secret": "{{env `MY_SECRET`}}",
  },
  "builders": [{
    "type": "amazon-ebs",
    "region": "us-east-1",
    "source_ami": "ami-de0d9eb7",
    "instance_type": "t1.micro",
    "ssh_username": "ubuntu",
    "ami_name": "packer-example {{timestamp}}"
    "subnet_id": "subnet-f0be1234",
    "security_group_id": "sg-9bf51234",
    "associate_public_ip_address": true,
    "ssh_keypair_name": "my-packer",
    "ssh_private_key_file": "./my-packer.pem"
  }],
  "provisioners": [{
    "type": "file",
    "source": "./debs/",
    "destination": "/tmp"
  },
  {
    "type": "shell",
    "inline": [
      "/sbin/ip a",
      "curl -s http://checkip.amazonaws.com",
      "ls -ahl /tmp",
      "echo {{user `my_secret`}} > /tmp/{{isotime \"2006-01-02-030405\"}}--my-secret.txt",
      "sudo dpkg -i --force-confnew /tmp/*.deb",
      "machine_state_validation.sh"
    ]
  }]
}

The access credentials could instead be environment variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

Post instantiation validation is a really handy safeguard as statistically something always goes wrong somewhere and it's far cheaper to find out with a quick test versus a system that loses data.

Packer and DigitalOcean

DigitalOcean is a relatively new player (compared to Linode and even AWS) but they provide a very fast and easy to use way of building boxes (a snapshot can be used like an AMI to spin up multiple instances).

{
  "_comment": "https://www.packer.io/docs/builders/digitalocean.html",
  "variables": {
    "digitalocean_api_token": "{{env `DIGITALOCEAN_API_TOKEN`}}",
    "newuser_name": "{{env `NEWUSER_NAME`}}",
    "newuser_password": "{{env `NEWUSER_PASSWORD`}}"
  },

  "builders": [{
    "type": "digitalocean",
    "api_token": "{{user `digitalocean_api_token`}}",
    "size": "512mb",
    "region": "lon1",
    "image": "ubuntu-16-04-x64",
    "droplet_name": "built-from-packer-{{timestamp}}",
    "snapshot_name": "built-from-packer-{{timestamp}}"
  }],

  "provisioners": [
    {
      "type": "shell",
      "inline": [
        "ip a",
        "curl -s http://checkip.amazonaws.com",
        "apt-get update",
        "sudo apt-get install -y vim curl wget byobu ntp",
        "timedatectl set-timezone Etc/UTC",
        "cat /etc/timezone",
        "date",
        "useradd -s /bin/bash -m {{user `newuser_name`}}",
        "usermod -a -G admin {{user `newuser_name`}}",
        "echo '{{user `newuser_name`}}:{{user `newuser_password`}}'|chpasswd",
        "cat /etc/passwd",
        "sed -i 's/Port 22/Port 2222/g' /etc/ssh/sshd_config",
        "sed -i 's/PermitRootLogin yes/PermitRootLogin no/g' /etc/ssh/sshd_config",
        "echo 'PasswordAuthentication no' >> /etc/ssh/sshd_config",
        "cat /etc/ssh/sshd_config",
        "mkdir -p /home/{{user `newuser_name`}}/.ssh",
        "mkdir -p /opt/www/html"
      ]
    },
    {
      "type": "file",
      "source": "authorized_keys",
      "destination": "/home/{{user `newuser_name`}}/.ssh/authorized_keys"
    }
  ]

}

This is a simple example that automates some of the security best practices of a non standard username, non standard ssh port, no ssh root login, no ssh password based login, etc.

NEWUSER_NAME=yourusername NEWUSER_PASSWORD=yourpassword DIGITALOCEAN_API_TOKEN=012345yourtoken /opt/packer build packer.json

Why not docker containers?

Actually I prefer docker containers as the artifact and deployment vehicle for services but it's not the only hammer in your toolbelt. And you have to setup the Docker hosts somehow, right?

(Unless you've already uploaded your soul into the matrix and are using Googazon's PaaS and never have to sully your container delicate fingers with a crude virtual machine again).

more info

https://www.packer.io/docs/installation.html
https://www.packer.io/docs/builders/amazon-ebs.html
https://www.packer.io/intro/getting-started/build-image.html
https://www.packer.io/docs/templates/configuration-templates.html
https://www.packer.io/docs/templates/user-variables.html

nginx with Docker

2015-06-13T08:00:00-07:00

nginx overview

http://nginx.org/en/ is one of the most popular performant web servers in the world (and it's pretty handy as a reverse proxy or load balancer too!). http://nginx.org/en/docs/http/load_balancing.html

https://github.com/nginx/nginx is written in c (very performant but often needs to be compiled , especially with any of the extra 3rd party modules https://www.nginx.com/resources/wiki/modules/).

A recent update means "...optionally load separate shared object files at runtime as modules" https://www.nginx.com/blog/dynamic-modules-nginx-1-9-11/

docker pull nginx

With Docker most of the time is spent in preparation, configuration, and testing. The advantage is that less time is wasted on compiling, packaging, etc. (for all those still eeking out another 1% in efficiency via esoteric flags and bundling all sorts of custom modules - good luck!)

One quick way to attempt to leverage nginx as a front end for your projects is using containers with Docker https://hub.docker.com/_/nginx/

sudo su
docker pull nginx:alpine
docker images

This will grab the latest image based on the very small alpine linux https://en.wikipedia.org/wiki/Alpine_Linux, around 13 MB vs 191 MB for the traditional nginx:latest which is based on debian jessie https://en.wikipedia.org/wiki/Debian

https://hub.docker.com/r/library/nginx/tags/ contains what other versions of nginx are provided by the vendor as Docker Images, the default build/image has quite a few modules http://nginx.org/en/docs/

all future references to docker commands will assume you are root or typing sudo first

docker nginx interactive container

docker run --rm --publish 127.0.0.1:80:80 nginx

starts an ephemeral container that binds the container port 80 to the local Host port 80 (binding to 127.0.0.1 prevents any other access except from the Host) , note by not explicitly sharing port 443 it is not connected/available

docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                 PORTS                           NAMES
3fccf15a3c24        nginx               "nginx -g 'daemon off"   3 seconds ago       Up 2 seconds        127.0.0.1:80->80/tcp, 443/tcp   pensive_elion

netstat -antp
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:80            0.0.0.0:*               LISTEN      5826/docker-proxy


curl 127.0.0.1:80

<h1>Welcome to nginx!</h1> is part of the default nginx home page, (success)

You will notice the access logs are being output to the console (where the docker container is running).

192.168.1.100 - - [01/Jan/1970:01:00:59 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.38.0" "-"

Control + C will terminate the container

running nginx manually in the interactive docker container

docker run --rm -i -t --publish 127.0.0.1:80:80 nginx /bin/bash

Starting as root inside the container

root@557ac197e2c1:/# which nginx
    /usr/sbin/nginx

nginx version and modules

root@557ac197e2c1:/# nginx -V
    nginx version: nginx/1.9.7
    built by gcc 4.9.2 (Debian 4.9.2-10) 
    built with OpenSSL 1.0.1k 8 Jan 2015
    TLS SNI support enabled
    configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-threads --with-stream --with-stream_ssl_module --with-mail --with-mail_ssl_module --with-file-aio --with-http_v2_module --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,--as-needed' --with-ipv6

Test your config file

nginx -t -c /etc/nginx/nginx.conf -g "daemon off;"

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful

run nginx directly

/usr/sbin/nginx -c /etc/nginx/nginx.conf

-g "pid /var/run/nginx.pid; worker_processes 2;" from https://www.nginx.com/resources/wiki/start/topics/tutorials/commandline/

nginx.conf

While there are quite a few ways to configure nginx one choice to make with Docker is to either

docker run --name some-nginx -v /some/nginx.conf:/etc/nginx/nginx.conf:ro -d nginx

run docker as a daemon with a specified Container Name and override the container nginx.conf file with /some/nginx.conf from the hose
Use a Dockerfile (based on the upstream nginx Docker image) to copy your own configuration file and other custom bits in and build your own custom Docker image (a highly recommended way of not being completely dependent on an upstream provider - especially if you push your Docker Image to your own private registry afterwards)

e.g. https://hub.docker.com/r/jwilder/nginx-proxy/~/dockerfile/ which also has "Foreman in Go lang" and makes use of expecting the Host to provide the SSL certificates

Quickly testing the nginx configuration file using an ephemeral docker container:

docker run --rm --publish 127.0.0.1:80:80 nginx /bin/bash -c "nginx -t -c /etc/nginx/nginx.conf"
    nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
    nginx: configuration file /etc/nginx/nginx.conf test is successful

Build your own nginx Dockerfile

If you require 3rd party modules then you will have to build nginx from source, e.g. https://github.com/openresty/headers-more-nginx-module#installation

That being the case you'll probably want to have a build process with 2 Docker files:

the first Dockerfile will contain build-essentials, gcc, make, etc. so that you can build the binary from source (with any 3rd party modules) http://nginx.org/en/docs/configure.html
The second docker image would be your "production container" where you copy in the custom nginx binary, install the dependencies (i.e. openssl), and setup the default config.

Converting an existing nginx 3rd party module to a dynamic module

Since NGINX 1.9.11 supports dynamic modules and attempts to maintain API compatibility then it is possible to sometimes convert a module into a dynamic module (i.e. build the shared library object)

./configure --add-dynamic-module=/opt/source/ngx_my_module/
make -f objs/Makefile modules

Look for .so files in the objs directory after compilation or the modules subdirectory during installation

https://www.nginx.com/resources/wiki/extending/converting/

Configuring nginx

There are entire books about how to configure nginx so I will just jot down some basics for myself.

/etc/nginx/nginx.conf

user nginx;
worker_processes 4;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    server {
        location / {
            root /var/www;
        }
    }
    # include /etc/nginx/conf.d/*.conf;
}

Run the service as the www-data user and define 4 worker processes

define an HTTP server that listens on port 80 by default

the root location will return the contents of /var/www

a best practice is to use multiple configuration files in the conf.d directory (as a really long complex configuration in a single is difficult to maintain)

BUT we must comment it out as in the installation there can be a default.conf that overrides our nginx.conf

sudo docker run -it --rm --publish 0.0.0.0:80:80 --volume /tmp/nginx.conf:/etc/nginx/nginx.conf:ro  nginx:alpine /bin/sh

debug the image interactively by overriding the default CMD with a shell (remember that Alpine is limited so apk update and apk add SOMENAME

sudo docker run --rm --publish 0.0.0.0:80:80 --volume /tmp/nginx.conf:/etc/nginx/nginx.conf:ro  nginx:alpine /bin/sh -c "nginx -t -c /etc/nginx/nginx.conf"

alternative method to just test your configuration file

sudo docker run --rm --publish 0.0.0.0:80:80 --volume /tmp/nginx.conf:/etc/nginx/nginx.conf:ro --volume /tmp/www:/var/www:ro  nginx:alpine

assumes you have a config file and some index file defined

echo "<html><body>hi</body></html>" > /var/www/index.html

That's it, now you have nginx serving static files! (curl localhost OR use a browser and visit localhost or http://hostfqdn)

of course /tmp is an insecure location so please store production nginx configuration files and web content from a secure directory in the docker host filesystem

nginx with ssl

It is hard to imagine running a production (or even test or dev server that should mirror production) without SSL since all traffic could be intercepted or hijacked. It takes a little more work but clearly it is an important step in running a service that others will use.

/etc/nginx/nginx.conf

user nobody;
worker_processes 4;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    server {
        listen 443 ssl;

        ssl_certificate /etc/nginx/server.crt;
        ssl_certificate_key /etc/nginx/server.key;

        location / {
            root /var/www;
        }
    }
}

openssl self signed certificates

openssl req -subj '/CN=example.com/O=My Company Name LTD./C=US' -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout /tmp/server.key -out /tmp/server.crt

mkdir -p /tmp/www

echo "<html><body>hi</body></html>" > /tmp/www/index.html

dockerized nginx with ssl

sudo docker run --rm --publish 0.0.0.0:443:443 --volume /tmp/nginx.conf:/etc/nginx/nginx.conf:ro --volume /tmp/server.crt:/etc/nginx/server.crt --volume /tmp/server.key:/etc/nginx/server.key --volume /tmp/www:/var/www:ro  nginx:alpine

curl --insecure https://localhost
firefox https://localhost

http://nginx.org/en/docs/http/configuring_https_servers.html

nginx with ssl and http/2

https://en.wikipedia.org/wiki/HTTP/2 is a new and more capable and performant standard for the venerable HTTP protocol. It has widespread vendor support so you can use most modern servers (e.g. nginx) and most modern browsers (e.g. chrome) and get the benefits immediately. (With of course all sorts of fallbacks for legacy clients)

/etc/nginx/nginx.conf

user nobody;
worker_processes 4;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    server {
        listen 80;
        location / {
            return 301 https://$host$request_uri;
        }
    }
    server {
        listen 443 ssl http2 default_server;

        ssl_certificate /etc/nginx/server.crt;
        ssl_certificate_key /etc/nginx/server.key;

        location / {
            root /var/www;
        }
    }
}

Assuming the previous nginx with ssl steps of creating a certificate and content, be aware that browsers permanently CACHE the 301 redirect so use Private Browsing mode otherwise you will never see a different result for localhost =[

sudo docker run --rm --publish 0.0.0.0:80:80 --publish 0.0.0.0:443:443 --volume /tmp/nginx.conf:/etc/nginx/nginx.conf:ro --volume /tmp/server.crt:/etc/nginx/server.crt --volume /tmp/server.key:/etc/nginx/server.key --volume /tmp/www:/var/www:ro  nginx:alpine

Verifying HTTP/2

curl

curl localhost

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.9.12</center>
</body>
</html>


curl --insecure --location localhost

<html><body>hi</body></html>

openssl s_client

openssl s_client -connect localhost:443 -nextprotoneg ''

CONNECTED(00000003)
Protocols advertised by server: h2, http/1.1

chrome browser

Browse directly to https://localhost/ with the chrome extension installed: https://chrome.google.com/webstore/detail/http2-and-spdy-indicator/mpbpobfflnpcgagjijhmgnchggcjblin/related?hl=en

The blue lightning symbol on the far far right (next to the "hamburger") indicates HTTP/2 is working.

browsers permanently CACHE the 301 redirect so connecting to http://localhost will forever see a different result for localhost =[

Browse directly to https://localhost/ (If using a self signed certificate accept any warnings about insecure SSL)
Chrome/Chromium Settings -> More tools -> Developer tools (aka Control + Shift + I)
Click on the Network section
Control + Shift + F5 to reload (or click on the arrow circling up)
Right click on the Name colum in the result so that you can add results for column heading, "Protocol"
Control + Shift + F5 to reload (or click on the arrow circling up)

Protocol: h2
https://www.nginx.com/blog/nginx-1-9-5/
https://blog.cloudflare.com/tools-for-debugging-testing-and-using-http-2/
http://tech.finn.no/2015/09/25/setup-nginx-with-http2-for-local-development/

nginx with php-fpm

Investigate php-fpm in Docker

This hack is fun but proves unnecessary when using Docker Compose later...

:::bash docker pull php:5.5-fpm-alpine ifconfig | grep Bc docker run -it --rm --publish 0.0.0.0:9000:9000 php:5.5-fpm-alpine /bin/sh route -n php-fpm --version sed -i 's/listen = 127.0.0.1:9000/listen = 0.0.0.0:9000/g' /usr/local/etc/php-fpm.d/www.conf php-fpm --fpm-config /usr/local/etc/php-fpm.conf

ifconfig and route -n are to discover the docker host IP Address via the default gateway 172.17.0.1

contains include=etc/php-fpm.d/*.conf

/usr/local/etc/php-fpm.d/www.conf

Configure nginx

vi /etc/nginx/nginx.conf

worker_processes 4;
pid /run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    server {
        location / {
            root /var/www/html;
            try_files $uri $uri/index.php;
        }
        location ~ \.php$ {
            root /var/www/html;
            try_files $uri $uri/;
            fastcgi_index index.php;

            # depends on Docker linking or hostfile for fpm to resolve
            fastcgi_pass fpm:9000;

            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            # https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2016-5385
            fastcgi_param HTTP_PROXY "";

            include fastcgi_params;
        }
    }

}

a basic nginx config that forwards .php requests to fpm on port 9000

docker run -it --rm --publish 0.0.0.0:80:80 --volume /tmp/nginx.conf:/etc/nginx/nginx.conf:ro --volume /tmp/www/html:/var/www/html:ro  nginx:alpine /bin/sh
route -n

This manual command proves unnecessary with Docker Compose but can be useful to debug (i.e. run the php-fpm container first, then run this one since it depends on fpm)

Some Test Content

Create the following files to test the various cases...

vi /tmp/www/html/foo.html

<html><body>hi</body></html>

vi /tmp/www/html/index.php

<?php
    print "<html><body>hello</body></html>";
?>

vi /tmp/www/html/bar.php

<?php
    print "bar";
?>

docker-compose.yml

# https://docs.docker.com/compose/compose-file/

nginx:
  image: nginx:alpine
  ports:
    - "80:80"
  volumes:
    - /tmp/nginx.conf:/etc/nginx/nginx.conf:ro
    - /tmp/www/html:/var/www/html
  links:
    - fpm

fpm:
  image: php:5.5-fpm-alpine
  ports:
    - "9000:9000"
  volumes:
    - /tmp/www/html:/var/www/html

# http://stackoverflow.com/questions/29905953/how-to-correctly-link-php-fpm-and-nginx-docker-containers-together
# http://stackoverflow.com/questions/35388590/issue-with-docker-compose-container-command-not-found

docker-compose up to start nginx and php-fpm

docker-compose up
docker-compose rm -f
curl localhost:80/foo.html
curl localhost:80/bar.php
curl localhost:80/index.php

Another tiny step forward in tying together a lot of moving pieces =]

Meeting Bjarne Stroustrup, creator of C plus plus, in the Atlassian Dev Den

2015-06-03T20:30:00-07:00

Bjarne Stroustrup visits Atlassian

One of the things I looked forward to when joining Atlassian was being a part of a culture that celebrated software development.

Today I had the opportunity to meet Bjarne Stroustrup http://www.stroustrup.com/, creator of C++. (yey)

To put this in perspective, C++ is one of the most popular programming languages of all time and is essentially like meeting the inventor of the hammer. (mindblown)

To summarize, he was one of the most intelligent, mindful, humble, and quietly passionate people I have ever met. Truly an inspiration to any engineer.

I have taken a tiny amount of free license due to my faulty memory and for readability.

Getting the book signed

I had dug out my old The C++ Programming Language (3rd edition from 1997) which filled me with nostalgia of how amazing it was to read a text book that was just full of succinct coding amazingness. So many textbooks were full of fluff, confusion, and downright goofiness but that book opened my mind to the difference it makes to have the original inventor writing it (Kernighan and Ritchie ;)

Lucky me, I was there about 15 minutes early and while they got the Audio/Video set up I bravely politely asked him to sign it. (yes, success!)

There was one other co worker there who was helping him prepare (and apparently also a huge fan, I took her picture with him) and I managed to ask some small talk questions (go me!)

Bjarne seemed distant at first and his driver/handler (former Chief Technology Officer somewhere important) was maybe a bit apprehensive but I suspect the fan-ness of it all relaxed a bit after we proved we weren't pyschos...

"This book was really amazing." He responded, "Well this one is a bit old, you should use a newer edition".

I quickly replied, "I don't actually use this in my day to day work so I keep it at home. It's a bit heavy to carry around all the time, I read a lot of things on my phone (easier to get the latest version of the books)."

I must have hit a topic of interest for him as he came over and I had flipped open the book to some sample code.

He ruminated, "Well, actually, I prefer this kind of reading with a book. The formatting and print... you have to work hard to make sure the examples don't get split across pages. And you have two pages." (Gesturing with his hands the extra width)

I agreed with him, "Yes, it is certainly a lot easier to read the diagrams and code examples in a physical book. The phone screen is really small. I guess the book is a perfected technology that's been around 1000 years."

"You know Atlassian isn't really a C++ shop, more Java with some Javascript... and Python products teams here in SF. Actually it's even a really argumentative culture sometimes around technology, though with you here I'm sure they'll all be quiet and respectful."

He actually smiled at that, "Back in AT&T we would ask: was there blood on the floor? That was how serious the discussions got. Not really any blood on the floor, just an expression."

"What do you enjoy most from your travels/touring?"

Such a wise response: "Meeting the people, learning about what they're thinking and what new ideas they might have".

Looking around the mostly empty room with 1 minute before the event starts, "Well I don't know where everyone else is, I'm sure there's lots of interest but the SF office is the Marketing and Business headquarters with only a few Development/Product teams". He responded, "Yesterday Facebook was standing room only."

I looked over at the monitors, "Well I'm sure there'll be attendance from the other offices too".

"What other locations does Atlassian have?"

"Austin, and of course Sydney has 800 Java Developers, and Poland, and Vietnam, it's a pretty globally diverse company".

I also warned him about the air conditioning intermittently coming on and being loud and often too cold. "I was baking hot yesterday so I guess if it's cold in here today I guess it will average out". (With his curious funny little smile)

And with the microphone on and an international audience he was introduced and here were the questions and answers...

Current challenges for programming languages

Since hardware improvements are through parallelization (unless quantum entanglements gets solved soon) developers must adapt, but concurrency is hard. (And no global variables, right?)

"Functional programming has good ideas but I did not want everything to be a recursive function"

Distributed systems are a new perspective problem: for a fair trade, an exchange in New York offering the same price to Sydney and San Francisco becomes difficult (the speed of light is still a limit).

Anyone heard of the leap second? There are at least 3 ways it is being dealt with:

Hold on for a really long time and then jump forward all at once.
Jump forward immediately.
Break the leap second into a lot of intervals and merge it in.

But if it's a distributed system how can everyone agree? And that's just one example.

Trends in programming?

Silicon Valley is an echo chamber but you probably know more than I do.

Mobile is very hot but it is a very small screen, and not everything (like a textbook) can be read on a small phone screen or even a tablet screen.

A second favorite language?

Bjarne was incredibly diplomatic about pointing out that any language he said he preferred might be tweeted and start a "language war"; language comparisons are useless without the context of the constraints of the problem domain.

For instance he had written C++ to be a performant language with the assumption of unix shell as the "other language".

But what about Rust "Zero Cost Abstractions"? "It's not good to pay for what you don't use."

He admitted that since he coined the phrase "Abstraction without overhead" that indeed he admired the goal of Rust (getting close to the hardware for maximum performance) but hadn't been following their recent developments. For garbage collection... you shouldn't litter.

http://blog.rust-lang.org/2015/05/11/traits.html

Languages are about solving problems so I am very familiar with the strengths and weaknesses of C++ and I know how to make it do quite a lot. But if I use another language then I am an amateur, not familiar with the idioms, and perhaps I will not use it so well.

Any features you regret putting in or leaving out of C++?

No, everything in there has a use and ... they have to stay or it would upset thousands of people. I wish it had better syntax, starting from C, well even they realized it was a mistake but it was already done so what are you going to do.

Anything you put in a language, especially a mistake, will stay with you. Maybe you can put something off to release a product and know you will get back to it in an iteration or two. But make a mistake with a language and you live with it for a couple of decades.

I do wish I had gotten some things right earlier. Templates. They solved the problem really well, general enough to do more than I imagined (that's the criteria for sucess). Nobody else at the time knew how to solve it but I wish the interface was better. Now, with the upcoming "Concepts", it will be better, but I wish I had done it right the first time.

You have to understand, at the time of creation C was constrained to 64K, and C++ to 256K, so there are simply things you cannot do then.

A complicated question about the N4047 modules proposal

It was a good proposal. Isolating code from other portions to make compilation faster.

C++ is maintained by people who actually pay 1280 to be on the committee... they don't make any money by doing it. Daveed Vandevoorde had to continue working and doing this proposal was something on the side.

If it is not in my top 20 then it's not an important problem. So a lot of good proposals do not make my top 20.

More recently teams in Microsoft, Google, GCC, and Clang are looking at this and some will even have implementations before it is released.

It should be including in release 17 and you may see a compilation speedup between %50 and 50x , maybe more usually 4x or 8x faster.

Language features are complicated as you have to coordinate between all of the major tool chains (you certainly can't have one of them do it one way and another differently).

How many languages?

When asked about whether a developer should focus on one language or learn many Bjarne had a very nuanced answer:

When he started programming it was not impossible to learn 25 languages. It is very valuable to learn the different ways of thinking and solving problems by programming in a different language. Now though, languages are more complex with a lot of libraries, idioms, and toolchain to learn.

So learning two languages, like the example before of 1 performant and 1 more general, would not be enough since that would not provide any comparison of why. Maybe 4, so that you could compare two general and two performant languages and understand how/why they did things differently.

My Question

I decided to mull over and try and ask a live question (with an audience this time).

"Given your comment about tooling before, how valuable do you find developer tools to a programming language (like an IDE)? Also, what are any major gaps you see in current tools?"

"Good question. I don't really use tools that much since I work on smaller examples that might need to be ported in a few places. On larger projects and code bases tooling is very useful as you can't keep all of it in your head anymore. I'm maybe not the best person to ask, you probably work on large code more than I. But just being able to click on a macro or identifier and jump to it in the code is very useful."

"I don't use a debugger very often. Other people might but I don't. If you are using a debugger it's maybe a bad sign about your code. But I would like to see a debugger that could follow concurrency. That would really help."

"Tests are really important, so tools that make it easy to run tests, and to write tests quickly, that would be very useful."

What is a professional?

If someone, your employer or your manager, asks you to do something that is shabby you should just resign. It doesn't happen very often but there are ethics.

I wish the Universities, that teach "Computer Science" and not programming, would spend some time on caching or real world coding and problems. They graduate being able to describe the Halting Problem but with magic constants throughout their code.

It would be nice if there was a more universal way to explain/teach being "professional" in software.

C++ in Education

Bjarne was asked and clearly had some strong feelings on the subject of why C++ wasn't taught in universities.

"Perhaps it's because C++ doesn't have a marketing budget" he quipped. More seriously he went on to point out "he had very successfully taught college freshman programming in C++ so it's quite possible. There isn't enough time to teach all of C first so you don't start with that: for instance pointers are chapter 17, not chapter 2."

More info

An interesting interview he gave for comparison: http://www.stroustrup.com/CVu263interview.pdf

Pragmatic testing, from Makefile to CI with Docker

2015-05-25T10:00:00-07:00

A colleague recently suggested "Hey, why don't you run those tests from outside of the target server under test?"

And I thought to myself, "Hmm.... why are we doing it that way? Was I just dumb when I did this the first time?

The answer is the journey of successfully testing a successful product and the pragmatic choices made along the way. I believe that engineering requires compromises because without achieving short term progress we would have never reached our bigger, long term goals.

Humble beginnings with Makefile

Our initial project was to deliver an OVA (open virtualization archive) that packaged all of the services (and dependencies) into an easy to deploy and maintain virtual appliance.

Will, as the project lead, had already spent some time setting up the build environment using vStudio and an Ubuntu ISO but one of our first issues after generating an OVA was to try and determine if it was "good". Could it even be deployed? Did it have the services installed correctly?

I leveraged the Makefile that we were already using for the build to create a "test": deploy the latest OVA artifact and check that it was basically sound.

This was a good start as, at the very least, we could quickly determine and bisect where we were producing bad builds.

Adding some tests, but where?

Once the initial spike ("tracer bullet") was done I began adding in selenium python based tests to verify the Web UI interface.

Since our Continuous Integration system was basically bash and Makefile (which was also our build system) I opted to run the tests from inside of each deployed Virtual Machine.

This allowed for isolation of the test execution from the build process and for each test run.

While not ideal for product acceptance testing it provided a basic safety net that allowed us to know if a breaking change occurred upstream and was the first automated verification (followed by lots of manual testing) of the exact OVA we were shipping to Customers in the Beta.

Bamboo Continuous Integration for visibility and the team

A further improvement was to improve visibility of the test results. In parallel, after the team discussion during "HipCon" and with motivation from Don and Sam, I setup a private Bamboo installation in our VM area and helped get Integration tests setup for our upstream backend code.

Once again I stuck with the tried and true "tests inside of the target virtual machine" pattern as the Bamboo server was only using "local agents" and I was concerned about trying to maintain a clean environment and the resources required as the number of test plans scaled.

Additionally I migrated all of the previous test plans from the "build factory" into Bamboo which really helped with failed test visibility and tracking over time.

The unit tests continued to run in a different SaaS version of Bamboo so I avoided scope creep and left things that were working alone.

Migrating to a managed service

One of the most asked questions whenever a new person joined was "Why are the tests spread across so many different servers/services"? I had to answer that question so many times! The answer basically boiled down to "the testing grew organically as different people in the team solved the problem they faced". Unsatisfactory? "Be the change you seek" didn't seem to get anyone else to solve the problem for me ;)

Using the opportunity presented by a service outage issue I pushed forward a plan I had to migrate all of the test plans into a newly provisioned Bamboo server run by Build Engineering. (awthanks)

There was an awesome team effort by a lot of people to make that happen (made even more challenging by doing it when the source service was out). (awesome)

This solved quite a few problems:

The Unit, Integration, and Product Acceptance tests were all finally combined under one roof... one UI to rule them all!
New people joining the team needing access to Bamboo: since I hadn't linked it to any directory/authentication I was adding users manually - we were finally able to leverage an Atlassian backend Directory system
I had been managing upgrades and maintenance of the Bamboo software (not fun and not my core expertise)
I began to worry about the resource consumption (since this was running on hardware that also provided for our Build Factory) - no longer a worry with lots of Remote Agents and Elastic Agents (all provided by Build Engineering)

Bonuses:

More plugins and capabilities and knowledge from the Bamboo server and Build Engineering expertise
Plan Templates to keep test plans in version control and macros for common functionality
A successful spike of using Docker for unit testing

With so much going on during the migration I avoided changing the testing paradigm, so tests continued to execute inside of each VM/EC2 Instance deployed. (shrug)

Adopting Docker and refactoring

Remember that question at the beginning? When something goes wrong, while it makes sense to separate "fixing it" from "improving it" I'm a big fan of taking advantage of having the hood open to go the extra mile and leave the campground cleaner =)

So some of the selenium based tests were failing and it occurred to me that some of how we were changing our dependency infrastructure at the operating system level could be the cause. After some unproductive poking around I tried to reproduce and isolate the issue by running the tests from outside of the VM.

Aha! In that moment I realized that this was actually the desired (original intention, honest!) way to run the blackbox product acceptance tests.

So I pulled up my sleeves and tried out the "hot fancy new silver bullet technology that solves every problem".

Why Docker?

Docker encourages design of modular, deterministic and defined, single purpose components that are easy to reuse and compose into larger services.

Not only are (Docker) Containers fast, one of the biggest advantages of Containers is the ability to reduce complexity. Docker can turn an application/service, it's dependencies, and even the OS level requirements into a single blackbox package (that you can still inspect inside if you really want to).

So I built a Docker Image containing python selenium and http://phantomjs.org/ (a headless javascript based browser) and other dependencies.

Sure enough I the tests passed when leveraging the previous Docker spike to run my new docker container. (success)

Refactoring the bamboo plan (since it was leveraging Plan Templates and the Groovy DSL macros) didn't take too long and with other stakeholders PR/approval we're moving full speed ahead towards the "ideal" solution. (It only took about 2 years).

Dogfooding the whole way

Something I should mention that has been an invaluable companion throughout the course of building the product: a dogfooding server.

"Eating our own dog food" is a wonderful way to experience the exact pain you are inflicting on your users. From the very begininng Will setup and we maintained a dogfood server which received every beta upgrade (and a few upgrades that never reached the customers), amazingly it's still alive and full of data!

Not only did I learn about bugs that would affect our oldest and most loyal users (who kept with us and kept upgrading), I also felt the User Experience pain of how long upgrades took, mysterious incomprehensible errors messages, and "partial upgrades". All of these learnings, along with being Developer on Support and assisting on support tickets, kept me honest and humble and allowed me to improve the product just as much as any fancy testing automation framework.

What does this all mean?

It's easy to draw up how things should work according to best practice. It's even easier if it's work that someone else has to do and there aren't any deadlines.

Success comes in stages. Overengineering and premature optimization cost way more in opportunity cost and thrown away work than doing things the "wrong way".

This story could be massaged to fit a parable of "Lean and Agile" but it's really just common sense about transparently understanding the cost/value tradeof of the work, solving the current needs, and moving forward onto something better (by keeping informed of new solutions) when the opportunity shows up.

HAProxy in Docker

2015-05-21T20:00:00-07:00

Not only are Containers fast, one of the biggest advantages of Containers is the ability to reduce complexity.

Docker can turn an application/service, it's dependencies, and even the OS level requirements into a single blackbox package (that you can still inspect inside if you really want to).

One thing I really like is less code. Seriously. Configuration over coding (whenever I don't need customization) means far less maintenance and bugs.

Here's a trivial example of how I can leverage the HAProxy Docker image/container to load balance two web servers. (aka "reverse proxy" http://en.wikipedia.org/wiki/Reverse_proxy)

    client -> all other sites
      |
reverse proxy (haproxy)
  /         \
BackendA  BackendB

There are new problems that go along with the benefits of any new technology, see the complicated networking/port coordination

Prerequisites

sudo docker pull haproxy:1.5

Some backend web servers

mkdir -p /tmp/BackendA
echo "foo" > /tmp/BackendA/foo.txt
cd /tmp/BackendA
python -m SimpleHTTPServer 8000 &

mkdir -p /tmp/BackendB
echo "bar" > /tmp/BackendB/bar.txt
cd /tmp/BackendB
python -m SimpleHTTPServer 8001 &

Clearly a trivial example (more likely two remote hosts in logical/geographic disparate areas if aiming for High Availability, or at least on different hosts to scale with more resources)

/opt/mydata/haproxy.cfg

global
        debug

defaults
        log global
        mode    http
        timeout connect 5000
        timeout client 5000
        timeout server 5000

listen http_proxy :8443
        mode tcp
        balance roundrobin
        server srv1 docker:8000 check
        server srv2 docker:8001 check

start haproxy.sh

#!/bin/bash
HOSTIP=`ip addr show | grep docker0 | grep global | awk '{print $2}' | cut -d / -f1`

sudo docker run -p 8443:8443 --add-host=docker:${HOSTIP} --rm -it -v /opt/mydata/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro --name myhaproxy haproxy:1.5

Now that the docker container's /etc/hosts file has the Host IP Address injected (with the name "docker") the haproxy config file probably makes more sense

running the container
the host port 8443 mapped to the container port 8443
injecting into the container /etc/hosts the Host IP Address as "docker"
ephemeral container (automatic cleanup on termination)
interactive
tty
readonly mapping of the /opt/mydata/haproxy.cfg file on the host to /usr/local/etc/haproxy/haproxy.cfg
name the container myhaproxy (each container name must be unique)
the container is using the haproxy version 1.5 Docker Image

./start-haproxy.sh

curl localhost:8443

:::html <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> Directory listing for /
Directory listing for /
- bar.txt
curl localhost:8443

:::html <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> Directory listing for /
Directory listing for /
- foo.txt

HAProxy Stats

sudo docker rm -f myhaproxy

/opt/mydata/haproxy.cfg

global
    debug

defaults
    log global
    mode    http
    timeout connect 5000
    timeout client 5000
    timeout server 5000

listen http_proxy :8443
    mode tcp
    balance roundrobin
    server srv1 docker:8000 check
    server srv2 docker:8001 check

# optional section to enable statistics for haproxy protected by basic auth (replace with your own user and password)
listen stats :1936
    stats enable
    stats uri /
    stats realm HAProxyStatistics
    stats auth user:password


./start-haproxy.sh

nginx forward proxy

client -> forward proxy (nginx) -> all other sites

docker pull nginx:alpine

nginx.conf

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;

    gzip  on;

    server {
        listen       8080;

        location / {
            resolver 8.8.8.8;
            proxy_pass http://$http_host$uri$is_args$args;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}



docker run -it --rm -p 8080:8080 -v /tmp/nginx.conf:/etc/nginx/nginx.conf:ro --name mynginx nginx:alpine

bind to port 8080 on the host and run an ephemeral container based on the alpine linux with nginx image that uses the /tmp/nginx config

http_proxy=127.0.0.1:8080 curl example.com

set the linux operating system proxy environment just for this one curl command and see the dockerized nginx forward proxy log show: 172.17.0.1 - - [06/May/2016:22:37:22 +0000] "GET HTTP://example.org/ HTTP/1.1" 200 1270 "-" "curl/7.35.0"

Configure your browser (firefox) to use 127.0.0.1:8080 as the proxy for all protocols and watch the log statements fly by when you test http://example.com

NOTE: this does not support HTTPS http://forum.nginx.org/read.php?2,15124,15256#msg-15256

Attempting https://example.com will return "CONNECT example.com:443 HTTP/1.1" 400 173 "-" "-" and the browser will show "Server not found"

** Warning ** do not use http_proxy=http://127.0.0.1:8080 as that will fail =[

You can permanently set the environment proxy with export http_proxy=127.0.0.1:8080

HAProxy as a limited outbound proxy

sudo docker pull haproxy:1.6-alpine

Now that you have the 10MB haproxy image...

Note there is a limitation to haproxy in that it always assumes a syslog facility (no direct logging to stdout or files)

/opt/mydata/haproxy.cfg

This is the more readable config style which separates the frontend from backend and this is haproxy 1.6

The logging probably does not work =(

global
        debug
        log /dev/log local0 info

defaults
        mode http
        timeout connect 5s
        timeout client 5s
        timeout server 5s

frontend myfrontend
        bind *:8443
        default_backend mybackend

backend mybackend
        server s1 example.com:443 ssl verify none

starting the haproxy docker container

docker run -p 8443:8443 --rm -it -v /home/admin/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro -v /dev/log:/dev/log --name myhaproxy haproxy:1.6-alpine

Control + C to quit

more info

Caching data and common gotchas and an intro to redis memcached and varnish

2015-03-26T00:00:00-07:00

Caching is when you use a copy of a data set rather than using the original source.

Caching often involves a "Key Value Lookup":

A request is received and the service checks the cache using a Key
The cache does not contain the Key
The service generates the result from the originating data source (i.e. database)
The service then stores the result in the cache with the Key as the index (and the result as the Value)
A subsequent request is received and the service checks (looks up) the cache using a Key
The cache does contain the Key
The service retrieves the Value from the cache and returns the result

A more concrete example would be to cache a User object by Email, so that whenever a request came in for a particular Users details the cache would contain their Name, Address, and Phone Number.

https://en.wikipedia.org/wiki/Associative_array

Why Cache?

A tradeoff of memory for cpu (or latency or some other business cost).

computation is expensive (in terms of cpu, time, or money)
accessing the data from source is too slow
access to data across a larger geographic distance
the data actually comes from multiple sources (complex and expensive to retrieve)
to reduce load on the service originating data
to reduce contention (i.e. reads not served from the same persistence that does writes)
server side caching can protect backend resources and improve throughput and performance
for a client-server architecture, caching on the client reduces the number of required connections to a server

Questions to ask when caching

Is the complexity of caching worth the performance gain? (a simpler implementation is often better, less chance of bugs!)
Does my cache need to be consistent? (meaning the cache and data source return identical results)
Can my cache be "eventually consistent"? (meaning a wrong answer for some specified period of time is ok)
Am I caching at a high level? (meaning aggregating a lot of work/responses from lower level systems)
Am I caching at a low level? (meaning inside of my Data Access Object pattern I'm protecting a single simple resource, i.e. a MySQL table, from being accessed too often)
How unique are my Keys in my cache (i.e. if multiple users can have the same identifier it would be very bad to return the wrong session to the wrong user)
Do I have the ability to operate or pay for a caching service?
What will happen if the cache is unavailable?

Cache Latency Times in Perspective

Taking "why cache" to another level, the relative speeds of different cache levels highlight why some applications or algorithms will fail if they do not leverage cache.

If your application uses a very large amount of data the network may actually be better than disk; optimization would probably not be focused on "loop unrolling"
If your application depends on data across the internet then network caching, routing algorithms, and data modeling (eventual consistency!) may be more important than "tail recursion vs iterative"

Action	nanoseconds	microseconds	milliseconds	human scale comparison
A typical cpu instruction	1 ns			1 second basis (approximations)
L1 cache fetch	0.5 ns
Branch misprediction	4 ns
L2 cache fetch	7 ns			7 seconds
Mutex lock/unlock	25 ns
RAM "main memory" fetch	100 ns	0.1 us		2 minutes
Read 4K randomly from SSD	100,000 ns	100 us		28 hours
Read 1 MB sequentially from memory	250,000 ns	250 us		3 days
Send 1000 bytes over 1 Gbps network	500,000 ns	500 us	0.5 ms	6 days
Read 1 MB sequentially from SSD	1,000,000 ns	1,000 us	1 ms	12 days
Spinning Hard Disk seek	8,000,000 ns	8,000 us	8 ms	3 months
Read 1 MB sequentially from disk	20,000,000 ns	20,000 us	20 ms	7.6 months
Packet Roundtrip SF to NY	70,000,000 ns	70,000 us	70 ms	2 years
Packet Roundtrip SF to NY	150,000,000 ns	150,000 us	150 ms	5 years

The L1 cache is the memory cache integrated into the CPU that is closest

Light travels 30 cm or about 1 foot in 1 nanosecond

ns = nanoseconds, us = microseconds, ms = milliseconds

Caches are another Operational component with Overhead

The best advice is to definitely avoiding caching until the last possible moment ("less is the best" and "premature optimization" and "be future flexible" and "defer architecture decisions")

Not only do you have to write code complexity for using a cache, there's the nitty gritty of running a cache (which can be a completely different expertise than programming)

Install
Deployment
Upgrades
Security
Monitoring
Metrics
Testing (i.e. synthetic smoke tests or load)

None of this operational cost is free, and there are plenty of issues when just implementing caching in the code...

"High Level" caching for the application versus "Low Level" caching for the persistence layer

Caching can be the most effective at the "highest" layer where the application is able to trivially service a request. (e.g. caching the final full web page that was requested)

This "protects" all of the underlying machinery from having to do work. It requires a good understanding of how dynamic or personalized the results are, designing the system to have pieces that can be cached, and is brittle to change.

Caching can be an "easy win" when applied at the "lowest" layer right near the persistence (i.e. redis "in front of" a database - that for example contains the new articles that will be served on a web page) as all of the components in layers above can benefit from improved performance without having to change the business logic (or perhaps even unaware of the caching).

Both can be used together but to paraphrase: "mo caching mo problems"

How to Cache

Cache on Write

Also known as "cache on write through", whenever new data is written a cache must also be updated.

Cache on Read

Also known as "cache on read through", whenever a query is made first the cache is checked.

If there is a "cache miss" then the data source is queried and the cache is updated and the result is returned.
If there is a "cache hit" and the data is in the cache then it is returned (and potentially a cache key expiration updated as this cache hit improved the cache efficiency).

Cache Warming

Pre-emptively adding data to the cache is "cache warming" in order to improve "cache hit" percentages and reduce the risk of "cold cache" issues.

Flush the Cache

Removing some or all data from the cache in order to invalidate a chunk of data (i.e. all users need to reset their passwords) or pre-emptively free up memory/space.

Common Gotchas

Caching is challenging because of the need for data consistency, parallel requests, and race conditions.

One good way to think about it is a banking system with money...

If two people both try to empty a bank account at an ATM at the same time how will your caching system handle it?

Cache on write gotchas

There may be a design mismatch as since data is only cached on write, if reads are occurring mostly on data written a long time ago they will be expired/pushed out and you will have poor cache efficiency (paradoxically adding caching will have resulted in more complexity and worse performance).
One implementation flaw is updating the cache first; if the update to the data source fails then some requests may have been given incorrect data.
The opposite order of updating the canonical source first can have a similar problem if the process fails before the cache can be updated. (Thus leaving the cache with old data).
- First invalidating the cache reduces the risk of a failure during a write creating an inconsistent state
- Next update the data persistence (which should be provided as an atomic operation by your vendor ;)
- Finally update the cache
- In the worst case the canonical source will be updated without the benefit of caching
- One might also have a "transaction" defined around both the update of the origin and cache with retries for failure
While "cache on write" is a sometimes band-aid for NoSQL "eventual consistency" when it fails (i.e. all applications should expect that a cache will not exist or have a cache miss) the result may be data inconsistency.
- One workaround is "check and set" (or "compare and set") where the cache will auto-invalidate if two conflicting entries are attempted.
- https://neopythonic.blogspot.com/2011/08/compare-and-set-in-memcache.html

This "gotcha" could be summarized as not handling rollback

Expiration: a cache full of stale junk

A naive implementation of caching will store every result in the cache forever...

While this seems like a good idea ("The cache application/service will just evict unused items based some algorithm") it is essentially forcing your cache to be full of potentially low value information on the hope that someone else will solve the problem.

Since some caching tools/framework do not set a default Time To Live or Expiration and in that case all of your data may quickly fill up the cache (not a bad thing per se), but then it will use whatever default or global "eviction policy" that is defined.

Even O(1) can be broken by a pathological data set, and keeping every item seems like a good way to find an edge case (i.e. hash collisions and chaining).

Applying business logic and empirical data to pick sane expiration values might not only improve cache performance but may protect your service from security issues or bugs due to serving really stale data.

e.g. for security reasons, caching a session "forever" is a bad idea as an attacker may get access to an old client cache or token and be able to impersonate a legimate user

Issues with Expiration Set Too Long:

Security concerns
Lack of control/non determinism for when and what items might be evicted
Poor performance, memory pressure, and possibly increased operational cost
Stale data
Large cache sizes may end up writing to disk (i.e. redis sync to disk may use copy on write)

Set a TTL or Expiration, whenever possible, that matches your domain (i.e. for a session 1 day or 1 week).

If the Time to Live is too short then the cache may have very poor efficiency (items expire before they can generate even one cache hit), meaning all of the coding and operational cost are for nothing =[

Cold Cache and the Thundering Herd

If the cache is "cold", i.e. has not been populated, then all queries will go directly to the source
If the source is not prepared for the "thundering herd" of requests (that were usually handled by the cache) then the source may become overloaded and bad things will happen
It is therefore best practice to "warm the cache" by seeding data from the source into the cache before significant load events

Cold cache not only can cause problems from the source but when lot of data is written simultaneously to the cache, if the cache uses underlying disk or some other IO resource, it may temporarily overwhelm the cache (system/framework).

Upgrading your application

In a sense the cache layer is an external persistence that has to stay in sync with the application code; they are logically and semantically bound together.

Modification to your application code, specifically the way it reads and writes to the cache, may return "bad" data.

Cache key "admins" stored a list of usernames of admin users for the application
Cache key "ausers" stored a list of usernames that begin with the letter "a"
An application upgrade occurs
Now the code has a bug that looks up "ausers" in order to give administrator permssions
(oops)

Spanning Multiple Keys

If you need to retrieve multiple items from a cache in order to fulfill a request you may run the risk of "torn reads" where the first item retrieved from cache is logically inconsistent with the second item.

Tools for caching

Much like encryption it is probably a good idea to use a time tested caching component over writing your own implementation.

A local in memory cache is a tried and true way of speeding up an application but it may not provide the transparency and visibility when there are bugs.

While it seems trivial to setup it will slow down your dev velocity on your high value focus area and every new feature you realize you need (automatic expiration, authentication, etc.) will create a distraction and eventual maintenance requirement.

Instead there are quite a few very popular battle tested options...

Memcached

docker run --rm -it --publish 11211:11211 --name mymemcached memcached:alpine

echo -e 'add foo 0 60 11\r\nhello world\r' | nc localhost 11211
telnet localhost 11211
get foo

VALUE foo 0 11 hello world

https://github.com/memcached/memcached/wiki/Commands

Redis Examples

Redis has surpassed memcached in terms of speed and functionality and if you need to store more than "just a string" you should experiment with it.

Besides having a cache to speed up lookups for your application or as a globally shared cache (be careful!) between multiple application serveers there can be a nice convenience as a "meta" persistence such that you can deploy a new version of your application and not lose all of the data in the cache.

One thing to think about is that local redis might be far more effective than remote over the network redis.

If your application can depend less shared state this is good because sharing is a nightmare for cache semantics and distributed computing.

When possible avoid a globally shared cache between multiple processes or servers, or invest in learning about atomic operations

Regardless of whether you securing your remote cache (or just depend on network isolation) you will always want to measure cache effectiveness.

Installing Redis

The simplest way is to use Docker, https://hub.docker.com/r/_/redis/

docker run --rm -it --publish 6379:6379 --name myredis redis:alpine docker run -it --link myredis:redis --rm redis:alpine redis-cli -h redis -p 6379 set message hello docker run -it --link myredis:redis --rm redis:alpine redis-cli -h redis -p 6379 get message

run an ephemeral docker container and then non-interactively use the same docker image to set and get a string key

If you prefer installing locally to your filesystem or server:

https://redis.io/topics/quickstart (compiling from source)
https://packages.ubuntu.com/trusty/redis-server (sudo apt-get install redis-server)

redis-cli -h localhost:6379 ping

PONG , aka verify a remote server connectivity

Interactive Redis Prompt

redis-cli
keys *

Non Interactive Redis Commands

redis-cli KEYS *:*

non-interactively get all of the keys that have subkeys

redis-cli KEYS "session:3:*" | xargs redis-cli DEL

non-interactively delete/remove all of the subkeys under the sub subkey

redis-cli KEYS session:1:*
redis-cli hgetall session:1:web
redis-cli hgetall session:1:web:presence

redis-cli KEYS session:1:*  | grep session:1:web-48
  session:1:web-48679:rooms
  session:1:web-48679:presence
  session:1:web-48679:message_ids
  session:1:web-48679

redis-cli zrange sessions:1 0 9
  1) "1:web"
  2) "1:web-48679"

redis-cli zrem sessions:1 1:web-48679
redis-cli del   session:1:web-48679:rooms
redis-cli del   session:1:web-48679:presence
redis-cli del   session:1:web-48679:message_ids
redis-cli del   session:1:web-48679

https://redis.io/commands

Redis Clients

pip install redis

http://redis.io/clients#python

import redis
r = redis.StrictRedis(host='localhost', port=6379)
r.flushall()

https://pypi.python.org/pypi/redis

Go go get github.com/garyburd/redigo

package main

import (
    "fmt"

    "github.com/garyburd/redigo/redis"
)

func main() {
    c, _ := redis.Dial("tcp", ":6379")
    defer c.Close()
    c.Do("SET", "message", "hi")
    s, _ := redis.String(c.Do("GET", "message"))
    fmt.Println(s)
}

A more complete example: https://bitbucket.org/johnpfeiffer/go-cache

Varnish

https://www.varnish-cache.org/about REST web caching

Subunit and Subunit2JunitXML to get JUnitXML test result output from UnitTest

2014-11-01T00:00:00-07:00

Test results from differing systems or multiple test runs need a common format.

JUnit XML is almost a de facto standard for test results given almost all major Continuous Integration products support it.

Setup

pip install python-subunit junitxml

assuming virtuelnv and myenv/bin/activate , junitxml is a hidden dependency :(

Do not use apt-get install subunit as even with 14.04 Ubuntu it has an older version does not contain timings and subunit2junitxml creates "skip" instead of "skipped"

Example UnitTest Class

import unittest

class john(unittest.TestCase):

    def test_success(self):
        self.assertTrue(True)

    def test_fail(self):
        self.assertTrue(False)

    @unittest.skipIf(True, 'always skip')
    def test_skip(self):
        self.assertTrue(False)

Example Usage

One Liner

python -m subunit.run foo | subunit2junitxml --no-passthrough --output-to test-results

forward = non-subunit output will be encapsulated in subunit

Intermediate Subunit Results File

python -m subunit.run test_some_filename_with_py_truncated > test-results.subunit

Do not use python -m subunit.run test_some_filename_with_py_truncated to stdout as it expects to have binary delimiters which screw up the console command line

subunit-ls < test-results.subunit
subunit-stats < test-results.subunit

python -m subunit.run foo >> test-results.subunit

append some more test results subunit-stats < test-results.subunit

subunit2junitxml --no-passthrough --output-to test-results.xml < test-results.subunit

no passthrough does not pass/convert any extraneous non subunit data/lines to the junit xml

<testsuite errors="0" failures="1" name="" tests="3" time="0.001">
    <testcase classname="john.john" name="test_fail" time="0.000">
        <failure type="testtools.testresult.real._StringException">_StringException: Traceback (most recent call last):
  File "john.py", line 9, in test_fail
    self.assertTrue(False)
  File "/usr/lib/python2.7/unittest/case.py", line 424, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true

        </failure>
    </testcase>
    <testcase classname="john.john" name="test_skip" time="0.000">
        <skipped>always skip</skipped>
    </testcase>
    <testcase classname="john.john" name="test_success" time="0.000"/>
</testsuite>

Twisted UnitTesting

`trial --reporter=subunit foo | subunit2junitxml --forward --output-to=junitxml-result.xml

Troubleshooting

ImportError: No module named 'junitxml'
- You may not have installed the junitxml module which subunit apparently sometimes depends on: pip install junitxml use sudo only if not using virtualenv
AttributeError: 'AutoTimingTestResultDecorator' object has no attribute 'errors'
- This occured becaused TestSomeClass(unittest.TestCase) definition had an errors property/attribute which resulted in a namespace collision =(
Empty results like this: <testsuite errors="0" failures="0" name="" tests="0" time="0.003">
- if you view/cat your results.subunit you will notice:

test: directory.path.foobar.FooBar.test_constructor

successful: directory.path.foobar.FooBar.test_constructor

- That is old subunit output (i.e. an old version of Twisted: trial --reporter=subunit), the new version 2 uses non printable characters instead of newlines (which sometimes ruins output to console)
- Resolution for old subunit version converted to new subunit version:
  
  trial --reporter=subunit foo | subunit-1to2 >> /tmp/results.subunit ; subunit2junitxml --no-passthrough --output-to test-results.xml < /tmp/test-results.subunit

More Info

Virtualenv Python Interpreter from source

2014-10-31T00:00:00-07:00

When building an application (including an external facing webapp or an internal test suite) it is critical to manage your dependencies.

Virtualenv is a tool that keeps all the dependencies in a file system based container (and overcomes permissions based issues as well).

To really isolate your application from the environment not only do you need a specific version of libraries (i.e. you know your application works fine with requests 2.4.3 and selenium 2.44) but additionally a specific version of the Python Interpreter.

Build python from source

wget https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz
tar -xf Python-2.7.8.tar.gz
cd Python-2.7.8
./configure --prefix=/home/ubuntu/python --enable-unicode=ucs4
make && make altinstall

/home/ubuntu/python/bin/python2.7 --version

altinstall ensures we do not try to override the existing /usr/bin/python binary which can be important if you want python 2.7.3 and python 2.7.8 to exist side by side

Optionally: echo 'alias py="/home/ubuntu/python/bin/python2.7"' >> ~/.bashrc

python3 from source

wget https://www.python.org/ftp/python/3.4.2/Python-3.4.2.tgz
tar -xf Python-3.4.2.tar.gz
cd Python-3.4.2
./configure --prefix=/opt/python3.4.2 && make -j$(nproc) && make altinstall
ls -ahl /usr/local/bin | grep 3.4

/usr/local/bin/pip3.4 install --upgrade virtualenv

If you mess up your OS level python

apt-get install python3

python3 depends on dh-python; however:
Package dh-python is not configured yet.

Look at the stack trace, reinstalling may not have put all of the helper directory and .py files in place

File "/usr/lib/python3.4/site.py", line 586, in <module>
ImportError: No module named '_sysconfigdata_m'

mv /usr/lib/python3.4 /usr/lib/python3.4-OLD
wget http://mirrors.kernel.org/ubuntu/pool/main/d/dh-python/dh-python_1.20140128-1ubuntu8_all.deb
dpkg -i --force-depends dh-python_1.20140128-1ubuntu8_all.deb
apt-get install python3

And in my case I needed to reinstall loads of python3 stuff: apt-get install xubuntu-destkop

Installing virtualenv

sudo pip install --upgrade virtualenv

getting the latest version of virtualenv as any OS packages are likely to be outdated alternatively you can go all out and just use virtualenv locally from source

virtualenv from source

wget https://pypi.python.org/packages/source/v/virtualenv/virtualenv-X.X.tar.gz
tar -xf virtualenv-X.X.tar.gz
cd virtualenv-X.X
/home/ubuntu/python/bin/python2.7 virtualenv.py myvenv

does not require sudo and works around path or permissions requirements

Example Usage

virtualenv myvenv

creates a local copy of required files like the python interpreter and its own version of pip

myvenv
| -- bin
     | -- activate
     | -- easy_install
     | -- pip
     | -- python
| -- include
| -- lib
     | -- python2.7
         | -- site-packages
              | -- pip
              | -- setuptools
| -- local

myenv/bin/pip install --upgrade requests

no sudo was required to add locally myvenv/lib/python2.7/site-packages/requests

myenv/bin/python
>>> import requests
>>> print requests.__version__

Using virtualenv with a specific python version or binary

virtualenv -p python3.5 venv
source venv/bin/activate
python --version
which pip

Python 3.5.0+ the shortest example to use the OS python3.5 binary when creating the venv directory with the virtual environment

virtualenv --python=/home/ubuntu/python/bin/python2.7 myvenv
myenv/bin/python

Python 2.7.8 the advanced example uses a python binary that was created from source to ensure the application does not suffer when the OS has a python upgrade (or your libraries need a newer version of python than provided)

virtualenv --version

just in case your version of virtualenv has a bug and needs to be upgraded first

activate and deactivate to update your environment temporarily

Rather than using the explicit paths (which is the most clear but cumbersome) you can override your shell Environment:

/usr/bin/python --version

2.7.3

source myenv/bin/activate python --version

2.7.8

pip install requests

no sudo was required to add locally myvenv/lib/python2.7/site-packages/requests

deactivate

More Info

When using git make sure .gitignore contains the "myenv" directory as you do not want to store these binaries in version control.

Typically Heroku or other PaaS allow you to specify a python interpreter version and library requirements in a configuration file.

Xubuntu hotkey shortcuts and Zoom (also for xfce)

2014-10-08T16:00:00-07:00

Interacting with the computer is so much faster with keyboard hotkey shortcuts and other tricks, these are applicable as of Xubuntu 14.04

Xubuntu Zoom (magnifier)

Alt + scrollwheel up to Zoom in

or on a laptop with a touchpad two finger swipe up

Alt + scrollwheel down to Zoom out

or on a laptop with a touchpad two finger swipe down

Hotkey Application Shortcuts

Mouse (start button) -> Settings -> Settings Manager -> Keyboard -> Application Shortcuts

Add -> sh -c "sleep 1 && xset dpms force off" -> OK (will open another popup)

press control + alt + q on the keyboard (the popup will go away and the hotkeys will be saved)

Double click on the command in the Command column to edit the command

Double click on the hotkeys in the Shortcut column to modify the hotkey combination

sleep 1 && xset dpms force off control + alt + q
exo-open --launch TerminalEmulator control + alt + t
gnome-calculator control + alt + g
/usr/bin/galculator control + alt + g
/usr/bin/leafpad --tab-width=4 control + alt + f
vi /usr/share/applications/leafpad.desktop Exec=leafpad --tab-width=4 %f
/usr/bin/chromium-browser control + alt + a
/usr/bin/filezilla control + alt + i
/opt/pycharm/bin/pycharm.sh control + alt + p
xfce4-screenshooter -r printscreen
select region (-f fullscreen , -w active window)
amixer set Master 5%- -q Alt + Down
amixer set Master 5%+ -q Alt + Up
xflock4 control + alt + delete
retext control + alt + r

/home/ubuntu/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-keyboard-shortcuts.xml

/etc/xdg/menus/xfce-applications.menu

XFCE Keyboard Shortcuts

Settings Editor -> xfce4-keyboard-shortcuts

commands -> custom... New

/commands/custom/f leafpad

/xfwm4/custom/d

show_desktop_key

/commands/custom/e

mousepad

exo-open --launch FileManager

Ubuntu Keyboard Shortcuts

ubuntu 12.04 keyboard shortcuts

System Settings -> Keyboard (may not be visible so type it in the search box) -> Shortcuts

Either modify an existing shortcut (i.e. disable one that is annoying)

OR Custom Shortcuts -> + (add a new one)

e.g.

name: chromium
command: /usr/bin/chromium-browser

Then highlight the row (clicking on the right area where it's "disabled") and type the key combination desired to trigger the shortcut (e.g. control + alt + a)

HINTS: sudo find / -iname "*chromium*"

sudo which chromium-browser

Static site pelican blog with GitHub Pages and Travis CI

2014-09-15T04:04:00-07:00

Setting up a static blog site I decided to follow some great Dev Ops principles:

Convention over customization
Minimal infrastructure to maintain
Leverage the cloud (from reliable vendors)

Also, being "an efficient engineer" I had the extra hard requirement of "free" =p

Github Setup

Create two new public repos: username.github.io and source.username.github.io

In source.username.github.io you will put the source markdown and theme etc. from the pelican static blog.

The username.github.io will be where the output pelican transformed .html is automatically pushed by Travis CI and is conveniently served by GitHub pages. https://pages.github.com/

Ensure https://travis-ci.org is authorized by clicking on the Configure button, https://docs.travis-ci.com/user/getting-started/

(You can review the OAuth apps with https://github.com/settings/applications)

While the Integration allows Travis CI to detect commits to your repositories it does not necessarily allow it to push changes into a repository, for that we'll use an OAuth Token.

Create a new personal oauth token: https://github.com/settings/tokens/new

scope should be public repos only

The long way is to use the GitHub WebUI -> Applications -> Personal Access Token -> public_repo (only)

Make sure you store the personal access token in a password manager or somewhere safe (i.e. not in plaintext in your email or published on your website ;)

Travis CI Setup

The beauty of these tightly integrated continuous integration systems is that when a commit is pushed into a specific github repo you can trigger some command execution, in this case to convert the markdown to html and then push it to a different repository. (The github pages special repository which is why it must be specifically username.github.io)

I use travis-ci.org which is free for public repos, travis-ci.com is the paid professional service for private repositories

From the Travis Side also "Authorize Application" using https://travis-ci.org/profile/yourusername
Find the list of repositories (you may have to first click "sync now" to see the list)
Slide to ON (checkmark) for the source.username.github.io repository
Click on the gear symbol next to the name of the source.username.github.io repository (should result in the URL https://travis-ci.org/username/source.username.github.io/settings)
Scroll down to Environment Variables - oh but maybe this last step isn't necessary because it is in the YAML file (travis.yml

Travis CLI with Docker to encrypt the OAuth Token

You do not want the unencrypted oauth token in your yaml file or even in the logs. Instead leverage the handy feature of encrypted environment variables by encrypting your oauth token using the Travis CLI (ruby based so...)

There is a special travis requirement of knowing the owner/repo , i.e. username/source.username.github.io

The easy way

docker run -it --rm ruby:alpine /bin/sh
apk add --no-cache build-base
gem install travis travis-lint

The slightly longer way with Ubuntu

docker run -it --rm ubuntu:trusty
apt-get update
apt-get install -y ruby1.9.3 build-essential
sudo gem install travis travis-lint

The actual travis commands...

travis help
travis pubkey -r username/source.username.github.io
travis encrypt GH_TOKEN=your_github_personal_oauth_token -r username/source.username.github.io

Alternatively make a fake yaml file and get the exact output added by the commands: - touch .travis.yml - travis encrypt GH_TOKEN=your_github_personal_oauth_token -r username/source.username.github.io --add env.global - less .travis.yml

.travis.yml

At the root of your source.username.github.io you'll need the Travis configuration file (yaml)

language: python
python:
    - "2.7"
before_install:
 - sudo apt-get update -qq
install:
    - pip install pelican==3.6.3 Markdown==2.6.7 beautifulsoup4==4.5.1
script:
    - rm -rf ./output
    - rm -rf ./cache
    - rm -rf ./plugins/*
    - git clone https://github.com/getpelican/pelican-plugins.git
    - mv ./pelican-plugins/* ./plugins
    - pelican ./content -o ./output -s ./publishconf.py
    - git clone --quiet https://${GH_TOKEN}@github.com/username/username.github.io.git > /dev/null
    - cd johnpfeiffer.github.io
    - git config user.email "me@john-pfeiffer.com"
    - git config user.name "John Pfeiffer"
    - rsync -rv --exclude=.git ../output/* .
    - git add -f .
    - git commit -m "Travis build $TRAVIS_BUILD_NUMBER"
    - git push -fq origin master > /dev/null
    - echo -e "Done\n"
env:
   global:
     secure: example126xOnLRCabGeZrxMUne9W0l5LTbN/hR5Wnq0P3nwrL4slWJ3rFAoi/wqivbINwZGOkU7e/OPVvjDCRivAIxeti61xtnKgyFL6rTvc7u5vAjCF6m4qx6+bXOx9YbXCEUdJmBd25qGBy3PIg4rt/524DOBZhZ9t4glt8Qo=

Use a python2.7 based travis builder with the pelican and its dependencies and the encrypted oauth token

Also helpful: https://lint.travis-ci.org (validate .travis.yml) or gem install travis-lint

The "target" repository is manually cloned using the encrypted oauth token, and the pelican output is then pushed to it. No humans involved!

Using a CNAME to have your own custom domain point to the GitHub Pages Pelican Blog

To ensure maximum coolness (and SEO points) make sure you have DNS control over the domain you have in mind so you can redirect it to your new static pelican-based blog (hosted for free by github pages).

Basically in the GitHub repo settings of source.username.github.io choose "Custom domain"
Then add a CNAME with your DNS provider (i.e. from Namecheap or Cloudflare I pointed blog.john-pfeiffer.com to johnpfeiffer.github.io)
https://help.github.com/articles/using-a-custom-domain-with-github-pages/
https://help.github.com/articles/adding-or-removing-a-custom-domain-for-your-github-pages-site/
https://help.github.com/articles/setting-up-a-custom-subdomain/

Mobile edit cloud execution of python code

2014-09-07T21:37:00-07:00

Haven't you just wanted to work through a coding kata http://codekata.com or puzzle on your phone?

Python is a great language for getting stuff done, and while there are some mobile apps often they are limited by the platform (eg ios sans file system).

Using the link from a Dropbox text file and a linode server (could be openshift red hat cloud?)...

start running the script on the remote server that waits for new code to execute python myflaskapp.py &
I can edit using Nocs
Nocs syncs from iOS to Dropbox
hitting a URL in my browser
- the python script downloads the latest version of the code (using shell to curl as Dropbox use javascript to authorize and return a link to the latest version)
- executes using the remote python environment
- returns the output

Note! This may be dangerous as hackers could exploit to run arbitrary code, use at your own risk.

*Also, clearly, downloading from Dropbox using curl is a hack with no guarantee of future support =]

from flask import Flask
import os
from subprocess import Popen, PIPE
import urllib2

app = Flask(__name__)

@app.route('/mysecreturl')
def update_and_run():
  output=''
  try:
      name = 'exercises.py'
      path = '/var/www/mystuff'
      urlpath = 'https://www.dropbox.com/s/ancdefgrandom/exercises.py?dl=0'
      os.system('curl --silent --location --insecure --output exercises.py https://www.dropbox.com/s/ancdefgrandom/exercises.py?dl=0')      
      output = Popen(["python",name], stdout=PIPE).communicate()[0]
  except Exception as error:
      return str(error)
  return output


if __name__ == '__main__':
    app.run('0.0.0.0', 8080, use_reloader=True)

Flask Application on OpenShift

git clone https://github.com/openshift/flask-example.git

Use the OpenShift WebUI to create an application

On the right of your application the WebUI has a note on how to clone the default repo:

git clone ssh://12345random@appname-domain.rhcloud.com/~/git/appname.git/
cd appname
git remote add upstream -m master git://github.com/openshift/flask-example.git
git pull -s recursive -X theirs upstream master
git push
cd wsgi/
virtualenv venv

vi myflaskapp.py

from flask import Flask
import os
from subprocess import Popen, PIPE
import urllib2


app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"


@app.route('/mysecreturl')
def update_and_run():
  output=''
  try:
      path = '/var/lib/openshift/12345appid/app-root/data/exercises.py'
      urlpath = 'https://www.dropbox.com/s/12345random/exercises.py?dl=0'
      os.system('curl --silent --location --insecure --output ' + path + ' ' + urlpath)
      output = Popen(["python",path], stdout=PIPE).communicate()[0]
      # output = Popen(["touch",path], stdout=PIPE).communicate()[0]
  except Exception as error:
      return str(error)
  return output


if __name__ == "__main__":
    app.run()

More Info

https://www.openshift.com/blogs/beginners-guide-to-writing-flask-apps-on-openshift

Fix Fn screen brightness Ubuntu 14.04 intel graphics

2014-08-26T22:36:00-07:00

I discovered post upgrade that Ubuntu 14.04 has a glaring bug with the Intel graphics card (which was working fine in 12.04), the Fn key no longer controlled the brightness.

sudo su

ls /sys/class/backlight

if it lists intel_backlight then this solution of adding the following should work for you too...

vi /usr/share/X11/xorg.conf.d/20-intel.conf

Section "Device"
    Identifier  "card0"
    Driver      "intel"
    Option      "Backlight"  "intel_backlight"
    BusID       "PCI:0:2:0"
EndSection

Log out and log back in, function keys should now control the brightness again (no more glaring bug!)

Ubuntu Bootable USB, apt-get and dpkg, and the best packages to install

2014-08-20T00:00:00-07:00

If you have a modern computer (BIOS) that can boot from USB it is well worth it since having the latest Ubuntu ISO on DVD tends to pile up extra plastic.

After setting up the Operating System you will need to install some software (packages).

And if you have an SSD drive you will want to optimize your OS to not wear it out unnecessarily.

Write an ISO to usb

Be very careful with sudo or using the root user as this can permanently remove files or render your operating system inoperable.

dd is a low level command that writes bytes directly without any prompts

sudo su
fdisk -l
umount /dev/sdc1

dd if=/home/ubuntu/Desktop/ubuntu-14.04.1-server-amd64.iso of=/dev/sdc

1171456+0 records in
1171456+0 records out
599785472 bytes (600 MB) copied, 260.364 s, 2.3 MB/s

fdisk allows you to see the device (e.g. an 8GB usb stick)

dd will overwrite from the "infile" to the "outfile" so make sure you get that target location correct!

Ubuntu Recovery mode

(which is access to a single root user command line)

boot in recovery mode by using the arrow keys during boot (down to select Recovery)

mount networking
root shell
- mount -o rw,remount /

mount --all # might be needed too

Now you can fix grub or /etc/passwd or free up some hard drive space

package management with apt commands

Debian has precompiled packages of binaries and libraries that can very easily be installed via the command line (or GUI) using Advanced Packaging Tool (APT)

https://wiki.debian.org/Apt

Since apt is a wonderful wrapper/manager of dpkg when you're in doubt most likely there is a dpkg command that will do what you need but it may take a lot of research and 8 parameters to do it ;]

https://en.wikipedia.org/wiki/Advanced_Packaging_Tool

Hint: Ubuntu is based upon Debian

apt-cache

apt-cache search ssh

to find packages with the name ssh

apt-cache search ssh | grep server

if there are too many results pipe to grep to filter down the results

apt-cache show ssh

to show the details about a specific package

apt-cache showpkg ssh

to show more general info about a package

apt-cache depends ssh

to show the package dependencies

Force Apt to use IPv4 to avoid lengthy IPv6 timeouts

sudo apt-get -o Acquire::ForceIPv4=true update
sudo apt-get -o Acquire::ForceIPv4=true install vim
sudo echo 'Acquire::ForceIPv4 "true";' > /etc/apt/apt.conf.d/99force-ipv4

update, then install vim, then save the persistent config to always use ipv4

apt-get

updating package indices with apt-get update

Apt contains indices that need to be updated from the upstream repositories

/etc/apt/sources.list is the main ubuntu repository listing

/etc/apt/sources.list.d is the directory where additional apt repositories can be added (usually from ppa or 3rd party vendors)

http://www.debian.org/mirror/mirrors_full for the Debian package mirror sites
http://packages.ubuntu.com/ for a web ui based search of package details

apt-get clean

/var/cache/apt/archive folder keeps a copy of the downloaded .deb files you will need an internet connection to download again any removed .deb files

rm -rf /var/lib/apt/lists/*

remove the indices in case they have gotten orphaned or corrupted, needs to be followed by apt-get update to repopulate

apt-get update

use /etc/apt/sources.list and /etc/apt/sources.list.d to update the package indices to determine if there are newer packages available

deb file:///file_store/archive trusty main universe

a snippet for how to configure apt to use a local repository (e.g. use reprepro to make a local mirror)

sudo apt-get update -o Dir::Etc::sourcelist="sources.list.d/example.list" -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0"

update only a single repository

apt-cache dump

shows all installed packages

To install netselect, a debian application that allows you to choose the "best" package mirror: sudo apt-get install netselect netselect-apt netselect-apt

installing and force installing with apt

sudo apt-get install --dry-run byobu

simulate what will happen but do not change the system

sudo apt-get install --download-only byobu

packages are retrieved but not installed

sudo apt-get install --yes byobu

install and pre-emptively answer yes to the yes/no prompt

sudo apt-get install --reinstall byobu

reinstall even if the package is installed

sudo apt-get install --fix-broken byobu

attempt to fix broken dependencies

DEBIAN_FRONTEND=noninteractive apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install --reinstall byobu

the most non interactive way to force install a package where all prompts are auto answered such that old configuration files are maintained

http://manpages.ubuntu.com/manpages/precise/man8/apt-get.8.html
https://help.ubuntu.com/community/AptGet/Howto

apt-get upgrade

upgrades to the latest version of existing packages, no new packages (so if the new version has new dependencies nothing happens)

apt-get dist-upgrade

upgrades to the latest version of existing packages and will try to grab any new dependencies as required

apt-get install update-manager-core

newer versions of ubuntu require a helper utility, http://packages.ubuntu.com/trusty/admin/update-manager-core

Before you do a major upgrade of Ubuntu you should bring all packages to the latest version... (apt-get update && apt-get dist-upgrade)

do-release-upgrade -f DistUpgradeViewNonInteractive

non interactive upgrade to a new version of Ubuntu (hold onto your seat!), often requires a reboot after for kernel upgrades

lsb_release -a
cat /etc/lsb_release
uname -a

verify that your system has been upgraded (kernel upgrades often require a reboot to become loaded in memory))

removing packages with apt

apt-get remove wget

uninstall a package

apt-get purge wget

remove the package and all files from disk

apt-get autoremove

attempt to clean up packages that are no longer needed (i.e. old versions of dependencies or unused kernel images)

apt-key

sudo apt-key update

if apt errors: WARNING: The following packages cannot be authenticated

dpkg really manages everything

Underneath apt is dpkg (and similar tools) which actually does all of the hard work but are sometimes hard to use =)

listing and finding packages with dpkg

dpkg -l

lists all of the packages installed (name, version, architecture, description)

dpkg -l | grep foobar

lists all of the packages but filters for something specific (i.e. a prefix or partial match)

dpkg -l packagename > myoutput.txt

lists whether a specific package is installed or not and redirects the output to a file

dpkg --get-selections

lists the package names and the state (installed, uninstalled, etc.)

dpkg-query -f '${binary:Package}\n' -W

lists just the package names, slightly more convenient is apt-cache pkgnames | sort

dpkg -S stdio.h

find a package that contains a specific file

dpkg -c packagename.deb

list the contents of the .deb file

https://wiki.debian.org/ListInstalledPackages

you can also manually inspect /var/lib/apt and /var/lib/dpkg

dpkg logs

vi /var/log/dpkg.log
tail -f /var/log/dpkg.log

in conjunction with apt-get upgrade -y

Installing and removing packages with dpkg

dpkg -i packagename.deb

install the .deb file, dpkg -i *.deb will install all of the .deb files in the current directory

dpkg -i --force depends packagename.deb

installs and turns a dependency error into a warning (i.e. libc6 circular dependency)

dpkg -L packagename

list the locations of the installed files

dpkg -s packagename

shows if the package is installed and information about it, dpkg -s | grep Version or dpkg -l | awk '$2=="packagename" { print $3 }' to only print the version (if it exists)

dpkg -r packagename.deb

remove a package but leave the configuration files, also known as dpkg --remove

dpkg --purge

remove a package and delete all configuration files (even if they have been customized by the user)

dpkg --force-help

to manually install a package (forcefully if synaptic and apt-get are stuck)

mv /var/lib/dpkg/info/postgresql.* /tmp/
dpkg --remove --force-remove-reinstreq postgresql-9.1

do the same for postgresql-common and other packages

apt-get install postgresql-9.1 apt-get purge postgresql-9.1 postgresql-client-9.1 postgresql-common postgresql-client-common

in order to have apt-get remove all of the binaries

Best Ubuntu Packages

as of Utopic 14.10

sudo apt-get update
sudo apt-get install -y byobu build-essential elinks unzip unrar vim wget curl ntp rcconf dialog git-core 
sudo apt-get install -y python-pip && sudo pip install --upgrade pip

pip is the package manager for python packages (different from the debian OS packages) so useful if you do any python development or run python applications An alternative to the usually stale OS pip version is to use the not entirely secure grab the .py file from the internet and run it... wget -qO- https://bootstrap.pypa.io/get-pip.py | sudo python

openssh-server libssl = the secure remote shell service and encryption dependency http://packages.ubuntu.com/search?keywords=openssh-server
build-essential = tools for compiling and building debian packages http://packages.ubuntu.com/lucid/build-essential
byobu = console terminal multi screen (survives network disconnects) http://byobu.co
wget and curl = utilities to download files over the network
elinks = cli browser (just in case your GUI dies and you need to research) http://kmandla.wordpress.com/2011/01/13/a-comparison-of-text-based-browsers
unzip and unrar = utilities to decompress compressed things
ntp = network time protocol client daemon to keep your clock in sync
rcconf = easier way to manage what services start at boot https://packages.debian.org/jessie/rcconf
dialog = user friendly dialog boxes for shell scripts (dependency for rcconf)
git-core = the distributed version control software that is eating the developer world
python-setuptools = Sometimes required to install pip http://pythonhosted.org/setuptools
nano = a simple text editor (much easier than vi/vim for just writing new text)

sudo apt-get install openconnect network-manager-openconnect network-manager-openconnect-gnome

openconnect = opensource compatible with cisco anyconnect vpn https://en.wikipedia.org/wiki/OpenConnect
openvpn = opensource vpn client https://openvpn.net

sudo apt-get install openjdk-8-jre

for just the java runtime (thank goodness not Oracle Sun)
jdk = java development kit
iced-tea-7-plugin = open source java 7 support for browsers
icedtea = open java (plugin = browser java)

openjdk-8-jdk

for the full java development kit - needed for some packages to run correctly

GUI

Xubuntu Desktop is my preferred "lightweight" GUI for Ubuntu: http://xubuntu.org

apt-get install -y chromium-browser pepperflashplugin-nonfree geany keepassx xdiskusage
apt-get install -y arandr rdesktop

chromium-browser = opensource branch/clone of google chrome browser,
- maybe srware.net with privacy badger and adblock plus (fanboy block lists) too?
geany = tabbed text notepad (with syntax highlighting)
keepassx = secure password inventory (has a mini version for iphone as well)
xdiskusage = graphical view of disk space usage by folder and file
arandr = multi display gui config
rdesktop = RDP client
grdesktop = gnome UI for rdesktop

sudo echo "autologin-user=ubuntu" >> /etc/lightdm/lightdm.conf.d/10-xubuntu.conf

Better yet use the UI and just choose auto login ;)

TODO: Disable guest user, *Disable crash reports: apport *

imagemagick = image transformation utility https://en.wikipedia.org/wiki/ImageMagick

convert -resize 50% source.png dest.jpg ; convert -rotate 90 source.jpg dest.jpg
dropbox = cloud file storage

deb http://linux.dropbox.com/ubuntu utopic main sudo apt-key adv --keyserver pgp.mit.edu --recv-keys 5044912E
```
deb http://downloads.hipchat.com/linux/apt stable main
wget -O - https://www.hipchat.com/keys/hipchat-linux.key | apt-key add -
```
apt-get update; apt-get install dropbox hipchat
filezilla = file transfer protocol client (that supports sftp = secure ssh ftp)

music and video

sudo apt-get install ubuntu-restricted-extras vlc

vlc = movies/music
ubuntu-restricted-extras = all of the encumbered with licenses packages to generally just watch or listen to stuff :(

Google Music Manager

If you are you using the web based https://play.google.com/music/listen#/all then you probably want the uploader/downloader:

wget https://dl.google.com/linux/direct/google-musicmanager-beta_current_amd64.deb
dpkg -i google-musicmanager-beta_current_amd64.deb

spotify in ubuntu 15.04

https://www.spotify.com/us/download/linux/

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys D2C19886

trust the spotify repository

libnss3-1d : Depends: libnss3 (= 2:3.17.4-0ubuntu1) but 2:3.19.2-0ubuntu15.04.1 is to be installed spotify is behind the times or only wants to support 14.04 and LTS releases =(

https://launchpad.net/ubuntu/vivid/amd64/libnss3/2:3.17.4-0ubuntu1 https://launchpad.net/ubuntu/wily/amd64/libnss3/2:3.19.2.1-0ubuntu0.15.10.1

dpkg -i libnss3_3.17.4-0ubuntu1_amd64.deb
apt-get install spotify-client
apt-get -f install

spotify: error while loading shared libraries: libgcrypt.so.11: cannot open shared object file: No such file or directory

what fun, the internet explains 15.04 (vivid) and 15.10 (wily) use the new libgcrypt20 so...

wget https://launchpad.net/ubuntu/+archive/primary/+files/libgcrypt11_1.5.3-2ubuntu4.2_amd64.deb
dpkg -i libgcrypt*.deb
apt-get install --reinstall spotify-client

pithos is an open source pandora client

sudo add-apt-repository ppa:pithos/ppa
sudo apt-get install pithos

more codecs and DVD playback

sudo apt-get install ffmpeg gstreamer0.10-plugins-bad lame libavcodec-extra
sudo /usr/share/doc/libdvdread4/install-css.sh

Packages you will probably want to remove

apt-get remove brltty

unless you are using braille on your computer

Other Useful Packages

Heroku CLI

deb http://toolbelt.heroku.com/ubuntu ./
wget -O- https://toolbelt.heroku.com/apt/release.key | apt-key add -
apt-get install -y heroku-toolbelt

Ruby and OpenShift CLI

https://gorails.com/setup/ubuntu/14.04

sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 \ libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libgdbm-dev libncurses5-dev automake libtool bison libffi-dev

source ~/.rvm/scripts/rvm
echo "source ~/.rvm/scripts/rvm" >> ~/.bashrc
rvm install 2.1.2
rvm use 2.1.2 --default
ruby -v
echo "gem: --no-ri --no-rdoc" > ~/.gemrc

gem install rhc

red hat client for openshift

SSD Optimization

Write Logs to tmpfs instead of disk

tmpfs ram (memory) virtual disk will just use memory (which I guess is overly abundant now) instead of wearing out the Solid State Drive

sudo vi /etc/fstab


# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda2 during installation
UUID=b7577587-22db-42f6-95d1-264a24f9dd90 /               ext4    noatime,errors=remount-ro 0       1
tmpfs    /tmp        tmpfs    defaults,noatime    0  0
tmpfs    /var/tmp    tmpfs    defaults,noatime    0  0
tmpfs    /var/log/apparmor    tmpfs    defaults,noatime    0  0
tmpfs    /var/log/apt    tmpfs    defaults,noatime    0  0
tmpfs    /var/log/cups    tmpfs    defaults,noatime    0  0
tmpfs    /var/log/dist-upgrade tmpfs    defaults,noatime    0  0
tmpfs    /var/log/installer tmpfs    defaults,noatime    0  0
tmpfs    /var/log/lightdm  tmpfs    defaults,noatime    0  0
tmpfs    /var/log/unattended-upgrades tmpfs    defaults,noatime    0  0

the tmpfs disks created in my fstab were discovered through trial and error and will differ based on what applications are actually running (Xubuntu!)

an older simpler example causes errors as applications create /var/log/SOMETHING directories during installation and then expect them on boot every time later

/dev/sda1 /     ext4    noatime,errors=remount-ro 0 1

tmpfs    /tmp        tmpfs    defaults,noatime    0  0
tmpfs    /var/tmp    tmpfs    defaults,noatime    0  0
tmpfs    /var/log    tmpfs    defaults,noatime    0  0

here is a list of directories that probably need to be generated for dir in apparmor apt cups dist-upgrade fsck gdm installer news samba unattended-upgrades ; do * mkdir -p /var/log/$dir done*

Customize Grub Boot Options

I prefer seeing my bootup screens so I remove some but add the SSD enhancement

vi nano /etc/default/grub
    GRUB_TIMEOUT=1
    #   GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
    GRUB_CMDLINE_LINUX_DEFAULT="elevator=noop"

sudo update-grub2
cat /boot/grub/grub.cfg
    root=UUID=f0ae2c59-83d2-42e7-81c4-2e870b6b255d ro quiet splash elevator=noop

only prompt for 1 second remove the quiet so the console displays all of the boot information the noop scheduler is a simple FIFO scheduler which is usually optimal for SSD or virtual machines https://en.wikipedia.org/wiki/Noop_scheduler since any OS attempt at optimization may cnoflict with more accurate information from the Disk or Hypervisor update-grub2 is to apply the update https://help.ubuntu.com/community/Grub2 manually verify the changes by examining all of the boot menu options (i.e. find the noop line)

cat /sys/block/sda/queue/scheduler
    [noop] deadline cfq

list what schedulers are available , http://www.linuxhowtos.org/System/iosched.htm, note that noop is selected

Note: the above command needs to be run as root, but sudo does not work with it on my system. Run sudo -i if you have a problem to get a root prompt.)

Good Digital Ocean Droplet Tips

useradd -s /bin/bash -m NEWUSERNAME
usermod -a -G admin NEWUSERNAME
passwd NEWUSERNAME
mkdir -p /home/NEWUSERNAME/.ssh
vi /home/NEWUSERNAME/.ssh/authorized_keys
visudo #includedir /etc/sudoers.d NEWUSERNAME ALL = (ALL) NOPASSWD: ALL

of course set the timezone to UTC and use network time protocol

dpkg-reconfigure tzdata
apt-get update
apt-get install byobu vim ntp
Secure SSH by removing root login with vi /etc/ssh/sshd_config: PermitRootLogin no , PasswordAuthentication no
Optionally change the SSH port to something different from the default: Port 22
/etc/init.d/ssh restart

Use the Digital Ocean Web UI to poweroff and take a snapshot of the fresh system

Revisting Apt Tips

apt list --installed | grep packagename

list what is installed but filter for just one name

apt-get update

refresh from the upstream sources what might be available for install or upgrade

apt install --only-upgrade packagename

upgrade one specific package

ListJS: Sort, Filters, Search and more for HTML lists and tables in Javascript

2014-08-08T21:37:00-07:00

Self contained javascript library to make lists of information awesome!

http://listjs.com/docs

listjs.html

<!DOCTYPE html>
<html>
<head>
<script src="http://listjs.com/no-cdn/list.js"></script>

<style type="text/css">
.list {
  font-family:sans-serif;
  margin:0;
  padding:20px 0 0;
}
.list > li {
  display:block;
  background-color: #eee;
  padding:10px;
  box-shadow: inset 0 1px 0 #fff;
}
.avatar {
  max-width: 150px;
}
img {
  max-width: 100%;
}
h3 {
  font-size: 16px;
  margin:0 0 0.3rem;
  font-weight: normal;
  font-weight:bold;
}
p {
  margin:0;
}

input {
  border:solid 1px #ccc;
  border-radius: 5px;
  padding:7px 14px;
  margin-bottom:10px
}
input:focus {
  outline:none;
  border-color:#aaa;
}
.sort {
  padding:8px 30px;
  border-radius: 6px;
  border:none;
  display:inline-block;
  color:#fff;
  text-decoration: none;
  background-color: #28a8e0;
  height:30px;
}
.sort:hover {
  text-decoration: none;
  background-color:#1b8aba;
}
.sort:focus {
  outline:none;
}
.sort:after {
  width: 0;
  height: 0;
  border-left: 5px solid transparent;
  border-right: 5px solid transparent;
  border-bottom: 5px solid transparent;
  content:"";
  position: relative;
  top:-10px;
  right:-5px;
}
.sort.asc:after {
  width: 0;
  height: 0;
  border-left: 5px solid transparent;
  border-right: 5px solid transparent;
  border-top: 5px solid #fff;
  content:"";
  position: relative;
  top:13px;
  right:-5px;
}
.sort.desc:after {
  width: 0;
  height: 0;
  border-left: 5px solid transparent;
  border-right: 5px solid transparent;
  border-bottom: 5px solid #fff;
</style>



<meta charset=utf-8 />
<title>Existing list</title>
</head>
<body>

<div id="users">
  <input class="search" placeholder="Search" />
  <button class="sort" data-sort="name">
    Sort by name
  </button>

  <ul class="list">
    <li>
      <h3 class="name">Jonny Stromberg</h3>
      <p class="born">1986</p>
    </li>
    <li>
      <h3 class="name">Jonas Arnklint</h3>
      <p class="born">1985</p>
    </li>
    <li>
      <h3 class="name">Martina Elm</h3>
      <p class="born">1986</p>
    </li>
    <li>
      <h3 class="name">Gustaf Lindqvist</h3>
      <p class="born">1983</p>
    </li>
  </ul>

</div>

<script type="text/javascript">
    var options = {
        valueNames: [ 'name', 'born' ]
    };

    var userList = new List('users', options);
</script>

</body>
</html>

ListJS with Pelican and Jinja2

ListJS with the Pelican elegant theme to list all the articles sortable/searchable, default pagination for ListJS set to 1000 items

{% extends "base.html" %}

{% block title %}
All Categories · {{ super() }}
{% endblock title %}

{% block head_description %}
All categories of the {{ SITENAME|striptags }} blog. 
{% endblock head_description %}
{% block content %}


<head>
    <script src="http://listjs.com/no-cdn/list.js"></script>

    <style type="text/css">
        h3 {
            font-size: 16px;
            margin:0 0 0.3rem;
            font-weight: normal;
            font-weight:bold;
        }
        p {
            margin:0;
        }

        input {
            border:solid 1px #ccc;
            border-radius: 5px;
            padding:7px 14px;
            margin-bottom:10px
        }
        input:focus {
            outline:none;
            border-color:#aaa;
        }
        .sort {
            padding:8px 30px;
            border-radius: 6px;
            border:none;
            display:inline-block;
            color:#fff;
            text-decoration: none;
            background-color: #28a8e0;
            height:30px;
        }
        .sort:hover {
            text-decoration: none;
            background-color:#1b8aba;
        }
        .sort:focus {
            outline:none;
        }
        .sort:after {
            width: 0;
            height: 0;
            border-left: 5px solid transparent;
            border-right: 5px solid transparent;
            border-bottom: 5px solid transparent;
            content:"";
            position: relative;
            top:-10px;
            right:-5px;
        }
        .sort.asc:after {
            width: 0;
            height: 0;
            border-left: 5px solid transparent;
            border-right: 5px solid transparent;
            border-top: 5px solid #fff;
            content:"";
            position: relative;
            top:13px;
            right:-5px;
        }
        .sort.desc:after {
            width: 0;
            height: 0;
            border-left: 5px solid transparent;
            border-right: 5px solid transparent;
            border-bottom: 5px solid #fff;
        }
    </style>

</head>
<body>


<div id="article-list">
    <button class="sort" data-sort="date">Sort by date</button>
    <button class="sort" data-sort="title">Sort by title</button>
    <input class="search" placeholder="Search" style="margin-top: 10px; height: 16px;"/>

    <ul class="list">
    {% for category, articles in categories %}
        {% for article in articles %}
        <li>
            <span class="date" style="padding-right: 10px;">
                <time pubdate="pubdate" datetime="{{ article.date.isoformat() }}">{{ article.locale_date }}</time>
            </span>
            <a href="{{ SITEURL }}/{{ article.url }}"><span class="title">{{ article.title }} {%if article.subtitle %}
                <small> {{ article.subtitle }} </small> {% endif %}</span> </a>
        </li>
        {% endfor %}
    {% endfor %}
    </ul>
</div>


{% endblock content %}
{% block script %}
{{ super() }}
<script  language="javascript" type="text/javascript">
    function uncollapse() {
            $(window.location.hash).collapse({
                toggle: true
            })
    }
</script>

<script type="text/javascript" language="JavaScript">
    uncollapse(); 
</script>

<script type="text/javascript">
    var options = {
        valueNames: [ 'date', 'title' ],
        page: 1000
    };
    var hackerList = new List('article-list', options);
    hackerList.sort('date')

</script>

{% endblock script %}

Intro to Amazon AWS Elastic Beanstalk

2014-08-08T20:00:00-07:00

While InfrastructureAsAService lightens the load of Operations, to truly push forward Developers require a frictionless place to deploy applications.

Platform-as-a-Service fills this niche: for this specific example AWS Elastic Beanstalk attempts to create a layer of abstraction on top of existing Amazon technologies. (updated in 2019)

PaaS = no OS management, no ssh required, no chef/puppet scripting, potentially easier (dynamic) scaling

A previous article about a Google PaaS https://blog.john-pfeiffer.com/google-app-engine-python

Installing dependencies (to use the CLI)

The AWS CLI relies heavily on Python. Originally this guide was written for python2 but now the world is all python3.

which pip3
pip3 --version
sudo apt install python3-pip --reinstall

(to fix when it is missing in ubuntu)

pip3 install --upgrade awscli

make sure to be on the latest AWS CLI

pip install awsebcli --upgrade

This might upgrade libraries from the OS that you prefer not to, so you could use --user

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/Welcome.html to ElasticBeanstalk
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-install-advanced.html
export PATH=$PATH:~/.local/bin
Optionally add this to ~/.bashrc , e.g. echo "export PATH=$PATH:~/.local/bin" >> ~/.bashrc

hopefully https://github.com/aws/aws-elastic-beanstalk-cli-setup listing all the dependencies can help if there are any issues

(deprecated) Read about how to use the CLI https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-api-cli.html

AWS account credentials to generate a CLI security file

There is the usual best practice of creating an IAM user (e.g. ebdemo) with a group that has the "AWSElasticBeanstalkFullAccess" attached policy...

And with those credentials...

vim ~/.ssh/awscli

AWSAccessKeyId=Write your AWS access ID
AWSSecretKey=Write your AWS secret key

chmod 600 ~/.ssh/awscli export AWS_CREDENTIAL_FILE=~/.ssh/awscli

Or alternatively use the usual AWS CLI: aws configure which puts things in ~/.aws/credentials

Initial directory and dependency setup

mkdir ebjohnexample
cd ebjohnexample
virtualenv venv --python python3.6
source venv/bin/activate
pip install flask==1.1.1
pip freeze > requirements.txt
vim application.py

A tiny flask python webapp

application.py

from flask import Flask

application = Flask(__name__)

@application.route("/")
def hello():
    return "hi"

if __name__ == "__main__":
    application.run()

python application.py

Serving Flask app "application" (lazy loading)
Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
Debug mode: off
Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

curl 127.0.0.1:5000

Creating an Elastic Beanstalk Project

While Java and .jar files are probably the obvious example I preferred to do this in python =)

The pricing/charges are "only" for the underlying resources, so at first an EC2 instance plus a load balancer...

echo "venv" > .ebignore ; cat .ebignore

ensure elastic beanstalk ignores the local dev dependencies

eb init -p python-3.6 ebjohnexample --region us-east-1

creates a python3.6 elastic beanstalk application named "ebjohnexample"

Ignoring setting up SSH since we really ought to never SSH into application servers , 12factor and all that

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb3-init.html

eb create development

This command can take around 5 minutes to create all the resources

Creating application version archive "app-191020_203345".
Uploading ebjohnexample/app-191020_203345.zip to S3. This may take a while.
Upload Complete.
Environment details for: development
  Application name: ebjohnexample
  Region: us-east-1
  Deployed Version: app-191020_203345
  Environment ID: e-zqag8vpi2p
  Platform: arn:aws:elasticbeanstalk:us-east-1::platform/Python 3.6 running on 64bit Amazon Linux/2.9.3
  Tier: WebServer-Standard-1.0
  CNAME: UNKNOWN
  Updated: 2019-10-21 03:33:48.839000+00:00

    Printing Status:
2019-10-21 03:33:47    INFO    createEnvironment is starting.
2019-10-21 03:33:49    INFO    Using elasticbeanstalk-us-east-1-538676797434 as Amazon S3 storage bucket for environment data.
2019-10-21 03:34:15    INFO    Created load balancer named: awseb-e-z-AWSEBLoa-1LEU58ECO2D9O
2019-10-21 03:34:16    INFO    Created security group named: awseb-e-zqag8vpi2p-stack-AWSEBSecurityGroup-HFU5QRINE35V
2019-10-21 03:34:31    INFO    Created Auto Scaling launch configuration named: awseb-e-zqag8vpi2p-stack-AWSEBAutoScalingLaunchConfiguration-1WRM6IN89BH24
2019-10-21 03:36:04    INFO    Created Auto Scaling group named: awseb-e-zqag8vpi2p-stack-AWSEBAutoScalingGroup-10LS20VBXQLOU
2019-10-21 03:36:04    INFO    Waiting for EC2 instances to launch. This may take a few minutes.
2019-10-21 03:36:20    INFO    Created Auto Scaling group policy named: arn:aws:autoscaling:us-east-1:538676797434:scalingPolicy:a0d4a40a-5060-406b-b3df-8ed43f8834d1:autoScalingGroupName/awseb-e-zqag8vpi2p-stack-AWSEBAutoScalingGroup-10LS20VBXQLOU:policyName/awseb-e-zqag8vpi2p-stack-AWSEBAutoScalingScaleUpPolicy-AQUPYQ179IYF
2019-10-21 03:36:20    INFO    Created Auto Scaling group policy named: arn:aws:autoscaling:us-east-1:538676797434:scalingPolicy:59cd3027-4bca-42a8-8feb-fe342e5b786a:autoScalingGroupName/awseb-e-zqag8vpi2p-stack-AWSEBAutoScalingGroup-10LS20VBXQLOU:policyName/awseb-e-zqag8vpi2p-stack-AWSEBAutoScalingScaleDownPolicy-CMAG2FZD4Y0B
2019-10-21 03:36:20    INFO    Created CloudWatch alarm named: awseb-e-zqag8vpi2p-stack-AWSEBCloudwatchAlarmLow-HWA2PL3FJCDQ
2019-10-21 03:36:20    INFO    Created CloudWatch alarm named: awseb-e-zqag8vpi2p-stack-AWSEBCloudwatchAlarmHigh-14EGWT2CXU238
2019-10-21 03:37:10    INFO    Application available at development.xyxifvqn9z.us-east-1.elasticbeanstalk.com.
2019-10-21 03:37:10    INFO    Successfully launched environment: development

Creates an "environment" (because we like to separate Dev, Test, and Production ... which means auto creating an S3 bucket, Elastic IP, and security group, etc.

You can examine the application in the WebUI:

By using AWS Web Console -> Services -> Elastic Beanstalk -> "development" , to see configuration/status

https://console.aws.amazon.com/elasticbeanstalk/home?region=us-east-1#/launchEnvironment?applicationName=ebjohnexample&environmentId=e-zqag8vpi2p

curl development.xyxifvqn9z.us-east-1.elasticbeanstalk.com

eb status --verbose

Verify the Status and Health

Environment details for: development
  Application name: ebjohnexample
  Region: us-east-1
  Deployed Version: app-191020_203345
  Environment ID: e-zqag8vpi2p
  Platform: arn:aws:elasticbeanstalk:us-east-1::platform/Python 3.6 running on 64bit Amazon Linux/2.9.3
  Tier: WebServer-Standard-1.0
  CNAME: development.xyxifvqn9z.us-east-1.elasticbeanstalk.com
  Updated: 2019-10-21 03:37:10.841000+00:00
  Status: Ready
  Health: Green
  Running instances: 1
      i-0365b4a823823ef5c: InService

Since this was written AWS have created CodeCommit and Deploy for source control and deployment pipelines so even simpler (tighter) integration with Amazon

Alternatively using the AWS Console UI

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/GettingStarted.CreateApp.html

Cleanup and Pricing

eb terminate development

terminate all instances, load balancers, etc. and delete the app from S3 entirely (since s3 costs money too)

Probably having a better name than just development might make it clearer if there are multiple applications in EB

Clearly there is very little operational management required: developers have AWS infrastructure at their fingertips.

The EC2 instances charged per hour cough-cough costs depend on what sizing you use, t2.micro for a low traffic demo project is near nothing...

But the load balancer at around $.02 an hour comes out to about 50 cents a day or at least ~$15 a month even if your application is doing nothing.

https://aws.amazon.com/elasticloadbalancing/pricing/

Logs

AWS Web Console -> Deployment -> Elastic Beanstalk -> App Name -> Logs

Snapshot Logs -> View Log File

notice that ElasticBeanstalk leverages CloudFormation under the hood /opt/elasticbeanstalk/hooks/preinit/03wsgiuser.s also uses virtualenv and pip

Credentials via Environment Variables

Environment Variables help keep configuration out of your code (e.g. access keys/passwords)

mkdir .ebextensions vi .ebextensions/01.config

configs are read and run sequentially

BUT oddly enough .config files in ElasticBeanstalk actually need to be committed to the repo (security fail!)

https://stackoverflow.com/questions/14206760/how-to-set-an-environment-variable-in-amazon-elastic-beanstalk-python

override these placeholders with actual secret values,

AWS Web Console -> Deployment -> Elastic Beanstalk -> App Name -> Configuration -> Software Configuration (Gear Symbol)

 option_settings:
  - option_name: PARAM1
    value: placeholder

More info

container_commands https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_django.html

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_custom_container.html https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html

Docker https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_eb.html

Publish a pelican blog using a Bitbucket POST Webhook

2014-08-08T06:00:00-07:00

Webhooks are an incredibly useful way to tie together disparate network parts, WHEN something happens in one place, it sends a POST HTTP request to another place.

Create the Bitbucket Webhook and Setup a Server to Receive the Webhook

Log in to the Bitbucket WebUI
Choose the repository
Choose to administer the repository (gear symbol) -> Hooks (left menu) , or simply https://bitbucket.org/username/reponame/admin/hooks
Select Hook Type (dropdown) , POST , Add Hook (Button)
Enter your target URL, SAVE
Setup a webserver (easiest might be Bamboo or Jenkins) somewhere
Ensure there is a URL that accepts POST requests
Ensure that when the POST is received it runs the pelican content generation commands to make the new output
Ensure new output is visible

You may notice any existing POST webhooks, i.e. a HipChat notification add-on, listed: https://hipchat-bitbucket.herokuapp.com/commit?client_id=f955ddb5

Flask and Bash source code to publish a pelican static web site

This custom solution requires running that flask app manually, i.e. python mypublish.py It also requires having two repositories, one for the pelican source content, the other repo (i.e. a bitbucket static web site) will only contain the output (.html files)

vi mypublish.py

from flask import Flask
import os
from subprocess import Popen, PIPE

app = Flask(__name__)

@app.route('/someuniquekeyhere', methods=['GET', 'POST'])
def mypublish():
  try:
      output = Popen(["./mypublish.sh"], stdout=PIPE).communicate()[0]
  except Exception as error:
      return str(error)
  return output


if __name__ == '__main__':
    app.run('0.0.0.0', 8443, use_reloader=True)

vi mypublish.sh

#!/bin/bash
git pull
GITMESSAGE=$(git log -n 1)
OUTPUT="../outputreponame.bitbucket.io"
./clean-output.sh "../sourcereponame.bitbucket.io"  # removes all of the old content
echo "$GITMESSAGE"
pelican content
cp -a ./output/* $OUTPUT

rm -rf ./output
rm -rf ./cache
rm -f *.pyc

cd "$OUTPUT"
git add --all ./content
git commit -m "source $GITMESSAGE"
git push

vi clean-output.sh

#!/bin/bash

rm -rf ./output
rm -rf ./cache
rm -f *.pyc

for ITEM in $SOURCE/*
do
  if [ -d "$ITEM" ]; then
    rm -rf "$ITEM"
  else
    rm -f "$ITEM"
  fi
done

More info

Using Vagrant to deploy instances on AWS

2014-07-16T21:12:00-07:00

Vagrant is an infrastructure tool that simplifies deployment, such as virtual machines or in this case Amazon EC2 instances.

Install Vagrant and the Vagrant AWS plugin

Download and install vagrant: https://www.vagrantup.com/downloads

wget https://releases.hashicorp.com/vagrant/1.7.4/vagrant_1.7.4_x86_64.deb
dpkg -i vagrant_1.7.4_x86_64.deb
vagrant --version
vagrant plugin install vagrant-aws

Quickstart Vagrant and VirtualBox with Ubuntu Trusty 14.04

The simple local VirtualBox method was:

virtualbox --help
vagrant init ubuntu/trusty64; vagrant up --provider virtualbox

    ==> default: Box 'ubuntu/trusty64' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
    ==> default: Loading metadata for box 'ubuntu/trusty64'
    default: URL: https://atlas.hashicorp.com/ubuntu/trusty64

this will do all the extra work for you of finding and downloading the "box" and starting it in VirtualBox

Vagrant and AWS EC2 with Ubuntu Precise 12.04

vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box

https://atlas.hashicorp.com/boxes/search

vagrant box list

vi Vagrantfile

Vagrant.configure("2") do |config|
    config.vm.box = "dummy"
    config.vm.provider :aws do |aws, override|
        aws.access_key_id = "YOURACCESSKEY"
        aws.secret_access_key = "YOURSECRETKEY"
        aws.keypair_name = "YOURKEYPAIRNAME"
        aws.ami = "ami-7747d01e"
        aws.instance_ready_timeout = 300
        aws.instance_type = "m4.large"
        aws.tags = {
            "Name" => "MyCloudInstance",
        }
        override.vm.box = "dummy"
        override.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"
        override.ssh.username = "ubuntu"
        override.ssh.private_key_path = "./YOURKEYPAIRNAME.pem"
    end
end

ubuntu/images/ubuntu-precise-12.04-amd64-server-20130204 - ami-7747d01e , no ebs storage - just instance storage , https://cloud-images.ubuntu.com/releases/ and https://atlas.hashicorp.com/boxes/search

vagrant will by default upload all folders and files in your "project" folder where the Vagrantfile is located

vagrant will start with the current working directory and look for a Vagrantfile, then go up one directory until it finds one: https://docs.vagrantup.com/v2/vagrantfile/

vagrant up --provider=aws --debug

--debug is interactive mode and requires pressing enter between every step

Bringing machine 'default' up with 'aws' provider...
==> default: HandleBoxUrl middleware is deprecated. Use HandleBox instead.
==> default: This is a bug with the provider. Please contact the creator
==> default: of the provider you use to fix this.
==> default: Warning! The AWS provider doesn't support any of the Vagrant
==> default: high-level network configurations (`config.vm.network`). They
==> default: will be silently ignored.
==> default: Launching an instance with the following settings...
==> default:  -- Type: m4.large
==> default:  -- AMI: ami-7747d01e
==> default:  -- Region: us-east-1
==> default:  -- Keypair: YOURKEYPAIRHERE
==> default:  -- Block Device Mapping: []
==> default:  -- Terminate On Shutdown: false
==> default:  -- Monitoring: false
==> default:  -- EBS optimized: false
==> default:  -- Assigning a public IP address in a VPC: false
==> default: Waiting for instance to become "ready"...
==> default: Waiting for SSH to become available...
==> default: Machine is booted and ready for use!
==> default: Rsyncing folder: /home/ubuntu/myproject/ => /vagrant

vagrant ssh

the default-easiest-interface way of getting SSH access into the machine

vagrant ssh-config
vagrant ssh-config > vagrant-ssh
ssh -F vagrant-ssh default

alternative interactive ssh session: Use the HostName or AWS EC2 WebUI or http://aws.amazon.com/cli to discover the remote machine IP address using ssh with the vagrant-ssh file output seems the simplest ssh -i YOURKEYPAIRHERE.pem ubuntu@1.2.3.4 ls -ahl /vagrant

exit

vagrant ssh -c "ls -ahl" for a non interactive listing of the home directory

vagrant ssh -c "pidof ntpd | xargs sudo kill -9"

vagrant up , vagrant reload , and vagrant provision will have the AWS provider use rsync to push data to /vagrant

https://docs.vagrantup.com/v2/synced-folders/rsync.html

vagrant stop

UnsupportedOperation => The instance 'i-1295bf39' does not have an 'ebs' root device type and cannot be stopped. (Fog::Compute::AWS::Error)

vagrant destroy

> default: Are you sure you want to destroy the 'default' VM? [y/N] Y
>
> ==> default: Terminating the instance...

vagrant destroy -f

non interactively destroy the instance and avoid the misleading error message: Vagrant is attempting to interface with the UI in a way that requires a TTY

Vagrant provisioning

Allows for automated installation of software bundled into the vagrant up command

vagrant up --provider=aws --no-provision to prevent any provisioning

config.vm.provision "shell", inline: "echo Hello, World"

config.vm.provision "shell", path: "script.sh"

config.vm.provision "shell", path: "https://example.com/script.sh"

http://docs.vagrantup.com/v2/provisioning

http://docs.vagrantup.com/v2/provisioning/shell.html

Advanced Vagrantfile example

# -*- mode: ruby -*-
# vi: set ft=ruby :

if ENV['UPDATEFQDN']
  updatedfqdn=ENV['UPDATEFQDN']
end

$fqdnscript = <<FQDNSCRIPT
echo "I am updating fqdn to #{updatedfqdn}..."
cat /etc/hosts | grep "#{updatedfqdn}" || sudo sed 's/127.0.0.1/127.0.0.1 #{updatedfqdn}/' -i /etc/hosts
hostname | grep "#{updatedfqdn}" || sudo hostname #{updatedfqdn}
FQDNSCRIPT

Vagrant.configure("2") do |config|
    config.vm.box = "dummy"
    config.vm.provider :aws do |aws, override|
        aws.access_key_id = "YOURACCESSKEY"
        aws.secret_access_key = "YOURSECRETKEY"
        aws.keypair_name = "YOURKEYPAIRNAME"
        aws.ami = "ami-7747d01e"
        aws.instance_ready_timeout = 300
        aws.instance_type = "m4.large"
        aws.tags = {
            "Name" => "MyCloudInstance",
        }
        if ENV['bamboo_aws_use_iops']
          aws.block_device_mapping = [{ 'DeviceName' => '/dev/sda1', 'Ebs.VolumeSize' => 100, 'Ebs.VolumeType' => 'io1', 'Ebs.Iops' => 3000 }]
        else
          aws.block_device_mapping = [{ 'DeviceName' => '/dev/sda1', 'Ebs.VolumeSize' => 16, 'Ebs.VolumeType' => 'gp2' }]
        end

        override.vm.box = "dummy"
        override.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"
        override.ssh.username = "ubuntu"
        override.ssh.private_key_path = "./YOURKEYPAIRNAME.pem"
        override.vm.synced_folder "/etc/sslcerts", "/tmp/sourcecode", type: "rsync", create: true, rsync__exclude: ".git/"
    end
    config.vm.provision :shell, :inline => "echo `hostname -f` >> /home/ubuntu/currenthostname.txt"
    config.vm.provision :shell, :inline => $fqdnscript
end

Bringing machine 'default' up with 'aws' provider...
==> default: Running provisioner: shell...
==> default: Running: inline script
stdin: is not a ttty
I am provisioning and updating hostname...

synced_folder is to sync other folders in your filesystem besides the folder with the Vagrantfile

config.vm.hostname does not appear to work on AWS EC2 so the workaround above (|| statements to prevent extra reconfiguration)

Vagrant and EC2 VPC (AMI that does not have/allow sudo)

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
    config.vm.box = "dummy"

    config.ssh.pty = true
    config.vm.synced_folder ".", "/vagrant", disabled: true

    config.vm.provider :aws do |aws, override|

        aws.access_key_id = "YOURACCESSKEY"
        aws.secret_access_key = "YOURSECRETKEY"
        aws.keypair_name = "YOURKEYPAIRNAME"
        aws.ami = "ami-7747d01e"
        aws.instance_ready_timeout = 300
        aws.instance_type = "m4.large"
        aws.tags = {
            "Name" => "MyCloudInstance",
        }
        aws.security_groups = [ "my_aws_security_group_id" ]
        aws.subnet_id = "my_aws_subnet_id"
        aws.associate_public_ip = true

        aws.user_data = "#cloud-boothook\n#!/bin/bash\ntouch /opt/.license/.eula\n"

        override.vm.box = "dummy"
        override.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"

        override.ssh.username = "ubuntu"
        override.ssh.private_key_path = "/home/ubuntu/my_aws.pem"
    end
end

vagrant up --provider=aws --no-provision --debug

INFO vagrant: `vagrant` invoked: ["up", "--provider=aws", "--no-provision", "--debug"]

Bringing machine 'default' up with 'aws' provider...
==> default: Launching an instance with the following settings...
INFO interface: info:  -- Type: m4.large
INFO interface: info: ==> default:  -- User Data: #cloud-boothook
INFO interface: info: ==> default:  -- Assigning a public IP address in a VPC: true
==> default: Waiting for instance to become "ready"...
==> default: Waiting for SSH to become available...

DEBUG ssh: == Net-SSH connection debug-level log END ==
INFO ssh: SSH is ready!
DEBUG ssh: Re-using SSH connection.
INFO ssh: Execute:  (sudo=false)
DEBUG ssh: pty obtained for connection
DEBUG ssh: stdout: export TERM=vt100

**JOHN: cloud-boothook script should have run by now in here**

DEBUG ssh: stdout: logout
DEBUG ssh: Exit status: 0
INFO run_instance: Time for SSH ready: 48.444087982177734
INFO interface: info: Machine is booted and ready for use!

which rsync    
DEBUG ssh: stdout: /usr/bin/rsync

INFO interface: info: Machine not provisioning because `--no-provision` is specified.

After all of that it is safe to either have vagrant provision or vagrant ssh -c "ls -ahl"

Using a pseudo tty is a required workaround if using an AMI that does not support tty / sudo (i.e. Amazon's default Linux AMI)
- https://docs.vagrantup.com/v2/vagrantfile/ssh_settings.html
- https://github.com/mitchellh/vagrant/issues/1482
Disabling the /vagrant synced project folder is nice if you don't automatically want the Vagrantfile and everything in there rsynced to your EC2 instance (and avoids the ugly mkdir -p /vagrant which requires sudo)
AWS User Data can be pushed in via Vagrant which allows for custom scripts / commands / package installation during the EC2 instance boot
- https://help.ubuntu.com/community/CloudInit
- https://cloudinit.readthedocs.org/en/latest/topics/format.html#cloud-boothook
- http://stackoverflow.com/questions/17413598/vagrant-rsync-error-before-provisioning
  
  One of the use cases for an aws.user_data #cloud-boothook script has been to add to /etc/sudoers.d/ (thus avoiding later sudo issues with rsync)

VirtualBox

A really easy A way to start an Ubuntu 14.04 box locally with VirtualBox, the shell provisioner is less elegant than chef/puppet/ansible but gets the job done (installs Docker and Docker Compose)

https://www.virtualbox.org/wiki/Linux_Downloads

Vagrantfile

VERSIONS = {
  'trusty' => {
    'box' => "canonical-ubuntu-14.04",
    'box_url' => "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box",
    'ami' => "ami-018c9568",
  },
}


Vagrant.configure("2") do |config|
    config.ssh.forward_agent = true

    version = VERSIONS[("trusty")]

    config.vm.provider "virtualbox" do |v, override|
      v.customize ["modifyvm", :id, "--memory", ENV['VM_MEMORY'] || 4096]
      v.customize ["modifyvm", :id, "--cpus", ENV['VM_CPUS'] || 2]

      override.vm.network :private_network, ip: "192.168.33.10"
      override.vm.box = version['box']
      override.vm.box_url = version['box_url']
    end

    config.vm.provision :shell, :inline => "sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9"
    config.vm.provision :shell, :inline => "sudo sh -c 'echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list'"
    config.vm.provision :shell, :inline => "sudo apt-get update"
    config.vm.provision :shell, :inline => "sudo apt-get install -y lxc-docker python-pip"
    config.vm.provision :shell, :inline => "sudo pip install --upgrade pip"
    config.vm.provision :shell, :inline => "sudo pip install --upgrade docker-compose"

end

more info

ps aux | grep vagrant

nothing to see here but there is still state for machines started...

vagrant global-status
vagrant global-status --prune
rm -rf .vagrant
rm -rf /home/ubuntu/.vagrant.d

If you use vagrant 1.7 don not be surprised if you see errors related to SSL, 1.6.3 FTW

https://github.com/mitchellh/vagrant-aws

Troubleshooting

vagrant up --provider=aws --debug

ERROR: The provider 'aws' could not be found, but was requested to back the machine 'default'. Please use a provider that exists.

RESOLUTION: try re-installing the vagrant-aws plugin again and immediately running the vagrant up command afterwards

ERROR: Timeout when waiting for SSH , SSH not up: ... The private key to connect to the machine via SSH must be owned

RESOLUTION: chown root:root and chmod 400

ERROR: INFO ssh: SSH not up: #<Vagrant::Errors::SSHAuthenticationFailed: SSH authentication failed! This is typically caused by the public/private keypair for the SSH user not being properly set on the guest VM

RESOLUTION: ensure the correct user, i.e. ec2-user or ubuntu etc. is used in the override.ssh.username to match what the AMI expects

ERROR: sudo: no tty present and no askpass program specified

RESOLUTION: your VM (or more likely, AMI) does not have tty or allow sudo so try using the config.ssh.pty = true (and make sure no provisioning commands require sudo)

Docker Intro install run and port forward

2014-07-10T17:00:00-07:00

Docker is a union file system based layer system (previously leveraging linux lxc containers) for ultra lightweight virtualization/compartmentalization.

Much like AWS cloud servers (api based dynamic deployment that should be tolerant of node failure) and automated deployment/configuration infrastructure (chef or puppet such that cloud servers are created idempotent, remotely and automatically managed at scale), Docker requires a change of mindset.

Docker encourages design of modular, deterministic and defined, single purpose components that are easy to compose into larger services.

As any tool, using it for managing complexity and packaging can be very helpful but it does expose other potential issues (composability, orchestration, security).

Images are the initial templates, each image has a unique ID https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/#content-addressable-storage
Containers are the running virtual machines, each container has a unique ID https://en.wikipedia.org/wiki/Operating-system-level_virtualization
From now on it is assumed you use sudo before any docker command!

Install Docker

https://docs.docker.com/engine/installation/linux/ubuntulinux/

apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
sudo sh -c "echo 'deb https://apt.dockerproject.org/repo ubuntu-trusty main' > /etc/apt/sources.list.d/docker.list"
sudo apt-get update
apt-get install linux-image-extra-$(uname -r)
apt-get install docker-engine
service docker status
docker info

Make sure it lists /var/lib/docker/aufs

OPTIONAL STEP IF YOU HAD AN OLD DOCKER INSTALLATION apt-get purge lxc-docker*

Installing on Mac or Windows

https://www.docker.com/docker-toolbox

Custom docker0 ip range

Fixing the Docker Bridge docker0 taking a huge 172.17.0.1/16 address space...

Private IP address space is not normally a thing to worry about, unless someone does something silly and grabs 65,534 addresses (stare) (docker)

Because the default docker0 bridge seems to cater to organizations that want to run thousands of containers simultaneously local developers need to do the following fix:

apt-get install bridge-utils
service docker stop
ip link set docker0 down
brctl delbr docker0
docker daemon --bip=192.168.239.1/24
ifconfig

Docker will be running interactively so you can see all the fun log messages

You should see docker0 "inet addr:192.168.239.1 Bcast:0.0.0.0 Mask:255.255.255.0"

Assuming you do not have some other purpose for the 192.168.239 range in which case you can change it to something else

To permanently have this custom ip range configuration for docker (assuming you have done the steps above):

Control + C to quit the previously interactive docker daemon
vi /etc/default/docker
DOCKER_OPTS="--bip=192.168.239.1/24"
service docker start
ifconfig

You should see docker0 "inet addr:192.168.239.1 Bcast:0.0.0.0 Mask:255.255.255.0"

More docs to further troubleshoot the docker0 bridge...

Quick Start Summary

docker run --rm busybox /bin/sh -c "echo 'hi'"
hi

docker run -it --rm -e MYVAR=123 busybox env

"run" will pull the image from Docker Hub by default, e injects an environment variable, overrides the Docker Image CMD with "env"

docker run -it --rm --entrypoint=/bin/sh python:latest

run the docker container (from the latest public python image) and start the shell prompt instead of the python interpreter
the entrypoint parameter overrides the Docker Image (in case they do not provide a helpfully overridable CMD)
https://docs.docker.com/engine/reference/builder/#entrypoint

Download a docker image

Official Images are he easiest to experiment with: https://hub.docker.com/explore/

sudo docker pull ubuntu:trusty

(grabs the latest, i.e. 14.04.1) or use a different tag to download a more specific version sudo docker pull ubuntu:12.04.3

CRITICAL WARNING! use the colon and a specific version! downloading all of the ubuntu images by accident sucks =(

sudo docker pull redis:latest

choose the latest for just messing around but... ALWAYS use a specific version to avoid having your dependencies change unexpectedly

Finding what versions of images (tags) you can pull requires using either the UI or the API:

https://hub.docker.com/r/library/redis/tags/
docker pull redis:3
docker pull redis:2.8
docker pull redis:2.6

Many of the tags are synonyms/symlinks, so latest is the same as 3 is the same as 3.0

Remove a docker image

docker rmi redis

docker rmi -f $(docker images --all --quiet | grep -v 5506de2b643b)

remove ALL images except one by taking the output (quiet means only image ids), excluding a specific one, and then force removing the images (by id)

docker images --quiet --filter "dangling=true" | xargs docker rmi

remove all images that do not have a tag and are not a parent of a tagged image

docker rmi -f $(docker images --all --quiet)

remove ALL images

du -sh /var/lib/docker/aufs

Summarize the amount of disk space taken by images and layers, e.g.: 72K /var/lib/docker/aufs/

Sometimes a docker image is still connected to a container (already exited or forgotten)

docker ps -a
docker rm name_or_id

docker rm a1b2
docker rmi image_id

it will "smart match" the first characters of the container ID the same as git short sha

Docker info

docker

get a helpful list of all the commands

docker --version

docker info

:::text
Containers: 1
Images: 23
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 25
Execution Driver: native-0.2
Kernel Version: 3.13.0-35-generic
Operating System: Ubuntu 14.04.1 LTS
WARNING: No swap limit support

du -sh /var/lib/docker/aufs

469M /var/lib/docker/aufs/

docker ps --all

no containers are running yet

docker ps --help

docker images --tree is a deprecated command to view the hashes and sizes of all of the parent images

Controlling Containers

Starting a container from an image

Create a container

First you must create a container from an image:

docker run

get a helpful list of how to run a container

docker run --rm -i -t ubuntu:14.04 /bin/bash

:::text
creates a container
--rm: automatically remove the container when it exits
-i: keep stdin open even if not attached
-t: allocate a tty, attach stdin and stdout
use the ubuntu 14.04 minimal image
Docker automatically gives the container a random name
Runs an interactive bash shell

This will continue to exist in a stopped state once exited (see "docker ps -a")

root@f5878ed6016e:/# cat /etc/issue
root@f5878ed6016e:/# uname -a
root@f5878ed6016e:/# df -h

Control-p then Control-q to detach the tty without exiting the shell

docker ps

docker run --detach --name myapp -p 127.0.0.1:5000:5000 training/webapp python app.py

detached with port 5000 available only to the host and executing the command python with parameter app.py

docker exec myapp ls -ahl

runs the ls command inside the container named "myapp"

docker run --rm -i -t --link myhipchat_mariadb_1 mariadb:5 /bin/bash -c "exec mysql --version"
docker run --rm -i -t --link myhipchat_mariadb_1:mysql mariadb:5 /bin/bash -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p'

an Image can contain both the server and client code so run a "client container" to connect to a running server Container

Start (resume) a container

After a container has already been created (which starts it so ironically this is actually a "restart")

docker start --interactive --attach container_id_or_name

Attaching to a running container

docker ps -a 
CONTAINER ID ... STATUS        NAMES
9e0ebf4421dd ... Up 6 seconds  myexample

docker attach 9e0ebf4421dd
docker attach myexample

since the above will expect the container to have /bin/bash it will reuse the instance of shell

sudo docker exec -i -t 9e0ebf4421dd /bin/bash
# ps aux
# exit

instead a new /bin/bash is executed inside creating a second shell - use the exit command to not leave it around

Running a container with admin privileges

If your container requires elevated privileges from the host then you need to provide the extra parameter:

docker run --rm -it --cap-add SYS_NET_ADMIN alpine

Alternatively give it all extended privileges and permissions:

docker run --rm -it --privileged alpine

https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

Stopping a container

Part of the efficiency in docker is that containers can run in the background automatically

docker run --detach --name myredis redis
docker ps
docker stop myredis

Another efficiency is that a docker container will only run as long as it takes to execute a command (and any changes are not forgotten)

docker run ubuntu:trusty uname -a

this runs the container only as long as it takes to execute the command

docker attach f5878ed6016e

Control + C (now we have exited the container and it will clean itself up)

docker ps -a

spun up another container but only long enough to run the command

CONTAINER ID IMAGE        COMMAND      CREATED      STATUS    PORTS         NAMES
e4b436320442 ubuntu:14.04 -uname -a  3 minutes ago              elegant_engelbart

Copying a file out of a container

docker cp <containerId>:/file/path/within/container /host/path/target

https://docs.docker.com/engine/reference/commandline/cp/

Deleting aka removing a container

docker rm e4b436320442

Alternatively: docker rm --force elegant_engelbart

docker rm -f $(docker ps -a -q)

Deletes forcibly all containers (be careful!)

Dockerfile to automate building an image

Dockerfiles allow automating the creation docker images.

One advantage to Dockerfile is that each command creates a separate layer so if a specific layer fails all of the previous intermediate images can be re-used.

Also the version style of imagename:tag allows for chaining of upgrades of child images

Containers as fast, reliable, and deterministic prod/qa/dev environments can also be extended to be just an improved experimentation sandbox (for those used to SSH and using Linux as a common base OS).

mkdir -p dockerfiles/trustyssh
vi dockerfiles/trustyssh/Dockerfile

one Dockerfile per directory

FROM ubuntu:trusty
MAINTAINER John Pfeiffer "https://bitbucket.org/johnpfeiffer"

RUN apt-get update -y
RUN apt-get install -y openssh-server byobu
RUN mkdir /var/run/sshd
RUN echo 'root:root' | chpasswd
RUN echo 'ubuntu:ubuntu' | chpasswd
RUN sed -ri 's/^PermitRootLogin\s+.*/PermitRootLogin yes/' /etc/ssh/sshd_config
RUN sed -ri 's/UsePAM yes/#UsePAM yes/g' /etc/ssh/sshd_config
EXPOSE 22
CMD    ["/usr/sbin/sshd", "-D"]


docker build --tag=newimagename --rm ./dockerfiles/ubuntu-trusty-ssh

Each RUN command creates an intermediate container, so make sure you use the -rm option

Sending build context to Docker daemon  2.56 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:trusty
 ---> 5506de2b643b
...
Removing intermediate container 135b686d82a6
Step 2 : RUN apt-get update -y
 ---> Running in 7fe06a9ef41b
Ign http://archive.ubuntu.com trusty InRelease
...

Since each run command creates a new layer it is best practice to consolidate commands for a single logical action when possible:

RUN echo "#!/bin/bash" > /root/build-and-run.sh && \
  echo "cd /opt/mydata/pelican-project" >> /root/build-and-run.sh && \
  echo "pelican content -r &" >> /root/build-and-run.sh && \
  echo "cd /opt/mydata/pelican-project/output" >> /root/build-and-run.sh && \
  echo "python -m SimpleHTTPServer" >> /root/build-and-run.sh && \
  chmod +x /root/build-and-run.sh


docker history ubuntu-utopic-pelican:latest

view the history of hashes (which can be run by themselves) and timestamps and sizes

IMAGE               CREATED             CREATED BY                                      SIZE
39ac388d3c0d        37 seconds ago      /bin/sh -c #(nop) CMD [/bin/bash /root/build-   0 B
218edf407f18        37 seconds ago      /bin/sh -c echo "#!/bin/bash" > /root/build-a   129 B
d9d774d344bd        2 weeks ago         /bin/sh -c #(nop) EXPOSE 8000/tcp               0 B
ae1733e0e1b9        2 weeks ago         /bin/sh -c pip install pelican Markdown beaut   20.64 MB
24561ed8052f        2 weeks ago         /bin/sh -c curl https://bootstrap.pypa.io/get   9.826 MB
1878a9a052eb        2 weeks ago         /bin/sh -c apt-get update && apt-get install    60.85 MB
5e5e0e9171da        2 weeks ago         /bin/sh -c #(nop) MAINTAINER John Pfeiffer "h   0 B
78949b1e1cfd        7 weeks ago         /bin/sh -c #(nop) CMD [/bin/bash]               0 B
21abcc4ef877        7 weeks ago         /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/   1.895 kB
f552c527d701        7 weeks ago         /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   215 kB
c4c77a6165f9        7 weeks ago         /bin/sh -c #(nop) ADD file:24ed1895f2e500dcec   194.2 MB
511136ea3c5a        22 months ago                                                       0 B

experimental: docker save 49b5a7a88d5 | sudo docker-squash -t ubuntu-utopic-pelican:squash | docker load

Add a port to a container

docker run --detach --publish 127.0.0.1:2222:22 --name johnssh trustyssh

run a detached container based on the trustyssh image that binds the private container port 22 to the host port 2222 (in this case the trustyssh image from above is running sshd on port 22)

docker port johnssh 22

127.0.0.1:2222

netstat -antp | grep 2222

tcp 0 0 127.0.0.1:2222 0.0.0.0:* LISTEN 24393/docker-proxy

ssh -o StrictHostKeychecking=no -p 2222 root@127.0.0.1

root@19ad0614b237:~#

command	output
`docker run --detach --publish 0.0.0.0:6379:6379 --name redis redis`	feeb79581810a8c182202c73d4e1c6b905960bcfc860e04285f1ae03c6a47f18
`docker port redis`	6379/tcp -> 0.0.0.0:6379
`docker port redis 6379`	0.0.0.0:6379
`redis-cli -h docker.example.com ping`	PONG
`docker run --detach --publish-all --name redis redis`	ff2f6d2e04d565f11d71664bf6cf23638656d9b633e4d3c94444c81b18b807bb
`docker port redis`	6379/tcp -> 0.0.0.0:49153
`docker port redis 6379`	0.0.0.0:49153

docker run --detach -P --name johnssh trustyssh

publish all EXPOSE'd ports to random ports on the host

Logs from the containers

docker logs container_name

view the latest logs of a specific container in stdout

docker logs -f container_name

tail the logs for

Host data with a Docker Container

Volumes are where Docker Containers can access storage (either from the Host or other Containers)

https://docs.docker.com/userguide/dockervolumes

docker run --interactive --tty --name mydata --volume /tmp/mydata:/opt/mydata trustyssh /bin/bash

create an interactive container named "mydata" that maps /tmp/mydata from the host onto /opt/mydata (warning: overriding any existing!)

Managing or limiting the resources available to a Container

https://docs.docker.com/engine/reference/run/#runtime-constraints-on-resources

docker run -i -t --rm --cpuset-cpu 0 --memory 512m ubuntu:14.04 /bin/bash

the usual ubuntu trusty bash prompt but anything we run will be pinned to cpu 0 (i.e. 25% of a 4 core system) and have at most 512 MB of RAM and 512 MB of swap available

Using Docker for a GUI application

Mostly outside of the vision of containerization (neither isolation nor performance exactly) is using Docker to run GUI applications without installing them on the Host.

Jessie Frazelle has done some excellent work pointing out how sharing the X11 socket from the host means lots of apps can run "without being installed" https://github.com/jfrazelle/dockerfiles

One tip she did not include was the part about XManager security, run the following if you run into an error

xhost local:root - https://www.netsarang.com/knowledgebase/xmanager/3898/xhost_and_how_to_use_it - http://www.x.org/archive/X11R6.8.0/doc/xhost.1.html

xhost local:root; docker run --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY --device /dev/snd jess/chromium - modify the xhost security to allow access to x windows - ephemeral so do not save anything - share the X11 unix socket - bind to the current display (i.e. :0.0) - allow sound from the Docker container

Saving a docker container as a new image

docker commit --help

docker commit for a container: ubuntu:trusty with git

docker run -i -t ubuntu:trusty /bin/bash
Control + p then Control + q (to detach the tty without exiting the shell)
docker ps -a (make a note of the ID or NAME)
docker attach ID_OR_NAME
apt-get update; apt-get install git -y
cd /root
git --version
git clone https://johnpfeiffer@bitbucket.org/johnpfeiffer/myrepo.git
exit
exit

now the Container will have git installed, a repo cloned, and will be stopped
docker commit container_name_here johnpfeiffer_git_repo

4a74440186d976caeccc52f5ed2bd44269beb84d472391a7ce26ee3db8ffc1e9
docker images

output

REPOSITORY               TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
johnpfeiffer_git_repo    latest              4a74440186d9        54 seconds ago      402 MB
ubuntu                   14.04               e54ca5efa2e9        3 weeks ago         276.5 MB

Misc

Python and Pelican static site generator on docker ubuntu:14.04

apt-get update; apt-get install python python-setuptools openssl wget -y
wget -qO- https://raw.github.com/pypa/pip/master/contrib/get-pip.py | sudo python
pip install pelican Markdown beautifulsoup4
docker run -i -t -p 127.0.0.1:8000:8000 --name john johnpfeiffer_git_repo:latest /bin/bash
control+p then control+q
docker attach john
cd /root/myrepo
pelican content (assuming pelicanconf.py is here)
cd output
python -m SimpleHTTPServer
control+p then control+q
docker ps -a
http://127.0.0.1:8000

Docker pre built Images from the Registry

Redis

docker search redis

search from the CLI but NAME, DESCRIPTION, STARS, OFFICIAL will only help you if you sort of already know what you are looking for

https://registry.hub.docker.com/_/redis/

docker run --detach --publish 127.0.0.1:6379:6379 --name redis redis

detached container based on the redis latest bound to the host on port 6379

the image already includes by default the expose port command: EXPOSE 6379... apt-get install redis-tools; redis-cli

connect from the host to the redis container to the redis interactive cli

docker run -i -t --link redis:db  johnssh /bin/bash

root@d95a758eaa6b:/#
apt-get install redis-tools
env
redis-cli -h $DB_PORT_6379_TCP_ADDR
ping

PONG

get mykey

"somevalue"

docker stop johnssh

docker start --interactive --attach johnssh

root@d95a758eaa6b:/#

Environment variables are tricky

sudo docker run --detach --publish 127.0.0.1:2222:22 --name johnssh trustyssh

if you SSH into your container or use byobu "env" will not show you the info

**From the host: ** sudo docker inspect --format '{{ .NetworkSettings.IPAddress }}' redis

172.17.0.72

ssh -p 2222 root@localhost
apt-get install redis-tools
redis-cli -h 172.17.0.72 keys *
1) mykey

** Add a route to the Host From inside a Container/Guest **

HOST:

ip addr show
ip addr show | grep docker0
ip addr show | grep docker0 | grep global | awk '{print $2}' | cut -d / -f1
HOSTIP=$(ip addr show | grep docker0 | grep global | awk '{print $2}' | cut -d / -f1)

sudo docker run -i -t --rm --add-host=docker:${HOSTIP} python:2 /bin/bash

root@1bbe25092f19:/# cat /etc/hosts

172.17.0.7      1bbe25092f19
127.0.0.1       localhost
 ::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

172.17.42.1     docker

root@1bbe25092f19:/# ping docker

PING docker (172.17.42.1): 56 data bytes
64 bytes from 172.17.42.1: icmp_seq=0 ttl=64 time=0.158 ms

https://docs.docker.com/engine/reference/commandline/run/

Docker Compose

Complex real systems have multiple dependencies and following the recommended Docker pattern of "do one thing per container" means needing a way to start/orchestrate a bunch of things at once.

While there are some amazing open source projects (http://kubernetes.io/ , https://mesos.apache.org/documentation/latest/docker-containerizer/) it is instructive to start with the simplest model provided directly from Docker, https://docs.docker.com/compose/

Either download the docker-compose binary directly (to /usr/local/bin): - https://github.com/docker/compose/releases

sudo curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose

Or use pip/python to install it...

sudo pip install --upgrade docker-compose

which docker-compose ; sudo docker-compose --version

app.py

from flask import Flask
from redis import Redis
import os
app = Flask(__name__)
redis = Redis(host='redis', port=6379)

@app.route('/')
def hello():
    redis.incr('hits')
    return 'Hello World! I have been seen %s times.' % redis.get('hits')

if __name__ == "__main__":
    app.run(host="0.0.0.0", debug=True)

requirements.txt

flask
redis

Dockerfile

FROM python:2.7
ADD . /code
WORKDIR /code
RUN pip install -r requirements.txt
CMD python app.py

docker-compose.yml

web:
  build: .
  ports:
   - "5000:5000"
  volumes:
   - .:/code
  links:
   - redis
redis:
  image: redis

dependencies like redis are "linked" for network access (ordering in the file is important)
"web" app (which comes from the current directory Dockerfile)
ports: exposes ports to the host (all of these are just like the docker run CLI parameters)
volumes: allows live editing of app.py
redis comes from the public docker registry image (not a Dockerfile nor a private registry) and is using the default tag of "latest"

docker-compose up

docker-compose up

this will docker pull redis and build from the Dockerfile and then start them all in the correct order

docker-compose ps

run this command in the directory with the docker-compose.yml to see the state of the system

         Name                      Command             State           Ports          
-------------------------------------------------------------------------------------
composeexample_redis_1   /entrypoint.sh redis-server   Up      6379/tcp               
composeexample_web_1     /bin/sh -c python app.py      Up      0.0.0.0:5000->5000/tcp

docker ps -a

docker-compose stop

http://localhost:5000/

Hello World! I have been seen 2 times.

https://bitbucket.org/johnpfeiffer/docker/src

Troubleshooting

Building images with Dockerfile

Building images often depends on the network and DNS

If you are using Wifi be careful as intermittent network connectivity may cause frustrating issues.

For DNS with docker installed onto ubuntu via apt-get, try changing to Google DNS by uncommenting in:

vi /etc/default/docker

sudo service docker restart

Building images often depends on dependencies look closely at error messages, i.e. make: not found and ensure that an early RUN statement has apt-get update && apt-get install -y build-essential

Discovering an Image Parent from History

To see all of the images in detail: docker images --all --digests=true

python      2.7-alpine      sha256:834b44717d3928266472066d82208241a2582ce7b3787f32abbfb0def9fb0324   c80455665c57        6 weeks ago         71.49 MB
alpine      latest          sha256:dfbd4a3a8ebca874ebd2474f044a0b33600d4523d03b0df76e5c5986cb02d7e8   88e169ea8f46        6 weeks ago         3.984 MB

And then you can brute force query to see if an image has child dependencies:

for i in $(docker images -q); do docker history $i | grep -q 7fb9bd20d612 && echo $i; done

More Info

Real World example of using Docker for behind the firewall delivery: https://bitbucket.org/atlassianlabs/ac-koa-hipchat-sassy/pull-request/6/readmemd-contains-instructions-on-how-to/diff
https://github.com/wsargent/docker-cheat-sheet
http://jpetazzo.github.io/2014/06/23/docker-ssh-considered-evil

Docker API and the Docker Hub Public Registry

Docker used to have an API endpoint at registry.hub.docker.com/v1 but in a fairly typical move for them they changed it so a lot of internet "documentation" examples are wrong.

Also the Docker Hub was deprecated as of 1.7 so this is the last docs on how to use it (because the service is still running or compatible)

curl https://index.docker.io/v1/_ping
true

It uses basic authentication and while you can use the API to sign up it may just be easier to use the web site: https://hub.docker.com/

curl https://username:password@index.docker.io/v1/users/
"OK"

just curl with basic auth in order to check the credentials, the trailing slash is IMPORTANT

Using a browser with the Docker REST API is often more convenient as it caches the Basic Authentication

https://docs.docker.com/v1.7/reference/api/docker-io_api/#users
https://docs.docker.com/v1.6/reference/api/registry_api/ (because the registry api was deprecated earlier?)

To see all of the images for a given repository (it is json formatted and there will be a lot of results!) curl https://username:password@index.docker.io/v1/repositories/python/images

https://docs.docker.com/v1.6/reference/api/registry_api/

If you docker search python and want to see the tags (i.e. you do not want to pull every python image ever made), then try curl with REST:

curl https://username:password@index.docker.io/v1/repositories/python/tags

[{"layer": "a2db1214", "name": "latest"}, {"layer": "edb21ec7", "name": "2"}, {"layer": "82b600dd", "name": "2-alpine"}, {"layer": "6a4e9662", "name": "2-onbuild"}, {"layer": "99b38a11", "name": "2-slim"}, {"layer": "fe724fa0", "name": "2-wheezy"}, {"layer": "edb21ec7", "name": "2.7"}, {"layer": "82b600dd", "name": "2.7-alpine"}, {"layer": "6a4e9662", "name": "2.7-onbuild"}, {"layer": "99b38a11", "name": "2.7-slim"}, {"layer": "fe724fa0", "name": "2.7-wheezy"}, {"layer": "c71c2739", "name": "2.7.10"}, {"layer": "f1f35fa4", "name": "2.7.10-onbuild"}, {"layer": "843123ac", "name": "2.7.10-slim"}, {"layer": "fde41dc3", "name": "2.7.10-wheezy"}, {"layer": "edb21ec7", "name": "2.7.11"}, {"layer": "82b600dd", "name": "2.7.11-alpine"}, {"layer": "6a4e9662", "name": "2.7.11-onbuild"}, {"layer": "99b38a11", "name": "2.7.11-slim"}, {"layer": "fe724fa0", "name": "2.7.11-wheezy"}, {"layer": "a87a2288", "name": "2.7.7"}, {"layer": "481b175a", "name": "2.7.8"}, {"layer": "fbb30ed2", "name": "2.7.8-onbuild"}, {"layer": "3cf7f142", "name": "2.7.8-slim"}, {"layer": "6a873836", "name": "2.7.8-wheezy"}, {"layer": "2d0d0130", "name": "2.7.9"}, {"layer": "10948f7c", "name": "2.7.9-onbuild"}, {"layer": "e86252d0", "name": "2.7.9-slim"}, {"layer": "a11d441b", "name": "2.7.9-wheezy"}, {"layer": "a2db1214", "name": "3"}, {"layer": "bb6cd371", "name": "3-alpine"}, {"layer": "80662aa6", "name": "3-onbuild"}, {"layer": "07bfefb9", "name": "3-slim"}, {"layer": "2edf9614", "name": "3-wheezy"}, {"layer": "7575f4a5", "name": "3.2"}, {"layer": "31b273f6", "name": "3.2-onbuild"}, {"layer": "ca0a0ed6", "name": "3.2-slim"}, {"layer": "f5644650", "name": "3.2-wheezy"}, {"layer": "7575f4a5", "name": "3.2.6"}, {"layer": "31b273f6", "name": "3.2.6-onbuild"}, {"layer": "ca0a0ed6", "name": "3.2.6-slim"}, {"layer": "f5644650", "name": "3.2.6-wheezy"}, {"layer": "84717b99", "name": "3.3"}, {"layer": "4de0c1a0", "name": "3.3-alpine"}, {"layer": "e0985e72", "name": "3.3-onbuild"}, {"layer": "3c0f39af", "name": "3.3-slim"}, {"layer": "a13ad718", "name": "3.3-wheezy"}, {"layer": "e663e96e", "name": "3.3.5"}, {"layer": "79d3367e", "name": "3.3.5-onbuild"}, {"layer": "84717b99", "name": "3.3.6"}, {"layer": "4de0c1a0", "name": "3.3.6-alpine"}, {"layer": "e0985e72", "name": "3.3.6-onbuild"}, {"layer": "3c0f39af", "name": "3.3.6-slim"}, {"layer": "a13ad718", "name": "3.3.6-wheezy"}, {"layer": "c7184f4f", "name": "3.4"}, {"layer": "e6310f15", "name": "3.4-alpine"}, {"layer": "c38d9f7b", "name": "3.4-onbuild"}, {"layer": "ab9f7f65", "name": "3.4-slim"}, {"layer": "c98c4a9d", "name": "3.4-wheezy"}, {"layer": "b504e00c", "name": "3.4.1"}, {"layer": "07e5901a", "name": "3.4.1-onbuild"}, {"layer": "ec50e6a0", "name": "3.4.2"}, {"layer": "ade8543e", "name": "3.4.2-onbuild"}, {"layer": "dd1dee45", "name": "3.4.2-slim"}, {"layer": "de6911d6", "name": "3.4.2-wheezy"}, {"layer": "48bc52cc", "name": "3.4.3"}, {"layer": "bf599bc6", "name": "3.4.3-onbuild"}, {"layer": "0b92f173", "name": "3.4.3-slim"}, {"layer": "b8845e5b", "name": "3.4.3-wheezy"}, {"layer": "c7184f4f", "name": "3.4.4"}, {"layer": "e6310f15", "name": "3.4.4-alpine"}, {"layer": "c38d9f7b", "name": "3.4.4-onbuild"}, {"layer": "ab9f7f65", "name": "3.4.4-slim"}, {"layer": "c98c4a9d", "name": "3.4.4-wheezy"}, {"layer": "a2db1214", "name": "3.5"}, {"layer": "bb6cd371", "name": "3.5-alpine"}, {"layer": "80662aa6", "name": "3.5-onbuild"}, {"layer": "07bfefb9", "name": "3.5-slim"}, {"layer": "c64596cb", "name": "3.5.0"}, {"layer": "c9744d7e", "name": "3.5.0-onbuild"}, {"layer": "ac60c7d8", "name": "3.5.0-slim"}, {"layer": "31ef8f64", "name": "3.5.0b3"}, {"layer": "7c5e081c", "name": "3.5.0b3-onbuild"}, {"layer": "0c47d2de", "name": "3.5.0b3-slim"}, {"layer": "a2db1214", "name": "3.5.1"}, {"layer": "bb6cd371", "name": "3.5.1-alpine"}, {"layer": "80662aa6", "name": "3.5.1-onbuild"}, {"layer": "07bfefb9", "name": "3.5.1-slim"}, {"layer": "bb6cd371", "name": "alpine"}, {"layer": "80662aa6", "name": "onbuild"}, {"layer": "07bfefb9", "name": "slim"}, {"layer": "2edf9614", "name": "wheezy"}]

Docker Engine internal API

If instead of the docker client you wish to interact more programatically...

Private Docker Registry

curl -i https://username:password@docker.example.com/v2/

attempt to login to a private registry

** Using the docker client to login to a private registry **

docker login docker.example.com:443
> Username: user@example.com
> WARNING: login credentials saved in /home/USER/.docker/config.json
> Login Succeeded

private registry basics via the browser

How to deploy your own docker registry: https://docs.docker.com/registry/deploying/

https://docker.example.com/_ping

{}

https://docker.example.com/info
https://docker.example.com/version

note these commands may not be enabled or available in your private registry version

https://docker.example.com/v1/repositories/library/ubuntu/tags

{"13.04": "5e47ac691989afcd10285ea4e67b46bc0fdc98d90844e57a6d4221c1e3ab4388"}

https://docker.example.com/v1/repositories/micros/baseimage-ubuntu/tags

{"latest": "5a14c1498ff4983793f6e5eddd16868dbad257195f0e85c66ece94d881ecb28f"}

https://docker.example.com/v1/repositories/micros/baseimage-ubuntu/images

list of the images available: [{"id":"8254ff58b098b72425854555204171352a69f5427ba83dee4642ba45d301d0b1"}]

https://docker.example.com/v1/repositories/micros/baseimage-ubuntu/json

inspect an image (what OS, kernel, etc.) {"arch": "amd64", "docker_go_version": "go1.3.3", "docker_version": "1.3.3", "kernel": "3.16.7-tinycore64", "last_update": 1426041024, "os": "linux"}

https://docker.example.com/v1/repositories/myuser/nginx/0348bf1e7cc54327b8c9ce8407c5b3eadade1ef1771d642d08ae16a6aad5bed5/json

inspect a very specific image (by id)

Searching a private docker registry

https://registry.hub.docker.com/v1/search?q=pfeiffer

the public docker registry search query, deprecated in APIv2 so no longer functional

docker search docker.example.com/myuser

the cli command returns a listing of all of the images for a user, deprecated in APIv2 so no longer functional

If there is a proxy in front: docker search user:password@docker.example.com/myuser

curl -s -X GET https://user:password@docker.example.com/v1/search

LIST ALL IMAGES: or use a browser https://docker.example.com/v1/search

curl -X GET https://user:password@docker.example.com/v1/search?q=ubuntu

https://docker.example.com/v1/search?q=ubuntu {"num_results": 4, "query": "ubuntu", "results": [{"description": null, "name": "example/ubuntu"}, {"description": null, "name": "library/ubuntu"}, {"description": null, "name": "micros/baseimage-ubuntu-ansible"}, {"description": null, "name": "micros/baseimage-ubuntu"}]}

https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-ubuntu-14-04

Markdown syntax cheatsheet

2014-07-02T20:21:00-07:00

Markdown Syntax

html <em> is markdown *emphasis* or _italics_ = emphasis or italics
html <strong> is markdown **strong* or __bold__ = strong or bold
html strikethrough is not supported but can just be <del>strikethrough</del> = ~~strikethrough~~
html <blockquote> is markdown >

at the start of each line
html unordered list <ul> = - item or alternatives: + item , * item
html <hr /> is markdown - - - or alternatives: * * * , *** , *****
html <a href= for hyper links is:

<http://blog.john-pfeiffer.com> converts into a link that is automatically turned clickable:

<a href="http://johnpfeiffer.bitbucket.org">http://johnpfeiffer.bitbucket.org</a>

[an example](http://example.com/ "ExampleTitle") an example

both absolute and relative links are supported, as well as reference links that are defined elsewhere: This is [an example][someid]

[someid]: http://example.com/ "Optional Title Here"

numbered list 1 at the beginning of each line
- any digit will do, the numbering is rendered in order
- ensure the numbered list is surrounded by empty lines

Inline code is markdown `backtick around the text` = backtick around the text

A code block is markdown indent 4 spaces or 1 tab

a blank line in the code block still needs to be indented

ensure the code block is surrounded by empty lines

:::text or :::bash at the top of a code block will control the syntax highlighting, see http://pygments.org/docs/lexers

H1

# H1

H6

###### H6

Tables are (sometimes) not supported but...

Table with left justified (GitHub Flavored Markdown)

|in|out|other|
|---|---|---|
|yes|no|maybe|

in	out	other
yes	no	maybe
`<em>`	`emphasis`	emphasis

Table with text center aligned

|short|long centered|
|:-:|:-:|
|y|n|

short	long centered
y	n

HTML Table

<table><th>header</th>
  <tr>
    <td>first column in row 1</td><td>2nd column</td>
  </tr>
</table>

more info

http://daringfireball.net/projects/markdown/syntax

Creating a static web site with Bitbucket

2014-06-30T20:20:00-07:00

create a bitbucket account with username
create a repo named username.bitbucket.org
mkdir username.bitbucket.org
cd username.bitbucket.org
echo "hi" > index.html
git init .
git add
git commit -m "first site index"
git remote add origin git@bitbucket.org:username/username.bitbucket.org.git
git push origin master
git branch --set-upstream master origin/master
git pull
https://username.bitbucket.org

https://confluence.atlassian.com/display/BITBUCKET/Publishing+a+Website+on+Bitbucket

Scroll down to the comments: You can setup a CNAME for your account . You cannot set up a CNAME for a static website hosted on Bitbucket. The account you setup a CNAME for may have a repository that represents a static website.

How to set up a Pelican static blog site

2014-06-21T20:21:00-07:00

Pelican is an open source project that converts static text files into an html site.

Why use a static site generator (pelican) instead of a hosted blog platform or a CMS (Content Management System)?

Because less is more and you should use the right tool for the right job

A static site made of html pages is very easy to maintain
It is also more secure and performance is good too =)
Allows for straightforward use of version control (git)
Developers prefer to be able to customize and add functionality (python and javascript)
Using widely adopted open source software reduces risk (more contributors, more testers, more bugfixes)
Using a widely adopted platform increases leverage (themes, plugins, tutorials, etc.)

Install Pelican

sudo pip install pelican Markdown beautifulsoup4

installing both the pelican and the Markdown packages

beautifulsoup4 is a dependency for the later step of the elegant theme TOC and search plugins

optionally use virtualenv venv; source venv/bin/activate

pelican-quickstart

Welcome to pelican-quickstart v3.3.0.
This script will help you create a new Pelican-based website.
Please answer the following questions so this script can generate the files needed by Pelican.

> Where do you want to create your new web site? [.]
> What will be the title of this web site? johnpfeiffer.bitbucket.io
> Who will be the author of this web site? john pfeiffer
> What will be the default language of this web site? [en]
> Do you want to specify a URL prefix? e.g., http://example.com   (Y/n)
> What is your URL prefix? (see above example; no trailing slash) https://johnpfeiffer.bitbucket.io
> Do you want to enable article pagination? (Y/n)
> Do you want to generate a Fabfile/Makefile to automate generation and publishing? (Y/n)
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n)
> Do you want to upload your website using FTP? (y/N)
> Do you want to upload your website using SSH? (y/N)
> Do you want to upload your website using Dropbox? (y/N)
> Do you want to upload your website using S3? (y/N)
> Do you want to upload your website using Rackspace Cloud Files? (y/N)
Done. Your new project is available at /home/ubuntu/BLOG

tree

.
├── content
├── develop_server.sh
├── fabfile.py
├── Makefile
├── output
├── pelicanconf.py
└── publishconf.py

2 directories, 5 files

Create Content

vi content/hello_world.md

Title: My first blog post
Date: 2014-06-21 20:20
Tags: python
Slug: my-first-blog-post
Author: John Pfeiffer
Summary: Short version for index and feeds

This is the content of my first blog post.

optional UI markdown editor: sudo apt-get install retext

Run a dev server to see the results locally

make devserver

...Starting up Pelican and pelican.server...

./develop_server.sh stop

stop the dev server (required if reloading the .conf file)

This only works with the basic first setup, after that it is better to manually use multiple screens:

make clean

make regenerate

auto detects any content changes and reloads itself

cd output; python -m SimpleHTTPServer

Serving HTTP on 0.0.0.0 port 8000 ... (Control + C to quit)

http://localhost:8000

http://docs.getpelican.com/en/latest/publish.html

Setup a static Pages directory

Besides lots of articles in categories it can be useful to have a few pages like About or Contact mkdir -p content/pages echo "Title: About Us" > content/pages/about.md

http://docs.getpelican.com/en/latest/content.html#pages

Publish

I just use the pelicanconf output rather than publishconf, and I use git with a bitbucket static html site.

Example pelican configuration file

Contains the elegant theme and tipue search plugin

vi pelicanconf.py

#!/usr/bin/env python
# -*- coding: utf-8 -*- #
from __future__ import unicode_literals

AUTHOR = u'john pfeiffer'
SITENAME = u'johnpfeiffer'
SITEURL = u'https://blog-john-pfeiffer.com'
OUTPUT_PATH = 'output/'
DEFAULT_DATE_FORMAT = '%Y-%m-%d'
TIMEZONE="America/Los_Angeles"

# Feed generation is usually not desired when developing
FEED_ALL_ATOM = None
CATEGORY_FEED_ATOM = None
TRANSLATION_FEED_ATOM = None

# clean urls for pages , trailing slash to support HTTPS
PAGE_URL = '{slug}/'
PAGE_SAVE_AS = '{slug}/index.html'
# clean urls for articles
ARTICLE_SAVE_AS = '{slug}/index.html'
ARTICLE_URL = '{slug}/'

DEFAULT_PAGINATION = 10

THEME = 'themes/pelican-elegant'
PLUGIN_PATHS = ['plugins']
PLUGINS = ['sitemap', 'extract_toc', 'tipue_search', 'post_stats']
MD_EXTENSIONS = ['codehilite(css_class=highlight)', 'extra', 'headerid', 'toc']
DIRECT_TEMPLATES = (('index', 'tags', 'categories','archives', 'search', '404'))
STATIC_PATHS = ['theme/images', 'images']

SITEMAP = {
    'format': 'xml',
    'priorities': {
        'articles': 0.5,
        'indexes': 0.5,
        'pages': 0.5
     },
    'changefreqs': {
        'articles': 'monthly',
        'indexes': 'daily',
        'pages': 'monthly'
    }
}

Hint: if you use SSL (e.g. cloudflare) then make sure your SITEURL is https and there are trailing slashes on your clean url helpers PAGE_URL and ARTICLE_URL otherwise you may get mixed content warnings or mixed http and https links in your output html

cp -a pelicanconf.py publishconf.py

Update publishconf.py with any final tweaks that should occur only in Production

Pelican Themes

https://github.com/getpelican/pelican-themes

git clone --recursive https://github.com/getpelican/pelican-themes ~/pelican-themes

http://pelican.readthedocs.org/en/latest/pelican-themes.html

mkdir themes
cp -a pelican-themes/elegant themes/

This is the lazy way, the correct way is to only copy in the theme you are using ;)

FYI For the elegant with tipuesearch you need to also install the tipuesearch plugin

http://moparx.com/2014/04/adding-search-capabilities-within-your-pelican-powered-site-using-tipue-search/

Depedency Management by Vendoring in Version Control

Dependencies are always hard and leveraging the work of others (themes and plugins) means you have to consider how to manage changes to those moving parts too.

Using your Build/CI/CD server to grab the latest themes/plugins is a mistake because while you would benefit from any new features and bug fixes you will also get burned (yes it has happened to me!) when you are publishing a blog post and suddenly your site doesn't work.

(Oh are you using Continuous Post Deploy Smoke Tests and Realtime Everything Monitoring to detect this on your blog?)

I highly recommend "vendoring" by copying pelican-elegant into the themes subdirectory in your version control.

This sort of pinning makes an explicit choice that will ensure your site will continue to work until you choose to next upgrade. Pinning in the Build/CI/CD server by git sha will also work but makes it harder to keep everything in one place for development.

The drawback of "vendoring" is the technical debt by the inevitable customization that occurs in this "directory separated" fork of the original

Pelican Plugins

https://github.com/getpelican/pelican-plugins

git clone https://github.com/getpelican/pelican-plugins
mkdir plugins
cp -a pelican-plugins/sitemap plugins/

note this is the lazy way, a more precise way would be to only copy in the plugins used

Hacking ListJS and per Article word count into a theme

Basically once you get the hang of the config file format (just python really) and the templating engine (HTML with some jinja2) you can add all sorts of interesting pieces.

In this example I leverage some javascript and another pelican plugin (python) to allow sorting and filtering on more data than the usual "archives.html" provides.

<div id="article-list">
    <button class="sort" data-sort="date">date</button>
    <button class="sort" data-sort="title">title</button>
    <button class="sort" data-sort="wordcount">word count</button>
    <button class="sort" data-sort="category">category</button>
    <input class="search" placeholder="Find by filter" style="margin-top: 10px; height: 16px;"/>
    &nbsp; from {{ dates| length }} articles

    <ul class="list">
    {% for category, articles in categories %}
        {% for article in articles %}
        <li>
            <span class="date" style="padding-right: 10px;">
                <time pubdate="pubdate" datetime="{{ article.date.isoformat() }}">{{ article.locale_date }}</time>
            </span>
            <a href="{{ SITEURL }}/{{ article.url }}"><span class="title">{{ article.title }} {%if article.subtitle %}
                <small> {{ article.subtitle }} </small> {% endif %}</span> </a>
            <em><span class="wordcount">({{ article.stats['wc'] }} words)</span></em>
            <em><span class="category">{{ article.category }}</span></em>
        </li>
        {% endfor %}
    {% endfor %}
    </ul>
</div>
...
...
<script type="text/javascript">
    var options = {
        valueNames: [ 'date', 'title', 'wordcount', 'category'],
        page: 1000
    };
    var hackerList = new List('article-list', options);
    hackerList.sort('date', { order: "desc" })
</script>

Advanced: skipping the Makefile

pelican --help
pelican ./content -o ./output -s ./publishconf.py
cd output ; python -m SimpleHTTPServer

Importing from Drupal with pelican-import

Hack the Drupal files to allow a lot more than 10 items per feed grep -r 'items per feed' .

learned from drupal-7.28/modules/system/system.module

vi modules/system/system.admin.inc

$form['feed_default_items']

Add to the dropdown choices of 10, 15, 30 etc. the option of 999

sudo apt-get install pandoc
sudo pip install feedparser
pelican-import -h
pelican-import --feed http://blog.example.com/rss.xml -o output/ -m markdown

more info

https://pelican.readthedocs.org/en/latest/settings.html
Tweaking default syntax highlighting:
- http://pygments.org/docs/lexers
- http://pygments.org/demo
https://bitbucket.org/johnpfeiffer/docker/src

Cumulus compatible S3, nginx, and HMAC signed requests

2013-11-26T17:10:00-08:00

With the exceptionally fast, reliable and popular web server nginx as a front end customers can use a browser to access their uploaded files via a simple URL, the same as the SaaS Amazon S3 implementation, without knowing about the Cumulus backend.

Unfortunately there were edge cases around the encodings of spaces, pluses, slashes, etc. where nginx + Cumulus was returning "Access Denied" when trying to GET a file.

Examining the relevant RFC's (http://tools.ietf.org/html/rfc3986#section-2.1), PHP (http://php.net/manual/en/function.rawurlencode.php) and Python (http://docs.python.org/2/library/urllib.html) references, and examining the logs, I could see the files were PUT correctly, s3cmd could retrieve the binary objects (files) from Cumulus fine... but the logs were showing a change in the URL's.

Increasing the debugging in nginx, digging into the Cumulus source code and nginx AWS Authentication Module (and adding more logging statements in Python and C respectively), I realized there was a mismatch in the REST URL signature process.

Since Cumulus was using the open source Python Boto library which is actually supported by Amazon (the de facto rulers of the S3 "standard"), I decided that their signing process was authoritative.

A lot of digging into nginx configs and source, along with learning a bit about nginx module development and hacking the source of the ngx_aws_auth module, I finally came up with a matching signature process, (success!)

ngx_aws_auth/ngx_http_aws_auth.c

/* uses the source and length to copy the uri, does not escape characters
(the argument signature is compatible with ngx_escape_uri)
*/
uintptr_t ngx_uri_extractor(u_char *dst, u_char *src, size_t size, ngx_uint_t type)
{
    while (size) {
        *dst++ = *src++;
        size--;
    }
    return (uintptr_t) dst;
}

/* customized to calculate the signature using the non escaped URI, compatible with cumulus boto
*/
static ngx_int_t
ngx_http_aws_auth_variable_s3(ngx_http_request_t *r, ngx_http_variable_value_t *v,  uintptr_t data)
{
    ngx_http_aws_auth_conf_t *aws_conf;
    unsigned int md_len;
    unsigned char md[EVP_MAX_MD_SIZE];
    aws_conf = ngx_http_get_module_loc_conf(r, ngx_http_aws_auth_module);

    /*
    * This Block of code added to deal with paths that are not on the root -
    * that is, via proxy_pass that are being redirected and the base part of
    * the proxy url needs to be taken off the beginning of the URI in order
    * to sign it correctly.
    */
    u_char *uri = ngx_palloc(r->pool, r->uri.len + 200); // allow room for escaping
    /*
    u_char *uri_end = (u_char*) ngx_escape_uri(uri,r->uri.data, r->uri.len, NGX_ESCAPE_URI);
    */
    u_char *uri_end = (u_char*) ngx_uri_extractor(uri,r->unparsed_uri.data, r->unparsed_uri.len, NGX_ESCAPE_URI);
    *uri_end = '\0'; // null terminate
    ...
}

Selenium headless browser automated testing with PhantomJS and Python

2013-09-17T21:04:00-07:00

Why Automated Testing

Automated Testing is critical to maintaining quality while increasing velocity (aka avoiding being crushed to death by technical debt). Web sites are inherently complex to test:

requires a client and server,
combination of UI and Logic,
lots of paths,
etc

A challenge facing the HipChat team is how to maintain quality while supporting both SaaS and BTF versions, expanding the team, increasing the functionality, and experiment with architecture (or even just normal bug fixing refactoring).

Choosing Selenium

Continued innovation by the Selenium project has created a stable and usable programmatic interface such that clicking on different components in a web page can be automated.

Selenium tests can be generated manually by a Firefox plugin https://docs.seleniumhq.org/projects/ide/ and then saved to a .html file which is helpful in that it can be done by non programmers. Eessentially selenium as a Domain Specific Language creates artifacts rather than tribal knowledge.

The Selenium Documentation is overly complicated by legacy versions just use the IDE, play with the record button and click on stuff, save the .html, and analyze and you'll quickly get the hang of it.

Example Selenium Test HTML

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head profile="http://selenium-ide.openqa.org/profiles/test-case">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<link rel="selenium.base" href="https://mysite.com/" />
<title>selenium-login</title>
</head>
<body>
    <table cellpadding="1" cellspacing="1" border="1">
    <thead>
    <tr><td rowspan="1" colspan="3">selenium-login</td></tr>
    </thead><tbody>
    <tr>
    <td>open</td>
    <td>/</td>
    <td></td>
    </tr>
    <tr>
    <td>type</td>
    <td>id=email</td>
    <td>admin@mysite.com</td>
    </tr>
    <tr>
    <td>type</td>
    <td>id=password</td>
    <td>examplepassword</td>
    </tr>
    <tr>
    <td>clickAndWait</td>
    <td>id=signin</td>
    <td></td>
    </tr>
    <tr>
    <td>assertLocation</td>
    <td>https://mysite.com/home</td>
    <td></td>
    </tr>
    <tr>
    <td>assertElementPresent</td>
    <td>link=Launch the web app</td>
    <td></td>
    </tr>
    </tbody></table>
</body>
</html>

To get started with the IDE automation framework for interactions with websites (like a programmatic browser) http://docs.seleniumhq.org/docs/02_selenium_ide.jsp#introduction

Selenium WebDriver

Characteristics of good test plans are focusing on "happy path" high value tests, avoiding fragility/brittleness (any change invalidates many tests), and of course avoiding manual intervention.

Having a huge suite of IDE based tests incurs potentially unsustainable technical testing debt (e.g. investigating/rewriting why/when many tests fail).

And requiring a testing machine with a UI with a browser etc. is a lot of overhead if you intend on (correctly!) running the tests really often.

Converting raw .html tests to a specific language binding (i.e. moving more to WhiteBox) can remove non essential parts of the test and increase flexibility (multiple browsers!).

sudo pip install selenium

installation assuming linux and python

Example Selenium Python Webdriver

The IDE helpfully not only can save a Test but can File -> Export Test Case As -> python

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
import unittest, time, re


class SeleniumLogin(unittest.TestCase):
def setUp(self):
    self.driver = webdriver.Firefox()
    self.driver.implicitly_wait(30)
    self.base_url = "https://mysite.com/"
    self.verificationErrors = []
    self.accept_next_alert = True

def test_selenium_login(self):
    driver = self.driver
    driver.get(self.base_url + "/")
    driver.find_element_by_id("email").clear()
    driver.find_element_by_id("email").send_keys("admin@mysite.com")
    driver.find_element_by_id("password").clear()
    driver.find_element_by_id("password").send_keys("examplepassword")
    driver.find_element_by_id("signin").click()
    self.assertEqual("https://mysite.com/home", driver.current_url)
    self.assertTrue(self.is_element_present(By.LINK_TEXT, "Launch the web app"

def is_element_present(self, how, what):
    try: self.driver.find_element(by=how, value=what)
    except NoSuchElementException, e: return False
    return True

def is_alert_present(self):
    try: self.driver.switch_to_alert()
    except NoAlertPresentException, e: return False
    return True

def close_alert_and_get_its_text(self):
    try:
    alert = self.driver.switch_to_alert()
    alert_text = alert.text
    if self.accept_next_alert:
        alert.accept()
    else:
    alert.dismiss()
    return alert_text
    finally: self.accept_next_alert = True

def tearDown(self):
    self.driver.quit()
    self.assertEqual([], self.verificationErrors)


if __name__ == "__main__":
unittest.main()

python selenium-login.py runs the tests (opens Firefox, executes all of the commands)

Docs for the webdriver interface:

useful for looking up things like driver.current_url

Selenium Performance with PhantomJS

Removing the UI requirement improves test performance and increases the options of where the test is run (Dev machines, headless VMs, cloud servers, etc.)

Installing the python selenium library, downloading the phantomJS binary running it was relatively easy with Ubuntu Linux since the project integrated GhostDriver too.

The phantomjs headless browser http://phantomjs.org Archived as of 2018-03 and no longer under active development

Installing PhantomJS

Archived as of 2018-03 and no longer under active development http://phantomjs.org/download.html

For example for Linux one might use:

wget <https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-x86_64.tar.bz2>
tar -xjvf phantomjs-1.9.8-linux-x86_64.tar.bz2

Example PhantomJS webdriver python script

mini script to just show usage

from selenium import webdriver

driver = webdriver.PhantomJS(executable_path='/opt/phantomjs-1.9.8-linux-x86_64/bin/phantomjs', port=9134)
driver.get("http://127.0.0.1")
print driver.current_url
driver.quit
print "done"

To run the phantomJS binary phantomjs-1.9.8-linux-x86_64/bin/phantomjs --webdriver=9134

ghostdriver included and running on port 9134

As an example I found PhantomJS to be at least twice as fast as Firefox ./phantomjs --webdriver=127.0.0.1:9134 --ignore-ssl-errors=true

skip verifying SSL

Advanced Python and PhantomJS example

more complete example with python unittest framework (used the Firefox Selenium IDE plugin -> Export)

logs in, asserts there is an Admin tab which when clicked shows Group Info

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
import unittest, time, re

class SeleniumAdminLogin( unittest.TestCase ):

    def setUp( self ):
        self.driver = webdriver.PhantomJS( '/opt/phantomjs-1.9.2-linux-x86_64/bin/phantomjs', port=9134 )
        self.driver.implicitly_wait(30)
        self.base_url = "https://myexample.org"
        self.verificationErrors = []
        self.accept_next_alert = True

    def test_selenium_admin_login( self ):
        driver = self.driver
        driver.get( self.base_url + "/" )
        driver.find_element_by_id( "email" ).clear()
        driver.find_element_by_id( "email" ).send_keys( "admin@example.org" )
        driver.find_element_by_id( "password" ).clear()
        driver.find_element_by_id( "password" ).send_keys( "mypassword" )
        driver.find_element_by_id( "signin" ).click()
        self.assertEqual("https://myexample.org/home", driver.current_url)
        self.assertTrue(self.is_element_present(By.LINK_TEXT, "Launch the web app"))
        self.assertTrue(self.is_element_present(By.CSS_SELECTOR, "a.admin > span"))
        driver.find_element_by_css_selector("a.admin > span").click()
        self.assertEqual("Group Info", driver.find_element_by_css_selector("h1").text)

    def is_element_present(self, how, what):
        try: 
            self.driver.find_element(by=how, value=what)
        except NoSuchElementException, e: return False
            return True

    def is_alert_present(self):
        try: 
            self.driver.switch_to_alert()
        except NoAlertPresentException, e: return False
        return True

    def close_alert_and_get_its_text(self):
        try:
            alert = self.driver.switch_to_alert()
            alert_text = alert.text
            if self.accept_next_alert:
                alert.accept()
            else:
                alert.dismiss()
            return alert_text
            finally: 
                self.accept_next_alert = True

    def tearDown(self):
        self.driver.quit()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

Basic Polling with Firefox and Mac

import datetime
import os
import sys
import time
import urllib2

from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary


FIREFOX_MAC_PATH = '/Applications/Firefox.app/Contents/MacOS/firefox-bin'
G_URL = 'https://g.example.com'

if __name__ == '__main__':

    if len(sys.argv) < 2:
        print('Usage error: requires a username')
        sys.exit(1)
    print(sys.argv)
    username = sys.argv[1]

    # https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior
    new_server_name = datetime.datetime.now().strftime('%d%b%H%M%S').lower()

    if os.path.exists(FIREFOX_MAC_PATH):
        driver = FirefoxBinary(firefox_path=FIREFOX_MAC_PATH)
    else:
        driver = webdriver.Firefox()

    driver.get(G_URL)
    print(driver.current_url)
    if driver.current_url.endswith('login'):
        driver.find_element_by_name('username').clear()
        driver.find_element_by_name('username').send_keys(username)
        driver.find_element_by_css_selector('input[type="submit"]').click()

    driver.get('{}/deploy/simple_server'.format(G_URL))
    driver.find_element_by_name('name').clear()
    driver.find_element_by_name('name').send_keys('{}-{}'.format(username, new_server_name))
    driver.find_element_by_name('hostname').clear()
    driver.find_element_by_name('hostname').send_keys('{}-{}'.format(username, new_server_name))
    driver.find_element_by_css_selector('input[type="submit"]').click()
    url = 'https://{}-{}.example.com'.format(username, new_server_name)

    print('requested build of {}'.format(url))
    for _ in xrange(20):
        status_code = 404
        try:
            connection = urllib2.urlopen(urllib2.Request(url))
            status_code = connection.getcode()
            content = connection.read()
        except Exception as error:
            print('waiting for {}'.format(url))

        if status_code == 200:
            break
        time.sleep(30)

    driver.get(url)
    print(driver.current_url)
    print(driver.title)
    driver.quit()
    print('done')

More Info https://realpython.com/headless-selenium-testing-with-python-and-phantomjs/

Attack of the Spiders, Bots, and Crawlers

2013-07-20T23:00:00-07:00

It is well known that search engines need to index what the contents of webpages are in order to return accurate results to Users. It may not be well known how much traffic that generates.

In this sampling from my logs (not representative of anything), 57% of the traffic in the log is from bots (I'm not turning them away, skip to the end on how to do that).

Drupal (CMS) is remarkably good at making content for bots to read so it's not surprising they're all slurping, and of course if you want to be popular you need all of those indexes to know about you...

How many hits from each kind of bot

cat john-pfeiffer.access | wc -l

1108 hits

cat john-pfeiffer.access | grep -v "Baiduspider" | grep -v "bingbot" | grep -v "YandexBot" | grep -v "Sogou" | grep -v "Mail.RU\_Bot" | grep -v "Googlebot" | grep -v "SISTRIX Crawler" | grep -v "MJ12bot" | wc -l  
637

cat john-pfeiffer.access | grep "Googlebot" | wc -l

105 hits from https://en.wikipedia.org/wiki/Googlebot

cat john-pfeiffer.access | grep "bingbot" | wc -l

96 hits from http://en.wikipedia.org/wiki/Bingbot

cat john-pfeiffer.access | grep "Baiduspider" | wc -l

93 hits from http://baidu.com/search/spider_english.html

cat john-pfeiffer.access | grep "SISTRIX Crawler" | wc -l

76 hits from http://crawler.sistrix.net

cat john-pfeiffer.access | grep "YandexBot" | wc -l

64 hits from http://www.botopedia.org/user-agent-list/search-bots/yandex-bot

cat john-pfeiffer.access | grep "Mail.RU\_Bot" | wc -l

23 hits from http://www.webmasterworld.com/search_engine_spiders/4520951.htm , http://www.botopedia.org/user-agent-list/search-bots/mailru-bot

cat john-pfeiffer.access | grep "MJ12bot" | wc -l

12 hits from http://www.majestic12.co.uk/projects/dsearch/mj12bot.php

cat john-pfeiffer.access | grep "Sogou" | wc -l

7 hits from http://www.botopedia.org/user-agent-list/search-bots/sogou-spider

More Info on Bots and Crawlers

http://www.incapsula.com/the-incapsula-blog/item/393-know-your-top-10-bots

http://searchenginewatch.com/article/2067357/Bye-bye-Crawler-Blocking-the-Parasites

(Besides robots.txt you are pretty much left with ban by User-Agent or IP Address Range.)

Git Basics and Hard to Scratch Itches

2013-07-07T12:34:00-07:00

Like "tabs vs spaces" there are tools and choices which create more controversy than others.

Here's my pragmatic take on how to get stuff done...

Git Cheat Sheet

git branch

see the current branch

git status

are there any files changed?

git stash

save and hide any modified files to a temporary cache

git checkout master

set the local working files to the shared canonical branch (aka "master")

git pull

get and merge any changes, many others prefer the more precise git fetch

git log

examine the history of changes made upstream (consider pre-emptively how the merge will affect things)

git checkout mybranch

change the focus back to the working dev branch

git pull

ensure the local working files from the branch are the latest (i.e. if other contributors or devices have made modifications)

git merge master

force reconciliation of master and the current working branch

RESOLVE CONFLICTS IN THE CODE MANUALLY

one thing to remember is that "merge" is best effort so sometimes code/lines will be added or removed that are not the desired semantic outcome

git commit

indicate a merge from master and any conflict resolution fixes

git push

never force , the changed branch may be worked on by others and a forked history means no reconciliation...

Extra Credit

cp -a FOOBAR /tmp

manually back up the files

git checkout -- ./name

reset a single changed file back to the previously committed version

git log ; git revert COMMITID

identify a mistaken commit and create a commit to undo all changes until the working files (and git history) match that specific commit id

git reset --soft HEAD^^

correct and recommit change A, then go back to recommitting change B

git reset HARD

reset all of the working files to a previously committed version

rm -rf FOOBAR; git clone

throw away entirely a locally copy and start again from the remote version

Simple Git Theory

Distributed Version Control

Directed Acyclical Graph

Any branch can be "master".

But in practice most developes use an agreed centralized location (i.e. good availability/uptime), especially a SaaS like github or bitbucket.

Basic Git Commands

git clone

SSH is far more convenient and secure

git clone

HTTPS is common for "read-only" downloads of an open source project

git add --all .

the extra parameter ensures new files are added (which I otherwise forget), and as a bonus will detect file/directory renaming

git log

Show the history of commits (and hashes)

git rebase -i

interactively rewrite git history, NEVER do this on a shared/collaborative branch/repository

Remove a large file from history

git filter-branch --tree-filter "rm -rf VeryLargeDirOrFileName" -f HEAD --all

https://git-scm.com/docs/git-filter-branch

git rev-list --objects --all | grep "$(git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -10 | awk '{print$1}')"

Split a subdirectory into its own repo

git subtree push --prefix examplesubdir https://bitbucket.org/USERNAME/examplereponame master

whoa, one line to take a subdirectory and push it into the waiting empty new repo

https://developer.atlassian.com/blog/2015/05/the-power-of-git-subtree/

Logic Puzzles

2013-05-13T05:31:00-07:00

How to have two 1 hour uneven burning fuses burn for 45 mins?

light only ONE end
burn both ends = 30 mins
when it finishes, light the other end

Three light bulbs are all off in one room, three light switches in another...

...If you can't see the bulbs from the switch room, how can you tell which connects to which if you can only enter the bulb room once?

Flip the first switch for one bulb for many minutes, then turn it off.

Flip the second switch on. Enter the bulb room, the bulb on connects to the second switch, the warmer of the two bulbs off is the first switch.

At night 4 people arrive at one side of a bridge...

...that can only support 2 people at a time and they have only one flashlight.

If each were alone here's how fast they could cross: the first in 15 minutes, the second in 10 minutes, the third in 2 minutes and the fourth in 1 minute.

If the flashlight only has 17 minutes how do they cross?

The 1 minute and 2 minute cross together (takes 2 minutes).
The 1 minute returns (total is now 3 minutes).
The 10 and 5 minutes cross together (total is now 13 minutes).
The 2 minute returns (the total is now 15 minutes).
Finally the 1 minute and 2 minute cross again (17 minutes and complete).

3 bags contain all oranges, all apples, or a mix of oranges and apples...

...BUT the label on each definitely is incorrect. How do you find out which bag has what and how many bags to you need to open?

Choose the one labeled "mixed" and if it contains apples, therefore the remaining bag which is labelled oranges must not contain all oranges and so must be the mixed one.

There are 3 red hats and 2 blue hats in a bag....

Andrew, Bob, and Carol each reach into the box and place one of the hats on their own head. They cannot see what color hat they have chosen. Then they sit down all facing in the same direction: Andrew <-- Bob <--Carol

Such that Carol can see what color Bob and Andrew are wearing.
Bob can see what color Andrew is wearing.
Andrew can't see anyone's hats.

The first person to say what color hat they are wearing wins. The winner of the game gets a billion dollars... But if they guess wrong, they die... so nobody wants to try "just guessing"...

SO... after they grab their hats and sit down, a long time passes...

Eventually somebody says "I have the answer" and successfully states what color hat they have.

Since Carol can see Andrew and Bob, if they both were wearing blue hats she would immediately know that she was wearing red, so unlikely that a long time would pass before she answered (and take the money)...
Therefore Andrew and Bob have one red hat and one blue hat. (And Carol knows there's a 33% chance she's got a blue hat and a 66% chance she's got a red hat).
Since Bob knows that Carol delayed in answering "a long time passes" AND he can see the color of hat that Andrew is wearing, then
Either Andrew is wearing a blue hat, and therefore Bob must be wearing a red hat.
Or Andrew is wearing a red hat, and therefore Bob must be wearing a blue hat.

Bob successfully answers...

(unless Carol is more interested in sadistic plots and even though she knows both Bob and Andrew have blue hats she just waits to watch Bob get it wrong...)

NOTE: this variation from the NY Times is slightly incorrect as it is essentially asking why Andrew wins... 
The clue in order for "Andrew" to know (the delay) is the same as what Bob uses, but Bob will be able to deduce it much faster. 
(Again, Carol could be deliberately waiting to watch Andrew fail too...)

http://tierneylab.blogs.nytimes.com/2009/03/16/the-puzzle-of-the-3-hats

One ball of 8 weighs more than the other 7, how can you identify it with two measurings of a balance scale?

Weigh 3 vs 3, if they match then weigh one of the "good" ones against one of the remaining ones, if they're unequal you have the heavier one, if they're equal then the last is the heavy one.

If the 3 vs 3 do not match then take two of the heavy side and weigh them against each other: if they're unequal you have the heavier one, if they're equal then the last of the heavy side is the heavy one.

One ball of 12 weighs more OR less than the other 11, how can you identify it and whether it's heavier or lighter with three measurings of a balance scale?

Weigh 4 vs 4, label them ABCD and EFGH. If they balance then weigh "good" ABC against WXY. If the second weighing balances then weigh Z against A to see if Z is light or heavy. If the second weighing has WXY heavier or lighter (make a note of which as we have solved whether the imbalance is heavy or light), then weigh W with X. If they balance then Y is the imbalance from step 2 (heavier or lighter). If they do not balance then whichever matches the imbalance from step 2 is the off ball.

IF at the first weighing ABCD was lighter than EFGH, in the second weighing rotate such that ABCZ is weighed against D with "good" XYZ. If ...

Note that this problem can also be solved by carefully measuring the results as the left and right groups of 4 are rotated.

12 becomes 4 v 4 v (4) - if the balance the last 4 are easy

if not balanced: 1,2,5 vs 3,6,12

5 pots of 10g coins BUT one contains 9g coins, which pot is off by measuring the weight once?

1 from pot 1, 2 from pot 2, 3 from pot 3, 4 from pot 4, and 5 from pot 5

Expected weight should be 150g: if it’s 149 then pot 1 , if it’s 148 then pot 2, etc.

How can you use a weighted coin (i.e. head more than tails) and still create a fair system of flips?

Aggregate the outcomes: treat two flips as a single result such that heads then tails = Heads, and tails followed by heads = Tails.

10 are heads of N coins, how to create 2 groups that have the same number of heads up?

Take any 10 from N and flip them (inverted will mirror the number of heads in the N - 10 group)

3 colors of socks: how many to take out until you have a pair?

4 (1 of each and the last 1 must match 1 of the first 3)

100 closed lockers, open all of the lockers on the first pass...

** ...close every 2nd locker on the second pass, on the third pass and for every 3rd locker open it if it's closed, close it if it's open. After the hundreth pass, how many lockers are open?**

i.e. after the hundreth the locker 12 was opened on pass 1, closed pass 2, opened pass 3, closed pass 4, opened pass 6, closed pass 12.

locker 13 (prime) was opened on pass 1 and closed on pass 13.

locker 16: opened on 1, closed on 2, opened on 4, closed on 8, opened on 16!

There are 9 perfect squares under one hundred: 1, 4, 9, 16, 25, 36, 49, 64, 81

Incomplete Puzzles

Minutes Degrees from 12:00 = mins * 6

Hours Degrees From 12:00 = hours * 30 + mins * .5

Find 1 duplicate in 100 numbers of 1 to 100 = add them up and subtract sum of 1 to 100

ALTERNATIVES: hashmap O(1), sort and then scan looking for a double entry (nLog(n) if mergesort? + n ), n\^2 if using brute force

Sorted List rotated: find min = naive if curr < prev is O(n)

Binary Search compare first and mid, if ordered then reset is in partition > mid (RIGHT)

find any elem: if > first and Ordered then LEFT

BAD: binary search against the "fixed" list using the min location as an offset with mod list size ?

nxn matrix of numbers in ascending order in both dimensions how would you go about finding if the number y is in the matrix.

5623 players in a tournament, how many matches must be played?

5622 (everybody loses at least once except the winner)

Unsolved Puzzles

N different flavored Cakes (each with a different Volume) for K people, each person should get an equal volume of Cake (but only a single flavor)

V’s 1, 2, 3, 4 for 5 people

add all volumes together / people = “theoretical best” (i.e. 10/5 = 2)

start with 2 cakes, 4 + 1, so V=1 for K, next 4 + 2 = doesn’t work, 4+3 doesn’t work

3 cakes: 4+3+2 = 9 BUT not divisible by 5

1 * 1, 2 * 1, 2 * 1 = waste 5 ( 10-5)

discard

Random Facts

World Population (estimated 2012) = 7 Billion
US Population (estimated 2012) = 316 Million
Earth circumference (equator) = 24,901.55 miles (40,075.16 kilometers)

Best Computer Science online and a More Complete Education

2013-05-09T16:52:00-07:00

Learning is a life long pleasure. Programming and Computer Science are challenging and take time but luckily the resources available today make it free to get access to top quality materials.

In the 6 years since I first wrote this article many of the links had to be updated (and some resources are no longer free, goodbye MOOC), hopefully I have managed to maintain this somewhat

Just starting with coding (mostly python)

I've been asked often enough about getting started in software programming that I decided prepend this "guide" from 2019 that I believe is a free and easy way to get bootstrapped.

Watch a couple of videos from https://www.coursera.org/learn/interactive-python-1 (pro tip, if possible watch at 1.25x or 1.5x speed and pause and go back to replay a specific part when necessary)
Do about 3 hackerrank exercises: e.g.- https://www.hackerrank.com/domains/python?filters%5Bsubdomains%5D%5B%5D=py-introduction (but stop after doing "Errors/Exceptions" and move onto the problem-solving track https://www.hackerrank.com/domains/algorithms?badge_type=problem-solving , always feel free to skip an exercise if the problem description is too confusing, the key is practice not perfection ;) , https://www.hackerrank.com/challenges/python-string-split-and-join/problem
Do one "chapter" from KhanAcademy (e.g. https://www.khanacademy.org/computing/computer-science/algorithms start from Binary search and you're "good enough" after finishing merge sort)
Read one "chapter" from the PythonDocs , e.g. start with https://docs.python.org/3.7/tutorial/introduction.html
Watch one video from MIT https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-0001-introduction-to-computer-science-and-programming-in-python-fall-2016/lecture-videos/ note this is the same as https://www.edx.org/course/introduction-to-computer-science-and-programming-using-python-3
Do one exercise from leetcode , https://leetcode.com/problemset/algorithms/?difficulty=Easy&listId=79h8rn6 (easy ones first, then go to mediums or the top100 list)

e.g.

(if statements for each edge case) https://leetcode.com/problems/valid-parentheses
(nested for loops and dicts/hashmaps) https://leetcode.com/problems/two-sum
(nested for loops) https://leetcode.com/problems/valid-sudoku

Then go back to #1 and repeat =]

Of course what I've suggested is really just the minimum. Some good auxiliary learning would be:

Introduction to Programming by Universities

Stanford

by Mehran Sahami (very fun and Java is an ok starting point - though the world has moved to Go)

https://see.stanford.edu/Course/CS106A Programming Methodology
Then algorithms with https://see.stanford.edu/Course/CS106B Julie Zelenski
And more algorithms https://www.coursera.org/learn/algorithms-divide-conquer Tim Roughgarden
And advanced algorithms https://www.coursera.org/learn/algorithms-graphs-data-structures Tim Roughgarden
https://see.stanford.edu/Course
https://online.stanford.edu/course/intro-computer-networking-winter-2014
https://www.youtube.com/playlist?list=PLvFG2xYBrYAQCyz4Wx3NPoYJOFjvU7g2Z

Harvard

MIT

https://www.edx.org/course/introduction-computer-science-python-mitx-6-00-1x now named https://www.edx.org/course/6-00-1x-introduction-to-computer-science-and-programming-using-python-3

Algorithms of course!

Princeton

Algorithms - https://www.coursera.org/learn/algorithms-part1 (aka https://online.princeton.edu/node/201) - https://www.coursera.org/learn/algorithms-part2 - https://www.coursera.org/learn/cs-algorithms-theory-machines

More Courses

Udacity Design of Computer Programs

https://www.udacity.com/course/cs101 became https://www.udacity.com/course/introduction-to-python--ud1110
https://www.udacity.com/course/ud036
https://www.udacity.com/course/cs212
https://www.udacity.com/course/cs253
https://www.udacity.com/course/cs313 Intro to Theoretical Computer Science

Udemy

https://www.udemy.com/learn-how-to-code/#about-course (paid)

Information and Models

MIT

University of Michigan

https://class.coursera.org/modelthinking-2012-002/class/index

Software Engineering

Vanderbilt

Software patterns - https://www.dre.vanderbilt.edu/~schmidt/Coursera/spring-2013-posa.html - https://www.youtube.com/playlist?list=PLZ9NgFYEMxp6CHE-QQ040tlDILNcBqJnc - https://www.youtube.com/watch?v=2GZttnHChHo&list=PLZ9NgFYEMxp4ZsvD10uXmClGnukcu3Uff&index=13 (patterns)

Online Masters in CS

There are even now recognized, accredited Masters degrees in Computer Science: https://www.omscs.gatech.edu/

PRACTICE, PRACTICE, PRACTICE

https://codingbat.com/python (beginner)
https://www.hackerrank.com
https://leetcode.com
https://projecteuler.net
http://acm.timus.ru
http://www.spoj.com/problems/classical
https://codeforces.com/problemset
https://tour.golang.org Go for a static language to complement Python
https://www.geeksforgeeks.org/javascript-tutorial/#basics

Interview Resources

Things Often Not Covered By Universities

For some reasons the "ivory tower" does not include all of the nitty gritty practicalities required to actually ship and run software in the real world.

Here are some of those topics I wish were covered in the first year. (Hint: they also help immensely in being gainfully employed)

Version Control

The fundamental tool of managing change which was strangely ignored for a very long time in the short history of programming

Quality and Testing

The practical answer to actually attempting to validate correctness in practice (rather than just logical proofs)

Build and Continuous Integration

Automation as a solution to the shortage of developer time and the exponential increase in software and complexity

Performance and DevOps and Operations

With hardware having kept up with Moore's Law and "the cloud" providing so much elastic compute, performance is now often an afterthought. Additionally, how to quickly and efficiently deliver software has coalesced into the term DevOps.

The big idea being that software that is not actually running is not very valuable ;)

I know it sounds crazy that Computer Science practitioners (or Developers) should sully their hands with "Operations" but to truly understand the problem domains watching the logs or responding to an outage can vastly change how we write code.

Importantly, seeing latency, traffic volume, and environmental issues makes us thankful when we do get back to the keyboard and can just focus on the theoretical aspects of a problem.

Jinja2, a web html template layout for everyone

2013-04-30T04:35:00-07:00

Web development used to be so hard (and static). (read <bold>)

Now everyone realizes (along with version control and automated testing) that decoupling views and displays from dynamic code makes everyone's life easier!

(Refresh your colors and layout without having to touch your backend logic! Re-engineer your persistence layer and business logic but leave your uber wow design intact! Allow front end and back end developers to work in parallel!)

Jinja2 is an excellent framework for python developers (to go with your uwsgi + django or webapp2) to get html templates (do not repeat yourself!) that can show off all of your backend python data magic with some pizazz.

It's fairly easy to do all sorts of powerful things (like accessing variables, loops, etc.) http://jinja.pocoo.org/docs

When combined with css and jquery (i.e. tablesorter) you can quickly throw together a decent looking interactive experience.

Here's my budding ode to sorting algorithms (yes, Google App Engine is free) http://john-pfeiffer.appspot.com/algorithms #TODO: finish using jquery tablesorter

A webapp2 project layout and source code example (please excuse the code formatting, you'll need to imagine the correct indents)...

File Layout

assets/css/style.css  
assets/css/tablesorter.css  
assets/images/css/asc.gif (bg.gif, desc.gif, etc.)  
assets/js/jquery-1.9.1.min.js  
assets/js/jquery.tablesorter.min.js

mainhandler.py  
templates/main.html  
app.yaml (only required if using AppEngine)  
main.py  
routes.py

app.yaml

application: john-jinja2  
version: 1  
runtime: python27  
api_version: 1  
threadsafe: true

handlers:  
- url: /favicon.ico  
static_files: favicon.ico  
upload: favicon.ico

- url: /css  
static_dir: assets/css

- url: /js  
static_dir: assets/js

- url: /images  
static_dir: assets/images

- url: /.*  
script: main.app


libraries:  
- name: webapp2  
version: "2.5.2"

- name: jinja2  
version: latest

main.py

# -*- coding: utf-8 -*-
import webapp2
from routes import entry_points

# must be named "application" for uwsgi webapp2, in AppEngine it should be "app"
application = webapp2.WSGIApplication( entry_points , debug = False )

routes.py

# -*- coding: utf-8 -*-
import webapp2
from mainhandler import MainHandler

# Map url's to handlers in the handlers module , optionally choosing specific target method and request type
entry_points = [ webapp2.Route( '/main', handler=MainHandler, handler_method='get', methods=['GET'] ), ]

mainhandler.py

# -*- coding: utf-8 -*-
import webapp2
import jinja2
import os
# weird hack to ensure we go up a directory level to correctly find the templates directory
jinja_environment = jinja2.Environment( loader = jinja2.FileSystemLoader( os.path.dirname(__file__) ) )

class MainHandler( webapp2.RequestHandler ):
    def get( self ):
        # a list of tuples
        result_list = [ ('apples','green'), ('bananas','yellow'), ('cherries','red') ] 
        template = jinja_environment.get_template( 'templates/main.html' )
        template_values = { 'title': 'fruits and colors', 'body_content': 'fruits and colors', 'result_list': result_list }
        self.response.content_type = 'text/html'
        self.response.out.write( template.render( template_values ) )

templates/main.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns="http://www.w3.org/1999/html">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>{{title}}</title>
    <link type="text/css" rel="stylesheet" href="/css/tablesorter.css" />
    <script type="text/javascript" src="/js/jquery-1.9.1.min.js"></script>
    <script type="text/javascript" src="/js/jquery.tablesorter.min.js"></script>
</head>

<body>
    {{body_content}}

    <div>
        <table id="results" class="tablesorter">
        <thead>
        <tr>
        <th>Fruit</th>
        <th>Color</th>
        </tr>
        </thead>
        <tbody>
        {% for item in result_list %}
        <tr>
        <td><a href="/{{item[0]}}">{{ item[0] }}</a></td><td>{{ item[1] }}</td>
        </tr>
        {% endfor %}
        </tbody>
        </table>
    </div><br/>
    <script type="text/javascript">
    $(document).ready(function() {
    $("#results").tablesorter( {sortList:[[0,1]]} ); # sort descending by the first element
    });
    </script>
</body>
</html>

Non Google App Engine /etc/init.d/uwsgi.sh might look like:

#!/bin/bash
# 2013-02-22 johnpfeiffer

start(){
    /usr/local/bin/uwsgi --pidfile /var/www/pidfile-uwsgi.pid --touch-reload=/var/www/pidfile-uwsgi.pid --logto2 /var/www/python-john/uwsgi.log --http :8080 --wsgi-file /var/www/main.py --pythonpath /var/www/ &
}

stop(){
    kill -INT `cat /var/www/pidfile-uwsgi.pid`
    sleep 1
}

status(){
    ps aux | grep uwsgi
}

case "$1" in
status)
    status
    exit 0
    ;;
start)
    start
    exit 0
    ;;
stop)
    stop
    exit 0
    ;;
reload|restart|force-reload)
    stop
    start
    exit 0
    ;;
**)
    echo "Usage: $0 {start|stop|reload}" 1>&2
    exit 1
    ;;
esac

Alternate Flask Jinja2 Example

https://bitbucket.org/johnpfeiffer/tddflask/src

Fix Byobu infinite scroll bug on Ubuntu 12.04 Precise Pangolin

2013-02-15T17:36:00-08:00

After installing Ubuntu Server 12.04 (Precise Pangolin) I was disappointed to see that one of my favorite utilities, byobu (an improvement on the classic screen = multi ssh screen with status and hotkeys), had an infinite scroll problem.

(Quick, type exit before your screen disappears entirely!)

Amazingly this bug shipped in the official Ubuntu Release even though byobu 5.17 lists it as a fixed.

Easy workaround is:

byobu-config
Toggle status notifications
Use the arrow keys to scroll down and space bar to disable the logo
Tab and Enter to Apply -> then Exit

Now you can safely use "byobu" on the command line in Ubuntu 12.04! (apparently not persisting in 12.04.2?)

p.s. Control + a (screen mode) and then Control + a , then c ... now you've got multiple screens!

(Control + a + 0 to go to screen 0, control + a + a to jump to the last used screen)

https://help.ubuntu.com/community/Byobu

Of course you can always use normal BASH navigation: http://www.gnu.org/software/bash/manual/html_node/Commands-For-Moving.html

A fix for Windows Putty users ... Window -> Translation -> UTF8 (the ISO-8859 + byobu UTF8 logo = ugh)

(An untested alternate workaround: .byobu/status::tmux_left : "logo" -> "#logo" )

UPDATE for 12.04.2!

byobu in Ubuntu 12.04 uses tmux as the backend. You can change this by running byobu-select-backend and selecting screen

Thanks for the tip Eric!

ALTERNATE: .byobu/status::tmux_left : "logo" -> "#logo" )

byobu installation and basics

installing a better tmux (multiple remote virtual console screens)

BYOBU = BETTER GUI FOR SCREEN + CPU/RAM USAGE + DATETIME https://help.ubuntu.com/community/Byobu

(an improvement on the classic "screen" = multi ssh screen with status and hotkeys)

sudo apt-get install byobu
byobu-config

Toggle status notifications -> spacebar to disable logo -> Apply -> Exit

byobu

F2 ... or control + a , then c to create a new screen
F3 to move to a previous screen
F4 to move to the next screen
control + a , then 0 = to go to screen zero, etc.
control + a , then a = to go to the last used screen

byobu-enable

will have it start on every ssh connection

byobu-disable

stops auto-starting of byobu

If byobu seems stuck when using vi

try Control + Q (or Control + S)

Moving the cursor in byobu (f7 and beyond)

F7 = scrollback mode , vi like commands to search and copy paste

h - Move the cursor left by one character
j - Move the cursor down by one line
k - Move the cursor up by one line
l - Move the cursor right by one character
0 - Move to the beginning of the current line
$ - Move to the end of the current line
G - Moves to the specified line (defaults to the end of the buffer)
/ - Search forward
? - Search backward
n - Moves to the next match, either forward or backword

screen - the previous generation of remote virtual console utility

sudo apt-get install screen

SSH into your machine and type:

screen           //to start your screen session.
ctrl + a + c     //to create a new session
ctrl + a + d     //to disconnect from the screen session, then log out of your SSH session
when you get disconnected/dropped by the network then log back in using SSH

screen -ls          //shows current screen sessions

There is a screen on:
        19894.pts-1.servername    (Attached)
1 Socket in /var/run/screen/S-username.


screen -r    //When you're ready to reconnect to the last screen session

screen -d 19894         //detach the screen from the other ssh session
screen -r 19894         //connect to the screen to see everything's ok

ctrl-a-d                //detach from the screen

who -u                  //find the id of the "lost" ssh session
sudo kill sessionid     //kill off the disconnected ssh session

screen -r 19894         //resume work

screen -r -d                //force detach an existing and attach a session

setting the config for no visual bell!

in your /home/USERNAME (or /root) directory

touch .screenrc

vbell off

Google App Engine Python

2013-01-26T15:01:00-08:00

Setting Up

"Google App Engine (often referred to as GAE or simply App Engine, and also used by the acronym GAE/J) is a platform as a service (PaaS) cloud computing platform for developing and hosting web applications in Google-managed data centers."

The Admin Dashboard is linked to your Google profile https://console.cloud.google.com

Download and extract the SDK (e.g. gae-python.zip)

cd google_appengine
cp -a new_project_template helloworld

Inside is the minimum file structucture required to have an application

app.yaml is the configuration file
index.yaml is how to override configuration for database indices
favicon.ico is the icon that appears in the browser tab or bookmark https://en.wikipedia.org/wiki/Favicon
main.py is the entrypoint for your application

A first helloworld/app.yaml

application: john-pfeiffer
version: 1
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /favicon\.ico
  static_files: favicon.ico
  upload: favicon\.ico

- url: .*
  script: main.app

libraries:
- name: webapp2
  version: "2.5.2"

yaml files are indentation based

A revised yaml file

runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: .*
  script: main.app

libraries:
- name: webapp2
  version: "2.5.2"

the newer requirement is to NOT include the application name or version, apparently that is passed as a CLI parameter

helloworld/main.py

#!/usr/bin/env python
import webapp2

class MainHandler(webapp2.RequestHandler):
    def get(self):
        self.response.write('Hi World!')

app = webapp2.WSGIApplication([('/', MainHandler)], debug=True)

View the app on your local machine

cd google_appengine
./dev_appserver.py ./helloworld

    INFO 2012-12-27 04:20:20,399 dev_appserver_multiprocess.py:655]
    Running application dev\~john-pfeiffer on port 8080:
    http://localhost:8080

    INFO 2012-12-27 04:20:20,399 dev_appserver_multiprocess.py:657] Admin console is available at: http://localhost:8080/\_ah/admin

Upload the app to Google App Engine

./appcfg.py update helloworld/

    07:44 PM Host: appengine.google.com
    07:44 PM Application: john-pfeiffer; version: 1
    07:44 PM Starting update of app: john-pfeiffer, version: 1
    07:44 PM Getting current resource limits.
    07:44 PM Scanning files on local disk.
    07:44 PM Cloning 1 static file.
    07:44 PM Cloning 3 application files.
    07:44 PM Uploading 1 files and blobs.
    07:44 PM Uploaded 1 files and blobs
    07:44 PM Compilation starting.
    07:44 PM Compilation completed.
    07:44 PM Starting deployment.
    07:45 PM Checking if deployment succeeded.
    07:45 PM Will check again in 1 seconds.
    07:45 PM Checking if deployment succeeded.
    07:45 PM Will check again in 2 seconds.
    07:45 PM Checking if deployment succeeded.
    07:45 PM Deployment successful.

Verify with curl http://john-pfeiffer.appspot.com

Download an existing application from Google App Engine

mkdir -p /tmp/myapp
appcfg.py download_app -A john-pfeiffer /tmp/myapp

This will use OAuth 2 to open a browser/give you a link where you can confirm the action Once confirmed it will download the latest version off your application code and files This is independent of and not a replacement for version control!

Updating the Application for routes and POST requests

It is easy to use the MVC pattern while inheriting from the framework https://webapp-improved.appspot.com/guide/handlers.html

app.yaml

Note that the version has to be explicitly updated in order to deploy something new

application: john-pfeiffer
version: 6
runtime: python27
api_version: 1
threadsafe: yes

handlers:
- url: /favicon.ico
  static_files: favicon.ico
  upload: favicon.ico

- url: /.*
  script: main.app

libraries:
- name: webapp2
  version: "2.5.2"

main.py

#!/usr/bin/env python
import webapp2

class MainHandler(webapp2.RequestHandler):
    def get(self):
        self.response.write('hello John Pfeiffer!')

class PageOne(webapp2.RequestHandler):
    def get(self):
        self.response.write("""
            PageOne
        """);

class PageTwo(webapp2.RequestHandler):
    def get(self):
        self.response.write('PageTwo')

    def post(self):
        get_values = self.request.GET
        post_values = self.request.POST
        self.response.write(str(get_values) + "\n")
        self.response.write(str(post_values))

app = webapp2.WSGIApplication([
        ('/', MainHandler),
        ('/page-one', PageOne),
        ('/page-two', PageTwo)
    ], debug=True)

def main():
    run_wsgi_app(application)


if __name__ == "__main__":
    main()

Update the google app engine application code that is running in the cloud

./appcfg.py update john-pfeiffer

update does not publish the new version yet

./appcfg.py set_default_version john-pfeiffer

or you can use the WebUI dashboard to change the default or delete a Version for an Application Instance

https://cloud.google.com/appengine/docs/python/config/appref

A simple single file CRUD app using the Google AppEngine Database

Google AppEngine applications can leverage the platforms NoSQL database https://cloud.google.com/appengine/docs/python/datastore/

The example application below also shows how to override the default 404 and 500 errors with custom jinja2 templates which would require installing the jinja2 dependency and an extra subdirectory named templates with the HTML

https://blog.john-pfeiffer.com/jinja2-a-web-html-template-layout-for-everyone/
https://cloud.google.com/appengine/docs/python/ndb/ has improved and deprecated the DB Datastore library used below

#!/usr/bin/env python
# 2013-01-20 johnpfeiffer

import os
import logging
import traceback
import cgi
import datetime
import webapp2
import jinja2

from google.appengine.ext import db

jinja_environment = jinja2.Environment( loader=jinja2.FileSystemLoader( os.path.dirname( os.path.dirname( __file__ ) ) ) )

class Node( db.Model ):
    id = db.StringProperty()
    name = db.StringProperty()
    parent_id = db.StringProperty()
    date = db.DateTimeProperty( auto_now_add = True )


# TODO: navigation bar
class MainPage( webapp2.RequestHandler ):
  def get( self ):
    self.response.out.write( '<html><body>' )
    self.response.out.write( """Welcome <br/>

         <form action="/listnodes" method="get">
             <input type="submit" value="List Nodes">
         </form>
         <br/>
         <form action="/createnodeform" method="get">
             <input type="submit" value="Create Node">
         </form>
         <form action="/deletenode" method="post">
             <label>id</label></td><td><input type="text" id="id" name="id" />
             <input type="submit" value="Delete Node">
         </form>

         """
    )
    self.response.out.write( "</body></html>" )


class ListNodes( webapp2.RequestHandler ):
  def get( self ):
    self.response.out.write( '<html><body>' )
    self.response.out.write( 'Welcome <br/>' )
    q = db.Query( Node )
    q.order( "name" )
    q.fetch( 100 )
    self.response.out.write( "%s , %s , %s ( %s )<br/>" % ( "name" , "id" , "parent_id" , "created" ) )
    for node in q :
        self.response.out.write( "%s , %s , %s ( %s )<br/>" % ( node.name , node.id , node.parent , node.date ) )

    self.response.out.write( "</body></html>" )


# TODO: use template system
class CreateNode( webapp2.RequestHandler ):
  def get( self ):
    self.response.out.write( '<html><body>' )
    self.response.write( 'Create uses POST' )
    self.response.out.write( "</body></html>" )

  def post( self ):
    self.response.out.write( '<html><body>' )
    post_values = self.request.POST

    # todo extract to helper for input validation and sanitization
    name = post_values[ "nodename" ]
    id = post_values[ "id" ]
    parent_id = post_values[ "parentid" ]

    if( name == None or id == None or parent_id == None ):
        self.response.out.write( 'ERROR: DEBUG:' , post_values )
    else:
        node = Node()
        node.name = name
        node.id = id
        node.parent_id = parent_id

        node.put()
        self.response.out.write( 'successfully created: ' + name )
        self.response.out.write( """
         <form action="/listnodes" method="get">
             <input type="submit" value="List Nodes">
         </form>
        """
        )

    self.response.out.write( "</body></html>" )


class CreateNodeForm( webapp2.RequestHandler ):
    def get( self ):
        self.response.out.write( '<html><body>' )
        self.response.out.write( """
         <form action="/createnode" method="post">
            <table>
                <tr><td><label>node name</label></td><td><input type="text" id="nodename" name="nodename" /></td></tr>
                <tr><td><label>id</label></td><td><input type="text" id="id" name="id" /></td></tr>
                <tr><td><label>parent id</label></td><td><input type="text" id="parentid" name="parentid" /></td></tr>
                <tr><td></td><td><input type="submit"></td></tr>
            </table>
          </form>
        """ )
        self.response.out.write( "</body></html>" )


# TODO: use template system
class DeleteNode( webapp2.RequestHandler ):
  def get( self ):
    self.response.out.write( '<html><body>' )
    self.response.write( 'Delete uses POST' )
    self.response.out.write( "</body></html>" )

  def post( self ):
    self.response.out.write( '<html><body>' )
    post_values = self.request.POST

    # todo extract to helper for input validation and sanitization
    id = post_values[ "id" ]

#    q = db.Query( Node )    # keys_only is faster and cheaper than retrieving the entities
#    q.filter( "id=" , id )
    q = Node.all( keys_only = True ).filter( "id=", id )
    node_to_delete = q.run()

#    db.delete( )
#    node_key =          # __key__

    self.response.out.write( "%s , %s , %s ( %s )<br/>" % ( "name" , id , "parent_id" , node_to_delete ) )


    self.response.out.write( "</body></html>" )



# url handler below -----------------------------

app = webapp2.WSGIApplication( [
  ( '/', MainPage ),
  ( '/listnodes' , ListNodes ),
  ( '/createnode' , CreateNode ),
  ( '/createnodeform' , CreateNodeForm ),
  ( '/deletenode' , DeleteNode )
], debug = True )


def handle_404 (request , response , exception) :
    template_dictionary = { 'title' : 'ERROR 404' , 'body_content' : exception.status }
    template = jinja_environment.get_template ( 'templates/error.html' )
    response.write ( template.render ( template_dictionary ) )
    response.set_status ( exception.status_int )


def handle_500 (request , response , exception) :
    logging.error ( traceback.print_exc ( ) )
    logging.error ( exception )
    template_dictionary = { 'title' : 'Meow' , 'body_content' : 'Meow.  Meow meow meow, meow meow.' }
    template = jinja_environment.get_template ( 'templates/error.html' )
    response.write ( template.render ( template_dictionary ) )
    response.set_status ( 500 )


app.error_handlers [ 404 ] = handle_404
app.error_handlers [ 500 ] = handle_500

Pointing DNS and your domain to your appengine app

If you want to have your appengine url be customized with a nice domain name you will need to have your registrar (like Namecheap =) point the DNS towards google:

https://cloud.google.com/appengine/docs/standard/python/mapping-custom-domains

Server Operations: Cloud versus Build Your Own

2012-11-08T17:06:00-08:00

Here's my response to Jeff Atwood's calculations and incorrect conclusion about Building Your Own Server: I'm not sure I know when an organization's Production deployment doesn't need reduced complexity/costs (people!), flexibility for load, and redundancy...

"Anyway, I want to make it clear that building and colocating your own servers isn't (always) crazy, it isn't scary, heck, it isn't even particularly hard. In some situations it can make sense to build and rack your own servers, provided ...

you want absolute top of the line server performance without paying thousands of dollars per month for the privilege
you are willing to invest the time in building, racking, and configuring your servers
you have the capital to invest up front
you desire total control over the hardware
you aren't worried about the flexibility of quickly provisioning new servers to handle unanticipated load
you don't need the redundancy, geographical backup, and flexibility that comes with cloud virtualization"

http://www.codinghorror.com/blog/2012/10/building-servers-for-fun-and-prof-ok-maybe-just-for-fun.html

Hi Jeff, long time fan, first time commenter... I love building servers too and I've managed a small group of servers, I personally use Linode, and my currently company uses AWS and some internal servers...

You would agree that in coding you pick the right tool for the job (scientific computing would use a different technology stack than a standard ecommerce startup website)...

AWS is elastic (you pay a premium for being to scale up or down - and there's value to the agility with which you can change or add new services)
AWS RDS is a huge improvement over managing MySQL replication, and they have Elastic Load Balancing and lots of other addons that take serious Ops chops to create and maintain
Server operations cost is not the raw hardware:
1. The biggest cost in Ops is people (same as coding), so leveraging Amazon saves on how many people you need to pay to manage your server farm (yes, SysAdmins take holidays and change jobs so cost = N+1 )... you can outsource half way by colocating but setting up the redundancy, monitoring, auto scaling, etc. becomes a physical pain (you want West Coast and East Coast servers, right?).
2. The infrastructure of cooling, UPS, network (bandwidth!), backups, etc. is also a big factor in Operations (does your server room have building security? a backup generator?)

My point is that for a stealth mode startup or any internal lab testing buying servers is a no brainer - do it with ESXi or OpenStack and hack away!

BUT for Production you'll need some Cloud strategy (AWS competitors: RedHat OpenShift, RackSpace Cloud, IBM, ATT Compute, Google AppEngine, etc. means lower prices and improved services)

Unless, as you've already mentioned, "if you happen to have hanging around a pile of cash and tech expertise that's underutilized..."

Tomcat deployment on Openshift for free

2012-09-11T13:48:00-07:00

openshift is the cloud PaaS offering from RedHat

Prerequisites and dependencies

sudo apt-get update
sudo apt-get install ruby1.9.3 git-core
//yay for ubuntu 12.04

sudo gem install rhc
//red hat openshift client

OpenShift Client tools setup

rhc setup

Created local config file: /home/ubuntu/.openshift/express.conf
The express.conf file contains user configuration, and can be transferred to different computers.
No SSH keys were found. We will generate a pair of keys for you.

2: No such file or directory
Created: /home/ubuntu/.ssh/id_rsa.pub

Your public ssh key must be uploaded to the OpenShift server. Would you like us to upload it for you? (yes/no) yes

rhc commands

rhc -h

rhc domain show

prompts for password

rhc app create -h

Valid application types are (nodejs-0.6, ruby-1.9, jbossas-7, python-2.6, jenkins-1.4, ruby-1.8, jbosseap-6.0, diy-0.1, php-5.3, perl-5.10)

rhc app create -a john -t diy-0.1

rhc app show -a john

rhc app cartridge list

Your local git repo

ON YOUR LOCAL MACHINE BROWSE TO WHERE YOU WANT TO STORE YOUR GIT REPO

git clone ssh://a261d0fc2932413694456e3473fdc972@APPNAME-DOMAIN.rhcloud.com/~/git/...

git status

git pull

REPO LAYOUT of ~/john/repo

.openshift/action_hooks/start - Script that gets run to start your application  
.openshift/action_hooks/stop - Script that gets run to stop your application  
.openshift/action_hooks/pre_build - Script that gets run every git push before the build  
.openshift/action_hooks/build - Script that gets run every git push as part of the build process (on the CI system if available)  
.openshift/action_hooks/deploy - Script that gets run every git push after build but before the app is restarted  
.openshift/action_hooks/post_deploy - Script that gets run every git push after the app is restarted

diy  
misc  
README  
static/ If it exists externally exposed static content goes here

CHANGES ARE ONE DIRECTIONAL FROM THE GIT CLONE TO THE OPENSHIFT BOX

mv diy/testrubyserver.rb ../misc

mv diy/index.html ../misc

git add -A

git commit -m "moved initial test stuff to /misc"

git push

if the app is running then a git push automatically ...

Counting objects: 6, done.
Delta compression using up to 4 threads.  
Compressing objects: 100% (4/4), done.  
Writing objects: 100% (4/4), 607 bytes, done.  
Total 4 (delta 1), reused 1 (delta 0)  
remote: Stopping application...  
remote: Done  
remote: ~/git/john.git ~/git/john.git  
remote: ~/git/john.git
remote: Running .openshift/action_hooks/pre_build  
remote: Running .openshift/action_hooks/build  
remote: Running .openshift/action_hooks/deploy  
remote: Starting application...  
remote: Done  
remote: Running .openshift/action_hooks/post_deploy

ssh 33d90ea45fd91e42096651d937e@john-pfeiffer.rhcloud.com

//note that HOME is /var/lib/stickshift/12315longidentifier

cd app-root/data

wget http://mirror.cc.columbia.edu/pub/software/apache/tomcat/tomcat-7/v7.0.29/bin/apache-tomcat-7.0.29.tar.gz

tar -xzvf apache-tomcat-7.0.29.tar.gz

expands to 13MB, note that JAVA is already installed by default

rm apache-tomcat-7.0.29.tar.gz

Openshift ports and proxy

Since OpenShift has a proxy setup that passes port 80 to your local server on port 8080, BUT

OpenShift does not allow users to bind to any port below 15000 other than 8080, so...

ssh 33d90ea45fd91e42096651d937e@john-pfeiffer.rhcloud.com

env

shows you all of the environment variables in the openShift

multitenant config

env | grep INTERNAL

OPENSHIFT_INTERNAL_PORT=8080
OPENSHIFT_INTERNAL_IP=127.13.29.1

vi app-root/data/apache-tomcat-7.0.29/conf/server.xml

escape key then :x (save and quit)

cd app-root/data/apache-tomcat-7.0.29/bin

sh startup.sh && tail -f ../logs/*

(this is how you can start it manually from within ssh, you'll have to stop it manually too!)

ADDING Tomcat to your default start and stop scripts (which are used during every git push)

In your local git repo there is a hidden directory ".openshift"

cd APPNAME/.openshift/action_hooks

vi .openshift/action_hooks/start

#nohup $OPENSHIFT_REPO_DIR/diy/testrubyserver.rb
$OPENSHIFT_INTERNAL_IP $OPENSHIFT_REPO_DIR/diy > $OPENSHIFT_LOG_DIR/server.log 2>&1 &

cd $OPENSHIFT\_DATA\_DIR/apache-tomcat-7.0.29/bin
nohup sh startup.sh
echo "completed tomcat7 startup"

vi .openshift/action_hooks/stop

cd $OPENSHIFT_DATA_DIR/apache-tomcat-7.0.29/bin  
nohup sh shutdown.sh  
echo "completed tomcat7 shutdown"  
exit 0

git add -A

`git commit -m "removed testrubyserver.rb and added tomcat to start/stop scripts"

git push

http://APPNAME-DOMAINNAME.rhcloud.com

Autodeploy the latest

MOVE YOUR WEBAPPS DIRECTORY TO THE GIT REPO SO THAT A GIT PUSH WILL AUTO DEPLOY THE NEWEST

mv $OPENSHIFT_DATA_DIR/apache-tomcat-7.0.29/webapps ~/john/repo/diy/webapps

ln -s ~/john/repo/diy/webapps webapps

Strongly advised to remove the manager and example apps (just deploy your .war's)

rhc app stop -a APPNAME -p YOURPASSWORD

yes, it uses your RHCloud account username and password for app management

ssh 33d90ea45fd91e42096651d937e@john-pfeiffer.rhcloud.com

mv ~/app-root/data/apache-tomcat-7.0.29/webapps/* app-root/repo/misc

mv app-root/repo/misc/ROOT ~/app-root/data/apache-tomcat-7.0.29/webapps

rhc app start -a APPNAME -p YOURPASSWORD

ONCE YOU'VE SSH'D IN...

help

ps|ls|
ctl_app start [stop|restart|status]
mysql | mongo | quota

NOTE: sometimes it's easier to use a UI https://openshift.redhat.com/app/console/applications

My Account -> Public Keys

My Applications -> APPLICATION_NAME ->

rhc app add-alias -a myapp --alias myapp.net

Future Thoughts

Eclipse + m2e (maven plugin) + jetty plugin for fast and easy dependency management -> mvn install + added custom script can put your .war into your local openshift repo for continuous deployment.

https://www.openshift.com/

Creating Economies of Scale in Software Development

2012-07-29T04:39:00-07:00

Planning to be a successful software company

Economies of Scale in Software Development is about planning to be successful.

Software development is rather cheap when compared to physical manufacturing but that doesn't mean it doesn't cost money to create a product or service.

The classic myth of the lone wolf hacker who creates a viral product runs counter to the reality of billions of dollars generated by the software industry.

Elements of a Software Team

Consider the many elements required to create a software product:

Developers (hiring, location, communication, etc.)
Developer tools (IDE, compiler, debugger, etc.)
QA (you do run tests of some sort, don't you?!?!)
Distribution/Release (shrink wrapped or delivered by Internet continuously, it still costs something)
Support (assuming you've actually got users)

Making Software Better, Faster, Cheaper

What can you do to make your software cheaper, faster, and most importantly, better?

1. Use Open Source

(like facebook, twitter, google, netflix). Leverage the crowd and reduce your costs (e.g. not buying software licenses, not paying someone to audit your software licenses, etc.)

Spending money on licenses becomes cost prohibitive for large successful organizations.

More importantly, get the quality of years of development by lots of different perspectives. The homogeneous gets wiped out by a single cause, get diversified!

Concerns about competitive advantage based on your supply chain (e.g. the myth "using Open Source tools doesn't give you an edge") are unfounded as:

a. If your competitive advantage is a "secret supplier" then your business will go bust as soon as anyone else finds out your secret.

b. If you're paranoid enough to worry your competitors are messing with your tools then you shouldn't trust a closed source vendor - keep it in the open where everyone's watching.

Giving back to open source projects you depend on has a multiplier effect: a healthy community project is far cheaper than paying full time to support your critical requirements and has built in support and marketing.

2. Standardization

Consider the trend of large clusters of commodity hardware. Using the same internal development tools means it's easier to focus on the real problems and not get lost in translation.

This doesn't mean sticking to old versions or being afraid to innovate, but make success a formula and not an accident.

Larger efficiencies can be generated by getting a decent level of determinism in the workflow.

This means use the same IDE, the same dependency libraries, communication channels, etc. If you have rock stars that can't learn the common tool or won't teach others why their method is better then you run the risk of a dysfunctional "all-star" time bomb.

This also applies to fairness: office perks, salaries, etc. Make it an organization about transparent achievable results, not a labyrinth of back room exceptions.

3. Reduced specialization (and silos)

System Administrator's whose sole purpose is to watch machines is a dying breed.

An army of QA who manually walk through the same test plan over and over is also legacy.

Start with "DRY - Do Not Repeat Yourself".

Develop automation early and make the process simple and obvious.

(i.e. Amazon's internal conversion to APIs http://apievangelist.com/2012/01/12/the-secret-to-amazons-success-internal-apis).

This also means avoiding purchasing hardware (and the associated inventory/maintenance/overhead).

The more virtual servers you buy the larger volume discount you can negotiate. It's not entirely about cost:

you're renting somebody else's implementation of best practice
you need to make your product number one, distractions from that reduce velocity.

Your feature set will continually grow. (Even with careful pruning).

By looking at software as the solution to quality/delivery/maintenance/etc. you can reduce your running cost to a fraction of what it would be AND be able scale up quickly when you become wildly successful.

Note: people are still irreplaceable but by having a lot more Developers (and less unique specialists) you can distribute the load more evenly.

Getting more cross functional individuals means less silos and less communication gaps.

The network mesh effect destroys productivity if everyone's a separate bottleneck.

4. Make it real

Code that lives solves problems and is valuable.

Uncommitted, unused, and otherwise un-useful code costs money to debug and deconstruct; much worse it costs time.

Continuous integration gets your stable unit tested code some real world bruises. Continuous deployment gets your code crunching data and making users happy.

Critical Leadership

There will be bugs.

Far more important are to ensure the product fits the market and have the ability to quickly fix what's broken - sometimes amputation due to business needs is critical (i.e. how Flickr the photo sensation was born http://en.m.wikipedia.org/wiki/Flickr)

Your team must be able to execute.

You lead them by example (both the late nights and the high fives) and these real people will deliver exponential success.

An Evolution of Questions and Answers

2012-06-21T17:46:00-07:00

Socrates oral questions (and answers) were immortalized by Plato's writings.

I love books, the distillation of the wisdom of the ages, and the libraries where they are enshrined and shared.

Visiting a library is an exciting insight into the culture and values of a location.

Encyclopedias, Dictionaries, and all manner of reference books were the compendium of answers; questions had to be researched (sometimes involving extensive legwork, detective work, and persistence).

Digitization of books (e.g. encyclopedia on cd-rom) enabled huge improvement in the access and distribution of information.

Physical bulletin boards and community centers are also places of information exchange; digital communication (modems!) transformed the reach and capacity.

Television and Video, despite many attempts, has long remained a one way medium. (Though too there's something inherently relaxing about passive entertainment).

The Internet (and especially Google search) means specialist websites and esoteric questions are a couple links away; it has also enabled crowd sourcing answers.

I'm amazed at the speed and accuracy of answers aggregated by Wikipedia, StackExchange, Quora, etc. (which admittedly still require filtering and research for correctness and comprehension).

It's gotten so that now I receive answers to questions which I didn't realize I had!

Mid 2012 technology and business prediction for 2013 and beyond

2012-05-31T16:00:00-07:00

The well known perceived future business model is constant (mobile) targeted (user profiling) advertising (everywhere) with instant purchase (online + mobile) and delivery (streaming)

So Apple (Android) + Facebook + Google + Amazon + (Netflix?) , right?

I think there's more to think about: the infrastructure is the gold; Hardware Manufacturers (CPU + RAM + SSD), Servers, Routers, ISP's, Content Producers, Transaction Processing, Data and User Platforms.

The new ecosystem is inherently more distributed (more users globally, more devices) and fragmented (more OS, Apps, Content Producers, etc.) than the "Microsoft Era".

Complexity increases with more data to aggregate/index/filter and many smaller transactions: consider previously purchasing a single desktop, monitor, (Windows bundled with it) and MS Word + one or two apps (Quickbooks or Photoshop).

Now it's a laptop + phone + tablet (Windows + Android + iOS), each with many apps (including online services like Facebook) and the challenge of maintaining a consistent view of security, data, and even application state.

Business purchases used to be larger, even grouped for volume (i.e. a company orders 1,000 desktops + monitors at $1,500 a piece); now it's bring your own device ($500) and many small app and content purchases for 99 cents to $10.

Companies that solve the multi platform and scalability problems (along with of course actually creating a product that fulfills a need or desire) will get bigger faster.

The opportunities are growing exponentially (network effect) for the talented, hard working teams that can execute.

Amazon S3 Bucket HTML Redirect

2012-04-13T23:57:00-07:00

Goal: More efficient and less error prone method to update a regularly changed downloadable file

Web page redirects enable you to change the URL of a web page on your S3 hosted website (e.g., from www.example.com/oldpage to www.example.com/newpage) without breaking links pointing to the old URL. Users accessing the old URL will automatically be redirected to the new one.

Redirect a single object

Amazon updated their functionality to allow per object meta data based redirects:

WebUI
Object properties
Metadata
Add Website Redirect Location
either /newpage.html (internal redirect) or http://example2.com/page.html (external redirect)

Or PUT the object (or a zero byte file) with the x-amz-website-redirect-location header set (i.e. http://example2.com/page.html)

Or use the universal static html redirect method:

So upload the following download.html that includes

<head>
    <meta http-equiv="refresh" content="0; url=http://example.com/file-v2.tar.gz">
</head>

Whenever you have a new version of your file you only have to upload a new "download.html" with an updated meta refresh header and any Users and links will always download the newest version of your file.

Note, javascript may help you open the download and then return to the original page but have strange interactions for a .txt file...

<script type="text/javascript">
<!--
window.location = "http://example.com/file-v2.tar.gz"
//-->
</script>

Redirect a domain

With two domains, example1.com and example2.com:

Create an s3 bucket for example1.com (static web hosting)
Set bucket property in the "Static Web Hosting" section, select "Redirect all requests to another host name" to redirect to example2.com
Configure Route53 (AWS DNS) for example1.com to have an A record that has an Alias Target as an S3 Website Endpoint (the bucket from step 1)
Register example1.com to use the Amazon name servers (from Route53)
Add any further subdomain redirects (e.g. www.example1.com) by repeating steps 1 and 2

more info

http://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html

LDAP and LDAPS with Apache Directory Studio and the Java Keystore

2012-03-29T21:44:51-07:00

The popularity of LDAP for directory service/lookup means Apache Directory Services + Apache Directory Studio is an excellent combination for getting started with identity and user management.

http://directory.apache.org/apacheds/

on windows it requires a 32bit JRE =(

Download then install: dpkg -i apacheds-1.5.7.deb

later on uinstall is possible via: dpkg -r apacheds

By default Apache DS listens on 10389 and 10636 (SSL)

You can connect to it with Apache Dir Studio with "LDAP -> New Connection -> hostname , port = 10389"

BindDN or user (click on Check Authentication)
uid=admin,ou=system
password= secret

ldapsearch -v -H ldap://ldap.domain.com:10389 -x -D "uid=admin,ou=system" -w "secret" "(objectclass=*)" -b "ou=system"

Self Signed Java Keystore

A Java Keystore is different from a normal cert + key combo. Example "self signed" java keystore, key, certificate combo

NOTE: CN should really be ldap.domain.com , not Zanzibar!!!!

keytool -genkey -keyalg "RSA" -dname "cn=zanzibar, ou=ApacheDS, o=ASF, c=US"  -alias zanzibar -keystore zanzibar.jks -storepass secret -validity 730

Press Enter to use the same password for certificate as we already entered above for the keystore

keytool -list -v -keystore zanzibar.jks -storetype jks -storepass secret

ApacheDS Config File

SSL is enabled by default but needs to be modified to use a separate certificate:

sudo vi /var/lib/apacheds-1.5.7/default/conf/server.xml

 <ldapServer id="ldapServer"

            allowAnonyn mousAccess="false"
            saslHost="ldap.example.com"
            saslPrincipal="ldap/ldap.example.com@EXAMPLE.COM"
            searchBaseDn="ou=users,ou=system"
            maxTimeLimit="15000"
            maxSizeLimit="1000"
   keystoreFile="/var/lib/ssl/zanzibar.jks"
   certificatePassword="secret">

    <transports>
      <tcpTransport address="0.0.0.0" port="10389" nbThreads="8" backLog="50" enableSSL="false"/>
      <tcpTransport address="0.0.0.0" port="10636" nbThreads="8" backLog="50" enableSSL="true"/>
    </transports>

Start and verify the new config

/etc/init.d/apacheds-1.5.7-default restart
/etc/init.d/apacheds-1.5.7-default status

netstat -an --inet

start the service and check ports listening, established, time wait (especially 10636)

JXplorer or ApacheDirectoryStudio connection "LDAP -> New Connection -> hostname , port = 10636" (SSL or LDAPS)

BindDN or user (click on Check Authentication)
uid=admin,ou=system 
password= secret

SELF SIGNED CERTIFICATE WARNING - you can view the certificate and choose to manually accept...

NOTE IF YOU SEE THE ERROR "The connection failed - Connection refused: connect (zanzibar:10636)"
javax.naming.CommunicationException: zanzibar:10636 [Root exception is java.net.ConnectException: Connection refused: connect]

ENSURE THAT YOUR TRANSPORT IS NOT ONLY ALLOWING localhost! (below is the locked down setting...)

<tcpTransport address="localhost" port="10636" enableSSL="true"/>

Modify the iptables firewall

Allow well known LDAP (389) and LDAPS (636) ports to work with ApacheDS

iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 389 -j REDIRECT --to-port 10389
iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 636 -j REDIRECT --to-port 10636

Test and verify connectivity from a remote machine

ldapsearch -H ldap://ldap.domain.com:389 -x -D "uid=example,ou=system" -W -b "ou=system" "(uid=example)"

Extract a Certificate from the Java Key Store

EXTRACT A CERTIFICATE FROM A JKS INTO A .DER, THEN CONVERT IT INTO A .PEM AND IMPORT INTO THE JVM cacerts (this allows a java based app to connect to your self signed cert LDAP server)

keytool -list -v -keystore zanzibar.jks -storetype jks -storepass secret
  Alias name: zanzibar

keytool -export -alias zanzibar -keystore zanzibar.jks -storepass secret > zanzibar.der
openssl x509 -in zanzibar.der -inform DER -out zanzibar.crt -outform PEM
openssl x509 -text -in zanzibar.crt | head        //verify you have a valid cert

openssl s_client -connect localhost:10636                           //Verify return code: 18 (self signed certificate)
openssl s_client -connect localhost:10636 -CAfile zanzibar.crt      //Verify return code: 0 (ok)

keytool -import -trustcacerts -alias zanzibar -file zanzibar.crt -keystore /etc/java-6-sun/security/cacerts -storepass changeit

Test and Verify the LDAPS connection

openssl s_client -connect ldap.domain.com:636

Verify return code: 18 (self signed certificate) openssl s_client -connect ldap.domain.com:636 -CAfile zanzibar.crt Verify return code: 0 (ok)

Use an existing certificate and key and create a Java Key Store

EXISTING CERTIFICATE AND KEY AND CREATE A .JKS

openssl pkcs12 -export -in cert.crt -inkey cert.key -out keystore.p12 -name serveraliashostname -CAfile intermediatebundle.crt -caname root
Enter Export Password: secret

keytool -list -v -keystore keystore.p12 -storetype pkcs12

CONVERT THE PKCS12 TO JKS FORMAT

keytool -importkeystore -srckeystore keystore.p12 -srcstoretype PKCS12 -deststoretype JKS -destkeystore keystore.jks

keytool -list -v -keystore keystore.jks -storetype jks

sudo vi /var/lib/apacheds-1.5.7/default/conf/server.xml 
      keystoreFile="/var/lib/ssl/keystore.jks"

still need to figure out how to include the intermediate CAcert root chain...!

IMPORTING/REPLACING A JKS KEYSTORE

keytool -importkeystore -deststorepass secret -destkeypass secret -destkeystore keystore.jks -srckeystore keystore.p12 -srcstoretype PKCS12 -srcstorepass secret -alias oldaliasname

GUI CONVERTER FROM PKCS12 to JKS (requires Java 1.6 or higher) http://portecle.sourceforge.net/ (open the .p12 , Tools -> Keystore type to JKS) (seems to have a funny message where it needed change the password to "password)")

keytool -import -trustcacerts -alias ldap.domain.com -file cert.crt -keystore zanzibar.ks -storepass secret
keytool -import -trustcacerts -alias root -file GeoTrust.crt -keystore zanzibar.ks

Both self signed and third party certificates can be supported with ApacheDS LDAPS.

LDAP with SSL is a little tricky and it's useful to use openssl and ldapsearch to verify.

 <ldapServer id="ldapServer"
            allowAnonymousAccess="false"
            saslHost="ldap.example.com"
            saslPrincipal="ldap/ldap.example.com@EXAMPLE.COM"
            searchBaseDn="ou=users,ou=system"
            maxTimeLimit="15000"
            maxSizeLimit="1000"
            keystoreFile="/var/lib/ssl/zanzibar.jks"
            certificatePassword="secret">

    <transports>
      <tcpTransport address="0.0.0.0" port="10389" nbThreads="8" backLog="50" enableSSL="false"/>
      <tcpTransport address="0.0.0.0" port="10636" nbThreads="8" backLog="50" enableSSL="true"/>
    </transports>

openssl s_client -connect domain.com:10636 -CAfile intermediate.crt

ldapsearch -H ldaps://ldap.domain.com:10636 -x -D "uid=example,ou=system" -w PASSWORD -b "ou=system" "(uid=example)"

ERROR: ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

ON A REMOTE MACHINE IN ORDER TO USE LDAPSEARCH WITH LDAPS 636...

sudo nano /etc/ldap/ldap.conf

TLS_CACERT /var/lib/ssl/intermediate.crt
TLS_CACERTDIR /var/lib/ssl/
TLS_CERT /var/lib/ssl/cert.crt

NOW THE BELOW WILL WORK

ldapsearch -H ldaps://ldap.domain.com:10636 -x -D "uid=example,ou=system" -w PASSWORD -b "ou=system" "(uid=example)"
ldapsearch -H ldaps://ldap.domain.com:636 -x -D "uid=example,ou=system" -w PASSWORD -b "ou=system" "(objectclass=*)"

Self configuration tests for scalability

2012-03-12T02:42:00-07:00

Goal: Customers (Users = system admins) able to self verify configuration.

The more a user can self verify with software, the less support required per customer deployment.

I used the Java API to quickly prototype a solution and exported a runnable .jar file...

https://code.google.com/p/atmos-java/

Java source code for a configuration validation

import com.emc.esu.api.EsuApi;
import com.emc.esu.api.rest.EsuRestApi;
import com.emc.esu.api.EsuException;
import com.emc.esu.api.ObjectId;
import com.emc.esu.api.ObjectInfo;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import org.apache.log4j.ConsoleAppender;
import org.apache.log4j.PatternLayout;

public class AtmosConnect
{
    static Logger rootLogger = Logger.getRootLogger();
    public static void main( String[] args )
    {
        if( !rootLogger.getAllAppenders().hasMoreElements() )
        {   rootLogger.setLevel( Level.INFO );
            rootLogger.addAppender( new ConsoleAppender( new PatternLayout(PatternLayout.TTCC_CONVERSION_PATTERN ) ) );
            rootLogger.info("Entering application");
        }

        if( args.length != 4 )
        {   System.out.println( args.length + " does not equal the 4 required arguments.");
            System.out.println( "version 0.1: java -jar AtmosConnect.jar HOST PORT SUBTENANTID/UID SECRETKEY" );
            System.exit( 1 );
        }

        String HOST = args[0];
        int PORT = Integer.parseInt( args[1] );
        String FULLTOKENID = args[2];
        String SECRETKEY = args[3];

        displayConnectionCredentials( HOST , PORT , FULLTOKENID , SECRETKEY );
        EsuApi myEsuAPI = null;
        try{        myEsuAPI = new EsuRestApi( HOST, PORT, FULLTOKENID, SECRETKEY );
        }catch( EsuException e )
        {   System.out.println( "EsuRestApi Constructor failed.");
            System.out.println( e.getMessage() );
            e.printStackTrace();
        }

        ObjectId myObjectId = null;
        myObjectId = createAtmosObject( myObjectId , myEsuAPI );
        displayAtmosObject( myObjectId , myEsuAPI );
        deleteAtmosObject( myObjectId , myEsuAPI );

        rootLogger.info("Application Successful");
    } // end main()

    private static void displayConnectionCredentials( String HOST , int PORT , String FULLTOKENID , String SECRETKEY )
    {
        System.out.println( "Connecting to Host: " + HOST );
        System.out.println( "Connecting on Port: " + PORT );
        System.out.println( "Full Token ID: " + FULLTOKENID );
        System.out.println( "Secret Key: **************" );
        //System.out.println( "Secret Key: " + SECRETKEY );
    }

    private static ObjectId createAtmosObject( ObjectId myObjectId , EsuApi myEsuAPI )
    {
        try
        {   myObjectId = myEsuAPI.createObject( null, null, null,"application/octet-stream");
            System.out.println( "Created object: " + myObjectId.toString() );
        } catch( Exception e )
        {
            System.out.println("Create Object failed.");
            System.out.println( e );
            //e.printStackTrace();
            // JUnit Tests: Invalid Host, Port, SubtenantID, UID, Shared Secret, etc.
        }
        return myObjectId;
    }

    private static void displayAtmosObject( ObjectId myObjectId , EsuApi myEsuAPI )
    {   ObjectInfo myObjectInfo = null;
        myObjectInfo = myEsuAPI.getObjectInfo( myObjectId );
        //System.out.println( "ObjectInfo: " + myObjectInfo.toString() );
        //System.out.println( "ObjectInfo as XML: " + myObjectInfo.getRawXml() );
    }

    private static void deleteAtmosObject( ObjectId myObjectId , EsuApi myEsuAPI )
    {
        try
        {   
            System.out.println( "Trying to delete Server Object: " + myObjectId.toString() );
            myEsuAPI.deleteObject( myObjectId );
            System.out.println("Test Object deleted on Server: " + myObjectId.toString() );
        } catch(Exception e)
        {   System.out.println("Delete Object " + myObjectId.toString() + " failed " + e);
            //e.printStackTrace();
        }
    }
} // end class

Verify the bash commands

Verify bash commands that will help extract the configuration:

/bin/grep -i 'emcIpAddress=' /var/lib/tomcat6/webapps/storagegateway/WEB-INF/app.properties | cut -f 2 -d '='

start on stopped rc RUNLEVEL=[2345]  
stop on runlevel [!2345]

respawn

exec /sbin/getty -n -l /etc/init.d/CONSOLEMENU.pl 38400 tty1

sudo vi /etc/init.d/CONSOLEMENU.pl

sub testatmosconnect()
{
    system("clear");
    my $sourcefilename = "/var/lib/tomcat6/webapps/storagegateway/WEB-INF/app.properties";
    my $applicationfilename = "/var/lib/tomcat6/oxygen-storagegateway/atmosconnect.jar";

    if( (-e $sourcefilename) and (-e $applicationfilename) )
    {
        my $emcipaddress = qx( /bin/grep -i 'emcIpAddress=' $sourcefilename | cut -f 2 -d '=' );
        chomp( $emcipaddress );
        my $emcportnumber = qx( /bin/grep -i 'emcPortNumber=' $sourcefilename | cut -f 2 -d '=' );
        chomp( $emcportnumber );
        my $emcuid = qx( /bin/grep -i 'emcUid=' $sourcefilename | cut -f 2 -d '=' );
        chomp( $emcuid );
        my $emcsharedsecret = qx( /bin/grep -i 'emcSharedSecret=' $sourcefilename | cut -f 2 -d '=' );
        chomp( $emcsharedsecret );

        my @status = qx( /usr/bin/java -jar $applicationfilename $emcipaddress $emcportnumber $emcuid $emcsharedsecret );

        print "@status\n";
    }else
    {   print "$sourcefilename or $applicationfilename does not exist.\n";
    }
} # end atmosconnect()

Goal Achieved

A user can self test if they've misconfigured the VM or there's missing conf files/app, etc. SUCCESS!

Next steps: further unit and system tests, debug, refactor