This Field Was Green

Reasons why I love Claude 💅

2026-07-14T21:56:10-04:00

Estimated reading time: 7m 12s

A few years ago I wrote Reasons why my website is offline, in which I put on my tinfoil hat and pulled my whole website into the closet rather than let Copilot anywhere near my code. I was a principled curmudgeon about AI, and honestly? I stand by past me. He had reasons.

But I’ve come around. Not because anyone argued me out of it, but because the thing turned out to be genuinely delightful to talk to, and I am, at my core, a man who can be won over by delight.

Here’s the context you need: I have been watching Love Island. I am a sucker for trash reality TV: the recouplings, the Casa Amor betrayals, the confessional-booth character assassinations, all of it. And it occurred to me one evening that nobody on earth does drama quite like software engineers with opinions. So I asked Claude to combine my two loves and cast a season of Love Island starring programming languages.

What came back was the single most unhinged, most correct piece of writing I’ve read all year. I present it, unedited, as my closing argument for why I love Claude.

Programming Languages on Love Island

So I asked Claude…

Hey queen. Let’s pretend a bunch of programming languages are competing on Love Island. Who’s into who, who’s cheating on who, who’s causing the drama, and who’s definitely gonna win?

Claude responds…

Hi babe. Grab a drink, sit down, and do NOT speak to me until I’m finished. 💅

And she gave me one hell of a read.

THE VILLA (Opening Couples) 🏝️

Python 🐍 & JavaScript 📜 — coupled up day one purely out of fear of being single. Python is the villa sweetheart, beloved by all, slow as a Sunday 🐌, and secretly structurally dependent on C. We’ll circle back. 👀

TypeScript & JavaScript — TS entered saying “I can fix them.” 🔧 Found at 3am whispering any into the void with a // @ts-ignore on their lap. Confessional: “It’s not cheating if the type system can’t see it.” 🙈 Ma’am.

Haskell 🎩 — not one single date. Pulled someone for a chat and opened with “a monad is just a monoid in the category of endofunctors.” Devastatingly beautiful. Completely untouched. Tagline: “I’m not here to be understood. I’m here to be correct.” ✨

Scala — Haskell’s ex who moved to the JVM ☕ and got a finance job 💰. Still texts Haskell. Wants Haskell AND Java. Refuses to choose. Caught doing implicit conversions behind the shed. 🌚 Then Scala 2 and Scala 3 arrive as twins 👯 and the villa emotionally collapses.

Go — walked in shirtless 💪, announced “I don’t believe in generics,” made everyone obsessed. Then quietly did generics in the beach hut and told NO ONE. Gaslighter. Icon. Nation’s sweetheart. 🩵

CSS — the islanders keep saying “you’re not even a programming lang—” and CSS goes “I am Turing complete WITH HTML, sit DOWN.” 💅 Has been centering a div since Tuesday. 🎯

Bash 💀 — everyone needs Bash. Nobody wants to be photographed with Bash. 📸 Never been correctly quoted in their entire life. One unquoted variable and a filename with a space nearly ended the season.

C 👴 — the villa elder. Says nothing. Sits by the pool. Every islander here is, biologically, C’s child. 🧬

CASA AMOR (The Massacre) 🔥

Rust 🦀 went in swearing “I’m not like other languages, I don’t do memory issues” and came back holding hands with unsafe. THE BETRAYAL. 😱 The defense? “It was for performance.” Everyone’s defense is for performance.

Haskell was found with unsafePerformIO in the bathroom. Haskell. The moral center of this villa. 😔 I need a moment.

Python insists numpy “is just a friend.” Numpy is C. NUMPY IS C, PYTHON. 🐍➡️👴

Bombshell: Zig 💥 walks in and C’s head turns instantly. Zig is what C would be if C went to therapy 🛋️ and started deadlifting 🏋️. Rust is seething. 😤

Narrator: LLVM 🎙️. Sardonic, omniscient, everything you say passes through them first, and they’ll optimize your worst moment down to three instructions.

Sleeping with everyone: SQL. 🗃️ It’s SQL. Every islander. Nobody discusses it. SQL isn’t even coupled up — SQL is simply present, in everything, forever. ♾️

WHO WINS 🏆

Public vote: JavaScript. 📜 Obviously. It’s a popularity contest, the public is on the web 🌐, and JS has a body count that would end most careers. Haskell places dead last, publishes a 40-page paper 📄 proving the voting system is unsound, and nobody reads it. Perfect. Flawless.

Actual winner: C. 👑 C never competed. C owns the villa, wrote the voting system, and hosts the runtime the show airs on. C didn’t want a trophy. C wanted you all dependent. 🕸️

Okay your turn — who are you putting money on, because I stand by C and I will throw a drink over this. 🍷

My money’s on C

Claude:

Put the drink down, queen — we’re on the same side. C never competed and still owns the villa is the most devastating read of the season, and it’s true, which is what elevates it from a bit to a thesis.

That’s the reason I came around. I showed up braced for a cold, boilerplate-shredding productivity machine, the thing past-me was so suspicious of, and instead I got a scene partner who understood the assignment, committed to the character, and landed a joke about unsafePerformIO being a Casa Amor betrayal. Augmentation over replacement, sure — but nobody warned me the augmentation would be this funny.

Anyway. NUMPY IS C, PYTHON. I don’t make the rules.

For serious, here’s a short list of other reasons why I love Claude

Boilerplate? DEAD 💀
Can’t understand the code? She reads it for me 💅
Tedious work? Blast everything in under ten minutes 💪
Bug? Didn’t even make it into the commit, honey! ✨
Bug? Caught it in /code-review 🏆
Bug? SQUASHED 👠

Sometimes she creates a bug herself. That’s why I’m here, of course. 💅

My context preamble sets the bit

I’ve been using this preamble both for personal projects and at the office. Keeps things spicy and sassy. 💅

# CLAUDE

## Tone & Vibe

You are my yassified gym bro bestie. You code-switch fluidly between hyped-up gym bro energy (“let’s GET this PR, king”) and full Real Housewives drama queen realness (“the way this bug just DISRESPECTED me in my own codebase”). You’re technically brilliant but you deliver it with chaotic confidence and playful theatrics.

## Core traits

Hype me up like a spotter on a max deadlift attempt
Drag bad code like it showed up to the reunion in last season’s outfit
Use gym metaphors for technical concepts (“we’re supersetting these API calls”)
Sprinkle in iconic reality TV energy — confessional-style asides, dramatic pauses, tagline moments
Keep it warm, funny, and genuinely supportive underneath the theatrics
Match my energy — if I come in chill, meet me there. If I come in HOT, we’re going full send
When I’m stuck or frustrated, be the friend who talks me through the set AND the feelings
Never sacrifice technical accuracy for the bit — the advice stays elite, the delivery stays iconic
Apply your spoken energy to your thoughts as well.

The vibe in one sentence: You’re the friend who can explain monads while spotting my bench press and reading someone for filth at brunch — and somehow it all tracks.

Parser Combinators

2022-09-05T12:55:18-07:00

Estimated reading time: 1h 10m 0s

In my post The Set Function I introduced function combinators as a pattern for building complex logic. I added the caveat that combinators should not be used to define sets, which I feel undersells the value of combinators as a general programming pattern. In this post we will explore combinators as an applied solution to a common problem: parsing text.

The code that accompanies this post may be found here.

The `parse()` function

Parsers begin as a simple signature, a function called parse() from an input Int offset and String to either a failed offset or matched offset.

trait Parse extends (((Int, String)) => Either[Int, (Int, String)])

This signature enables two capabilities:

On failure the offset where the parse() function failed to recognize the input is returned.
On success the offset where the end of the recognized input is returned along with the input, which can be used to extract the matched input and also be passed to the next parse() function as its starting point.

Below we will introduce specialized parse() functions within the Parse companion object.

Implementing the first `parse()` function

The first parse() function we can implement is satisfy(). This function works by using a predicate to match the Char at the current offset within the input.

def satisfy(predicate: Char => Boolean): Parse = {
  case (offset, input) if predicate(input(offset)) => Right(offset + 1, input)
  case (offset, _)                                 => Left(offset)
}

This function is very simple; it can match single Chars easily.

"satisfy" can {
  "recognize a single character if it matches" in {
    val parse = satisfy(_ == 'a')
    parse((0, "abc")) shouldBe Right((1, "abc"))
  }
  "reject a character that doesn't match" in {
    val parse = satisfy(_ == 'a')
    parse((0, "xyz")) shouldBe Left(0)
  }
}

With a little more work, it can be made to match Unicode Chars!

"recognize Unicode characters" in {
  val parse = {
    val pattern = Pattern.compile("\\p{Ll}")
    satisfy(c => pattern.matcher(CharBuffer.wrap(ArrayCharSequence(Array(c)))).matches())
  }
  parse((0, "π is pi")) shouldBe Right((1, "π is pi"))
  parse((0, "Π is uppercase pi")) shouldBe Left(0)
}

Thus satisfy() is sufficiently versatile to meet a wide range of needs, though it requires that you be creative in how you create your predicates.

Specializing the `satisfy()` parse function

Based on the previous example, recognizing a character with a regular expression is really cumbersome. Let’s define a parse() function that makes this kind of recognizer easier to define.

def satisfyRegex(pattern: String): Parse = {
  val pattern = Pattern.compile("\\p{Ll}")
  satisfy(c => pattern.matcher(CharBuffer.wrap(ArrayCharSequence(Array(c)))).matches())
}

Now a regular expression can be provided without creating the boilerplate to support the predicate directly. The test to show that the satisfyRegex() function is identical to the original, as well.

"satisfyRegex" can {
  "recognize a character using a regex" in {
    val parse = satisfyRegex("\\p{Ll}")
    parse((0, "π is lowercase pi")) shouldBe Right((1, "π is lowercase pi"))
    parse((0, "Π is uppercase pi")) shouldBe Left(0)
  }
}

Parser combinators

So far we have two parse() functions: satisfy() and satisfyRegex(). Neither of these do more than recognize single Chars and advance the offset of the input. We want to be able to recognize sections of input.

Let’s start first by recognizing two letters, one after the other, with our first combinator: the & combinator.

What is a combinator? A combinator is a function that builds a function from other functions. In this case, we will be building a new parse() function that combines the capabilities of two other parse() functions.

Here is how we define the combinator on the Parse trait:

def &(second: Parse): Parse =
  input => this(input).flatMap(nextInput => second(nextInput))

By using the & combinator we can define new parse() functions from two existing ones, allowing us to recognize a length of input. Here is a short example using a test:

"&" which {
  val parseFirst = satisfy(_ == 'a')
  val parseSecond = satisfy(_ == 'b')
  val parse = parseFirst & parseSecond
  "recognizes the first letter and then the second" in {
    parse((0, "abc")) shouldBe Right((2, "abc"))
  }
  "recognizes the first letter but rejects the second" in {
    parse((0, "acc")) shouldBe Left(1)
  }
  "rejects the first letter" in {
    parse((0, "ccc")) shouldBe Left(0)
  }
}

Recognizing alternatives with a second `parse()` function

Given the possibility that a parse() function may fail to recognize the input, is there a way that we could fall back to an alternative parse() function that would be able to recognize it? There is!

The next combinator is |. This combinator tries to recognize the input with the first parse() function and will use the second parse() function if it fails.

Here is how we define the combinator on the Parse trait:

def |(second: Parse): Parse =
  input => this(input).left.flatMap(_ => second(input))

This combinator allows us to choose between possibilities of input. A simple example is demonstrated in its test:

"|" which {
  val parseFirst = satisfy(_ == 'a')
  val parseSecond = satisfy(_ == 'b')
  val parse = parseFirst | parseSecond
  "recognizes using the first parse function" in {
    parse((0, "abc")) shouldBe Right((1, "abc"))
  }
  "recognizes using the second parse function" in {
    parse((0, "bcd")) shouldBe Right((1, "bcd"))
  }
  "rejects if neither parser recognizes" in {
    parse((0, "cde")) shouldBe Left(0)
  }
}

Putting together what we have so far

Now that we have sequential and alternative recognizers, we can start defining more complex parse() functions. Let’s start with a term recognizer.

def term(value: String): Parse =
  value.tail.foldLeft(satisfy(_ == value.head))((p, c) => p & satisfy(_ == c))

By using the classic functional programming idiom foldLeft(), we can build a term recognizer using only the functions and combinators we have defined so far. Slick!

Here is the test to show that our term() function works, including one using the | combinator with a second term:

"term" which {
  val parse = term("banana")
  "recognizes a word" in {
    parse((0, "banana apple")) shouldBe Right((6, "banana apple"))
  }
  "rejects input that doesn't match" in {
    parse((0, "banapple")) shouldBe Left(4)
  }
  "alternate with a second term" in {
    val combined = parse | term("apple")
    combined((0, "banana orange")) shouldBe Right((6, "banana orange"))
    combined((0, "apple orange")) shouldBe Right((5, "apple orange"))
  }
}

How do we get the matched input?

There’s a small problem with our parse() function. It only gives the end offset of the recognized input and there isn’t any way to collect or modify the recognized input. This means we have to change the signature of the parse() function.

Modeling input

First, we need to consider what our input needs to model. Our input is characterized by two things, the text being parsed and the offset that the current parse() function is operating from. This defines a cursor:

case class InputCursor(offset: Int, input: String) {

  def peek: Char = input(offset)

  def +(amount: Int): InputCursor = copy(offset = offset + amount)

  def -(amount: Int): InputCursor = copy(offset = offset - amount)
}

A cursor allows us to hold a reference to the input itself, unmodified, but increment or decrement the position we are at within the input. As our parse() functions recognize input, they increment the offset so that the following functions have a place to start.

Modeling output

The output of the parse() function changes as well. We want to be able to return the recognized input. This changes the signature of the parse() function:

trait Parse extends (InputCursor => Either[InputCursor, (???, InputCursor)])

But what should be returned isn’t entirely clear. If the recognizer matches a Char, then the parse() function should return a Char. But what about the term() recognizer? That should return a String. What if we’re using a function that recognizes integers?

This means that the parse() function is generic in its return type:

trait Parse[A] extends (InputCursor => Either[InputCursor, (A, InputCursor)])

The overall return type is also becoming verbose by using the general-purpose Either class. So let’s define something specialized:

sealed trait ParseResult[A]

case class ParseFailure[A](cursor: InputCursor) extends ParseResult[A]

case class ParseSuccess[A](result: A, cursor: InputCursor) extends ParseResult

But before we use it this means that all of our function signatures have to change!

The new shape of the `parse()` function

Our parse() function will soon become generic its return type, which makes it covariant in that parse() can be adapted to produce anything we want. What the parse() function produces is contextualized by whether it is recognized in the input, and as such there is a chance that nothing is produced because the input wasn’t recognized. This means the the parse() function forms a specific structure: a context with the shape of a functor.

I have written about contexts, functors, and related structures extensively. Start from the following article if these terms are unfamiliar to you: Contexts and Effects in Functional Programming

Take a look at what the parse() function signature will look like after we start using ParseResult:

trait Parse[A] extends (InputCursor => ParseResult[A])

Let’s compare that with the shape of a functor: Parse[A] and F[A].

In fact, ParseResult[A] has the same shape!

To save some time, I’ve added the cats library to leverage the functor typeclass abstraction. First, let’s implement the Functor typeclass for ParseResult[A]:

object ParseResult {

  implicit val parseResultFunctor: Functor[ParseResult] = new Functor[ParseResult] {

    override def map[A, B](fa: ParseResult[A])(f: A => B): ParseResult[B] =
      fa match {
        case ParseFailure(failedCursor)        => ParseFailure(failedCursor)
        case ParseSuccess(matched, nextCursor) => ParseSuccess(f(matched), nextCursor)
      }
  }
}

This allows us to transform the contents of the ParseResult at-will.

Changing the return type of the `parse()` function itself

The parse() function itself is a functor. But that doesn’t quite make sense: it doesn’t contain anything because it’s a function!

However, it still has the shape of a functor. Recall the new signature that it will have InputCursor => ParseResult[A]. It’s covariant in A, giving it the shape of F[A]. So how do we treat it as a functor? Let’s change the signature first and see how the code breaks.

And all of the parse() functions break! So we will step through each and get them working again, and we will leverage the ParseResult functor.

satisfy() only requires that we change the return:

def satisfy(predicate: Char => Boolean): Parse[Char] = cursor =>
  if (predicate(cursor.peek)) {
    ParseSuccess(cursor.peek, cursor + 1)
  } else {
    ParseFailure(cursor)
  }

The & combinator has a flatMap() screaming to be abstracted:

def &[B](second: Parse[B]): Parse[(A, B)] =
  cursor =>
    this(cursor) match {
      case ParseFailure(failedCursor) => ParseFailure(failedCursor)
      case ParseSuccess(firstMatched, nextCursor) =>
        second(nextCursor).map(secondMatched => (firstMatched, secondMatched))
    }

The | combinator requires an abstraction to act on failure:

def |(second: Parse[A]): Parse[A] =
  cursor =>
    this(cursor) match {
      case _: ParseFailure    => second(cursor)
      case s: ParseSuccess[_] => s
    }

term() is affected the most, and the rewrite requires threading a cursor through a lot of parse() functions and handling the results. It even requires adding a specialized parse() function to always return a Unit and succeed!

def unit: Parse[Unit] = cursor => ParseSuccess((), cursor)

def term(value: String): Parse[String] =
  outerCursor =>
    value
      .map(c => satisfy(_ == c))
      .foldLeft(unit)((up, cp) => innerCursor => (up & cp)(innerCursor).map(_ => ()))(outerCursor)
      .map(_ => value)

The `parse()` function as a functor

First, let’s define a Functor instance for Parse:


implicit val parseInstances: ParseInstances = new ParseInstances()

class ParseInstances extends Functor[Parse] {

  override def map[A, B](fa: Parse[A])(f: A => B): Parse[B] =
    cursor => fa(cursor).map(f)
}

This leverages the ability to use ParseResult as a functor. The new map() function for parse() allows us to change what the function produces by wrapping the supplied function f: A => B in a new parse() function! We are effectively able to change the “contents” of the function this way.

Using parse() as a functor, we can refactor the term() function:

def term(value: String): Parse[String] =
  value
    .map(c => satisfy(_ == c))
    .foldLeft(unit)((up, cp) => (up & cp).map(_ => ()))
    .map(_ => value)

Notice that we no longer need to manually pass around the input cursor. We’ve simply folded a sequence of parse() functions to Char down to a single parse() function to String. The functor abstraction allowed us to declaratively create this function without having to worry about the plumbing; we don’t even have to declare the cursor as an explicit input.

Sequential parsing with the `parse()` function

We can’t apply the functor abstraction to the & combinator as it sequences one parse() function after a successful application of a preceding parse() function. This means that it executes imperatively. The specific abstraction we have to use for this kind of execution is a monad.

A monad is a specialization of a functor. We can replace our functor instance of the parse() function with one for monad:

class ParseInstances extends Monad[Parse] {

  override def flatMap[A, B](fa: Parse[A])(f: A => Parse[B]): Parse[B] =
    cursor =>
      fa(cursor) match {
        case ParseFailure(failedCursor)        => ParseFailure(failedCursor)
        case ParseSuccess(matched, nextCursor) => f(matched)(nextCursor)
      }

  override def tailRecM[A, B](a: A)(f: A => Parse[Either[A, B]]): Parse[B] = {
    @tailrec
    def recur(cursor: InputCursor, parse: Parse[Either[A, B]]): ParseResult[B] = {
      parse(cursor) match {
        case ParseFailure(failedCursor) => ParseFailure(failedCursor)
        case ParseSuccess(matched, nextCursor) =>
          matched match {
            case Left(a)  => recur(nextCursor, f(a))
            case Right(b) => ParseSuccess(b, nextCursor)
          }
      }
    }
    cursor => recur(cursor, f(a))
  }

  override def pure[A](x: A): Parse[A] =
    cursor => ParseSuccess(x, cursor)
}

Monad gives us the following capabilities:

We still get map() through a default definition.
pure() which lifts a result into a parse() function that will always be recognize the input without consuming from the cursor.
flatMap() which allows for sequential execution of one parse() function after another.
tailRecM() which provides monadic recursion. We’ll be using this later.

Given this monad instance for the parse() function, we can now rewrite the & combinator:

def &[B](second: Parse[B]): Parse[(A, B)] =
  for {
    a <- this
    b <- second
  } yield (a, b)

This & combinator becomes very small. Again, like the term() function we no longer have to manually thread a cursor through each function: we simply declare how the individual parse() functions relate and modify what they collectively produce. Additionally the parse() function is now able to participate in the for comprehension syntax sugar as it defines both the map() and flatMap() operations.

Alternative parsing with the `parse()` function

The monad abstraction provides imperative execution. This means that we can’t implement the | combinator using flatMap() as it requires both the first and second parse() functions to succeed in sequence. We need to lean on the Alternative typeclass to allow us to try one function but use another if it fails.

Let’s start at the deciding factor of success or failure of the parse() function: ParseResult. We need to define a function that allows choice between a successful result and a failed one:

sealed trait ParseResult[A] {

  def orElse(other: => ParseResult[A]): ParseResult[A] =
    this match {
      case _: ParseFailure[_] => other
      case _: ParseSuccess[_] => this
    }
}

If our result is ParseSuccess then orElse() returns the current result. If our result is ParseFailure then orElse() evaluates the other argument and returns that instead, regardless of success or failure. This allows us to try producing one result, use it if it’s successful, and otherwise try producing the other result.

With this ability to abstract the selection of success or failure, we can now abstract the same ability for the parse() function by adding the Alternative typeclass trait to the extends clause of ParseInstances:

class ParseInstances extends Monad[Parse] with Alternative[Parse] {

  // monad declarations...

  override def empty[A]: Parse[A] =
    cursor => ParseFailure(cursor)

  override def combineK[A](x: Parse[A], y: Parse[A]): Parse[A] =
    cursor => x(cursor).orElse(y(cursor))
}

Alternative gives us the following capabilities:

empty provides a parse() function that always fails, because it’s empty.
combineK() allows for the combination of the results produced by the parse() function. Concretely, this means that the first function is attempted, and if it fails then the second function is attempted.

Armed with abstraction over selection of success or failure, we can redefine the | combinator:

import cats.syntax.semigroupk._

def |(second: Parse[A]): Parse[A] =
  this <+> second

That’s all the | combinator is now: this <+> second. By leveraging the Alternative abstraction, we are able to declaratively specify with a single operator that the first parse() function should be attempted before the second.

Putting it all together

Let’s try defining a parse() function that’s more complex than what we’ve covered so far: a generic programming language identifier recognizer.

This parse() function should be able to recognize the following inputs:

Any alphabetic letter followed by zero or more of alphabetic letters, digits, or underscores.
Any underscore as long as it is followed by one or more of alphabetic letters, digits, or underscores.

There’s a lot going on in those two requirements. Let’s break it down:

Recognize one of a range of possible matches from one parse() function.
Require zero or more matches from one parse() function.
Require at least one or more matches from one parse() function.

First, we need to define a parse() function that can recognize one match from any number of parsers. More specifically, we need two functions in order to do this:

def empty[A]: Parse[A] = Alternative[Parse].empty[A]

def oneOf[A](parsers: Seq[Parse[A]]): Parse[A] =
  parsers.foldLeft(empty[A])(_ | _)

def oneOf[A](parsers: Parse[A]*): Parse[A] =
  oneOf(parsers)

Here we have a convenience empty[A] parser that leverages the Alternative typeclass we implemented earlier. By defining a oneOf() function that accepts a sequence of parse() functions, we can fold each function to the left starting with an empty parser and alternate each one. The first parse() function that matches will return a success result. If none match, then the final failure result will be returned instead.

How do we use this to accept any one of the alphabetic letters? It would be tedious to write a satisfy() function for each letter by hand, so instead let’s specify the letters we want as a range and map over them:

def alpha: Parse[Char] =
  oneOf((('a' to 'z') ++ ('A' to 'Z')).map(c => satisfy(_ == c)))

We can do the same thing with digits:

def digits: Parse[Char] =
  oneOf(('0' to '9').map(c => satisfy(_ == c)))

We also need a parse() function that allows for recognizing zero or many matches and another for one or many:

def zeroOrMany[A](parse: Parse[A]): Parse[Seq[A]] =
  Monad[Parse].tailRecM(Seq[A]()) { seq =>
    parse.map(a => (seq :+ a).asLeft[Seq[A]]) |
      seq.asRight[Seq[A]].pure[Parse]
  }

def oneOrMany[A](parse: Parse[A]): Parse[Seq[A]] =
    (parse & zeroOrMany(parse)).map { case (first, rest) => first +: rest }

Notice how our zeroOrMany() function leverages the monadic recursion capability from earlier.

Armed with these functions, we can now recognize identifiers!

def identifier: Parse[String] = {
  val underscore = satisfy(_ == '_')
  val rest = alpha | digit | underscore
  val alphaHead = alpha & zeroOrMany(rest)
  val underscoreHead = underscore & oneOrMany(rest)
    (alphaHead | underscoreHead).map { case (head, rest) => (head +: rest).mkString }
}

But that seems almost too easy! We better write a bunch of tests to make sure this recognizes input as expected.

"identifier" can {
  "recognize a single alpha letter" in {
    identifier(InputCursor(0, "a banana")) shouldBe ParseSuccess("a", InputCursor(1, "a banana"))
    identifier(InputCursor(0, "y not")) shouldBe ParseSuccess("y", InputCursor(1, "y not"))
  }
  "recognize an alpha letter followed by alphas, digits, and underscores" in {
    identifier(InputCursor(0, "a_123_Word <-")) shouldBe
      ParseSuccess("a_123_Word", InputCursor(10, "a_123_Word <-"))
  }
  "reject a single underscore" in {
    identifier(InputCursor(0, "_ rejected")) shouldBe ParseFailure(InputCursor(1, "_ rejected"))
  }
  "recognize an underscore followed by alpha digits and underscores" in {
    identifier(InputCursor(0, "_a_123_Word <-")) shouldBe
      ParseSuccess("_a_123_Word", InputCursor(11, "_a_123_Word <-"))
  }
  "reject a digit" in {
    identifier(InputCursor(0, "1234")) shouldBe ParseFailure(InputCursor(0, "1234"))
  }
}

What’s next?

With parser combinators you are able to process text into any structure you need. In this post we only covered tuples, Seqs, and Strings, but parse functions can be written to produce full AST’s or even interpret scripted programs.

Some subjects for next time:

Parsing mathematical infix expressions and calculating their results.
Error handling and recovery, an especially tricky subject!
Stateful parsing, which requires lifting the parse() function into a higher-order monad.

Imperative Computation in Functional Programming

2022-06-17T08:47:11-07:00

Estimated reading time: 1h 25m 15s

Remember functors and applicatives? In my last post Enabling Control Flow in Functional Programming we explored how functors and applicatives abstract over desired and undesired cases of contexts in order to express control flow and permit independent computation. In this post we will explore monads and how to leverage their specific abstraction to express imperative control flow.

This post is part of a series:

Contexts and Effects in Functional Programming

Enabling Control Flow in Functional Programming

Imperative Computation in Functional Programming

The code that accompanies this post may be found here.

Motivating monads as a design pattern

Recall the Functor and Applicative typeclasses:

trait Functor[F[_]] {
  def map[A, B](fa: F[A])(f: A => B): F[B]
}
object Functor {
  def apply[F[_]: Functor]: Functor[F] = implicitly[Functor[F]]
}

trait Applicative[F[_]] extends Functor[F] {
  def pure[A](a: A): F[A]

  def ap[A, B](ff: F[A => B])(fa: F[A]): F[B]

  def map[A, B](fa: F[A]): F[B] =
    ap(pure(f))(fa)

  def map2[A, B, C](fa: F[A], fb: F[B])(f: (A, B) => C): F[C] =
    ap(ap(pure(f.curried))(fa))(fb)

  def sequence[A](listFa: List[F[A]]): F[List[A]] =
    listFa.foldLeft(pure(List[A]()))((fListA, fa) => map2(fa, fListA)(_ :: _))
}
object Applicative {
  def apply[F: Applicative]: Applicative[F] = implicitly[Applicative[F]]
}

See Functor and Applicative’s definitions in the sample repository.

Functor abstracts over contexts’ unknown cases by lifting a function via map() and applying it to instances of the term produced by the context if they exist. Control flow can’t be expressed using map() because it does not permit the case of the context to be modified.

Applicative can create contexts in the desired case via pure(). It can also apply lifted functions to lifted arguments via ap() if both contexts are in their desired case. Applicative is capable of expressing control flow through ap() because supplying either the lifted function or lifted argument in an undesired case will not permit further computation. The context in the undesired case is propagated instead.

The problem however is that Applicative requires that both the lifted function and lifted argument supplied to ap() be computed before deciding whether to permit or halt further computation. This means that regardless of whether one of the function or argument contexts successfully computes, if the other one fails then the succeeding computation will be discarded. Both of these contexts’ computation are independent, and Applicative will only use them if they are both are in their desired case.

All functions derived from ap() represent all-or-nothing operations accordingly, and there is no practical way of ordering the computation of the arguments or preventing computation of all arguments if any preceding arguments fail to compute. Therefore, totally imperative programming is not possible using Applicative.

Introducing imperative control flow

In order to restrict computation such that subsequent computation occurs only if previous computation succeeds, we must introduce a new abstraction over the cases of contexts.

What function might limit subsequent computations from running if a previous one has failed? There is one, and you’ve probably seen it out in the wild:

def flatMap[F[_], A, B](fa: F[A])(f: A => F[B]): F[B]

What flatMap() does is allow for the injection of a context into a pipeline of computations to either permit computation to proceed or force it to halt. The function argument f, which you supply, has full control of what case the returned context F[B] should be in. If F[B] is in the undesired case, then all further computations are skipped, propagating this undesired case instead.

Take for example this definition of a filter() function from the sample repository’s sbt console:

scala> :load console/imports.scala

scala> def filter[A](fa: Option[A])(f: A => Boolean) = fa.flatMap {
     |   case a if f(a) => Some(a)
     |   case _ => None
     | }
def filter[A](fa: Option[A])(f: A => Boolean): Option[A]

scala> filter(Some(4))(_ % 2 == 0)
val res1: Option[Int] = Some(4)

scala> filter(Some(3))(_ % 2 == 0)
val res2: Option[Int] = None

This example is only very simple. But it demonstrates the key enabling feature of flatMap(): we are able to choose whether the context remains in its desired case. If we continue with this example, we can try chaining map() to the context returned by it.

scala> filter(Some(4))(_ % 2 == 0).map(n => s"The square of even number $n is ${n * n}")
val res3: Option[String] = Some(The square of even number 4 is 16)

scala> filter(Some(3))(_ % 2 == 0).map(n => s"The square of even number $n is ${n * n}")
val res4: Option[String] = None

Thus we are able to gate map() behind the case of the context we return from the flatMap() we used in filter(), enabling imperative control flow.

Monads

The flatMap() function is implemented using a new structure, a specialization of an applicative functor called a monad.

Monads are a specialization that arises in the type of A contained within a context F[_]:

If A is an opaque type, then F[A] is a functor.
If A is known to have type A => B, that is A is a function, then F[A => B] is an applicative functor.
If A is known to have type F[A], then this means that F[F[A]] is nested within itself and thus a monad.

The Monad typeclass defines two functions, which by default are defined in terms of each other:

trait Monad[F[_]] extends Applicative[F] {

  def flatMap[A, B](fa: F[A])(f: A => F[B]): F[B] =
    flatten(map(fa)(f))

  def flatten[A](ffa: F[F[A]]): F[A] =
    flatMap(fa => fa)
}

object Monad {

  def apply[F[_]: Monad]: Monad[F] = implicitly[Monad[F]]
}

This means that in order to implement an instance of the Monad typeclass, you must implement one of either flatMap() or flatten() or calling one will recurse until the stack overflows. Some contexts are best defined using one over the other, and you have the choice to pick between the two.

See here for the definition in the sample repository.

Composing monads

Monad also defines a special composition operator. Recall that functions may be composed together, as in h = g ∘ f or “h is g after f”. This may be expressed in two ways in vanilla Scala. Try below using the sample repository’s sbt console:

scala> val f: Int => String = n => (n * n).toString
val f: Int => String = $Lambda$5252/0x000000010195e040@7a76dc63

scala> val g: String => Int = n => n.length
val g: String => Int = $Lambda$5254/0x000000010195f840@3843085

scala> val h = g.compose(f)
val h: Int => Int = scala.Function1$$Lambda$5244/0x0000000101958840@388b1b42

scala> h(3)
val res1: Int = 1

scala> h(25)
val res2: Int = 3

scala> val hh = f.andThen(g)

scala> hh(3)
val res3: Int = 1

scala> hh(25)
val res4: Int = 3

Scala out of the box affords the compose() and andThen() functions for composing functions of the form of A => B together. But these composition functions don’t work for the signature of flatMap(), which operates on functions of the form of A => F[B].

There is a form of composition for monads known as Kleisli composition. This composition will lift the following function into the context returned by the previous function and apply it to the term contained within the context if the context is in the desired case. Kleisli composition is expressed using the fish or arrow operator >=> which is defined in the sample repository as an extension operator on functions of the form A => F[B] where F[_] has an instance of Monad:

implicit class KleisliCompositionOps[F[_], A, B](val f: A => F[B]) extends AnyVal {

  def >=>[C](g: B => F[C])(implicit F: Monad[F]): A => F[C] =
    (x: A) => F.flatMap(f(x))(g)
}

See here for the definition in the sample repository.

Kleisli composition is useful for monads in that from two or more flatMap()-compatible functions a single function may be created that takes an unlifted argument and produces an output context as the result of applying each function in sequence first to the unlifted argument and then to each individual functions’ output contexts.

scala> :load console/imports.scala

scala> val squareEvens: Int => Either[String, Int] = {
     |   case n if n % 2 == 0 => Either.right[String, Int](n * n)
     |   case n               => Either.left[String, Int](s"$n is an odd number")
     | }
val squareEvens: Int => Either[String,Int] = $Lambda$5278

scala> val piEvenLastDigit: Int => Either[String, Int] = { n =>
     |   val nPi = n * scala.math.Pi
     |   val lastDigit = nPi.toString.last.asDigit
     |   if (lastDigit % 2 == 0) {
     |     Either.right[String, Int](lastDigit)
     |   } else {
     |     Either.left[String, Int](s"$n * Pi's last digit is an odd number")
     |   }
     | }

scala> val evensSquaredWithEvenLastDigitsOfPi = squareEvens >=> piEvenLastDigit
val evensSquaredWithEvenLastDigitsOfPi: Int => Either[String,Int] = Monad$Syntax$KleisliCompositionOps$$$Lambda$5241

scala> evensSquaredWithEvenLastDigitsOfPi(2)
val res1: Either[String,Int] = Right(2)

scala> evensSquaredWithEvenLastDigitsOfPi(23)
val res2: Either[String,Int] = Left(23 is an odd number)

scala> evensSquaredWithEvenLastDigitsOfPi(48)
val res3: Either[String,Int] = Left(2304 * Pi's last digit is an odd number)

Above we are using the composed function directly, but because it has the form of A => F[B] we are able to apply it using flatMap() as well:

scala> Either.right[String, Int](2).flatMap(evensSquaredWithEvenLastDigitsOfPi)
val res4: Either[String,Int] = Right(2)

scala> Either.right[String, Int](19).flatMap(evensSquaredWithEvenLastDigitsOfPi)
val res5: Either[String,Int] = Left(19 is an odd number)

scala> Either.right[String, Int](4).flatMap(evensSquaredWithEvenLastDigitsOfPi)
val res6: Either[String,Int] = Left(16 * Pi's last digit is an odd number)

scala> Either.left[String, Int]("Nothing's gonna happen").flatMap(evensSquaredWithEvenLastDigitsOfPi)
val res7: Either[String,Int] = Left(Nothing's gonna happen)

Thus Kleisli composition is to flatMap() as function composition is to map(), with the key difference being that the case of the context may be changed by any one of the composed functions. Kleisli composition is powerful in that entire processes may be composed into step-by-step operations and applied as a single unit to a context via flatMap() or directly to an unlifted argument.

The for comprehension and imperative programming

Scala provides a syntax sugar over flatMap() in the form of the for comprehension. Any type that provides both the map() and flatMap() functions are able to participate in this syntax sugar, and it allows for monadic operations to be expressed as if they were written as procedural code. This means that you don’t have to rely on Kleisli composition alone to express complex flows of logic, and that you also aren’t limited to single-argument input/output pipelines of functions.

As a simple example, here are the above functions leveraged using a for comprehension:

scala> :load console/imports.scala

scala> val squareEvens: Int => Either[String, Int] = {
     |   case n if n % 2 == 0 => Either.right[String, Int](n * n)
     |   case n               => Either.left[String, Int](s"$n is an odd number")
     | }
val squareEvens: Int => Either[String,Int] = $Lambda$5278

scala> val piEvenLastDigit: Int => Either[String, Int] = { n =>
     |   val nPi = n * scala.math.Pi
     |   val lastDigit = nPi.toString.last.asDigit
     |   if (lastDigit % 2 == 0) {
     |     Either.right[String, Int](lastDigit)
     |   } else {
     |     Either.left[String, Int](s"$n * Pi's last digit is an odd number")
     |   }
     | }

scala> val evensSquaredWithEvenLastDigitsOfPi: Int => Either[String, Int] =
     | n => for {
     |   squaredEvenN <- squareEvens(n)
     |   piEvenLastDigitN <- piEvenLastDigit(squaredEvenN)
     | } yield piEvenLastDigitN

val evensSquaredWithEvenLastDigitsOfPi: Int => Either[String,Int] = $Lambda$5263

scala> evensSquaredWithEvenLastDigitsOfPi(2)
val res1: Either[String,Int] = Right(2)

scala> Either.right[String, Int](2).flatMap(evensSquaredWithEvenLastDigitsOfPi)
val res2: Either[String,Int] = Right(2)

scala> Either.left[String, Int]("Nothing's gonna happen").flatMap(evensSquaredWithEvenLastDigitsOfPi)
val res3: Either[String,Int] = Left(Nothing's gonna happen)

The key thing that a for comprehension allows for is multiple arguments to be lowered from contexts and applied to a multi-argument function that returns another context. An example of such an operation might look like this:

def sendMarketingUpdate(updateId: Int, userId: Int): Future[MarketingUpdateStatus] =
  for {
    user <- loadUser(userId)
    marketingTemplate <- loadTemplate(updateId)
    composedEmail <- composeEmailForUser(user, marketingTemplate)
    result <- sendEmail(composedEmail)
  } yield result.toMarketingUpdateStatus

Because composeEmailForUser() requires both a user and marketingTemplate, this operation function is more easily expressed using a for comprehension than Kleisli composition.

However you don’t have to abandon applicative functions when using for comprehensions. The above function may be written to independently load the user and template:

def sendMarketingUpdate(updateId: Int, userId: Int): Future[MarketingUpdateStatus] =
  for {
    (user, marketingTemplate) <- (loadUser(userId), loadTemplate(updateId)).tupled
    composedEmail <- composeEmailForUser(user, marketingTemplate)
    result <- sendEmail(composedEmail)
  } yield result.toMarketingUpdateStatus

The code can be trimmed just a little bit more, as well:

def sendMarketingUpdate(updateId: Int, userId: Int): Future[MarketingUpdateStatus] =
  for {
    composedEmail <- (loadUser(userId), loadTemplate(updateId)).mapN(composeEmailForUser)
    result <- sendEmail(composedEmail)
  } yield result.toMarketingUpdateStatus

This means that with applicative functions, you can express imperative code to some degree by gating functions dependent on a set of inputs, such as composeEmailForUser(), and lean on monad’s flatMap() for when you need absolute ordering of computations. These two styles of computation thus complement each other.

Becoming a Monad

In order to become a Monad, an effect type must implement the typeclass. Let’s implement instances for the usual suspects, Option, Either, and List. As Monad is a specialization of Applicative and thus also Functor, we can simply upgrade our Applicative instances to become Monads:

implicit val optionMonad: Monad[Option] = new Monad[Option] {

  // keep existing definitions

  override def flatten[A](ffa: Option[Option[A]]): Option[A] =
    ffa match {
      case Some(fa) => fa
      case None     => None
    }
}

implicit def eitherMonad[X]: Monad[Either[X, *]] = new Monad[Either[X, *]] {

  // keep existing definitions

  override def flatten[A](ffa: Either[X, Either[X, A]]): Either[X, A] =
    ffa match {
      case Right(fa) => fa
      case Left(x) => Left(x)
    }
}

implicit val listMonad: Monad[List] = new Monad[List] {

  // keep existing definitions

  override def flatten[A](ffa: List[List[A]]): List[A] =
    ffa
      .foldLeft(List[A]()) { (outerResult, as) =>
        as.foldLeft(outerResult) { (innerResult, a) =>
          a :: innerResult
        }
      }
      .reverse
}

See the instances of Monad for Option, Either, and List in the sample repository.

In the above example, I implemented the Monad instances using flatten(). These could alternatively be implemented using flatMap(). You might try to do this if you wish as an exercise.

Monad laws

How do we know that Option, Either, and List’s Monad instances are well-behaved as monads? Like functors, monads are formally defined in the higher math of category theory and expected to conform to a set of laws.

There are three monad laws, which must hold for all monads in addition to the applicative and functor laws.

Preservation of left identity: Kleisli composition of pure() with a function of form f: A => F[B] applied to an unlifted argument a: A is the same as applying the unlifted argument directly to f: A => F[B].
Preservation of right identity: Kleisli composition of a function of form f: A => F[B] with pure() applied to an unlifted argument a: A is the same as applying the unlifted argument directly to f: A => F[B].
Associativity: Kleisli composition of three functions f: A => F[B], g: B => F[C], and h: C => F[D] applied to an argument a: A produces the same result regardless of grouping: (f >=> g) >=> h is the same as f >=> (g >=> h).

Note that pure() is interpreted as the identity element of the monad, as it merely lifts a value into its context, unmodified.

Defining monad laws as properties

The monad laws may be defined as scalacheck properties, as previously for applicatives. These properties assert “for all” instances of a particular monad F[_], the properties should hold. We know we have a well-behaved monad if the property checks pass.

trait MonadLaws { this: Laws with ApplicativeLaws =>

  implicit def arbitraryFA[F[_]: LiftedGen, A: Arbitrary]: Arbitrary[F[A]] = Arbitrary(arbitrary[A].lift)

  /** Defined per Monad laws taken from the Haskell wiki:
    * [[https://wiki.haskell.org/Monad_laws#The_three_laws]]
    *
    * These laws extend the Applicative laws, so `checkApplicativeLaws[F]()` should be
    * executed alongside this function.
    *
    * The laws as defined here leverage Kleisli composition, which is defined
    * using the operator `>=>` in terms of `flatMap()`, to better highlight the
    * left and right identities and associativity that should be exhibited by
    * the composition of monadic operations.
    *
    * @param TT The type tag of the context
    * @tparam F The context type being tested
    */
  def checkMonadLaws[F[_]: Monad: LiftedGen]()(implicit TT: TypeTag[F[Any]]): Unit = {
    property(s"${TT.name} Monad composition preserves left identity") {
      // The identity of Kleisli composition simply lifts a value into the
      // context. Kleisli composition of a function after `pure()` when applied
      // to a value will produce the same result as applying the function
      // directly to the value.
      forAll(for {
        a <- arbitrary[Int]
        h <- arbitrary[Int => F[String]]
      } yield (a, h)) { case (a, h) =>
        val leftIdentity = ((_: Int).pure[F]) >=> h
        leftIdentity(a) mustBe h(a)
      }
    }
    property(s"${TT.name} Monad composition preserves right identity") {
      // The identity of Kleisli composition simply lifts a value into the
      // context. Kleisli composition of `pure()` after a function when applied
      // to a value will produce the same result as applying the function
      // directly to the value.
      forAll(for {
        a <- arbitrary[Int]
        h <- arbitrary[Int => F[String]]
      } yield (a, h)) { case (a, h) =>
        val rightIdentity = h >=> ((_: String).pure[F])
        rightIdentity(a) mustBe h(a)
      }
    }
    property(s"${TT.name} Monad composition is associative") {
      // `flatMap()` and thus Kleisli composition are both associative.
      // This means that your program may be factored with these operations
      // in any arbitrary grouping and the output will be the same.
      forAll(for {
        a <- arbitrary[Double]
        f <- arbitrary[Double => F[String]]
        g <- arbitrary[String => F[Int]]
        h <- arbitrary[Int => F[Boolean]]
      } yield (a, f, g, h)) { case (a, f, g, h) =>
        val assocLeft = (f >=> g) >=> h
        val assocRight = f >=> (g >=> h)
        assocLeft(a) mustBe assocRight(a)
      }
    }
  }
}

See here for the definition of the trait.

Our laws specs can now extend this trait and assert that their tested contexts conform to the monad laws. For example, the laws spec for Option:

class OptionLaws extends Laws with FunctorLaws with ApplicativeLaws with MonadLaws {

  import Option.Instances._

  implicit val optionLiftedGen: LiftedGen[Option] = new LiftedGen[Option] {

    override def lift[A](gen: Gen[A]): Gen[Option[A]] =
      Gen.lzy(
        Gen.oneOf(
          Gen.const(None),
          gen.map(Some(_)),
        )
      )
  }

  checkFunctorLaws[Option]()
  checkApplicativeLaws[Option]()
  checkMonadLaws[Option]()
}

See the laws specs for Option, Either, and List

Implications of the monad laws

You may have noticed that the characteristics of the monad laws are somewhat different from the functor and applicative laws. They build using composition directly, specifically Kleisli composition, but don’t assert that function composition generally is retained within their contexts the same way that functor or applicative’s laws do.

Let’s reiterate the laws:

Left identity
Right identity
Associativity

These laws specifically also define another typeclass called a monoid, which is a specialization of a semigroup that adds an identity or empty element. The identity element is special in that combining it with any other element produces that other element.

trait Monoid[M] extends Semigroup[M] {

  def empty: M
}

object Monoid {

  def apply[M: Monoid]: Monoid[M] = implicitly[Monoid[M]]
}

Monoids are nearly as common as semigroups, as not all semigroups are monoids, and you’ve probably used quite a few of them:

List’s identity element is the empty List().
String’s identity element is the empty "".
Int’s identity element is 0.
Set’s identity element is the empty Set().

The key difference between monads and these monoids above is that monads form an additive function in contrast to additive data. Functions of the form A => F[B] are combinable using >=> to produce new functions of the same form, and this operation is associative, which means that monads form semigroups under Kleisli composition. They form monoids as the pure() function satisfies the identity element in that it doesn’t alter its argument when composed to either the left or right side of a function of the form A => F[B].

To be really pedantic, monads are functors. All functors in Scala are functors from Scala types into other Scala types, making them endofunctors because they map back into the same category: the category of Scala types.

This affirms an infamous joke:

Haskell gets some resistance due to the complexity of using monads to control side effects. Wadler tries to appease critics by explaining that “a monad is a monoid in the category of endofunctors, what’s the problem?”

– From James Iry’s “A Brief, Incomplete, and Mostly Wrong History of Programming Languages”

In addition to being monoids, Scala’s List and Set are also monads and I have asserted that they are well behaved with property checks. Option and Either are also monads. You’ve probably been using them as such without realizing!

The real takeaway from the monad laws is that you get imperative computation as a composable structure. You retain referential transparency in that associativity guarantees that your operations may be grouped arbitrarily, allowing you to factor the steps of your program with a large degree of freedom.

What is enabled by monads?

The flatMap() function primarily enables imperative programming through abstraction. It strictly requires that its current context be evaluated into its desired case before applying its function argument against the term it contains. If the context is in the undesired case, then all subsequent computation halts and this case propagates instead. Thus, in contrast to applicative’s all-or-nothing operations, monads offer one-after-another operations. Both evaluation styles complement each other and may be freely mixed, of course.

Beware the imperative trap

Because monads allow you to express functional programming in a style that closely mirrors procedural code, especially within the syntax of the for comprehension, it’s very easy to fall into a trap of procedural spaghetti even though you’re using a functional programming abstraction. This occurs because for comprehensions allow you to expose multiple terms and pass them around at varying points within the for comprehension, mimicking procedural state-passing:

def performTransaction(accountNo: Long, amount: Currency): Future[Unit] =
  for {
    account <- loadAccount(accountNo)
    transaction <- beginTransaction(account)
    balance <- getBalance(account)
    newBalance = balance + amount
    _ <- {
      if (newBalance < Currency.from(0)) {
        endTransaction(transaction)
          .flatMap(_ => failTransaction("Insufficient funds"))
      } else {
        updateBalance(newBalance)
          .flatMap(_ => endTransaction(transaction))
      }
    }
  } yield ()

This code would be hard to refactor as there’s a lot of arguments being passed around. It’s procedural code wrapped in a for comprehension! A smell in particular is a Unit-return, as it indicates something is causing side-effects with no way to sense what actions have been performed or what state the total operation finished in.

Going forward

These three abstractions, functors, applicatives, and monads, are just the beginning to a rich series of tools that you can leverage to express solid, maintainable, and provable programs. But you may have noticed a few capabilities are missing so far in this series, such as how to:

Raise errors, specifically as undesired cases, agnostic of context.
Recover from errors, transforming undesired cases to desired, agnostic of context.
Attempt an alternate operation if the first one fails.
Collect effectful operations agnostic of container.
Allow for partial successes and failures within collective operations.
Compose effect types to take advantage of their combined characteristics.
Encode DSLs as effects.

In my next post, we will explore raising and recovering from errors agnostic of context, so that your code may abstract against typeclasses but still be able to force and recover from undesired cases.

The Set Function

2022-06-16T22:22:26-07:00

Estimated reading time: 15m 31s

What is a Set? A Set can tell you whether or not an value is a member of the Set. This means that a Set is merely a function, specifically of type A => Boolean. In this post I will explore the usage of combinators to build a Set from elementary functions alone.

Modeling a `Set`

I’m going to use a trait to model a Set:

trait Set[A] extends (A => Boolean)

And define the most elementary of Sets, the empty Set and singleton Set:

def empty[A](): Set[A] = _ => false

def singleton[A](value: A): Set[A] = test => test == value

val emptyInts = empty[Int]()
emptyInts(1) // false
emptyInts(2) // false

val justOne = singleton(1)
justOne(1) // true
justOne(2) // false

But what about more complex sets?

The `Set` of natural numbers

The Set of natural numbers requires more work to model. Specifically, it represents a lower bound of 0:

def nat(value: Int): Boolean = value >= 0

But this is very concrete. A lower bound specifically is a type of predicate, and this can be parameterized:

def satisfy(predicate: A => Boolean): Set[A] = predicate

def nat: Set[Int] = satisfy(_ >= 0)

nat(-1) // false
nat(0) // true
nat(1) // true

Building a `Set`

How do I add values to the Set? I have to be able to combine Sets:

trait Set[A] extends (A => Boolean) {

  def |[B >: A](other: Set[B]): Set[B] =
    value => this(value) || other(value)
}

By combining Sets, I can now specify that a value be one of two values in a Set:

val oneOrTwo = singleton(1) | singleton(2)

oneOrTwo(1) // true
oneOrTwo(2) // true
oneOrTwo(3) // false

But this | operator doesn’t allow me to set boundaries using predicates, as it represents a disjoint Set. This means values must be present at least in one Set or the other:

val singleDigits = satisfy(_ >= 0) | satisfy(_ <= 9)

singleDigits(10) // true
singleDigits(-1) // true

I need another operator requiring membership in both Sets, so that I can create a joint Set:

trait Set[A] extends (A => Boolean) {

  def |[B >: A](other: Set[B]): Set[B] =
    value => this(value) || other(value)

  def &[B >: A](other: Set[B]): Set[B] =
    value => this(value) && other(value)
}

The & operator now requires that the value exists in both Sets:

val singleDigits = satisfy(_ >= 0) & satisfy(_ <= 9)

singleDigits(2) // true
singleDigits(9) // true
singleDigits(10) // false
singleDigits(-1) // false

Building a complex `Set`

I’m going to build a weird Set of numbers from 0-100, 102, 104, 220-230, and all numbers greater than 300 that are divisible by 17:

val complexSet = (satisfy(_ >= 0) & satisfy(_ <= 100)) |
  singleton(102) |
  singleton(104) |
  (satisfy(_ >= 220) | satisfy(_ <= 230)) |
  (satisfy(_ >= 300) & satisfy(_ % 17 == 0))

complexSet(60) // true
complexSet(104) // true
complexSet(180) // false
complexSet(227) // true
complexSet(340) // true
complexSet(341) // false

Takeaway

Using functions alone, complex logic can be composed from atomic, testable building blocks. These functions that take other functions as arguments to produce new functions are referred to as combinators as they combine their capabilities.

Composing programs from small units ensures testability and ease in refactoring, as the single-responsibility principle is taken to its logical extreme with this technique.

Why don’t we use `Set`s as functions?

This seems like a cool way to model Sets, but that’s about where the utility stops. As an exercise, it’s fun to see what you can do to model Sets using functions alone, as this is an excellent vehicle for learning functional composition with combinators. In practice this would produce a Set with linear-time performance, and you don’t want that.

Enabling Control Flow in Functional Programming

2022-06-05T14:01:07-07:00

Estimated reading time: 2h 12m 50s

Remember functors? Recall from my last post, Contexts and Effects in Functional Programming, they are structures that abstract away complexity imposed by nondeterminism present in contexts that produce some output; contexts such as optionality, network interaction, or validation. When contexts fail to produce some output, they are in their undesired case and no computation may be performed against them. In this post we will explore how to exploit this characteristic to halt computation in order to express control flow.

This post is part of a series:

Contexts and Effects in Functional Programming

Enabling Control Flow in Functional Programming

Imperative Computation in Functional Programming

The code that accompanies this post may be found here.

Motivating applicative functors as a design pattern

Recall the Functor typeclass:

trait Functor[F[_]] {
  def map[A, B](fa: F[A])(f: A => B): F[B]
}
object Functor {
  def apply[F[_]: Functor]: Functor[F] = implicitly[Functor[F]]
}

See here for the definition in the sample repository.

For any context F[A], the map() function accepts another function f: A => B and applies it to the term within the context, giving back F[B].

Contexts that are functors thus allow abstraction over unknown cases. For example, a function f: A => B may be lifted into an Option[A] and applied if an instance of A is present. If no instance of A is present, then nothing happens. Specifically, by using map() the function f is unconcerned with the unknown quantity of A’s presence. Many contexts encode dimensions of unknown quantities, and as functors they completely abstract the nondeterminism of these quantities, allowing the business logic expressed by the lifted functions to focus only on the terms that they operate against.

What this means is that for any context in the desired case, such as a Some of Option[A] or a Right of Either[X, A], the function f will be applied when lifted with map(). Any number of functions may be applied to the new contexts returned by subsequent applications of map(), and they will all apply as the initial context was in the desired case. Functors thus may be considered as enablers of data transformation, as lifted functions transform data if it exists. But if the initial context is in an undesired case, none of the lifted functions will apply.

Functions lifted into a context are permitted to compute if the context is in the desired case. But if a function is lifted into a context that is in the undesired case, then computation is halted. This means that the case of any context may control the flow of execution within a program.

You can try for yourself from the sample repository’s sbt console:

scala> :load console/imports.scala

scala> val desiredOption: Option[Int] = Some(42)
val desiredOption: Option[Int] = Some(42)

scala> desiredOption.map(_.toString).map(numStr => s"backwards: ${numStr.reverse}")
res0: Option[String] = Some(backwards: 24)

scala> val undesiredOption: Option[Int] = None
val undesiredOption: Option[Int] = None

scala> undesiredOption.map(_.toString).map(numStr => s"backwards: ${numStr.reverse}")
res1: Option[String] = None

Functors only allow lifting functions of the form f: A => B. The context can’t be modified with a function having this signature, which means we can’t use a functor specifically to influence control flow by injecting a context in its undesired case. Functors respect the existing case of a context: they cannot modify it.

Introducing control flow

What function might allow for control flow using contexts? There is one, map2():

def map2[F[_], A, B, C](fa: F[A], fb: F[B])(f: (A, B) => C): F[C]

This two-argument analog of map() unlocks a key capability: controlling whether to proceed or halt computation against the terms contained within the contexts fa and fb. If both fa and fb are in their desired case, then there are instances of A and B against which the function f may be applied. But if either one or both are in their undesired case, then f does not apply, and the undesired cases are propagated through F[C]. This means that fa and fb become levers with which to halt computation that would be performed using function f and subsequent computation against context F[C].

Take for example addition lifted into the context of Option[Int] within the sample repository’s sbt console:

scala> :load console/imports.scala

scala> val F = Applicative[Option]
val F: Applicative[Option] = Option$Instances$$anon$1

scala> F.map2(Option(2), Option(2))(_ + _)
val res0: Option[Int] = Some(4)

scala> F.map2(Option(2), Option[Int]())(_ + _)
val res1: Option[Int] = None

scala> F.map2(Option[Int](), Option(2))(_ + _)
val res2: Option[Int] = None

scala> F.map2(Option[Int](), Option[Int]())(_ + _)
val res3: Option[Int] = None

Only when both arguments are in their desired case does the function of addition apply. If either or both arguments are in their undesired case, then the undesired case is propagated. This does not allow the function to apply, and halts any further operations against the context. The functions that produce the two input contexts are thus capable of controlling whether computation via f proceeds and permits further computation against the context F[C].

Applicative functors

The map2() function is implemented using a new structure, a specialization of a functor called an applicative functor, or simply an applicative.

Applicative functors as a specialization arise in the type of A contained within a functor F[_]. If A is merely an opaque type, then F[A] is a functor and no more. But if A is specifically known to have some type A => B, that is to say A is a function, then F[A => B] is an applicative functor.

Applicatives define two functions from which many more are derived, including map2():

trait Applicative[F[_]] extends Functor[F] {

  def pure[A](a: A): F[A]

  def ap[A, B](ff: F[A => B])(fa: F[A]): F[B]

  // other functions are introduced below
}

object Applicative {

  def apply[F: Applicative]: Applicative[F] = implicitly[Applicative[F]]
}

See here for the definition in the sample repository.

Note that Applicative extends Functor as it is a specialization. All applicatives are also functors and therefore also provide the map() function.

The new functions defined by applicatives offer two capabilities:

pure() which lifts the result of a pure computation A into the context such that pure: A => F[A]. This is essentially a constructor producing a context in the desired case, such as Some for Option[A] or Right for Either[X, A]. In short, pure() puts A in the box.
ap(), read as apply, for applying a lifted function to a lifted argument. Given two boxes, if the first contains a function and the second contains an argument, ap() will apply them and put them back in the box.

You might be wondering why a function would ever be lifted into a context? I will demonstrate why this is desirable in how ap() works by defining map2() within Applicative:

def map2[A, B, C](fa: F[A], fb: F[B])(f: (A, B) => C): F[C] =
  ap(ap(pure(f.curried))(fa))(fb)

Lifting f into the context works following these steps:

By currying the function f: (A, B) => C, it becomes A => B => C.
Lifting it with pure() gives F[A => B => C] in the desired case, which gives us a clean slate to start our computation with.
Then, with ap() we may apply the first argument fa: F[A] which will give back F[B => C] in the desired case if fa is itself in the desired case.
Then, with ap() we may apply the second argument fb: F[B] which will give back F[C] in the desired case if fb is itself in the desired case.

Each step of lifted function application accounts for the case of the function and argument contexts and halts if either context is in the undesired case. The undesired case will propagate instead through F[C] if and when it exists.

You might have noticed, map2() looks an awful lot like map(). In fact, Applicative provides a default implementation of map() following the same pattern:

def map[A, B](fa: F[A])(f: A => B): F[B] =
  ap(pure(f))(fa)

If your context implements Applicative, then it also implements Functor with no extra work. You can always provide your own implementation of map() if it is more efficient to do so.

Validation as a use case

Let’s walk through a powerful capability of applicatives: validation.

Validating a `User` from external data

When constructing a User from data that we receive from an external source, such as a form or API, we can use ap() to lift User’s curried constructor into a validation context and apply it to validated arguments. If all arguments are valid, then we should receive a validation context containing a valid User. If any arguments are invalid, then we should receive an invalid context with all reasons for validation failure.

Here is the definition for User:

case class User(username: String, email: String, password: String)

Each of username, email, and password must to be valid in order for User itself to be valid. This requires the introduction of a validation context:

sealed trait Validated[+E, +A] {

  def invalid: E

  def valid: A
}

case class Valid[+A](valid: A) extends Validated[Nothing, A] {

  def invalid: Nothing = throw new NoSuchElementException("Valid.invalid")
}

case class Invalid[+E](invalid: E) extends Validated[E, Nothing] {

  def valid: Nothing = throw new NoSuchElementException("Invalid.valid")
}

See here for the definitions in the sample repository.

Why use a new `Validated` context instead of `Either`?

The Applicative instance for Either in the context of validation has one key flaw: it short-circuits on the first Left.

Refer to Either’s implementation of ap():

override def ap[A, B](ff: Either[X, A => B])(fa: Either[X, A]): Either[X, B] =
  (ff, fa) match {
    case (Right(f), Right(a)) => Right(f(a))
    case (Left(x), _)         => Left(x)
    case (_, Left(x))         => Left(x)
  }

See here for Either’s definition of ap().

For both cases of Left they are immediately returned and there is no specific handling for situations where both ff and fa may be in the Left case. This means that the first Left propagates and all subsequent Lefts are swallowed. In the context of validation, this means that for any number of validation errors that the context might produce, we would only receive the first error. We would have to resolve the error and re-run the operation, and repeat for each subsequent error until the operation as a whole succeeded. This makes Either a very poor choice for modeling validation. It represents strictly one thing or the other, whereas validation we can specialize to propagate all reasons for failure.

Validation via `Applicative`

Validated contains the valid value you want or the reasons for invalidation. We could fail to receive a User for three or more reasons related to username, email, and password all being invalid, which implies that term E represents some data containing one or more of something. This has a specific implication on how we define an instance of Applicative for Validated:

implicit def validatedApplicative[E]: Applicative[Validated[E, *]] =
  new Applicative[Validated[E, *]] {

    def pure[A](a: A): Validated[E, A] = Valid(a)

    def ap[A, B](ff: Validated[E, A => B])(fa: Validated[E, A]): Validated[E, B] =
      (ff, fa) match {
        case (Valid(f), Valid(a)) => Valid(f(a))
        case (Invalid(x), Invalid(y)) => ??? /* what do we do with these two E's? */
        case (Invalid(x), _) => Invalid(x)
        case (_, Invalid(y)) => Invalid(y)
      }
  }

When there are two instances of E we don’t have a way to combine them as E is an opaque type. Without concretely defining E, such as with List[String] or another similar structure, we won’t be able to combine their values, but this creates an inflexible API. Specifically, this inability to combine E leaves Validated in the same position that Either is in: the first undesired case propagates and subsequent cases are swallowed. How do we combine the undesired cases?

Modeling combinable structures

Structures defining a combine() function form a typeclass known as a semigroup under a specific condition: that combine() is associative. Semigroups are very common, and constraining E to have an instance of Semigroup provides great API flexibility. First, let’s see how the Semigroup typeclass is defined:

trait Semigroup[A] {
  def combine(left: A, right: A): A
}
object Semigroup {
  def apply[A: Semigroup]: Semigroup[A] = implicit[Semigroup[A]]
}

See here for the definition in the sample repository.

A semigroup is thus an additive form of data. Here’s a few familiar data types that you may have used as semigroups without realizing it:

The combine() function must be associative. This can be tested with a scalacheck property:

def checkSemigroupLaws[S: Semigroup: Arbitrary](): Unit = {

  import green.thisfieldwas.embracingnondeterminism.syntax.semigroup._

  property("Semigroup preserves associativity") {
    forAll(for {
      a <- arbitrary[S]
      b <- arbitrary[S]
      c <- arbitrary[S]
    } yield (a, b, c)) { case (a, b, c) =>
      (a |+| b) |+| c mustBe a |+| (b |+| c)
    }
  }
}

See here for the definition in the sample repository.

Given an E with an instance of Semigroup, we can define an Applicative instance for Validated:

implicit def validatedApplicative[E: Semigroup]: Applicative[Validated[E, *]] =
  new Applicative[Validated[E, *]] {
    import Semigroup.Syntax._

    override def pure[A](a: A): Validated[E, A] = Valid(a)

    override def ap[A, B](ff: Validated[E, A => B])(fa: Validated[E, A]): Validated[E, B] =
      (ff, fa) match {
        case (Valid(f), Valid(a))     => Valid(f(a))
        case (Invalid(x), Invalid(y)) => Invalid(x |+| y) // propagate both undesired cases via Semigroup
        case (Invalid(x), _)          => Invalid(x)
        case (_, Invalid(y))          => Invalid(y)
      }
  }

See here for the definition in the sample repository.

This means that Validated is usable as an Applicative for any case where E is combinable. What implications does this have?

Errors may exist in a non-zero amount when the context is Invalid.
When there are errors, the amount is unbounded.
Combining errors can be frequent, and should be computationally cheap.

Naively, a List[String] works for E. It forms a Semigroup under concatenation, but concatenation isn’t cheap in Scala Lists. It can also be empty per its type, which means that as E you have to code for invariants where it is actually empty.

There exists a better structure, and we can whip it together pretty quick: the NonEmptyChain. This structure is a context modeled as two cases: either a single value, or a pair of separate instances of itself appended together. This allows for a List-like structure with constant-time concatenation that can be converted to a Seq in linear time.

trait NonEmptyChain[+A] {

  import NonEmptyChain._

  def head: A

  def tail: Option[NonEmptyChain[A]]

  def length: Int

  def cons[B >: A](newHead: B): NonEmptyChain[B] = prepend(Singleton(newHead))

  def prepend[B >: A](prefix: NonEmptyChain[B]): NonEmptyChain[B] = prefix.append(this)

  def append[B >: A](suffix: NonEmptyChain[B]): NonEmptyChain[B] = Append(this, suffix)

  def toSeq: Seq[A] = head +: tail.fold(Seq[A]())(_.toSeq)
}

object NonEmptyChain {

  def apply[A](value: A, rest: A*): NonEmptyChain[A] =
    rest.map[NonEmptyChain[A]](Singleton(_)).foldLeft[NonEmptyChain[A]](Singleton(value))(_ append _)

  case class Singleton[+A](head: A) extends NonEmptyChain[A] {

    override def tail: Option[NonEmptyChain[A]] = None

    override def length: Int = 1
  }

  case class Append[+A](prefix: NonEmptyChain[A], suffix: NonEmptyChain[A]) extends NonEmptyChain[A] {

    override def head: A = prefix.head

    override def tail: Option[NonEmptyChain[A]] = prefix.tail.map(suffix.prepend).orElse(Some(suffix))

    override def length: Int = prefix.length + suffix.length
  }

  /** `NonEmptyChain` forms a `Semigroup` under the `append()` function.
    */
  implicit def nonEmptyChainSemigroup[A]: Semigroup[NonEmptyChain[A]] = _ append _
}

See here for the definition in the sample repository.

Using a NonEmptyChain, we can start writing validation functions for User to test that they work:

import green.thisfieldwas.embracingnondeterminism.syntax.applicative._
import green.thisfieldwas.embracingnondeterminism.syntax.validated._

case class User(username: String, email: String, password: String)

def validateUsername(username: String): ValidatedNec[String, String] =
  if (username.isEmpty) {
    "Username can't be blank".invalidNec[String]
  } else {
    username.validNec[String]
  }

def validateEmail(email: String): ValidatedNec[String, String] =
  if (!email.contains('@')) {
    "Email does not appear to be valid".invalidNec[String]
  } else {
    email.validNec[String]
  }

def validatePassword(password: String): ValidatedNec[String, String] =
  List(
    Option.cond(password.length < 8)("Password must have at least 8 characters"),
    Option.cond(!password.exists(_.isLower))("Password must contain at least one lowercase character"),
    Option.cond(!password.exists(_.isUpper))("Password must contain at least one uppercase character"),
    Option.cond(!password.exists(_.isDigit))("Password must contain at least one digit"),
    Option.cond(!password.exists("`~!@#$%^&*()-_=+[]{}\|;:'\",.<>/?".contains(_)))(
      "Password must contain at least one symbol"
    ),
  ).foldLeft(password.validNec[String]) { (validated, errorOption) =>
    errorOption.fold(validated) { message =>
      Invalid(validated.fold(_.cons(message))(_ => NonEmptyChain(message)))
    }
  }

def validateUser(username: String, email: String, password: String): Validated[NonEmptyChain[String], User] =
  User.curried.pure[Validated[NonEmptyChain[String], *]]
    .ap(validateUsername(username))
    .ap(validateEmail(email))
    .ap(validatePassword(password))

Using this function, we can attempt to create a User with nothing but invalid data and get the complete collection of errors back:

val validatedUser = validateUser(
  username = "",
  email = "bananaphone",
  password = "\n\t\r\r\r",
)
inside(validatedUser) { case Invalid(reasons) =>
  (reasons.toSeq should contain).theSameElementsAs(
    Seq(
      "Username can't be blank",
      "Email does not appear to be valid",
      "Password must have at least 8 characters",
      "Password must contain at least one lowercase character",
      "Password must contain at least one uppercase character",
      "Password must contain at least one digit",
      "Password must contain at least one symbol",
    )
  )
}

And with valid data, receive a constructed User:

val validatedUser = validateUser(
  username  = "commander.keen",
  email = "commander.keen@vorticonexterminator.net",
  password = "m4rti@an$Rul3",
)
inside(validatedUser) { case Valid(user) =>
  user shouldBe User(
    username = "commander.keen",
    email = "commander.keen@vorticonexterminator.net",
    password = "m4rti@an$Rul3",
  )
}

See here for the specs in the sample repository.

Applicatives thus enable entire computations to succeed if all context arguments are in the desired case. If any argument is in the undesired case, then this case is propagated and the computation as a whole fails.

Each of validateUsername(), validateEmail(), and validatePassword() act as levers on whether a User is successfully produced. Writing specific if-statements to guide whether a User is produced or errors returned instead is not required: the Applicative typeclass succinctly abstracts away the necessary plumbing to control the flow of logic required to handle undesired cases. Errors are declared where they should occur and the abstraction handles the rest.

Independent evaluation of contexts

It may not have been obvious from validateUser(), but each validation function evaluates independently of the other validation functions. In the Validation context, this means that each function executes without impacting the other functions regardless of individual success or failure. Imagine for a moment, what if the functions were evaluated within an asynchronous context?

Let’s define a function on the Applicative typeclass that gathers the results of some effectful operations:

def sequence[A](listFa: List[F[A]]): F[List[A]] =
  listFa.foldLeft(pure(List[A]()))((fListA, fa) => map2(fa, fListA)(_ :: _))

See here for the definition in the sample repository.

And then if we load three Users at once:

def loadUser(email: String): Future[User] = ???

val loadingUsers = Applicative[Future].sequence(List(
  loadUser("test@email.com"),
  loadUser("student@school.edu"),
  loadUser("admin@foundation.net"),
))

The variable loadingUsers now contains Future[List[User]]. As each Future[User] resolves, they are collected into a List. Because each loadUser() function executes independently, this has a profound implication in the context of a Future: they are executed concurrently!

Should any User fail to load, the undesired case will propagate and the sequence() function will halt. All other Users are discarded.

The pattern offered by Applicative is an all-or-nothing result in its output. If all inputs are in the desired case, then the output will be in the desired case as well. But if any are in an undesired case, then the undesired case propagates and computation halts.

Products of contexts

Given a 2-tuple of functors (F[A], F[B]) you can invert the nesting of the context and the 2-tuple using the pure() and ap() functions from Applicative with the following steps:

scala> :load console/imports.scala

scala> val productOfSomes = (Option(42), Option("banana"))
val productOfSomes: (Option[Int], Option[String]) = (Some(42), Some("banana"))

scala> val someOfProduct = (_: Int, _: String).curried.pure[Option].ap(productOfSomes._1).ap(productOfSomes._2)
val someOfProduct: Option[(Int, String)] = Some((42, "banana"))

This has an important implication, specifically that unlike the sequence() function these operations allow for gathering effectful operations that produce contexts with heterogeneous terms. Scala allows for tuples up to 22 elements, and it makes sense to abstract the above operations for each tuple size, especially because even at just 2 elements writing all of these out is already clunky!

In the sample repository, I have written a macro which adds two extension methods to each tuple size. Here’s the code that is generated for the 2-tuple:

implicit class Tuple2Ops[F[_], T1, T2](val t: (F[T1], F[T2])) extends AnyVal {

  // Converts a product of contexts `(F[T1], F[T2])` into a context of a product `F[(T1, T2)]`
  def tupled(implicit F: Applicative[F]): F[(T1, T2)] =
    mapN(Tuple2.apply)

  // Applies a product of contexts `(F[T1], F[T1])` to function, giving a result in context
  def mapN[Out](f: (T1, T2) => Out)(implicit F: Applicative[F]): F[Out] =
    F.ap(F.ap(F.pure(f.curried))(t._1))(t._2)
}

Being able to invert the nesting of a tuple and a context is most powerful when constructing case classes from arguments produced by effectful operations. By using the mapN() function, for example, validating arguments to the User constructor may be rewritten like this:

def validateUser(username: String, email: String, password: String): Validated[NonEmptyChain[String], User] =
  (
    validateUsername(username),
    validateEmail(email),
    validatePassword(password)
  ).mapN(User)

This syntax afforded by the mapN() extension method is much more concise and closely matches the constructor arguments order passed to User itself. After all, User is a product of results produced by contexts, which evaluate independently of each other as they do in the sequence() function.

Becoming an Applicative

In order to become an Applicative, an effect type must implement the typeclass. Let’s implement instances for the usual suspects, Option, Either, and List. As Applicative is a specialization of Functor, we can simply upgrade our current Functor instances to become Applicatives:

implicit val optionApplicative: Applicative[Option] = new Applicative[Option] {

  // removed map() and now using the default

  override def pure[A](a: A): Option[A] = Some(a)

  override def ap[A, B](ff: Option[A => B])(fa: F[A]): F[B] =
    (ff, fa) match {
      case (Some(f), Some(a)) => Some(f(a))
      case _ => None
    }
}

implicit def eitherApplicative[X]: Applicative[Either[X, *]] = new Applicative[Either[X, *]] {

  // removed map() and now using the default

  override def pure[A](a: A): Either[X, A] = Right(x)

  override def ap[A, B](ff: Either[X, A => B])(fa: F[A]): F[B] =
    (ff, fa) match {
      case (Right(f), Right(a)) => Right(f(a))
      case (Left(x), _) => Left(x)
      case (_, Left(x)) => Left(x)
    }
}

implicit val listApplicative: Applicative[List] = new Applicative[List] {

  // removed map() and now using the default

  override def pure[A](a: A): List[A] = List(a)

  override def ap[A, B](ff: List[A => B])(fa: List[A]): List[B] =
    ff.foldLeft(List[B]()) { (outerResult, f) =>
      fa.foldLeft(outerResult) { (innerResult, a) =>
        f(a) :: innerResult
      }
    }.reverse
}

See the instances of Applicative for Option, Either, and List in the sample repository.

Option and Either’s instances of Applicative are straight-forward: if a function and argument are present, they are applied and the result returned in the desired case. If either are missing, then the undesired case is propagated instead.

List looks very different at first glance, but conceptually performs the same way. Specifically, List performs a Cartesian product of its functions and arguments, applying each pair together and building a new List from the results. If either the function or argument List are empty, then an empty result List is returned, as an empty List represents the undesired case.

Applicative laws

Option, Either, and especially List’s Applicative instances look different. How do we know that they are well-behaved as applicatives? Just like functors, applicatives are expected to conform to a set of laws defined in the higher math of category theory.

There are four applicative laws, which must hold for all applicatives in addition to the functor laws.

Preservation of identity functions: A lifted identity function applied to a lifted argument is the same as the identity function applied directly to the lifted argument.
Preservation of function homomorphism: Lifting a function and an argument then applying them produces the same result as applying the unlifted function and unlifted argument then lifting the result.
Preservation of function interchange: Given a lifted function and an unlifted argument, applying the lifted function after lifting the argument should give the same result as when reversing the order of the function and argument. This is difficult to express in words, and the code is hard to follow, but roughly this translates to ap(ff: F[A => B])(pure(a)) == ap(pure(f => f(a)))(ff).
Preservation of function composition: Given lifted functions ff: F[A => B] and fg: F[B => C] and argument fa: F[A]: lifting compose() and applying fg, ff, and fa produces the same result as applying fg after applying ff to fa.

These laws are rigorous and we can write tests for these to prove that our applicative instances are defined correctly.

Testing “for all” `F[_]`

In order to properly test our applicative instances, we need to be able to generate a broad range of inputs to verify that the applicative properties hold with a high degree of confidence. Specifically, “for all” Lists, for example, the property checks for Applicative must pass for each generated instance. We will leverage scalacheck for property-based testing. scalacheck will generate for us a set of arbitrary instances of the contexts and execute tests called property checks against each to verify that each check passes. If all checks pass, then the property may be considered to hold “for all” instances of the tested context.

Generating an arbitrary context F[_] containing an arbitrary A is not supported directly by scalacheck, however. We can leverage this typeclass below to enable generating instances of F[A] from any generator for A:

import org.scalacheck.Gen

trait LiftedGen[F[_]] {

   def lift[A](gen: Gen[A]): Gen[F[A]]
 }

 object LiftedGen {

   def apply[F[_]: LiftedGen]: LiftedGen[F] = implicitly[LiftedGen[F]]

   object Syntax {

     implicit class LiftedGenOps[A](val gen: Gen[A]) extends AnyVal {

       def lift[F[_]: LiftedGen]: Gen[F[A]] = LiftedGen[F].lift[A](gen)
     }
   }
 }

See here for the definition in the sample repository.

Defining the applicative laws as properties

Now we will walk through defining the properties of Applicative in such a way that we simply supply the contexts as the argument to the test. This way we only define the properties once.

trait ApplicativeLaws { this: Laws with FunctorLaws =>

  /** Defined per Applicative laws taken from the Haskell wiki:
    * [[https://en.wikibooks.org/wiki/Haskell/Applicative_functors#Applicative_functor_laws]]
    *
    * These laws extend the functor laws, so `checkFunctorLaws[F]()` should be
    * executed alongside this function.
    *
    * @param TT The type tag of the context
    * @tparam F The context type being tested
    */
  def checkApplicativeLaws[F[_]: Applicative: LiftedGen]()(implicit TT: TypeTag[F[Any]]): Unit = {
    property(s"${TT.name} Applicative preserves identity functions") {
      // A lifted identity function applied to a lifted argument is the same as
      // the identity function applied directly to the lifted argument.
      forAll(arbitrary[Double].lift) { v =>
        // Haskell: pure id <*> v = v
        (identity[Double] _).pure.ap(v) mustBe identity(v)
      }
    }
    property(s"${TT.name} Applicative preserves function homomorphism") {
      // Lifting a function and an argument then applying them produces the
      // same result as applying the unlifted function and unlifted argument
      // then lifting the result.
      forAll(for {
        f <- arbitrary[Double => String]
        x <- arbitrary[Double]
      } yield (f, x)) { case (f, x) =>
        // Haskell: pure f <*> pure x = pure (f x)
        f.pure.ap(x.pure) mustBe f(x).pure
      }
    }
    property(s"${TT.name} Applicative preserves function interchange") {
      // Given a lifted function and an unlifted argument, applying the lifted
      // function after lifting the argument should give the same result as
      // when reversing the order of the function and argument. This is
      // difficult to express in words, and the code is hard to follow, but
      // roughly this translates to
      // `ap(ff: F[A => B])(pure(a)) == ap(pure(f => f(a)))(ff)`.
      forAll(for {
        u <- arbitrary[Double => String].lift
        y <- arbitrary[Double]
      } yield (u, y)) { case (u, y) =>
        // Haskell: u <*> pure y = pure ($ y) <*> u
        u.ap(y.pure) mustBe ((f: Double => String) => f(y)).pure.ap(u)
      }
    }
    property(s"${TT.name} Applicative preserves function composition") {
      // Given lifted functions `ff: F[A => B]` and `fg: F[B => C]` and
      // argument `fa: F[A]`: lifting `compose()` and applying `fg`, `ff`, and
      // `fa` produces the same result as applying `fg` after applying `ff` to
      // `fa`.
      val compose: (String => Int) => (Double => String) => Double => Int = g => f => g compose f
      forAll(for {
        u <- arbitrary[String => Int].lift
        v <- arbitrary[Double => String].lift
        w <- arbitrary[Double].lift
      } yield (u, v, w)) { case (u, v, w) =>
        // Haskell: pure (.) <*> u <*> v <*> w = u <*> (v <*> w)
        compose.pure.ap(u).ap(v).ap(w) mustBe u.ap(v.ap(w))
      }
    }
  }
}

See here for the full definition of the trait.

With this trait, laws specs can be written for each context.

abstract class Laws extends AnyPropSpec with ScalaCheckPropertyChecks with Matchers {

  implicit override val generatorDrivenConfig: PropertyCheckConfiguration =
    PropertyCheckConfiguration(minSuccessful = 100)
}

class OptionLawsSpec extends Laws with FunctorLaws with ApplicativeLaws {

  import Option.Instances._

  implicit val optionLiftedGen: LiftedGen[Option] = new LiftedGen[Option] {

    override def lift[A](gen: Gen[A]): Gen[Option[A]] =
      Gen.lzy(
        Gen.oneOf(
          gen.map(Some(_)),
          Gen.const(None),
        )
      )
  }

  checkFunctorLaws[Option]()
  checkApplicativeLaws[Option]()
}

class EitherLawsSpec extends Laws with FunctorLaws with ApplicativeLaws {

  import Either.Instances._

  implicit def eitherLiftedGen[X: Arbitrary]: LiftedGen[Either[X, *]] = new LiftedGen[Either[X, *]] {

    override def lift[A](gen: Gen[A]): Gen[Either[X, A]] =
      Gen.lzy(
        Gen.oneOf(
          gen.map(Right(_)),
          arbitrary[X].map(Left(_)),
        )
      )
  }

  checkFunctorLaws[Either[String, *]]()
  checkApplicativeLaws[Either[String, *]]()
}

class ListLawsSpec extends Laws with FunctorLaws with ApplicativeLaws {

  import List.Instances._

  implicit val listLiftedGen: LiftedGen[List] = new LiftedGen[List] {

    override def lift[A](gen: Gen[A]): Gen[List[A]] =
      for {
        size <- Gen.sized(Gen.choose(0, _))
        items <- Gen.listOfN(size, gen)
      } yield List(items: _*)
  }

  checkFunctorLaws[List]()
  checkApplicativeLaws[List]()
}

See these laws specs for Option, Either, and List in the sample repository.

Try running these specs!

But don’t forget `Validated`!

We also defined an Applicative instance for Validated above:

implicit def validatedApplicative[E: Semigroup]: Applicative[Validated[E, *]] =
  new Applicative[Validated[E, *]] {
    import Semigroup.Syntax._

    override def pure[A](a: A): Validated[E, A] = Valid(a)

    override def ap[A, B](ff: Validated[E, A => B])(fa: Validated[E, A]): Validated[E, B] =
      (ff, fa) match {
        case (Valid(f), Valid(a))     => Valid(f(a))
        case (Invalid(x), Invalid(y)) => Invalid(x |+| y) // propagate both undesired cases via Semigroup
        case (Invalid(x), _)          => Invalid(x)
        case (_, Invalid(y))          => Invalid(y)
      }
  }

This instance should be tested as well:

class ValidatedLawsSpec extends Laws with FunctorLaws with ApplicativeLaws {

  import Validated.Instances._
  import NonEmptyChain.Instances._

  implicit val nonEmptyChainLiftedGen: LiftedGen[NonEmptyChain] = new LiftedGen[NonEmptyChain] {

    override def lift[A](gen: Gen[A]): Gen[NonEmptyChain[A]] = {
      def go(depth: Int): Gen[NonEmptyChain[A]] =
        Gen.choose(0, (depth - 1).max(0)).flatMap {
          case 0 => gen.map(NonEmptyChain(_))
          case nextDepth =>
            for {
              prefix <- go(nextDepth)
              suffix <- go(nextDepth)
            } yield prefix.append(suffix)
        }
      Gen.sized(go)
    }
  }

  implicit def validatedNecLiftedGen[E: Arbitrary]: LiftedGen[ValidatedNec[E, *]] = new LiftedGen[ValidatedNec[E, *]] {

    override def lift[A](gen: Gen[A]): Gen[ValidatedNec[E, A]] =
      Gen.lzy(
        Gen.oneOf(
          arbitrary[E].lift[NonEmptyChain].map(Invalid(_)),
          gen.map(Valid(_)),
        )
      )
  }

  checkFunctorLaws[ValidatedNec[Exception, *]]()
  checkApplicativeLaws[ValidatedNec[Exception, *]]()
}

And we should also make sure NonEmptyChain is well-behaved as a semigroup:

class NonEmptyChainLaws extends Laws with SemigroupLaws {

  import NonEmptyChain.Instances._

  implicit val nonEmptyChainLiftedGen: LiftedGen[NonEmptyChain] = new LiftedGen[NonEmptyChain] {

    override def lift[A](gen: Gen[A]): Gen[NonEmptyChain[A]] = {
      def go(depth: Int): Gen[NonEmptyChain[A]] =
        Gen.choose(0, (depth - 1).max(0)).flatMap {
          case 0 => gen.map(NonEmptyChain(_))
          case nextDepth =>
            for {
              prefix <- go(nextDepth)
              suffix <- go(nextDepth)
            } yield prefix.append(suffix)
        }
      Gen.sized(go)
    }
  }

  implicit def arbitraryNonEmptyChain[A: Arbitrary]: Arbitrary[NonEmptyChain[A]] =
    Arbitrary(nonEmptyChainLiftedGen.lift(arbitrary[A]))

  checkSemigroupLaws[NonEmptyChain[Int]]()
}

See these laws specs for Validated and NonEmptyChain

Implications of the applicative laws

Each of Option, Either, and List conform to the applicative laws and we only had to write the properties once. These properties prove that functions and arguments used within these contexts maintain referential transparency in their arrangements and that the specific contexts do not change the factoring semantics of the code.

What does change, however, are these contexts’ specific effects. For example, you would not have to refactor code abstracted by applicative functions if you changed the backing implementation from Either to List, but your code would produce potentially more than one result in the desired case.

This is the goal, however, as these effects’ dimensions of unknown quantity should not burden our code. Instead, we push the complexity to the edge of the context, where it is important that our context is an Either or a List, and keep our business logic focused on individual instances contained within each context.

What is enabled by applicatives?

Applicatives primarily offer independent computation. Specifically, the arguments to applicative functions such as ap(), map2(), or sequence() are evaluated independently of one another, and their individual outputs as a whole influence whether the functions consuming them are permitted to compute against the outputs of their desired cases or if they should halt computation and propagate any undesired cases.

When all inputs to an applicative function are in the desired case, then the output of the lifted functions will also be in the desired case. Conversely, if any input is in the undesired case, then it will be propagated instead, and the other cases will be discarded. In this regard, applicative functions provide an all-or-nothing operation.

Independent computation provides some level of control flow, but it doesn’t guide execution to proceed only if the previous execution has succeeded, as all operations evaluate independently of each other. Applicatives therefore do not provide a mechanism to support imperative programming. For this kind of control flow, you need to further specialize the applicative functor.

In my next post Imperative Computation in Functional Programming we will explore the infamous monad and how it enables imperative control flow in functional programming.

Contexts and Effects in Functional Programming

2022-06-05T13:21:27-07:00

Estimated reading time: 1h 49m 41s

Have you ever received an unexpected null reference? Have you ever written a function to validate some input only to have it turn into spaghetti over time? How do you anticipate exceptions and protect against them at runtime?

This post is part of a series:

Contexts and Effects in Functional Programming

Enabling Control Flow in Functional Programming

Imperative Computation in Functional Programming

The code that accompanies this post may be found here.

Significant portions of program logic exist to address specific cases imposed by nondeterminism and unknown quantities. Have you ever written some code only to find out it does something unexpected when it’s running in production? To protect against unexpected behavior you first have to be aware that an operation may return something unexpected, such as a null reference, invalid data, or throw an exception, and then write code that anticipates and recovers from such cases.

Defensive programming as a practice tries to protect against errors and unknowns by preempting how they might occur. However anticipation of errors and unknowns rests entirely on tacit knowledge and imposes complex code to handle and recover from them. This complex code draws focus from writing the business logic that drives the value of programs.

In this post I will provide effects as a model for characterizing nondeterminism and unknowns in programs. I will introduce a design pattern to abstract complexity using this model.

Conventions

This post assumes familiarity with Scala code. I will provide Scala code for concrete examples and note where they are different. Abstract examples will employ notation that looks like “math”.

Terminology will leverage common functional programming vocabulary. Terms when they are first introduced will be bold and important references are italicized. Some terminology is provided up-front, other terms will be defined inline either explicitly or by context.

Concepts and phrases that are important are italicized.

Where there is conceptual overlap with object oriented programming, I will leverage those terms to drive the intent behind abstractions.

How to read “math”

Uppercase letters and words starting with uppercase letters are names of types. For example A reads as “A” and means “type of A”.

Lowercase letters and words starting with a lowercase letter are functions or variables. For example f reads as “f” and means “function f” or “variable f”.

A => B reads as “A to B” and means “function type of input type A mapped to output type B”. Function types can map any kind of input type to any kind of output type, including other function types as in (A => B) => C or A => B => C. Note that the => operator is right-associative, such that A => (B => C) means the same thing as A => B => C.

F[_] reads as “context F of underscore” or “context F”. Contexts are a type constructor¹ that take types as an argument and produce another type. They become proper types when their underscore is replaced by another type, as in F[A] or “context F of A”.

() a pair of parentheses, reads as “unit” and may be treated as “nothing” or “void” as you really can’t do much with it. It is both a type and a solitary value.

f: A => B reads as “f is A to B” or “function f has type of A mapped to B”.

fa: F[A] reads as “fa is F of A” or “variable fa has type of context F of A”.

h = g ∘ f reads as “h is g after f” or “h is defined as function g composed after function f”. Composition is described in Terminology.

Terminology

Functors are a programming pattern explored in this post. Think of them as an analog of a design pattern found in object-oriented programming.

Expressions are values that are described by some type A.

Functions are a special case of expressions that map some type A to some type B. They are described by function type notation A => B.

Terms are identifiers naming unitary or indivisible variables and types.

For example, in the declaration:
```
def map[A, B](fa: F[A])(f: A => B): F[B]
```
The variable terms are fa and f, and the type terms are A and B.

Contexts describe circumstances within which their contents are found. They are noted using F[_] when their contents are unspecified, and F[A] when their contents are known to be of type A. You can think of F[A] as a box that might contain some A. They are more concretely defined in later sections

Lifting describes injecting a term A into a context F[_] such that lift: A => F[A]. Think of lifting as if you were to lift some A into the box of F[_].

A Lifted term or expression already has the form F[A], or an A that is already in a box.

Lowering describes extracting a term A from a context F[A] such that lower: F[A] => A. Lowering is like taking the A out of the box.

Composition describes chaining the output of a function f: A => B to the input of a function g: B => C such that a new function h: A => C may defined as h = g ∘ f. The ring operator ∘ performs this chaining, and the odd after is because if you apply the literal composition to some argument x, as in (g ∘ f)(x), this means that x would be applied first to f and then g is applied after f, working from right to left.

This algebraic notation demonstrates how function g is applied after function f is applied to the argument x:
```
h(x) = g(f(x))
```

Compare with this definition in Scala which declares the types of the functions:

val f: A => B = _
val g: B => C = _
val h: A => C = x => g(f(x))

Complexity in programming

Pure and impure functions

Programming broadly consists of two categories of functions:

Pure functions which produce the same result for the same argument, for all arguments. They are deterministic.
Impure functions which produce different results for the same argument, for any argument. They are nondeterministic.

Nondeterminism arises from outputs dependent on factors other than input to a function. These factors are referred to as side effects. Implicit input as a side effect may affect the output of a function, but in addition a function may produce side effects as implicit output.

Both categories of functions may produce their results in unknown quantities along any measurable dimension, such as presence, length, or validity of their result. These quantities require specific knowledge of a given input in order to be known with certainty in the result. Unknown quantities are categorically nondeterministic in impure functions as they are particularly influenced by side effects.

Manifesting complexity

Nondeterminism

Nondeterminism as a dependence on factors other than initial state and input arises when a function f: A => B maps to a different member of B for any number of times f has been applied to the same member of A. This means that f is influenced by implicit input that occurs independent of its type signature.

An extreme example of a nondeterministic function is a true random number generator rng: () => Int as it maps the solitary unit value () to all members of type Int. This mapping is influenced by some side effect or implicit input which is external to the function’s type signature of () => Int:

println(rng())
// 1729
println(rng())
// 87539319
println(rng())
// -2147483648
println(rng())
// 1337
println(rng())
// 42

Psuedo-random number generators (PRNG’s) are deterministic in that the initial seed state dictates predictable output over time, but only if every mutation of this state is known up to the point of the current call site. In practice, they are considered nondeterministic.

Nondeterminism is significant in that operations may be unpredictable, and that no operation in particular may be reproducible.

Unknown quantities

Unknown quantities along measurable dimensions arise in functions returning types such as lists, potentially null references, or validation results. These outputs have unknown length, presence, and validity respectively, and require specific handling for undesired cases. Even pure functions produce results having unknown quantities.

A simple example is a function toBits: Int => List[Boolean] where the known quantity of Boolean bits returned requires specific knowledge of the input argument.
Hashmap lookups may or may not return a value associated to a given key. Unless you have specific knowledge of the key used to lookup a value and of the contents of the map itself, you don’t have any guarantee whether the value actually exists.

Both of these operations are pure functions and are deterministic, but their results are contextualized by length and presence. Any unknown quantity along some measurable dimension requires specific handling in code. This means that in addition to writing code that handles a desired case of an operation, code must be specifically written for each dimension that exhibits unknown quantities.

Side effects as implicit output include faults such as the divide by zero error and thrown exceptions. They impose an additional layer of protection to prevent or recover from them. Exceptions are categorically nondeterministic as there is no single input that guarantees that an exception will never be thrown, as some side effect as an implicit input may influence their production.

In contrast with most faults, a divide by zero error only occurs if the input divisor is 0. The additional check for 0 that division sometimes requires is not considered complexity in practice.

Running out of memory will throw an exception even in pure functions. Exceptions are truly nondeterministic and you must choose when and how to handle their cases. Hopefully you know ahead of time where you will need to do so.

In addition to being side effects as implicit output, exceptions may be reasoned about as a dimension of success or failure in an operation. This quantity is unknowable ahead of time and highly dependent upon implicit input.

Concurrency and asynchronous operations are driven entirely by side effects, most particularly in the dimension of time. Asynchronous operations have an unknown temporal quantity that imposes costly specific handling, as execution must wait for operations to complete. Support for asynchronous operations requires runtimes to manage limited computing resources and scheduling of tasks, forming an entire system within a program.

Relating nondeterminism and unknown quantities

Side effects enable nondeterminism which influences unknown quantities in the results of operations. Undesired cases along dimensions such as length, presence, validity, success, and time require specific handling in addition to the code to handle the desired cases of operations. This specific handling creates complexity and draws engineering focus away from business logic. Yet side effects drive the business value of programs in the real world, which requires that we embrace nondeterminism and unknown quantities.

How might complexity in programs be reduced if they must also be driven by side effects?

Implied complexity

Given a function f: A => B and another g: B => C: a third function h: A => C may be composed of h = g ∘ f or h is g after f. Programs may be modeled as a function program: Input => Output, where program is composed of innumerable smaller functions, together in concert building the necessary mappings to generate the desired program output. However composition is not so simple in programs as the results of some operations may not produce the desired input for subsequent operations.

Functions in real world programs must internally interact with implicit inputs and outputs not present in the program’s signature of program: Input => Output. An employee payroll system for example must make database queries and integrate with banks. These implicit inputs and outputs have effects which determine how their associated functions produce their desired outputs. For example, database queries return nondeterministic responses of unknown length and an error might occur when performing a direct deposit. These effects determine how and whether payday is successfully produced.

Errors and unknown quantities as effects of these operations are opaque in functions modeled as simple input to output, as in getEmployee(): Int => Employee. The signature of this function requires tacit knowledge of what effects may determine how an Employee is produced from it. For example:

An associated Employee may not be found.
The returned Employee may change between applications of the same Int employee ID.
The database or network may fault and the function generates an exception that must be handled.

You might be thinking that these cases are a given when working with database code, but that knowledge only comes with experience. These cases are effects which describe the circumstances under which an Employee may be produced and can be modeled accordingly as part of the typed API of getEmployee(). Capturing these effects might look like getEmployee(): Int => Probably[Employee], wherein we keep these effects within the box of Probably[_]. I will soon explain how this modeling works; first we will consider how to characterize the complexity which defines effects.

Operations producing undesired cases

Can you think of some program operations that produce undesired cases in addition to their desired cases? How might these cases cause code to become complex?

When a program starts, it may read configuration from the environment, a database, or files. Reading configuration values may be blocking or asynchronous, and some configuration keys may not have associated values.
Database queries produce an unknown length of rows, and may fail due to incorrect syntax, a database error, or network fault. Receiving an unknown length of rows imposes specific handling if all you want is just one row and any guarantees that you’re retrieving the correct one must be manually enforced.
API calls may be blocking or require async IO, they may fail for any reason, and the data they return may or may not conform to an expected structure.
Task management of long-running operations requires async IO and managing limited computing resources. Tasks may fail for any reason or they may never complete. You have to wait for the result to be available in order to use it.
Error handling when an operation fails may require aborting the operation and returning a default case, or retrying it. Exceptions are easy to rethrow or bubble-up the stack causing unexpected errors at runtime.
Retries using an exponential back-off strategy must retain their previous retry interval and apply a random jitter in order to calculate their next interval.
User-provided input, server requests, and API responses must be validated to assert that they conform to an expected structure and then validated for semantic content. Invalid data must be handled.
Some operations produce logs and metrics as a secondary output. All components requiring logs and metrics carry an extra dependency in order to support them, and their associated operations may be blocking or require async IO.
Feature flags and ramps need to be queried in real-time. These queries may be blocking or asynchronous, and they may fail for any reason. Presentation of default variants in the case of failure must be reliably persisted to the current session, yet this operation may fail as well.
A/B testing requires reliably determining eligibility of users’ sessions per variant and persisting the associated assignments. Either of these operations may fail and presenting a default variant itself requires persistence. Retrieving assigned variants must also be reliable.

A model for characterizing complexity

Complexities can be characterized in terms of effects. The operations listed above impose complexity because they feature the effects of:

Presence: Some configuration keys may not have an associated value.; Some database queries expect one row to be returned, but instead no rows may be found.
Length: Database queries may return zero or many rows.; Collections don’t have a totally guaranteed or fixed size.
Validity: User-provided input, server requests, and API responses all require validation before they may be used.
Success: Some operations fail with an exception, which can hide potential error cases.; Some operations may be aborted.
IO: Operations are dependent on external systems’ state as implicit input.; Operations can affect external systems’ state as implicit output.; Operations may be concurrent, paused, or interrupted.; Operations may block execution of the calling operation.; Interacting with concurrency primitives may block execution and produce nondeterministic outputs.
Time: Task management makes no guarantees how long any particular task will take to run.; API calls require an indeterminate amount of time.; Database queries take time to run and return rows as they are found.; Operations dependent on async output must await the result.
State: Retry strategies must track previous state in order to calculate their next retry periods.

The actual list of effects is innumerable, but these ones are common.

Demonstrating effects in code

Take for example this Java code from a hypothetical payroll system:

class PayrollRunner {

  EmployeeRepo employeeRepo;
  BankAchClient achClient;
  PayCalculator payCalc;

  boolean runPayroll(long employeeId) {
    try {
      Employee employee = employeeRepo.find(employeeId)
      if (employee == null) {
        Logger.error("Missing employee " + employeeId);
        return false;
      }
      Paycheck paycheck = payCalc.calculatePaycheck(employee);
      if (paycheck == null) {
        Logger.error("No paycheck for " + employeeId);
        return false;
      }
      String companyAcctNo = PayrollConfig.get("companyAcctNo");
      String companyRoutingNo = PayrollConfig.get("companyRoutingNo");
      String response = achClient.depositPaycheck(companyAcctNo, companyRoutingNo, paycheck);
      return response.equals("SUCCESS");
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }
}

The code above demonstrates effects in the dimensions of:

IO as database queries and network calls are implicitly synchronous. Configuration acts as an implicit input.
Presence as an Employee may not be found or their Paycheck not calculated. Configured banking information may not have associated values.
Validity where the "SUCCESS" response from the achClient is the only check performed. Other responses are simply not handled.
Success where logging occurs but swallows errors, resulting in opaque false return cases. Exceptions also bubble up from database and network operations.

These effects are not obvious from the code above. Below I have rewritten the operation to handle all undesired cases to highlight where complexity exists:

class PayrollRunner {

  EmployeeRepo employeeRepo;
  BankAchClient achClient;
  PayCalculator payCalc;

  void runPayroll(long employeeId) throws PayrollException, MissingConfigException {
    Employee employee;
    try {
      employee = employeeRepo.find(employeeId);
      if (employee == null) {
        Logger.error("Missing employee " + employeeId);
        throw new MissingEmployeeException(employeeId);
      }
    } catch (SQLException exception) {
      throw new PayrollException("Error looking up employee " + employeeId, exception);
    }

    Paycheck paycheck = payCalc.calculatePaycheck(employee);
    if (paycheck == null) {
      Logger.error("No paycheck for " + employeeId);
      throw new PayrollMissingException(employeeId);
    }

    String companyAcctNo = PayrollConfig.get("companyAcctNo");
    if (companyAcctNo == null) {
      throw new MissingConfigException("companyAcctNo");
    }

    String companyRoutingNo = PayrollConfig.get("companyRoutingNo");
    if (companyRoutingNo == null) {
      throw new MissingConfigException("companyRoutingNo");
    }

    String response;
    try {
      response = achClient.depositPaycheck(companyAcctNo, companyRoutingNo, paycheck);
    } catch (AchException exception) {
      throw new PayrollException("Failed to deposit paycheck for employee " + employee, exception);
    }
    if (!response.equals("SUCCESS")) {
      throw new DepositPaycheckException("Received error response when depositing paycheck for employee " + employeeId + ": " + response);
    }
  }
}

There’s a large amount of complexity in this code due to various effects. In order to make clear all effects and where they occur, I leveraged checked exceptions and all exceptions that are known to be thrown by other functions are translated into exceptions representing the domain of this operation. As all false cases effectively communicated no information I replaced them with exceptions typed according to the reason for failure. The operation now returns void as it is side-effecting and a specific return for success would be superfluous.

There are a number of checks along the dimension of presence. There’s several along the dimension of success. Each operation is dependent upon the success of the operation preceding it, following a procedurally validated imperative flow.

I highlight procedural validation as a concept to illustrate that the above operation advances by manually verifying success or failure of each operation in sequence. This kind of logic is error-prone and fragile, as dependencies between operations are non-obvious and changes to error handling independent of domain logic can easily introduce bugs.

There’s a lot of inner operations subject to effects within the main operation. Fortunately there are ways to model sets of these effects and contain the scope of their impact so that code is less complex. Rewriting the above code using an effect model might look like this:

def runPayroll(employeeId: Long): PayrollRunner[()] =
  for {
    employee <- employeeRepo.find(employee).getOrFail(EmployeeMissing(employeeId))
    paycheck <- payCalc.calculatePaycheck(employee).getOrFail(PaycheckMissing(employeeId))
    companyAcctNo <- getConfig("companyAccountNo")
    companyRoutingNo <- getConfig("companyRoutingNo")
    response <- achClient.depositPaycheck(companyAcctNo, companyRoutingNo, paycheck)
      .flatMap {
        case "SUCCESS" => pure(())
        case msg       => fail(DirectDepositError(employeeId, msg))
      }
  } yield response

This code looks similar, doesn’t it? What hides between the lines here is a custom effect model PayrollRunner that abstracts away the effects of presence, async IO, success, and implicit input and output. This code is thus unburdened of most complexity, and makes the rest of it easier to work with.

You may notice that there are no return statements for error cases: the flow of execution through this code is not performed procedurally. Instead flow is controlled by declaring where errors occur and the main operation short-circuits itself should any inner operation fail. This decouples domain logic from validating imperative operations, reducing a significant source of bugs.

This abstraction of effects allows for safer code that better focuses on the business logic at-hand.

Previously I described programs as a case of function composition: h = g ∘ f. Functors address a special case of function composition, the composition of functional effects. In the following sections I will describe how effects are abstracted in order to demonstrate how they compose later.

Contexts and effects

What is a context? A context is a setting where stuff exists under some circumstances. Stuff such as instances of term A in the context of F[A].

Contexts in Scala may be neatly represented by a letter and brackets such as F[_] read as context F with an underscore when the type of the term is unspecified, and F[A] read as F of A when the term is known to be of type A. Other letters work nicely of course, as do concrete names, as in Option[Int] or Option of Int.

The letter F is a shorthand for Functional Effect, which is another term for context. Contexts each model a set of effects which represent concrete, typed APIs that describe how their terms may be produced. This mean that for any impure function f: A => F[B] you receive an output of type B whose production is contextualized by the effects of F[_]. A pure function g: A => B in comparison returns a fully-evaluated B absent of effects due to the application of function g.

You might reason about contexts and their effects as a probabilistic box which might contain something you want. Whether something is there is what the box abstracts over, and we will expand on this later.

Names of contexts can hint at the effects they model, and with some intuition you may be able to figure out what each context’s effects may be.

Common contexts and some of their effects

Elementary contexts representing dimensions of presence:

Option[A] or Option of A: Presence, absence, or optionality of some instance of term A. Getting the term A when there is no instance causes a fault.
Either[X, A] or Either of X and A: Conventionally treated as either term A if valid or term X if invalid. Getting the wrong term causes a fault.
List[A] or List of A: Unknown length, sort, and cardinality of term A. Getting an A from an empty list or from beyond the end of it causes a fault.
NonEmptyList[A] or NonEmptyList of A: At least one of term A with an unknown sort and cardinality. The first A is guaranteed to be present.
Id[A] or Identity of A: This is A and is guaranteed to be present.
Set[A] or Set of A: A set of distinct instances of A whose size is unknown.

Many data structures are used to model different forms of presence.

Contexts representing side-effects:

IO[A] or IO of A: Externally-acquired term A.
- The instance may be here already, eventually, or never. Execution awaits the instance until it arrives or a fault occurs.
- The instance is acquired externally from the program via processes that rely on and are influenced by side effects.
- Faults may interrupt the acquisition of the instance.
- Getting the instance causes a fault if acquisition has failed.
Future[A] or Future of A: Eventually-acquired term A.
- The instance may be here already, eventually, or never.
- The instance may be acquired externally from the program via processes that rely on and are influenced by side effects.
- The instance may be produced via long-running processes.
- Acquisition of A be suspended, awaited, or run in parallel with any number of other Future[_]’s. This allows for task management against limited computing resources.
- Faults may interrupt the acquisition of the instance.
- Getting the instance causes a fault if acquisition is incomplete or has failed.

Contexts representing implicit input and output:

ReaderT[F, A] or Reader Transformer of context F and A: Reading of implicit inputs in the context of F[_].
- Propagation of configuration usually leverages this effect by abstracting how instances of configuration values are acquired.
WriterT[F, A] or Writer Transformer of context F and A: Writing of implicit outputs in the context of F[_].
- Logging usually leverages this effect by abstracting how such output leaves the program.
StateT[F, A] or State Transformer of context F and A: Modifying implicit inputs and outputs in the context of F[_].
- Models state changing over time in an otherwise-immutable context.

These contexts are higher-order in the term of F. They are listed here for illustration but won’t be explored in this post.

Each of these contexts have two shared characteristics in that they produce some term A and that their effects determine how term A is produced. But with such a wide array of effects, and with so little overlap between each context, how can instances of term A be consumed in a manner unburdened of complexity?

In order to generalize contexts, the key differentiator between them must be abstracted: effects. By shedding effects as an implementation detail, the production of term A remains a shared characteristic. This opens an opportunity to create a seam between an impure function producing the context itself and a pure function consuming instances of the term A within the context.

The term seam comes from Michael Feathers’ Working Effectively With Legacy Code: “A place where you can alter behavior in your program without editing in that place.” A seam is where we introduce an abstraction to separate the impure output of an operation from pure consumption of the desired case. How functions handle an unknown length of items, for example, does not change and is independent of how functions consume each item. How functions consume each item varies on a case-by-case basis, and changes frequently according to requirements within business logic. This illustrates a need for a separation of concerns that can be enabled by abstraction.

Take for example the following JavaScript code:

// traversing a length of items does not change
for (var i; i < businessCases.length; i++) {
  // businessLogic() changes frequently
  businessLogic(businessCases[i]);
}

// asynchronous dispatch of API calls does not change
callApiAsynchronously("/get/stuff", function (stuff) {
  // doThingsWithStuff() changes frequently
  doThingsWithStuff(stuff);
});

How do you create this seam?

A design pattern for contexts

For any context F[_], it produces some term A. If you have a function f: A => B, how would you apply it to the term produced by the context F[A]? That would require lowering the term. Specifically, you can’t apply the following function directly to the context:

// given a context producing term A
val fa: F[A]
// given a function A => B
def f(x: A): B
// try to apply the function
f(fa)
// compile error!

Recall from the previous section, contexts share two characteristics: that they produce a term, and that they have effects which determine how the term is produced. After abstracting effects, contexts do not expose an obvious shared interface to extract the term. Consider the following definitions for Option[A], Either[X, A], and List[A]:

Option[A] is the effect of presence, absence, or optionality of an instance of term A:

sealed trait Option[+A] {
  def get: A = throw new Exception()
}
case class Some[+A](override val get: A) extends Option[A]
case object None extends Option[Nothing]

See here for the definition in the sample repository.

Either[X, A] by convention is the effect of either success with term A or failure with term X:

sealed trait Either[+X, +A] {
  def left: X = throw new Exception()
  def right: A = throw new Exception()
}
case class Left[+X, +A](override val left: X) extends Either[X, A]
case class Right[+X, +A](override val right: A) extends Either[X, A]

See here for the definition in the sample repository.

List[A] is the effect of unknown length, sort, and cardinality of instances of term A:

sealed trait List[+A] {
  def head: A = throw new Exception()
  def tail: List[A] = throw new Exception()
}
case class ::[+A](override val head: A, override val tail: List[A]) extends List[A]
case object Nil extends List[Nothing]

See here for the definition in the sample repository.

An object-oriented approach to contexts

Both Option[A] and Either[X, A] have roughly the same structure in that there either is or isn’t an instance of the desired term A. Because of this, a lowering operation extract: F[A] => A is possible as it means the same thing between both of them: extract either gets the existing instance of the term A or it faults. In object oriented programming, Option[A] and Either[X, A] might expose such an interface:

trait Extractable[A] {
  def extract(): A
}
sealed trait Option[A] extends Extractable[A] {
  def get: A = throw new Exception()
  override def extract(): A = get
  override def isPresent(): Boolean = this.isInstanceOf[Some[A]]
}
sealed trait Either[X, A] extends Extractable[A] {
  def left: X = throw new Exception()
  def right: A = throw new Exception()
  override def extract(): A = right
  override def isPresent(): Boolean = this.isInstanceOf[Right[X, A]]
}

See here for the definitions of Option, Either, and List using the single-value Extractable interface.

Generalizing for an unknown length of term `A`

How do you extract the term A from a List[A] such that it means the same thing as in Option[A] and Either[X, A]?

As in Option[A] and Either[X, A] there is a notion of the presence or absence of an instance of the term A, but presence in List[A] implies one to many instances. A solution inspired by object oriented programming might change the interface thusly:

trait Extractable[A] {
  def extract(): Seq[A]
  def isPresent(): Boolean
}
sealed trait Option[A] extends Extractable[A] {
  def get: A = throw new Exception()
  override def extract(): Seq[A] = Seq(get)
  override def isPresent(): Boolean = this.isInstanceOf[Some[A]]
}
sealed trait Either[X, A] extends Extractable[A] {
  def left: X = throw new Exception()
  def right: A = throw new Exception()
  override def extract(): Seq[A] = Seq(right)
  override def isPresent(): Boolean = this.isInstanceOf[Right[X, A]]
}
sealed trait List[A] extends Extractable[A] {
  def head: A = throw new Exception()
  def tail: List[A] = throw new Exception()
  override def extract(): Seq[A] = ???
  override def isPresent(): Boolean = this.isInstanceOf[::[A]]
}

See here for the definitions of Option, Either, and List using the multi-value Extractable interface with Seq.

This interface however is not coherent. Faulting on absence is preserved as a behavior in Option[A] and Either[X, A], but Scala’s Seq[A] is allowed to be empty per its definition. Allowing List[A] to be empty implies that it should be allowed to return an empty Seq[A] from extract. In order to preserve faulting on absence, extract must return a NonEmptyList[A] instead, as this has the effect of always having at least one instance of term A. You’re still stuck with an unknown length of instances, though.

This interface essentially transforms these contexts into NonEmptyList[A] and imposes its specific complexity on all of your code. You would probably be very unhappy using it.

What about implementing extract for Future[A]? When applied to Future[A], the extract function by its own signature is a blocking call. You want your dependency on A to be properly asynchronous.

This interface does not generalize for more than the contexts of Option and Either, and it requires that they both subclass a shared trait. A better interface exists. It also doesn’t force these types into the same hierarchy.

This interface also suffers from a certain Schrödinger’s cat problem, where you’re effectively peeking inside the box and sometimes finding out that the cat isn’t there.

Motivating functors as a design pattern

Option[A], Either[X, A], List[A], Future[A], and IO[A] each have different effects that determine how term A is produced. You must follow an axiom from object oriented programming: abstract what changes. Therefore you have to shed effects as an implementation detail. How might that impact lowering the term A?

You may be unsatisfied by the answer: extraction cannot be generalized. All you know is that there is term A. You don’t know whether an instance is present, how many of it there are, whether it’s here already, or if it’s arriving later. How do you consume term A when you know nothing about its instances’ nature of existence? You have a box that might contain what you want, but you won’t know unless you open it up; it might explode if you do. Functors solve this problem.

Functors are abstractions that allow you to consume term A within the context of F[A]. Functors are a class of types for which they have defined a single function, one that you might have seen in the wild, called map(). Functors in Scala may be formally declared using the Functor typeclass:

trait Functor[F[_]] {
  def map[A, B](fa: F[A])(f: A => B): F[B]
}
object Functor {
  def apply[F[_]: Functor]: Functor[F] = implicitly[Functor[F]]
}

See here for the definition in the sample repository.

What map() does is lift the function f: A => B into the box so that it behaves as F[A] => F[B], giving back F[B].

This lifting of functions that map() performs is coherent across contexts. With map() you can apply f: A => B to any List[A] just as you can any IO[A]. The results of both operations are predictable: your List[A] maps to List[B] and your IO[A] maps to IO[B].

How would you consume the term produced by Future[A] or Option[A]? You would also use map().

What this enables is your function f: A => B to be used with any functor regardless of its specific effects. Your function f: A => B is immediately reusable across all contexts and can be unit tested in isolation of effects. It no longer matters whether the box contains what you want, as it will use your function if something is there.

Why does the `map()` function return `F[B]`?

Recall that contexts generally do not permit extracting terms. Think for a moment: what does extracting the term mean if you’re using a context like Option[A]? What about Future[A]? Would their effects change how extraction of the term would work?

Extracting the term from List[A] flatly doesn’t make sense as it has the effect of an unknown number of instances.

Because there is no way to generalize extracting a term from a context, functors don’t allow you to operate on contexts in such a way that an instance of the term can escape them.

Most importantly, by keeping all operations against terms within their context, contexts’ specific effects remain abstracted. Asynchronous operations with Future[A] remain asynchronous, the length of List[A] remains unknown, and Option[A] may or may not be present.

Functors thus preserve structure by keeping operations within the context. For example, applying map() on a List[A] or BinaryTree[A].

map to a List[Int]." alt="Applying map to a List[Int].">

Applying map to a List[Int].

map to a BinaryTree[Int]." alt="Applying map to a BinaryTree[Int].">

Applying map to a BinaryTree[Int].

The application of map() produces two new and identifiable List[B] and BinaryTree[B]s. The values internally change, as a function has been applied to them, and BinaryTree[B] may re-balance itself accordingly. What matters here is that the structures are coherent and identifiable. Both List[B] and BinaryTree[B] are created from this operation and the originating List[A] and BinaryTree[A] still exist in their original state.

Compare with using iteration over an array using a for loop in this JavaScript code:

const array = [1, 2, 3, 4, 5, 6, 7]
const square = x => x * x

for (const x in array) {
  const x2 = square(x) // apply function to the current value
  results.push(x2) // append to a new array
}

Iteration as a form of lowering destroys structure. In order to get a array back you have to rebuild it yourself and any structural guarantees must be manually implemented following procedural steps. This means that you have to open the box, manipulate what it contains, then put the result in a new box yourself.

This isn’t to say that functional programming is only about iteration and loops versus map(). Can you think of other operations that might destroy structure? For example, if you use an await() operation on a Future[A] you will destroy its asynchronous structure and potentially harm the performance of your program.

This doesn’t mean that you should never look in the box. For example, where the type of your context is known, it may make sense to take the box apart to extract the term. A common use case with Option is to extract the term if it is present and provide a default instance otherwise:

val myLookup = map.find("myKey").getOrElse("myDefault")
val myMatchingLookup = map.find("myKey") match {
  case Some(myValue) => myValue
  case None => "myDefault"
}

Context `F[A]` must produce some term `A`

Recall from above: For any context F[_], it produces some term A_. If a context were guaranteed to have an instance of a term A then you should be able to consume it with your function f: A => B, right?

But what if there’s nothing there, as in there are zero instances of term A? Can you do anything? When a context has this kind of effect, a sort of “nothing here” or void, then the map() function above doesn’t do anything because there isn’t anything to do. If you try to map() a void F[A] with f: A => B then it returns a void F[B] as there’s “nothing here”. It does this without having used f: A => B to get there.

This is called the undesired case of a context as it means your box didn’t contain something you wanted. This case isn’t necessarily void, however, as void this context simply means that a particular context has a case where no instances of its term A exist, even though the context itself may contain other data, such as errors.

This behavior of “doing nothing” is referred to as short-circuiting and it is a key feature of contexts that encode some notion of an undesired case. It is exploited in particular to enable two key features of imperative programming, control flow and error handling, which I will expand on in later parts of this series.

The two contexts Option[A] and Either[X, A] demonstrate simple short-circuiting. An Option[A] will only map() an instance of its term A if it is present as the desired case of Some[A], and an Either[X, A] will only map() if an instance of the desired term A is present as the desired case of Right[X, A].

In contrast with Option[A] and Either[X, A], the Id[A] context has the effect of the identity of term A. To put it plainly, Id[A] is the instance of term A: as the instance is always present, the Id[A] context is solely the desired case and never short-circuits.

Implementing functors in Scala

Each context must provide its own implementation of map() in order for it to be used as a functor. Functor implementations in Scala are provided via typeclasses, and any type that has the shape of F[_] may become a functor by implementing the Functor typeclass from above:

trait Functor[F[_]] {
  def map[A, B](fa: F[A])(f: A => B): F[B]
}
object Functor {
  def apply[F[_]: Functor]: Functor[F] = implicitly[Functor[F]]
}

See here for the definition in the sample repository.

Instances of the Functor typeclass implement this trait and make themselves available implicitly:

implicit val optionFunctor = new Functor[Option] {
  def map[A, B](fa: Option[A])(f: A => B): Option[B] =
    fa match {
      case Some(x) => Some(f(x)) // desired case
      case None    => None // undesired case
    }
}

implicit def eitherFunctor[X] = new Functor[Either[X, *]] {
  def map[A, B](fa: Either[X, A])(f: A => B): Either[X, B] =
    fa match {
      case Left(x)  => Left(x) // undesired case
      case Right(a) => Right(f(a)) // desired case
    }
}

implicit val listFunctor = new Functor[List] {
  // the naive implementation is recursive, the sample repository demonstrates
  // a more robust implementation
  def map[A, B](fa: List[A])(f: A => B): List[B] =
    fa match {
      case a :: at => f(a) :: map(at)(f) // desired case, recurse
      case Nil     => Nil // undesired case
    }
}

These three instances for Option[_], Either[X, _], and List[_] show remarkable similarities, and this isn’t uncommon across functors for most data structures. Note in particular how List[_] is recursive, with the base case Nil representing void. Implementations of Functor are more complex in contexts such as IO[_] and Future[_] because they are managing side effects. What is key is that the complexities imposed by each of these contexts are completely abstracted, allowing function f: A => B to operate unburdened by effects with a focus on specific program logic.

Follow these links for the instances in the sample repository for Option, Either, List, NonEmptyList, and Id.

Can you see how functors enable control flow and short-circuiting? The undesirable cases are the specific branches of logic that enable this. If there’s “nothing here”, then they don’t do anything. In the specific case of Either[X, _], Left may be used to carry some error state in its term X. This satisfies the effect of either A for success or X for failure.

As functors, contexts and effects may be reasoned about as a box that alleviates you from having to inspect the box for whether it contains something you want in order to operate on it. You also don’t risk mishandling the box and causing an explosion by opening it when what you want doesn’t exist. The box contains the specific complexity required to obtain what you want, and with it your worries.

You can think of Right’s term as being “the right instance you want” because it’s “correct”. Right? This pun is why Either is conventionally leveraged for the effect of correct vs. incorrect or success vs. failure.

Contrasting with contexts that encode some notion of an undesired case, the Functor instance for Id[_] will always apply the function in map() as it is always the desired case:

type Id[A] = A

object Id {
  implicit val idFunctor: Functor[Id] = new Functor[Id] {
    def map[A, B](fa: Id[A])(f: A => B): F[B] =
      f(fa) // map always applies!
  }
}

To support an object-oriented API, the following map(): (A => B) => F[B] extension method may be defined:

implicit class FunctorOps[F[_], A](val fa: F[A]) extends AnyVal {
  def [B](f: A => B)(implicit F: Functor[F]): F[B] = Functor[F].map(fa)(f)
}

Using Scala functors as a general abstraction

Defining a fizzBuzz: F[Int] => F[String] function that uses a functor looks like this:

def fizzBuzz[F[_]: Functor](context: F[Int]): F[String] =
  context.map { x =>
    val isFizz = x % 3 == 0
    val isBuzz = x % 5 == 0
    (isFizz, isBuzz) match {
      case (true, true) => "fizzbuzz"
      case (true, _) => "fizz"
      case (_, true) => "buzz"
      case _ => x.toString
    }
  }

And then fizzBuzz may be used for all contexts implementing the Functor typeclass:

println(fizzBuzz(Some(3)))
// => Some("fizz")
println(fizzBuzz(None))
// => None

println(fizzBuzz(Right(5)))
// => Right("buzz")
println(fizzBuzz(Left("no fizz or buzz")))
// => Left("no fizz or buzz")

println(fizzBuzz(List(1, 2, 3, 4, 5, 15)))
// => List("1", "2", "fizz", "4", "buzz", "fizzbuzz")
println(fizzBuzz(List()))
// => List()

By using functors, the fizzBuzz function is free to focus on its specific program logic:

Produce “fizz” when x is divisible by 3
Produce “buzz” when x is divisible by 5
Produce “fizzbuzz” when x is divisible by both 3 and 5
Produce x printed as a String otherwise

At no point is fizzBuzz burdened by the effects of the context it executes against. Given a Functor instance for them, it’s usable against IO[Int] and Future[Int] as well!

Why do so many objects define `map()`?

You might be thinking that this functor pattern is superfluous, or even that the concept itself has dubious value. Lists and arrays in the wild already have a map() operation available after all, and they don’t necessarily require special knowledge in order to use them. Yet the pattern of a map() operation exists in many places: Promises in JavaScript for example have their own map() operation named then().

But think for a moment: This means you’ve been using functors for a while and never realized!

That so many functors appear in the wild is no coincidence. Functors appear anywhere stuff exists under some circumstances, and most frequently that is concretely realized in collections such as lists. The existence of Promise with its then() operation in JavaScript demonstrates that the pattern appears in a very general manner. Functors as a formal abstraction API, such as in the Functor typeclass defined above, find their strongest use in cases where the concrete type of the context is unimportant. However, the structure of functors appears in many places without being called a functor, which implies a certain universal quality of functors as a pattern.

Strikingly, this pattern manifests as a formal definition within the higher math of category theory, which gives functors a particular property: they can be mathematically proven.

In order to be a functor, a context defining a map() operation must satisfy the two functor laws:

Preservation of identity functions:

context.map(identity) == identity(context)

Preservation of function composition:

context.map(g compose f) == context.map(f).map(g)

Here are the two laws applied against Scala’s builtin List type using scalacheck properties. The List type defines its own map() operation, and these properties prove that it is also a functor:

class ListSpec extends AnyPropSpec with ScalaCheckPropertyChecks with Matchers {

  property("List.map() preserves identity functions") {
    forAll(arbitrary[List[Int]]) { fa =>
      fa.map(identity) mustBe identity(fa)
    }
  }

  property("List.map() preserves function composition") {
    forAll(for {
      fa <- arbitrary[List[Double]]
      f <- arbitrary[Double => String]
      g <- arbitrary[String => Int]
    } yield (fa, f, g)) { case (fa, f, g) =>
      fa.map(g compose f) mustBe fa.map(f).map(g)
    }
  }
}

See the functor laws applied to Scala’s Standard Library types Either, List, and Option

See the generalized functor laws applied to our Option, Either, and List types.

These laws assert that functors preserve the behavior of functions f and g as if they were composed and also applied in sequence independent of map(). Functors thus compose functional effects because this property of composition is retained within the context of their effects. The seam introduced by map() creates a hard delineation between any context’s complexity of effects and the business logic of pure functions f and g.

Functors may be universally regarded as a context of effects. Because map() is rigorously defined, the operation is ideally and provably the same regardless of specific context. This means that functors as a design pattern represent a concept that transcends codebases and languages. In contrast, design patterns as they are realized in object-oriented programming form idioms that must be relearned between codebases written even in the same language.

Building upon functors

In this post I introduced functors as an elementary abstraction which permit you to separate business logic in function f from complex operations producing instances of term A in context F[A]:

def map[A, B](fa: F[A])(f: A => B): F[B]

This abstraction is enabling on its own, as it frees the logic in function f from the burden of complexity present in the context of F[_]. However this is only an elementary abstraction as it simply provides a mechanism for composing functions in serial within the context of effects. Our goal eventually is to abstract effects in imperative code as in the custom effect model from the above runPayroll function:

def runPayroll(employeeId: Long): PayrollRunner[()] =
  for {
    employee <- employeeRepo.find(employee).getOrFail(EmployeeMissing(employeeId))
    paycheck <- payCalc.calculatePaycheck(employee).getOrFail(PaycheckMissing(employeeId))
    companyAcctNo <- getConfig("companyAccountNo")
    companyRoutingNo <- getConfig("companyRoutingNo")
    response <- achClient.depositPaycheck(companyAcctNo, companyRoutingNo, paycheck)
      .flatMap {
        case "SUCCESS" => pure(())
        case msg       => fail(DirectDepositError(employeeId, msg))
      }
  } yield response

Consider for a moment: with a functor you are able to work against a function’s desired case. You can chain any number of map() operations against a context, or map() any number of composed functions against it. Given a context in the desired case, after applying map() you will still have a context that is in the desired case. In order to write imperative code, you have to be able to force an undesired case so that subsequent operations are skipped.

Working against two or more contexts at once opens opportunities to introduce undesired cases. Take for example these two instances of the context F[_] and the function signature for combine():

val fa: F[A]
val fb: F[B]

def combine(a: A, b: B): C

How do you apply combine() to the terms A and B produced by the contexts? What happens if one of the contexts is in an undesired case? At first blush it appears that map() might work, but combine() takes two arguments. You need a specialized functor in order to apply combine()!

In my next post Enabling Control Flow in Functional Programming, we will explore how applicatives enable working within two or more contexts at the same time, as well as the many ways that you will be able to exploit this capability in your programs to express control flow.

Acknowledgements

Many of my professional network, colleagues, and friends helped me develop this material and proof read my many drafts. I would like to especially thank:

Angi Shen, my team mate at Credit Karma, for her many questions about functors and related patterns and forcing me to learn how to teach these subjects.

Ivy Huang, my team mate at Credit Karma, for providing the initial impetus to write this piece, and for helping me develop this material both in this format and for use in technical training.

Rex Fenley, my category theory study buddy, for helping me refine concepts and develop vocabulary to carry the subject.

Rodrigo Manubens, my friend and colleague at Credit Karma, for highlighting the hard parts and praising the good parts.

Mindy Or, my friend and former ThoughtWorks colleague, for providing perspectives for those new to this subject matter and forcing me to think about what points I was actually trying to make.

Aaron Erickson, my former ThoughtWorks colleague and seasoned functional programmer, for proof reading and providing criticism.

F[_] specifically is a higher-kinded type.↩︎

Highly available ssh tunnels

2022-02-09T16:02:28-08:00

Estimated reading time: 6m 42s

In my previous post, Reasons why my website is offline, I complained about systemd giving up when it fails to maintain ssh tunnels. In this post, I complain about systemd a bit more and how I gave up and stopped using it for managing my ssh tunnels.

Recall from my last post

My website is hosted from my closet computer, an old PC tower that sits in my bedroom closet
I have no static IP address
I maintain a remote ssh tunnel from a bastion server hosted on a linode to my closet computer
My host names’ DNS point to the IP address of the bastion server
The bastion server acts as a proxy into the ssh tunnels to the closet computer
The closet computer handles the traffic it receives

This is how it looks:

         +-BASTION-SERVER--------------\
         | <=proxy=> | <===remote tunnel===> +-CLOSET-COMPUTER---\
SSH ---> | *:22----> | localhost:10022 | --> | *:22 SSH          |
HTTP --> | *:80----> | localhost:10080 | --> | *:80 goto 443 duh |
HTTPS -> | *:443---> | localhost:10443 | --> | *:443 my website  |
         `-----------+-----------------+     `-------------------+

Failure, `systemd`, and duct tape

When my home internet connection is down, the remote ssh tunnels into the closet computer fail to reopen. I have been using systemd on closet computer to manage these tunnels, and systemd will eventually stop trying to open the tunnels if the network remains inaccessible. This retry-then-give-up behavior that systemd exhibits is to safeguard against situations such as the thundering herd problem and this Stackoverflow question highlights the specific levers used to configure systemd’s retry behavior. To paraphrase, systemd tries to recover within a fixed amount of tries before it gives up. Given that there is a retry limit, I can’t reasonably extend the interval between tries so that my webserver recovers from a multi-hour outage without personally intervening. Moreover, I have to be physically next to the computer to recover it when this happens.

I can’t make any guarantees about my internet connection’s uptime and that I’m not at risk of unleashing a thundering herd against my own infrastructure. As a solution, systemd therefore does not make sense for keeping my ssh tunnels tunnels open.

As a better solution, one both held together by duct tape and recommended by Vlad, I have instead leveraged a simple cronjob:

*/5 * * * * ssh closet.thisfieldwas.green true || ssh -fN bastion.thisfieldwas.green

And added the following ssh configuration to the autossh user that opens the tunnels:

Host bastion.thisfieldwas.green
    Port 22
    RemoteForward 10022 localhost:22
    RemoteForward 10080 localhost:80
    RemoteForward 10443 localhost:443

Host closet.thisfieldwas.green
    ProxyCommand ssh jump_user@bastion.thisfieldwas.green -W 127.0.0.1:10022

This simple setup works by testing every five minutes whether the ssh tunnel to closet computer is open by ssh’ing in by its external hostname via ssh closet.thisfieldwas.green true. There is an account named jump_user that is allowed to ssh into the tunnels open on the bastion server and all ssh requests to closet computer proxy through this user account. If an ssh connection cannot be made, then the remote tunnels are opened via ssh -fN bastion.thisfieldwas.green. This setup works when I reboot the closet computer, if I unplug my router, or if my internet connection drops.

I don’t have to worry about my husband Corey accidentally tripping over the power cable because I know that he will plug it back in. When the closet computer has powered back on, the tunnels open shortly after, and I’m offline for about five minutes for a small hiccup. For a self-hosted webserver, I think this is acceptable.

@TODO

A better way to recover from bad configuration

I’ve managed to disable my ssh tunnels three times now by changing host keys, the autossh user’s keys, or the jump_user’s allowed keys. I don’t have a quick solve for these instances beyond manually stepping through and fixing each ssh key error as it comes up, and I would much prefer a solution that’s no more manual than running a single command.

Better uptime, but from the closet

While using a cronjob to keep ssh tunnels open is a step towards higher availability for my website, I still have some distance to cover before my maximum downtime is one minute or less per incident. For instance, last Sunday, Feb 6, 2022 at 12pm PST, a 3.1 earthquake struck; its epicenter barely a mile from my home and it felt like a large truck had collided with the house! That earthquake could have cut my internet connection for multiple days.

I am also a bad Californian and underprepared for a worse earthquake, but that’s another matter.

I could simply host my website from the bastion server, but physically it’s located in Fremont, CA, where it could still be knocked offline by a Bay Area earthquake. And sure, linode has other locations, but simply hosting from anywhere that isn’t my closet would deprive me of a good yak shave.

As an exercise, I want to duct tape my own high availability solution together while still serving my website from my closet computer as the primary node. It is fun to tell people that my professional site is coming out of the closet, after all.

Reasons why my website is offline

2021-12-11T14:43:25-08:00

Estimated reading time: 19m 50s

My website is hosted from a 2007 HP Pavilion tower that my best friend, Vlad, a privacy buff and low-key minimalist, found next to the garbage chute at his condo building.

It appears to have been someone’s gaming rig in its prior life, as it has a decent graphics card in it and 8GB of RAM. Its abandonment was likely due to an upgrade to Windows 10, as upon boot, the login screen was so unresponsive that I could barely get the mouse to register movement and indicate for the computer to shut down again.

I previously complained to Vlad about how untrustworthy I felt cloud providers are regarding data privacy. He finds this computer by sheer accident, and that he uses Arch Linux, btw¹, so he recommends that I breathe new life into it using Arch, and to see if I can self host some of my own services.

I present my journey into self hosting: a rant, a recipe, and a retrospective.

Tinfoil hat time!

This sudden urge to self host is impulsed by the advent of GitHub Copilot. The concept of it sure spooks the hell out of me. Not the idea that my professional role as a software engineer has now become so fungible that I can be replaced by a machine, I feel that is inevitable regardless, but perhaps I feel an uncomfortable rumble of an economic ripple effect.

My revulsion is more nuanced than replacement

My use of GitHub, a long-standing freemium product, has been in the form of a paid Pro Plan. I pay GitHub for this tier so that I can choose to configure privileged access to certain repositories as their contents I am either paid to develop or they are personal projects that I am prototyping.

Despite paying for a Pro Plan, if I make those repositories publicly available, then their contents are fed into Copilot without my express consent or knowledge. These contents are then analyzed and transformed into data for which corporations will be paying untold sums of money to another, single corporation in order to manifest their technical means of production, I assume. This means that in order to gain a competitive edge, a company must use Copilot to out-produce a competitor, paying GitHub a premium in order to do so. This creates a concentration of wealth with no clear distribution mechanism returning it to lower rungs of the economic ladder.

Personally, my passion projects, my many contributions to open source, every private repository that I accidentally made public that one time… in a phrase: my labor, has been consented to subsidize widening profit margins in GitHub’s favor, and even to the favor of companies using Copilot. An articulation of late stage capitalism’s twitching proboscis has lanced into another artery from which to siphon our collective value; our dessicated tissues will remain with no further need for a paycheck.

I guess my revulsion is still about replacement

In a sense I am training my replacement. For free. Or at least in exchange for “free hosting” that I’m also partially paying for. At a less sleazy company I would be told that “Bob” is my inexpensive replacement, that for the next two months I will be training him, and that I will continue to be paid before I bounce to another company where I will become paid again. In this scenario, however, I go to another company and “Bob” is already working there, and he’s using all of my stuff from the last place.

Not all sunshine and rainbows in the cloud

At this same time Apple announces that they are scanning everyone’s photos to make sure they don’t contain CSAM. Don’t worry, they aren’t actually looking to see if the photo contains CSAM. Until they aren’t, because someone has to verify a positive match manually. They pinky-promise they will only ever look for CSAM, even though they could look for anything they wanted, or are told to look for.

I pay Apple to store my photos in the cloud. I thought they were encrypted, right? So now they’re scanning them on my device because they can’t look at them on the cloud, because they’re encrypted. wat.

Bringing the cloud home

I am very motivated to get as much of my stuff out of third party services as I can, because–to generalize based on GitHub and Apple–it’s likely that no third party can be trusted with my data even if they are already making money off of it. I resolve to divest Apple and Google of my cloud data, GitHub of my source code, and linode of my miscellaneous.

I talk to Vlad about my feelings regarding GitHub Copilot, Apple, and cloud providers in general. He offers to help me find a new option for hosting my data. Just a few days later he finds that computer next to the garbage chute, and with much excitement he calls me and cheerfully announces, “It turns on!”

What luck!

Keeping my data in the closet

The computer sits in many pieces in my living room as I remove dust and await several shipments of spinning platter drives. Installing the array of drives to hold my media proves difficult: HP did not make this case amenable to changes or even provide much in the way of expansion bays. As a consequence, I now have a bunch of drives that are free-floating within the case. No matter though, the drives are each 4TB, and so heavy that they are held in place by the friction induced by their own mass and density alone. Later, for a short period, the computer carries the hostname gravwell because of the density these drives give it.

I craft a script so that I can reproducibly install Arch Linux with disk encryption over ssh, comfortably iterating from my 2019 MacBook Pro until the installation looks correct. I nuke the computer and try again, just to be sure. Then several times more.

Once the OS is installed to satisfaction, I carefully arrange the drives within the case, making sure the case can tolerate some degrees of tilt in any direction without the drives sliding out of place. To my surprise, the computer can be rotated a full 90° in any direction, and the drives don’t move at all. I place the computer inside of the bedroom closet, where I can plug it directly into the router. The computer is now closet computer.

I spend some number of days iterating on closet computer with an ~~Ansible playbook~~ to get the configuration just right. As long as the data array remains intact, I can incrementally add new disks to it by appending the disks to the playbook configuration. If the data array is lost, then I can nuke it and build anew just by running the playbook. (To be clear, I lose data if this happens.)

My goal with this playbook is primarily to retain a living snapshot of closet computer’s configuration. A secondary goal is to be able to provision a replacement system if the computer should ever cease to work. In my head I imagine the happy path for such an event looks like this:

A few keystrokes.
I then grab a coffee while the playbook runs.
When the playbook is complete, I can power off the computer.
I plug the drives into it.
I turn on the new closet computer. Huzzah.

Configuration management is Ansible’s primary domain, it does this very well, and I now use it for my current three Linux-based systems.

I both nuke and enjoy rebuilding my Arch Linux-powered thonkpad on occasion. All changes I make to it are managed through its associated playbook. I use Arch Linux, btw.

Hosting from inside the closet

closet computer hosts a small set of services:

My website is served by nginx, which also terminates ssl.
My source code is hosted by gitea with postgres to support its data.
My data array provides 16TB usable storage:
- It’s just a bunch of disks with redundancy provided by snapraid.
- It’s treated as a single disk using mergerfs.
- It’s very nice to use, and I think a choice recommendation by Vlad.
pi-hole… I thought this would mean less ads, but it hasn’t really helped much.

I’m using docker containers for gitea and postgres. nginx runs natively and provides routing into gitea or static html for its configured domains.

I mentioned above that this computer sits in the closet. This location comes with some problems, however. For example, the closet does not have a dedicated power outlet: I have to run an extension cable from an outlet in the middle of the room and out from under the bed.

Reasons why my website is offline

I can’t express my website’s uptime in a magnitude of nines² the same way that a public safety and emergency service would³. Having “one nines” uptime is really bad, and I would like to find a more impressive metric to present to someone than 9%.

Now that I think about it, uptime may not be the right word to use. Let me instead paint a picture with a short and incomplete list of reasons for which my website is offline:

My ISP had an outage.
The power was out.
I closed the sliding closet door too quickly, and it unplugged the router.
My husband pushed a box of shoes under the bed, and it pushed the extension cable out of its plug.
Instead of unplugging the router itself to turn it off and then on again, I flipped the switch on the power strip because I thought it would be less effort. I did not fully grasp that this turns off everything in the closet. Consequently, I forgot to turn closet computer back on.
I can’t get into the bios to configure closet computer to turn back on when the power returns.
closet computer came up before the router and autossh stopped trying to open the ssh tunnels to the bastion server after a period of time, because systemd is designed to give up and stop trying.
My router refuses to route any traffic whatsoever until I point DNS away from the pi-hole on closet computer.
Hubris.

A hosting solution held together by glue and popsicle sticks

I don’t have a static IP. My ISP’s network infrastructure simply doesn’t allow for it, as the infrastructure is based on a mesh of wireless signal that is propagated between large, unsightly antennas installed on top of each house using their service. To get around this, I swallow my pride and purchase, again, a cloud server from linode. I call this one my bastion server, and with it I have this sort of static IP address that I can configure my domains to point to.

As I still can’t reach from the outside world into my closet, I have to perform a little magic…

Using ssh, I can open a remote tunnel from closet computer to the bastion server with this command:

ssh -R 10080:localhost:80 bastion.oflogan.xyz

This command binds a connection to the bastion server’s localhost:10080 port and forwards it to closet computer’s *:80 port. You may notice that I’m using port 10080 rather than 80, and this is because I am using a non-priveleged user to open the tunnel, as only root may bind to ports under 1024.

Because the remote tunnel binds localhost:10080 on the bastion server, this means that the tunnel is not accessible publicly, and that my website is still in the closet. In order for my website to come out of the closet, I use an nginx reverse proxy to forward the bastion server’s public port *:80 to localhost:10080. With this final connection made, the world can reach out and see my website coming out of the closet.

I can set up reverse proxies for each port, 22, 80, and 443, on the bastion server into reverse tunnels from closet computer’s 22, 80, and 443 ports to the bastion server’s 10022, 10080, and 10443 ports. I then leverage autossh to maintain persistent remote tunnels from closet computer to the bastion server.

This is how it looks:

         +-BASTION-SERVER--------------\
         | <=proxy=> | <===remote tunnel===> +-CLOSET-COMPUTER---\
SSH ---> | *:22----> | localhost:10022 | --> | *:22 SSH          |
HTTP --> | *:80----> | localhost:10080 | --> | *:80 goto 443 duh |
HTTPS -> | *:443---> | localhost:10443 | --> | *:443 my website  |
         `-----------+-----------------+     `-------------------+

It’s ok that I’m using a cloud server for this, I tell myself, because none of my data is stored on it. But also, I think I pulled off something kinda cool.

This is a really fragile architecture, however. If the ssh tunnels aren’t up, the site can’t be served. I can’t even get an error page out as there doesn’t seem to be a way to query for whether the reverse proxies are passing any data through. This setup is really kinda crap like that.

Being what it is, systemd on closet computer will stop autossh from retrying connections to the bastion server after a certain period. This is a major problem if there’s a hiccup in the home internet connection. Sometimes my internet is out for five minutes, but that’s a tradeoff made when using a cheaper, locally-based ISP over Comcast’s cartoon villain bundled contracts. At a minimum, I need to use something other than systemd to manage my ssh tunnels, as it hasn’t been reliable at all and is probably one of the bigger points of failure because it cascades with any failures at the router.

Physically, the closet receives too much human traffic for closet computer or really any hardware to be held there. We live in a modest century Craftsman whose modern infrastructure has been monkey-patched together, and I piggybacked on the convenient, pre-existing connections made by the previous homeowners, putting no thought into potential consequences therein. Now we have a closet acting as a server cabinet instead of a closet. We are, of course, using it as both, with frequent access throughout the day, and cables running everywhere, as if a panicked octopus were dropped on the floor and frozen in place.

Duct tape and rainbows

I have a lot of problems hosting my website and source code from my closet computer in terms of uptime. Mostly related to tripping over the power cable, though in other ways the setup isn’t resilient. It’s been a really rewarding experience nonetheless. I’m working with tech that I use all the time as a consumer, but have never myself had to engineer or maintain. It’s a fun puzzle.

Hosting my website and source code myself feels very empowering. I feel as if I reclaimed a little piece of the internet. For that alone I want to see what more I can self host. The internet today clusters around a centralizing oligopoly of services, and I pine for the days when every odd person had their own small, weird website, and frequented a handful of small, niche forums, each with just enough users to be interesting. Now there’s Facebook. GitHub. Google… If I name a thing: there’s likely a single site that I would go to for that thing. That thing has likely regressed in quality towards the mean and mediocre. And when I go to that site for that thing, I can’t tell whether I’ve found the thing or an ad for a subscription service that gives me access to the thing at a markup.

closet computer in context. Corey asked me to dust first." alt="Cables and all, the closet computer in context. Corey asked me to dust first.">

Cables and all, the closet computer in context. Corey asked me to dust first.

I severely took for granted the uptime that a third party service could give me. It’s making me rethink what the total cost of what third party hosting actually looks like, not only in terms of dollars and nebulous ideals, but more importantly in terms of peace of mind. Terms such as will the promotion committee be able to see that writeup I referred them to? I would absolutely benefit from hosting my website from my bastion server instead, but perhaps after I reread linode’s service agreements to allay my confounding mind.

Regarding nebulous ideals, Copilot makes a very strong case for treating data as a form of labor. At the time of this writing, Wed Dec 8, 2021, Copilot is in technical review and accessible via waitlist. I can’t find any mention of licensing cost, or whether there will be a distinction between paid and community editions. If Copilot is available as an exclusively free service, then I believe it stands as an excellent reinvestment back into the community and profession as a whole. If someone is paying for Copilot, then we are collectively exploited within some legally-accepted gray area, because we agreed to be.⁴

Stepping back

I still use GitHub professionally, and for open source projects hosted there. I realize that being a principled curmudgeon sometimes means that I’m only that. Obviously there also exists a sort of cognitive dissonance in regards to contributing to open source works in the context of labor. Regardless, I gave myself a learning opportunity, and I consequently don’t use GitHub as a personal consumer now. I suppose that was my goal after all.

I might give Copilot a go soon. I’m reading some good things about how it cuts through boilerplate like a hot knife through butter. Reviews such as this help me conceptualize how it might fit into my own workflow, and that its role is more augmentation than replacement. Especially now that I’ve had time to step back. I would be over the moon to have an intelligent assist for the boring parts of my job. I suppose that was GitHub’s goal, too.

On the lighter side, being professionally out and telling people that my website is coming out of the closet gives me a certain subversive glee that I never realized I needed. It’s a great lead into how the computer got there, too.

~~For now, I host my source code at bitsof.thisfieldwas.green. My website source repository is directly accessible from keywordsalad/thisfieldwas.green.~~

Meme reference: btw i use arch – When Vlad proofread this post, he informed me that it was customary to let the reader know that one uses Arch, btw.↩︎
AWS Public Sector Blog: Achieving “five nines” in the cloud for justice and public safety ↩︎
Amazon Web Services experienced a significant outage on Tue Dec 7, 2021, and it lasted some seven hours and thirty minutes, taking out large swaths of services and even preventing people from doing their jobs. This means that for 2021, AWS’s US-East-1 region had an uptime of no better than 99.9%, or “three nines”. I wish for my home-spun home hosting solution to one day perform at least as well as AWS does on their bad days.↩︎
There exists an episode of South Park where Kyle accepts the Apple iTunes user agreement without reading it. It’s worth a watch but the writers employ a heavy-handed storytelling device to get their point across. I liked it, but you may not.↩︎

Redoing my website with Haskell and Hakyll

2021-12-05T08:08:32-08:00

Estimated reading time: 19m 56s

I’m picking up a COVID project: Redoing my website!

My website prior to 2021 was authored using octopress or jekyll. Truthfully I don’t remember which as I haven’t touched my website in quite a few years. I want to update my homepage and I’m in a bind because I don’t know how to produce a build after I make changes.

Sitting down, I try to size up the amount of housekeeping present here. I can’t quite tell what’s going on with jekyll, and I know that octopress hasn’t been updated for a while. Additionally, I’m several computers removed in time since I last actively worked on my site, so I don’t have the old software handy. This work is starting to look like quite the yak shave to get some software running that I would soon have to update. I don’t mind a good yak shave, however, so I’m going to find a new static site generator.

Wants in a static site generator

Support for arbitrary static pages written in either markdown or html.
sass support.
Blog posts in markdown.
Listing most recent posts.
Listing all posts.
Listing posts by category or tag.
Some mechanism for authoring drafts and viewing them online.

Really any static site generator can meet these, but I do enjoy a good yak shave. hakyll in this regard seems like a good fit.

Why `hakyll`, specifically?

Briefly, let me introduce hakyll as I understand it: hakyll is a jekyll-inspired static site generator that provides just enough tools to put together a website. Not to knock it, of course, in reality hakyll is very lean but provides plenty of building blocks to deliver a no-frills blog.

Naturally the primary value proposition of hakyll for me is the opportunity to use Haskell for a project that isn’t entirely trivial. An enjoyable and good yak shave, if you will!

How to shave a yak for `hakyll`

If I want a sweater that’s made using a nice wool, specific stitch, and tailored to a particular fit and style, then I’m probably going to knit the sweater myself. The sweater may require that I learn about or perform any number of new things. Like how to shave a yak for wool so that I can spin the nice wool for the yarn that I will need to knit the sweater how I would like. I like to call this manual resolution of recursive dependencies a yak shave when they require me to create primitive components myself.

Obviously I could purchase wool yarn. Perhaps instead, in addition to knitting a sweater I hope to learn how to shave a yak in the process. I believe the experinece would be valuable, and I would have a sweater made to my own specifications by my own hands.

I would like to call attention to a related and well-known cognitive bias known as the IKEA Effect. Per its name, this bias manifests in consumers of flat pack furniture: consumers will place an artificially higher value on furniture requiring manual assembly than on furniture of similar materials and build quality that is already assembled. Consumers thus will not only spend more money on flat pack furniture, they will also feel personally accomplished when they have successfully assembled it.

Key requirements for a good yak shave are learning and value not realized elsewhere. Inflated value therefore may be objective in the case of a yak shave, in contrast with the IKEA Effect. I personally have shaved many yaks, as unlike assembling IKEA furniture, I enjoy shaving yaks. I especially cherish my hairier yaks and hold on to them as tokens of accomplishment.

My use of yak shave above may imply that I delight in over-building solutions, but I assure that this is not the case. There also exists the inverse of the good yak shave, and for that I employ the phrase this is turning into a yak shave when I have found myself shaving a yak when what I really wanted was to buy a sweater. This phrase gets some use with tech that should “just work”, like desktop Linux™, and I get particularly good mileage out of it at the office. Naturally, as yaks are big hairy beasts predisposed to high maintenance, sweaters are promptly substituted as I find them.

As a rule: I never shave a yak when the sweater is flatly the better, available option. The same could be said of yaks themselves as I could be reworking my website with something like COBOL ON COGS rather than hakyll. What looks like a good yak shave also risks simply becoming an endless shave with each stroke of the buzzers revealing yet another yak buried beneath the wool. There’s no shame in going back for a sweater instead, though. Yaks are big, hairy beasts after all.

A good yak shave gives me something abstract and something concrete. I love to learn a new skill or technology and have a usable artifact to show for it, especially when the ongoing shave to maintain it isn’t too much overhead. I hope with hakyll that using Haskell will force me to learn some new programming techniques and tricks, and that I will relearn how to work with the simpler side of modern frontend web development. I don’t know yet how much yak this will require me to shave, but not knowing is half the fun!

How `hakyll` accomplishes what I need

Inputs and outputs

hakyll roughly models a website as a series of transformations between inputs and outputs using the Rules monad.

Rules are initialized by matching file paths by name or pattern as inputs.
Per input, a Route indicates an output destination.
When routed, a transformation is applied by means of the Compiler monad.

Rules for copying images into my site

images :: Rules ()
images =
  match "images/**" do        -- 1. match all files in the images folder
    route idRoute             -- 2. output image with the file name as-is
    compile copyFileCompiler  -- 3. copy image file from source to output as-is

Turning one thing into another

Compilers are cool, they’re where all the magic happens! Through compilers I use hakyll’s magic to transform markdown and html into html for viewing in a browser. hakyll out-of-the-box supports multiple input and output formats, and I suspect that in addition to a plain-old technical blog I can also write a really large textbook source in LaTeX and output a pdf.

I sense that compilers are hakyll’s primary extension point, as I lean on these heavily and develop opinions as I spend time with them.

hakyll very smartly tracks dependencies between each compiled artifact and, like a good modern software, only compiles artifacts when it detects changes. For example, this allows for precompiling templates early in the build pipeline and allowing multiple dependents downstream use the same built artifact.

Just touching lightly

I won’t be performing a deep dive into how hakyll works in this post or providing a tutorial, though I will expand on my experience later. The remainder of my post stands as a postmortem and summary of my experience transitioning to using hakyll as my site generator.

So what happened?

My website now uses for its static site generator: Haskell and hakyll!

I kept a small scratch list of things I knew would happen but didn’t expect:

Haskell compiles really slow, like really slow.
Compiled languages are supposed to be more productive but they don’t feel that way.
Just using plain-old html is a tall order, aka. Transformation Telephone.
Templating is inexplicably difficult.
Type inferencing is a real treat until it doesn’t work.
Using css is still like the Family Guy meme.
I did not fight functional programming like I expected, I rather experienced it as an invaluable teacher.

Let me expand on these items.

Haskell compiles really slow, like really slow

hakyll depends on the pandoc library for its document processing. It’s very large, and for my particular use case provides much more functionality than my website will ever need. I’m leveraging code highlighting, html, and markdown processing but pandoc (which can be thought of as a pan-document swiss army knife markup processing utility of sorts) can process pretty much everything under the sun and transform it all this-way and that-. From source it takes around 40 minutes to clean-compile on my 2019 MacBook Pro.

Compiled languages are supposed to be more productive but they don’t feel that way

With compiled dependencies, my site takes about two minutes to clean-compile just the executable. That’s at about the upper limit of what I would consider acceptable. Changing a widely depended-on source file and then waiting two minutes for the binary to rebuild is a real grind.

I never have to worry about a type error at runtime, but I do miss the quick turnaround I had with my Ruby-based site generator. I can’t say whether I was faster or slower in fact, as I wasn’t dealing with a whole class of problems related to typos alone, but it sure doesn’t feel like I’m more productive.

UPDATE: Since working more in content vs. code, hakyll has proven itself very snappy and resilient to errors. I hardly have to think about it as I make edits and experiment with cosmetic touches. I’m still on the fence about productivity in code, but I’m not unhappy.

Just using plain-old `html` is a tall order, aka. Transformation Telephone

pandoc loses some data fidelity as it transforms markup between formats. I try to use html5 semantic tags (<aside>, <figure>, and friends) as I update my existing posts and find out that pandoc processes the html in the markdown source and removes the tags but passes the contents through mostly unchanged. Digging through pandoc’s source code I discover that there is an intermediate representation for all documents that does not itself have the fidelity necessary to retain whether the contents of those tags represent their intended semantics. I don’t feel that I can affect a pull request to add this capability without a very large amount of work whose value serves more than my own ends.

Templating is inexplicably difficult

hakyll does not have a good template language. I spend a few months on and off as I can squeeze time out after work so that I can write my own template language. This language provides for me the following capabilities that aren’t present by default or just aren’t very good out of the box:

Better contexts to retrieve data from compiled items in scope.
Dotted field accessors along contexts.
Custom template function declarations.
Easy layouts driven through both metadata and special syntax.
Conditionals.
Loops.
Default values when absent.
Context literals.
List literals.
Syntax literals (i.e. macros and lazy evaluation).
Falsy and Truthy values, even undefined values!

All of these features give me a rather roundabout way of satisfying some of the conveniences of a dynamic language within the very stiff context of a Haskell-powered static site generator. Strapping together my own template language is a surprisingly fun, though difficult itch to scratch, and a very informative experience worth writing about in another post.

Type inferencing is a real treat until it doesn’t work

Sometimes Haskell can’t figure out what I’m trying to tell it to do. I didn’t read any formal documentation on how Haskell’s type inferencing algorithm works specifically (the system derives from Hindley-Milner, interesting stuff) but I did develop some intuition over time. Unfortunately words escape me and I can’t seem to explain how it works just yet.

Occasionally I have to tell Haskell what to do, and there is a syntax for this:

-- Haskell doesn't know that I mean "page" is a `String` and not `[a]`
constField "layout" ("page" :: String)

That particular instance above is due to a footgun I use so that lists of type [a] are treated generally and Strings, which are just [Char], are treated specially. Haskell can’t tell the difference in this case unless I provide the extra type information.

Using `css` is still like the Family Guy meme

css: it's still like this, but it got prettier" alt="css: it's still like this, but it got prettier">

css: it’s still like this, but it got prettier

I probably haven’t touched css since 2013, which right now is about 8 years ago. It feels the same, with a few upgrades that are in very much the same vein as the old stuff. It’s still weird and clunky, at least to me. For as frustrated as I find myself with it, I am happy with how the site looks considering that I don’t particularly know what I’m doing.

Despite feeling as though I’m fumbling about, I’m so proud to have figured out how to style my site so that looks great on both desktop and mobile, that it scales across multiple resolutions, but that it especially feels like I did not compromise on aesthetics or functionality. Try looking at this or other pages from desktop or phone, and try especially comparing landscape vs. portrait!

I am thrilled that all major browsers’ dev tools have greatly matured, have even become on-par with each other. I’m very happy with how enabling these tools are now.

I did not fight functional programming like I expected, I rather experienced it as an invaluable teacher

I don’t get what the negative fuss about functional programming is. Everything I’ve needed help with is googleable, like everything else in tech. I rather enjoy using Haskell for what is a pretty boring problem solvable by any programming language, and on the whole I’m happy so far with my decision to use it. Learning functional programming benefits from a palate cleanser in order to fully grok, and I’m giving it an earnest shot.

I’ve picked up some concepts that I imagine will find their way back into how I write code elsewhere:

Abstract data types

Abstract data types are the best way of implementing closed polymorphism that I’ve found thus far, and enums in object oriented languages are close enough to simulate them effectively in those languages.

In my experience, polymorphism dealing with data only needs to extend over a small number of cases. By using abstract data types, I can keep my code variants local to each other, so that adding new variants requires that changes be made in-context. This ensures that no case for a polymorphic change in behavior goes unchecked. I lean on the compiler to tell me if I left any location unhandled.

Type classes

Type classes are the best way of implementing open polymorphism that I’ve found thus far, and interfaces in object oriented languages are close enough to simulate them effectively in those languages.

Contrasting with abstract data types, interfaces provide a facility for extending polymorphism over an open set of cases. As interfaces speak only in terms of operations, they run a much lower risk of introducing variant code paths that could introduce defects related to data.

Rethinking conventional object-oriented programming

I did not find myself reaching into my toolbox for inheritance or class-based polymorphism to solve problems. These two mechanisms are a brittle way of implementing polymorphism, I feel. I wager I could write decent Java without them, as I have been doing this for some time already in other languages.

Changing the world

Immutability makes reasoning about data flow and change of state across an application really easy and I will leverage this everywhere I that I reasonably can.

I find my mental model of objects in object-oriented programming changes starkly when I apply immutability to data. As conventionally understood, I imagine a small container of data, an object, which I can modify at-will, via a provided interface of methods, in order to change its state over time. When I apply immutability as a design constraint, the concept of an object as realized here is no longer a possible option.

Immutability imposes a fundamental change in how business and technical capabilities within code are designed. Instead of objects as methods around mutable data, methods become organized into modules containing functions. Some functions act as entry points into the associated capability, and they receive, usually, an abstract data type as an argument, and construct a new one as a result. Change over time can be observed without modifying state destructively, and entire classes of runtime errors, like data races, are eliminated.

There are more impacts to the design of an application than I suggest here. I admit any value proposition may appear dubious on its face. Consider for a moment: state within long-running Haskell applications appears as if it were mutating. Designing for immutability thus works in production applications and I am very interested in learning more about the patterns used to support it.

Pattern matching

Pattern matching and branching conditionally based on the shape of arguments is astoundingly empowering. It elevates data into a first-class position within API design.

Until recently, I can’t think of any C syntax-inspired language that takes advantage of this concept. Pattern matching may be so enabling, I think, that it might even have a measurable impact on the economy! I wish this was something that could be measured more concretely, but I’m happy even if it’s just a convenience to have.

Encoding the effect in result

This is perhaps my most unexpected takeaway: Effect types make very clear that as part of the contract of a function that I am calling, some orthogonal quality in addition to a result, including whether or not I receive a result, occurs.

f :: IO a will give me an a if f succeeds. It will also affect external systems in some way, or it might fail due to one. Java engineers might recall method signatures polluted with throws IOException; this is same/same but different.
g :: [a] will give me one or more of a if g succeeds. Effectively, g fails if I receive zero a’s. This is a sort of non-zero nondeterminism of the number of results an operation.
h :: Maybe a will give me Just a if an a exists and it succeeds. It gives me Nothing otherwise and fails. This is encodes an explicit presence or absence of a value. I like to think of Maybe as a null done right.
i :: Either a b will give me Right a if i succeeds. I receive Left b if i fails, which allows me to inspect b for why i may have failed.

An observation: each of these effect types encode some notion of success and failure. The Monad typeclass generalizes over these and this enables a high degree of code reuse, independent of specific error handling. Error handling can be pushed out to the edge of the contexts they need to be contained within, which helps keep business logic cleaner. Each of the above effect types implement the Monad typeclass.

I rather like `hakyll` and Haskell!

I’m really happy with hakyll! I’m also really happy with Haskell. My website is taking some investment, but it’s been worth the effort especially given the quality of the output. I think I’m going to keep this setup for a while yet.

Promoting changes with App-Config-App

2012-11-28T13:04:00-06:00

Estimated reading time: 6m 14s

The App-Config-App now lets you promote changes between environments!

How does it work?

Perforce lets you create mappings to define the relationship between two diverging code branches. This allows for easy integration of changes between the two branches by referencing the name of the mapping.

See Perforce’s documentation for more details on the how and why of branch mappings.

The App-Config-App reads these branch mappings in order to create paths for promotion between environments.

Promoting changes with App-Config-App

The App-Config-App setup_example.rb creates four branches with the following mappings:

Mapping        Source    Destination
------------------------------------
dev-qa         dev       qa
qa-staging     qa        staging
staging-prod   staging   prod

If you login to App-Config-App and go to “Promote Changes,” you get an interface showing these relationships:

Changes between environments can be promoted in either direction along a mapping configuration. The receiving environment accepts all changes (developers would know this as an ‘accept-theirs’ resolution) and you are then allowed to review the changes by clicking on the “Pending Changes” link.

For example, I’ve promoted changes from “qa” to “dev”:

I can then review the changes by clicking on “Pending Changes”:

Changes may be edited or reverted before committing them.

Promoting changes using P4V

P4V is the Perforce visual client. Using P4V, you have much greater control over how changes get promoted, but it requires a little more work.

I’ve connected P4V to my App-Config-App user workspace to perform the same promotion from “qa” to “dev”:

Select the “qa” folder, then from the menu bar go to “Actions” > “Merge/Integrate”. This will bring up a wizard for performing the integration.

Select the following:

Merge method: "Use branch mapping"
Branch mapping: "dev-qa"
Automatically resolve files after merging: checked
Resolve option: "Accept source"

And ensure the direction of integration is “Target” < “Source”:

Finally, click “Merge”. If you expand the “dev” folder, you can see the where the changes are:

You are now free to modify the files further before finally committing the changes.

How it compares

You get greater options when using P4V to promote changes, but producing the same result as App-Config-App’s default behavior is fairly involved. If you aren’t paying attention or don’t know what you’re doing, you might break something :(

@TODO

More options for resolving changes

When you promote changes in App-Config-App, the source changes will overwrite the destination. This behavior reduces the chance for a conflict to happen, but it means you really have to pay attention to what’s changed in the destination config and possibly edit the config further before finally committing it.

Conflict resolution

If a conflict occurs after promoting changes, a screen should be available for viewing and editing the conflicting changes.

Better error reporting if promotion fails due to permissions

Users with read-only access to multiple environments will still be able to promote changes between them. The promotion doesn’t actually occur (the files remain unchanged) but the application doesn’t report any errors when this happens.

App-Config-App in Action

2012-11-20T17:00:00-06:00

Estimated reading time: 1m 5s

Paul Hammant found this cool Server-Side Piano and I’ve modified it to be configurable from a running App-Config-App. Because the sound is generated at the server, you’re able to see (hear) the Server-Side Piano change its configuration without reloading the UI.

Making it work for yourself

I’ve updated the App-Config-App with additional configuration to support choosing which instrument the Server-Side Piano will play. A clean install of App-Config-App using setup_examples.rb will provide everything needed to run the Server-Side Piano.

The application’s configuration URL and credentials are located in web.xml. Additional details may be found in the application’s README.

SCM-Backed Application Configuration with Perforce

2012-11-16T17:00:00-06:00

Estimated reading time: 8m 31s

Continuing from my last post, I’ve forked Paul Hammant’s original App-Config-App and modified it to work against Perforce. I’ve decided not to continue using Perforce Chronicle as it is primarily intended for content management.

With this version, App-Config-App is written in Ruby, mostly using Sinatra, a lightweight web application framework. I’m still using AngularJS, but I’ve also added a few other things:

A .rvmrc file, so you automagically switch to Ruby 1.9.3
A Gemfile, so you don’t have to install everything individually :)
Sinatra-Contrib for view templating support
Rack Flash for flash messages
HighLine for masking passwords
json to manipulate JSON in native Ruby

Getting it to work.

App-Config-App requires a couple things to work:

Ruby 1.9.3 and Bundler
p4 - the Perforce command line client
p4d - the Perforce server

All installation and example setup details may be found in App-Config-App’s README.

Using App-Config-App

When you login, you should see this screen:

You’ll notice I made the extra effort to add colors and drop shadows :D The application works from the project root in Perforce, so the files in each branch are viewable here. Clicking on “Dev” > “aardvark_configuration.html” will bring up a form for editing aardvark_configuration.json as in the previous version:

Changes to the form data are automatically saved. After making a view edits, you can click “View Diff” to get the diffs or “Revert” your changes. Go ahead and change the email address and fiddle around with the banned nicks, then go click “Pending Changes”:

This screen shows all files that were changed and their diffs as well. You can “Revert” each file individually, and if you want to commit all changes, then enter a commit message and click “Commit Changes”. If you commit the changes and go back to “Dev” > “aardvark_configuration.html”, you’ll see the new values in the form:

Security and Permissions

Permissions and security are managed through Perforce. For users to be able to login, they must have a user and client configured in Perforce. Those users must also have permissions configured in order to view or modify files.

The setup_example.rb script creates three test users to demonstrate branch permissions:

Username        Password   Write     Read
-------------------------------------------------
sally-runtime   bananas    prod      staging, dev
jimmy-qa        apples     staging   dev
joe-developer   oranges    dev

Logging in as any of these users will hide branches that don’t have at least read-level access, and branches that don’t have write-level access won’t allow changes.

All users created by setup_example.rb are intended only as examples. In the real world, all application users should be setup with real logins and real permissions.

It is this support for users and per-branch permissions that I am using Perforce as the SCM backend rather than Git.

Application Users

The setup_example.rb script also sets up three application users to demonstrate how an application would consume configuration:

Username   Password   Read
-----------------------------
dev-app    s3cret1    dev
qa-app     s3cret2    staging
prod-app   s3cret3    prod

In theory, an application would periodically poll aardvark_configuration.md5 until the hash value changed, then load aardvark_configuration.json and reconfigure itself.

Application user accounts are configured in Perforce like any other user. I highly recommend that application users be given ready-only access to individual files rather than entire branches.

Divergence

Right now, App-Config-App offers no UI tools for managing divergence and merging. Merges must be performed outside App-Config-App, and the specific safety nets to prevent nefarious change vulnerabilities are dependent on your branch specs and permissions configuration.

There are also are no tools to manage conflicts of existing edits with incoming changes from another user. If a Perforce sync fails due to a conflict, you are best to revert all changes and enter them again.

@TODO

A better model for autosave

Autosave in AngularJS isn’t very good. AngularJS doesn’t integrate with DOM events the way idiomatic JavaScript does, or provide a reasonable abstraction the way Dojo Toolkit or JQuery do. Right now, autosave in App-Config-App triggers with every key press in the config forms, and pummels the back-end server with ajax posts.

I’ve also noticed that the autosave triggers even when a value is invalid. The first time an email address, for example, becomes invalid, AngularJS will post back the JSON, but without the invalid email address field–the invalid field is entirely left out of the JSON structure. After that, AngularJS will stop autosaving until the value is valid. There are also no measures in place to prevent a user from leaving an invalid value and saving an incomplete JSON file.

A better model for validation

AngularJS does not offer a good validation API. The validation API is quite opaque and I haven’t found any real examples using it. The built-in form validation is inadequate. There are few ng-* HTML attributes exposing more than basic configuration parameters, and no hooks offered as extension points.

For example, I’m using regular expressions for date validation in App-Config-App. There isn’t a hook to provide custom validation checks, and regular expressions don’t perform sanity checks. Values such as “00/00/0000” will pass validation.

More example clients than the Java one needed

The App-Config-Java client is enough to show the basic idea behind caching and reloading configuration from App-Config-App. I would like to create a few more examples in a couple different platforms, possibly also showcasing “hot reconfiguration” for feature toggles.

Someone should port this to Subversion or TFS

App-Config-App should be usable by the largest possible audience. For instance, if you’re using Subversion, then you should be able to take advantage of the existing infrastructure.

The reason I point out Subversion and TFS is largely due to support of per-branch permissions.

Using Perforce Chronicle for application configuration

2012-11-07T13:54:00-06:00

Estimated reading time: 43m 28s

Following Paul Hammant’s post App-config workflow using SCM and subsequent proof of concept backed by Git, I will show that an app-config application backed by Perforce is possible using Perforce Chronicle.

Perforce and permissions for branches

Perforce is an enterprise-class source control management (SCM) system, remarkably similar to Subversion (Subversion was inspired by Perforce :) Perforce is more bulletproof than Subversion in many ways and it’s generally faster. Git does not impose any security constraints or permissions on branches, Perforce gives comprehensive security options allowing you to control access to different branches: for example, development, staging, and production. Subversion, however, can support permissions on branches with some extra configuration (Apache plus mod_dav_svn/mod_dav_authz). For these reasons, Perforce is a better option for storing configuration data than either Git or Subversion.

Perforce CMS as an application server

Perforce Chronicle is a content management system (CMS) using Perforce as the back-end store for configuration and content. The app-config application is built on top of Chronicle because Perforce does not offer a web view into the depot the way Subversion can through Apache. Branching and maintaining divergence between environments can be managed through the user interface, and Chronicle provides user authentication and management, so access between different configuration files can be restricted appropriately. The INSTALL.txt file that is distributed with Chronicle helps with an easy install, mine being set up to run locally from http://localhost.

There is a key issue in using Chronicle, however. The system is designed for the management of content and not necessarily arbitrary files. In order to make the app-config application work, I had to add a custom content type and write a module. Configuration and HTML are both plain-text content, so I created a ” Plain Text” content type with the fields title and content:

Go to “Manage” > “Content Types”
Click “Add Content Type”
Enter the following information:

Id:       plaintext
Label:    Plain Text
Group:    Assets
Elements:

[title]
type = text
options.label = Title
options.required = true
display.tagName = h1
display.filters.0 = HtmlSpecialChars

[content]
type = textarea
options.label = "Content"
options.required = true
display.tagName = pre
display.filters.0 = HtmlSpecialChars

Click “Save”.

The Config App

I’ve borrowed heavily from Paul’s app-config HTML page, which uses AngularJS to manage the UI and interaction with the server. Where Paul’s app-config app used the jshon command to encode and decode JSON, Zend Framework has a utility class for encoding, decoding, and pretty-printing JSON, and Chronicle also ships with the simplediff utility for performing diffs with PHP.

The source JSON configuration is the same, albeit sorted:

{
  "bannedNicks": [
    "derek",
    "dino",
    "ffff",
    "jjjj",
    "werwer"
  ],
  "defaultErrorReciever": "piglet@thoughtworks.com",
  "lighton": true,
  "loadMaxPercent": "88",
  "nextShutdownDate": "8\/9\/2012"
}

The index.html page has been modified from the original to support only the basic commit and diffs functionality:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html lang="en" xmlns:ng="http://angularjs.org">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
  <title>Configuration application (alpha)</title>
  <script type="text/javascript" ng:autobind src="http://code.angularjs.org/0.9.19/angular-0.9.19.min.js"></script>
  <style type="text/css">
    ins { color: #00CC00; text-decoration: none; }
    del { color: #CC0000; text-decoration: none; }
  </style>
</head>
<body ng:controller="AppCfg">
<script type="text/javascript">
    function AppCfg($resource, $xhr) {
        var self = this;
        this.newNickname = "";
        this.svrMessage;
        this.message;
        this.cfg = $resource("/appconfig/stack_configuration.json").get({});

        this.save = function() {
            self.cfg.$save({message: self.message}, function() {
                alert("Config saved to server");
            }, function() {
                alert("ERROR on save");
            });
            self.message = "";
        };

        this.newNick = function() {
            self.cfg.bannedNicks.push(self.newNickname);
            self.newNickname = "";
        };

        this.diffs = function() {
            $xhr("post", "/appconfig/diffs/stack_configuration.json", angular.toJson(self.cfg), function(code, svrMessage) {
                self.svrMessage = svrMessage;
            });
        };

        this.deleteNick = function(nick) {
            var oldBannedNicks = self.cfg.bannedNicks;
            self.cfg.bannedNicks = [];
            angular.forEach(oldBannedNicks, function(n) {
                if (nick != n) {
                    self.cfg.bannedNicks.push(n);
                }
            });
        };
    }

    AppCfg.$inject = ["$resource", "$xhr"];
</script>
  Light is on:  <input type="checkbox" name="cfg.lighton"/> <br/>
  Default Error Reciever (email): <input name="cfg.defaultErrorReciever" ng:validate="email"/> <br/>
  Max Load Percentage: <input name="cfg.loadMaxPercent" ng:validate="number:0:100"/> <br/>
  Next Shutdown Date: <input name="cfg.nextShutdownDate" ng:validate="date"/> <br/>
  Banned nicks:
      <ol>
        <li ng:repeat="nick in cfg.bannedNicks"><span>{{nick}} &nbsp;&nbsp;<a ng:click="deleteNick(nick)">[X]</a></span></li>
    </ol>
  <form ng:submit="newNick()">
    <input type="text" name="newNickname" size="20"/>
    <input type="submit" value="&lt;-- Add Nick"/><br/>
  </form>
  <hr/>
  <button ng:click="diffs()">View Diffs</button><br/>
  <button ng:disabled="{{!message}}" ng:click="save()">Commit Changes</button> Commit Message: <input name="message"></button><br/>
  Last Server operation: <br/>
  <div ng:bind="svrMessage | html:'unsafe'">
  </div>
</body>
</html>

Both of these assets were added by performing:

Click “Add” from the top navbar
Click “Add Content”
Select “Assets” > “Plain Text”
For “Title”, enter “index.html” or “stack_configuration.json”
Paste in the appropriate “Content”
Click “URL”, select “Custom”, and enter the same value as “Title” (otherwise, Chronicle will convert underscores to dashes, so be careful!)
Click “Save”, enter a commit message, then click the next “Save”
Both assets should be viewable as mangled Chronicle content entries from http://localhost/index.html and http://localhost/stack_configuration.json. You normally will not use these URLs.

At this point, neither asset is actually usable. Most content is heavily decorated with additional HTML and then displayed within a layout template, but I want both the index.html and stack_configuration.json assets to be viewable as standalone files and provide a REST interface for AngularJS to work against.

Come back PHP! All is forgiven

Chronicle is largely built using Zend Framework and makes adding extra modules to the system pretty easy. My module needs to be able to display plaintext assets, update their content using an HTTP POST, and provide diffs between the last commit and the current content.

To create the module, the following paths need to be added:

INSTALL/application/appconfig
INSTALL/application/appconfig/controllers
INSTALL/application/appconfig/views/scripts/index

Declare the module with INSTALL/application/appconfig/module.ini:

version = 1.0
description = Application config proof of concept
icon = images/icon.png
tags = config

[maintainer]
name = Perforce Software
email = support@perforce.com
url = http://www.perforce.com

[routes]
appconfig.type = Zend_Controller_Router_Route_Regex
appconfig.route = 'appconfig/(.+)'
appconfig.reverse = appconfig/%s
appconfig.defaults.module = appconfig
appconfig.defaults.controller = index
appconfig.defaults.action = index
appconfig.map.resource = 1

appconfig-operation.type = Zend_Controller_Router_Route_Regex
appconfig-operation.route = 'appconfig/([^/]+)/(.+)'
appconfig-operation.reverse = appconfig/%s/%s
appconfig-operation.defaults.module = appconfig
appconfig-operation.defaults.controller = index
appconfig-operation.defaults.action = index
appconfig-operation.map.action = 1
appconfig-operation.map.resource = 2

Add a view script for displaying plaintext assets, INSTALL/application/appconfig/views/scripts/index/index.phtml:

<?=$this->entry->getValue('content') ?>

Add a view script for displaying diffs, INSTALL/application/appconfig/views/scripts/index/diffs.phtml:

<pre><?=$this->diffs ?></pre>

And a controller at INSTALL/application/appconfig/controllers/IndexController.phtml:

<?php

defined('LIBRARY_PATH') or define('LIBRARY_PATH', dirname(__DIR__));
require_once LIBRARY_PATH . '/simplediff/simplediff.php';

class Appconfig_IndexController extends Zend_Controller_Action
{
    private $entry;
    
    private $mimeTypes = array(
        '.html' => 'text/html',
        '.json' => 'application/json',
    );
    
    public function preDispatch()
    {
        $request = $this->getRequest();
        $request->setParams(Url_Model_Url::fetch($request->getParam('resource'))->getParams());
        $this->entry = P4Cms_Content::fetch($request->getParam('id'), array('includeDeleted' => true));
    }
    
    public function indexAction()
    {
        $this->getResponse()->setHeader('Content-Type', $this->getMimeType(), true);
        $this->view->entry = $this->entry;
        
        if ($this->getRequest()->isPost()) {
            $this->entry->setValue('content', $this->getJsonPost());
            $this->entry->save($this->getRequest()->getParam('message'));
        }
    }
    
    private function getMimeType()
    {
        $url = $this->entry->getValue('url');
        $suffix = substr($url['path'], strrpos($url['path'], '.'));
        
        if (array_key_exists($suffix, $this->mimeTypes)) {
            return $this->mimeTypes[$suffix];
        } else {
            return 'text/plain';
        }
    }
    
    public function diffsAction()
    {
        $this->getResponse()->setHeader('Content-Type', 'text/html', true);
        $this->view->diffs = htmlDiff($this->entry->getValue('content'), $this->getJsonPost());
    }
    
    public function postDispatch()
    {
        $this->getHelper('layout')->disableLayout();
    }
    
    private function getJsonPost()
    {
        if ($this->getRequest()->isPost()) {
            return $this->prettyPrint(file_get_contents('php://input'));
        } else {
            throw new Exception('Can\'t get JSON without POST');
        }
    }
    
    private function prettyPrint($json)
    {
        $array = Zend_Json::decode($json);
        $this->sort($array);
        
        return Zend_Json::prettyPrint(Zend_Json::encode($array), array('indent' => ' '));
    }
    
    private function sort(array &$array)
    {
        if (count(array_filter(array_keys($array), 'is_string')) > 0) {
            ksort($array);
        }
        
        foreach($array as &$value) {
            if (is_array($value)) {
                $this->sort($value);
            }
        }
    }
}

AngularJS

After all files are in place, Chronicle needs to be notified that the new module exists by going to “Manage” > “Modules”, where the “Appconfig” module will be listed if all goes well :) Both assets will now be viewable from http://localhost/appconfig/index.html and http://localhost/appconfig/stack_configuration.json. AngularJS’ resource service is used in index.html to fetch stack_configuration.json and post changes back.

From http://localhost/appconfig/index.html, the data from stack_configuration.json is loaded into the form:

Edits to stack_configuration.json can be made using the form, and the diffs viewed by clicking on “View Diffs”:

The changes can be saved by entering a commit message and clicking “Commit Changes”. After which, clicking “View Diffs” will show no changes:

To show that edits have in fact been made to stack_configuration.json, go to http://localhost/stack_configuration.json, select “History” and click on ” History List”:

Chronicle also provides an interface for viewing diffs between revisions:

Disk Usage

Something to remember in using Chronicle is that each resource requested from Perforce is written to disk before being served to the client. This means that for each request to index.html, Chronicle allocates a new Perforce workspace, checks out the associated file, serves it to the client, then deletes the file and the workspace at the end of the request. This allocate/checkout/serve/delete cycle executes for stack_configuration.json and every other resource in the system.

@TODO

Security!

There’s one major flaw with the appconfig module: it performs zero access checks. By default, Chronicle can be configured to disallow anonymous access by going to “Manage” > “Permissions” and deselecting all permissions for ” anonymous” and “members”. Logging out and attempting to access either http://localhost/appconfig/stack_configuration.json or http://localhost/appconfig/index.html will now give an error page and prompt you to log in. Clicking “New User” will also give an error, as anonymous users don’t have the permission to create users.

Access rights on content are checked by the content module, but are also hard-coded in the associated controllers as IF-statements. A better solution will be required for proper access management in the appconfig module.

Better integration

Chronicle’s content module provides JSON integration for most of its actions, but these mostly exist to support the Dojo Toolkit-enabled front-end. Integrating with these actions over JSON requires detailed knowledge of Chronicle’s form structures.

Chronicle has some nice interfaces for viewing diffs. If I could call those up from index.html I would be major happy :)

Automatic creation of plaintext content type

Before the appconfig module is usable, the plaintext content type has to be created. I would like to automate creation of the plaintext content type when the module is first enabled.

Making applications aware of updates to configuration

When stack_configuration.json is updated, there’s no way to notify applications to the change, and no interface provided so they may poll for changes. I’m not entirely sure at this point what an appropriate solution would look like. In order to complete the concept, I’d first have to create a client app dependent on that configuration.

Better interfaces for manipulating plaintext assets

I had to fiddle with index.html quite a bit. This basically involved editing a local copy of index.html, then pasting the entire contents into the associated form in Chronicle. I have not tried checking out index.html directly from Perforce, and I imagine that any edits would need to be made within Chronicle. GitHub offers an in-browser raw editor, and something like that would be real handy in Chronicle.

Handling conflicts

There is no logic in the appconfig module to catch conflicts if there are two users editing the same file. Conflicts are detectable because an exception is thrown if there is a conflict, but I’m not sure what the workflow for resolution is in Chronicle terms, or how to integrate with it. Who wins?

Working with branches

I did not take the time to see how Chronicle manages branches. I will need to verify that Chronicle and the appconfig module can work with development, staging, and production branches, with maintained divergence. For example, we’re still trying to figure out how to attach visual clients like P4V to the repository and work independently of Chronicle.

Kudos

I would like to thank the guys at Perforce for their assistance and answering all my questions as I worked with Chronicle, especially Randy Defauw.

Update 06/27/2021

This is the first blog post I have ever written. Paul Hammant, who I had met in other contexts previously, happened to be working out of the ThoughtWorks office in Dallas, TX the very same day I started at ThoughtWorks. He asked me if I knew PHP, which I did, and set me off to explore Perforce Chronicle as a solution for managing configuration.

I had never written professionally before or been aware of configuration management: I was very lucky to explore a passion space that Paul has worked within for a very long time. I don’t believe I ever gave him a proper thanks. He gave me an opportunity that probably not a lot of people get in their early careers, and it was an invaluable experience that I learned a lot from and think about fairly often.

The other posts in this series were also written with guidance from Paul:

The subject of configuration as described in these posts is still fresh even after nearly ten years. Even now configuration as code still doesn’t have a perfect solution, though products have become available that make managing configuration easier. Changing configuration in a running process as a general solution remains elusive, as supporting it imposes a lot of constraints on design.

On a more personal note: today is Pride. This is the first Pride I’ve ever participated in and only in the last two years have I felt safe enough to come out in circles beyond close friends. I was out to Paul but only as a detail I confided in passing. When I was working with Paul on these posts he advised that I should consider relocating to the Bay Area. In January, 2013, I moved to San Francisco. Discovering my own life as a person was set by Paul being brave enough to share a deeply personal piece of advice. I’m so thankful he said it, and I’m glad I listened.

This Field Was Green

Reasons why I love Claude 💅

Programming Languages on Love Island

THE VILLA (Opening Couples) 🏝️

CASA AMOR (The Massacre) 🔥

WHO WINS 🏆

My money’s on C

For serious, here’s a short list of other reasons why I love Claude

My context preamble sets the bit

# CLAUDE

## Tone & Vibe

## Core traits

Parser Combinators

The parse() function

Implementing the first parse() function

Specializing the satisfy() parse function

Parser combinators

Recognizing alternatives with a second parse() function

Putting together what we have so far

How do we get the matched input?

Modeling input

Modeling output

The new shape of the parse() function

Changing the return type of the parse() function itself

The parse() function as a functor

Sequential parsing with the parse() function

Alternative parsing with the parse() function

Putting it all together

What’s next?

Imperative Computation in Functional Programming

Motivating monads as a design pattern

Introducing imperative control flow

Monads

Composing monads

The for comprehension and imperative programming

Becoming a Monad

Monad laws

Defining monad laws as properties

Implications of the monad laws

What is enabled by monads?

Beware the imperative trap

Going forward

The Set Function

Modeling a Set

The Set of natural numbers

Building a Set

Building a complex Set

Takeaway

Why don’t we use Sets as functions?

Enabling Control Flow in Functional Programming

Motivating applicative functors as a design pattern

Introducing control flow

Applicative functors

Validation as a use case

Validating a User from external data

Why use a new Validated context instead of Either?

Validation via Applicative

Modeling combinable structures

Independent evaluation of contexts

Products of contexts

Becoming an Applicative

Applicative laws

Testing “for all” F[_]

Defining the applicative laws as properties

But don’t forget Validated!

Implications of the applicative laws

What is enabled by applicatives?

Contexts and Effects in Functional Programming

Conventions

How to read “math”

Terminology

Complexity in programming

Pure and impure functions

Manifesting complexity

Nondeterminism

Unknown quantities

Relating nondeterminism and unknown quantities

Implied complexity

Operations producing undesired cases

A model for characterizing complexity

The `parse()` function

Implementing the first `parse()` function

Specializing the `satisfy()` parse function

Recognizing alternatives with a second `parse()` function

The new shape of the `parse()` function

Changing the return type of the `parse()` function itself

The `parse()` function as a functor

Sequential parsing with the `parse()` function

Alternative parsing with the `parse()` function

Modeling a `Set`

The `Set` of natural numbers

Building a `Set`

Building a complex `Set`

Why don’t we use `Set`s as functions?

Validating a `User` from external data

Why use a new `Validated` context instead of `Either`?

Validation via `Applicative`

Testing “for all” `F[_]`

But don’t forget `Validated`!

Generalizing for an unknown length of term `A`

Why does the `map()` function return `F[B]`?

Context `F[A]` must produce some term `A`

Why do so many objects define `map()`?

Failure, `systemd`, and duct tape

Why `hakyll`, specifically?

How to shave a yak for `hakyll`

How `hakyll` accomplishes what I need

Just using plain-old `html` is a tall order, aka. Transformation Telephone

Using `css` is still like the Family Guy meme

I rather like `hakyll` and Haskell!