Schematic Schematic Pup docs
Schematic website
Introduction

Welcome to Pup#

Pup is Schematic’s command-line tool for checking behaviors your code must always preserve. Supertests express those behaviors as precise claims over every possible input in scope, not just selected examples. Pup looks for problems ordinary tests can miss. When it finds one, it explains what went wrong.

Write a supertest#

A supertest looks much like an ordinary test. It calls your code and uses assertions to state what should happen.

In Rust, the #[supertest] attribute turns a test function into a supertest.

In Python, the @supertest decorator turns a test function into a supertest. If you use import schematic, write @schematic.supertest instead.

In Ruby, the Schematic.supertest DSL defines a test block as a supertest.

A supertest takes inputs as parameters. Pup checks the claim across all possible values in scope. If one breaks the claim, Pup returns it as a concrete counterexample.

Consider collapse_spaces, a function that should replace every run of consecutive spaces with one space. Once text has been normalized, running the function again should not change it. We can express that claim as a supertest:

supertests/normalize_spaces.rs
use schematic::supertest;

#[supertest]
fn normalizing_twice_changes_nothing(text: String) {
    let once = collapse_spaces(&text);
    let twice = collapse_spaces(&once);

    assert_eq!(twice, once);
}
supertests/normalize_spaces.py
from schematic import *

@supertest
def normalizing_twice_changes_nothing(text: str) -> None:
    once = collapse_spaces(text)
    twice = collapse_spaces(once)

    assert twice == once
supertests/normalize_spaces.rb
require "schematic"

Schematic.supertest(
  "normalizing_twice_changes_nothing",
  text: String
) do |text|
  once = collapse_spaces(text)
  twice = collapse_spaces(once)

  assert twice == once
end

The input text covers every possible string, not just a few examples. That includes easy-to-miss cases such as empty strings, strings without spaces, and long runs of spaces.

Your language’s test runner does not execute the supertest. Pup does not run it over those strings one by one either. Instead, Pup reasons about the supertest as one claim that must hold for every possible input.

Check the supertest with Pup#

Start a check by passing the supertest file to Pup:

Terminal
$ pup check supertests/normalize_spaces.rs
text-tools · a81d7c2 · supertests/normalize_spaces.rs

● normalizing_twice_changes_nothing · checking · ref 43
Terminal
$ pup check supertests/normalize_spaces.py
text-tools · a81d7c2 · supertests/normalize_spaces.py

● normalizing_twice_changes_nothing · checking · ref 43
Terminal
$ pup check supertests/normalize_spaces.rb
text-tools · a81d7c2 · supertests/normalize_spaces.rb

● normalizing_twice_changes_nothing · checking · ref 43

The row identifies one check for this supertest at commit a81d7c2. Checks are asynchronous. You can leave the terminal and inspect it later, or pass --wait to keep the command open. Once checking completes, Pup returns the result and, if the claim is false, the exact input that breaks it:

Terminal
$ pup check status
text-tools · supertests/normalize_spaces.rs::normalizing_twice_changes_nothing

    COMMIT  · STATUS · WHEN   · REF
› × a81d7c2 · failed · 1m ago ·  43

Problem P1
  Normalization is not idempotent
  collapse_spaces replaces only one pair of spaces per call.

Confirmed counterexample
  text   = "hello   world"
  once   = "hello  world"
  twice  = "hello world"
Terminal
$ pup check status
text-tools · supertests/normalize_spaces.py::normalizing_twice_changes_nothing

    COMMIT  · STATUS · WHEN   · REF
› × a81d7c2 · failed · 1m ago ·  43

Problem P1
  Normalization is not idempotent
  collapse_spaces replaces only one pair of spaces per call.

Confirmed counterexample
  text   = "hello   world"
  once   = "hello  world"
  twice  = "hello world"
Terminal
$ pup check status
text-tools · supertests/normalize_spaces.rb::normalizing_twice_changes_nothing

    COMMIT  · STATUS · WHEN   · REF
› × a81d7c2 · failed · 1m ago ·  43

Problem P1
  Normalization is not idempotent
  collapse_spaces replaces only one pair of spaces per call.

Confirmed counterexample
  text   = "hello   world"
  once   = "hello  world"
  twice  = "hello world"

This is a concrete counterexample, not a prediction. Running the implementation with that input reproduces the failure.

Fix the implementation and check again#

The counterexample reveals the bug: the implementation replaces only one pair of spaces per call. Repeating the replacement until no consecutive spaces remain fixes it:

src/text.rs
fn collapse_spaces(text: &str) -> String {
    text.replace("  ", " ") 
    let mut normalized = text.to_owned(); 
    while normalized.contains("  ") { 
        normalized = normalized.replace("  ", " "); 
    } 
    normalized 
}
src/text.py
def collapse_spaces(text: str) -> str:
    return text.replace("  ", " ") 
    normalized = text 
    while "  " in normalized: 
        normalized = normalized.replace("  ", " ") 
    return normalized
lib/text.rb
def collapse_spaces(text)
  text.gsub("  ", " ") 
  normalized = text.dup
  while normalized.include?("  ") 
    normalized = normalized.gsub("  ", " ") 
  end
  normalized 
end

Once the fix is committed, running the same command checks the updated source:

Terminal
$ pup check supertests/normalize_spaces.rs
text-tools · c24f901 · supertests/normalize_spaces.rs

● normalizing_twice_changes_nothing · checking · ref 44

$ pup check status
text-tools · supertests/normalize_spaces.rs::normalizing_twice_changes_nothing

    COMMIT  · STATUS     · WHEN    · REF
› ✓ c24f901 · 0 problems · 12s ago ·  44
  × a81d7c2 · failed     · 4m ago  ·  43
Terminal
$ pup check supertests/normalize_spaces.py
text-tools · c24f901 · supertests/normalize_spaces.py

● normalizing_twice_changes_nothing · checking · ref 44

$ pup check status
text-tools · supertests/normalize_spaces.py::normalizing_twice_changes_nothing

    COMMIT  · STATUS     · WHEN    · REF
› ✓ c24f901 · 0 problems · 12s ago ·  44
  × a81d7c2 · failed     · 4m ago  ·  43
Terminal
$ pup check supertests/normalize_spaces.rb
text-tools · c24f901 · supertests/normalize_spaces.rb

● normalizing_twice_changes_nothing · checking · ref 44

$ pup check status
text-tools · supertests/normalize_spaces.rb::normalizing_twice_changes_nothing

    COMMIT  · STATUS     · WHEN    · REF
› ✓ c24f901 · 0 problems · 12s ago ·  44
  × a81d7c2 · failed     · 4m ago  ·  43

Understand check results#

0 problems means the standard check completed without detecting any actionable issues. In Pup, a Problem is actionable analysis to inspect, while failed means Pup confirmed a counterexample.

For stronger assurance, pass --prove. This may take longer while Pup works to mathematically prove that the claim always holds. A verified result means Pup proved the claim for the checked commit. If Pup cannot prove the claim or confirm a failure, the result is inconclusive.

Next steps#

Get started covers installation, browser login, linking your first repository, and your first check. To go deeper, learn more about writing supertests or see the pup check command reference.