Introduction

Writing supertests#

A supertest is a claim about behavior that must hold for every permitted input. It consists of typed inputs, optional assumptions, and one or more assertions.

Declare a supertest#

Keep supertests in your repository. A top-level supertests/ directory makes them easy to review and select, but Pup can discover them anywhere inside the linked root.

use schematic::supertest;

#[supertest]
fn sorting_preserves_length(values: Vec<i32>) {
    let original_length = values.len();
    let mut sorted = values;
    sorted.sort();
    assert_eq!(sorted.len(), original_length);
}

from schematic import *

@supertest
def sorting_preserves_length(values: list[int]) -> None:
    assert len(sorted(values)) == len(values)

require "schematic"

Schematic.supertest(
  "sorting_preserves_length",
  values: Array
) do |values|
  assert values.sort.length == values.length
end

Give each supertest a clear, reviewable name that states one behavior. Prefer sorting_preserves_length over a generic name such as sorting_test.

Python import forms#

Pup recognizes both qualified and directly imported decorators. These declarations are equivalent:

import schematic

@schematic.supertest
def identity(value: int) -> None:
    assert value == value

from schematic import supertest

@supertest
def identity(value: int) -> None:
    assert value == value

Wildcard imports also work and are used throughout these guides:

from schematic import *

@supertest
def identity(value: int) -> None:
    assert value == value

Supertest inputs cover every value#

Each parameter introduces a universally quantified input. Pup checks the assertion for every value in the supported domain of that type, not a random sample.

Use precise application types where possible. They communicate the intended domain and let the supertest use the same validation and behavior as the application.

Narrow the scope with assumptions#

An assumption defines which inputs are in scope. Put assumptions before calculations or calls that depend on them.

Suppose a claim refers to the first value returned by a sorting function. That claim only applies to nonempty inputs, so the assumption comes before indexing the result:

use schematic::{assume, supertest};

#[supertest]
fn sorted_first_is_minimum(values: Vec<i32>) {
    assume(!values.is_empty());

    let sorted = sort_values(&values);
    let first = sorted[0];

    assert!(sorted.iter().all(|value| first <= *value));
}

from schematic import *

@supertest
def sorted_first_is_minimum(values: list[int]) -> None:
    assume(len(values) > 0)

    result = sort_values(values)
    first = result[0]

    assert all(first <= value for value in result)

require "schematic"

Schematic.supertest(
  "sorted_first_is_minimum",
  values: Array
) do |values|
  Schematic.assume !values.empty?

  result = sort_values(values)
  first = result[0]

  assert result.all? { |value| first <= value }
end

Do not use assumptions to hide behavior the application must handle. If callers can supply an input, either include it in the claim or write a separate supertest for its expected rejection behavior.

Keep each supertest focused#

A supertest may contain several assertions when they describe one behavior. Split unrelated obligations so each reported Problem or confirmed failure remains focused.

Keep supertests deterministic. Do not depend on wall-clock time, random numbers, network services, mutable global state, or resources outside the linked repository. Pup reports unsupported behavior instead of treating the check as complete.

Select supertests from the CLI#

Pass a file, a directory, or one named supertest to pup check. Pup creates or reuses one independent check for every selected declaration at the chosen commit:

$ pup check supertests/sorting.rs
$ pup check supertests/
$ pup check 'supertests/sorting.rs::sorting_preserves_length'

$ pup check supertests/sorting.py
$ pup check supertests/
$ pup check 'supertests/sorting.py::sorting_preserves_length'

$ pup check supertests/sorting.rb
$ pup check supertests/
$ pup check 'supertests/sorting.rb::sorting_preserves_length'

Selector	Meaning
`path/to/file`	Every supertest declared in the file
`path/to/directory/`	Every discovered supertest below the directory
`path/to/file::name`	One named supertest in the file
`.`	Every discovered supertest in the linked repository

Paths are resolved from the current working directory. Quotes around a named selector prevent shells from interpreting punctuation unexpectedly.

How Pup evaluates supertests#

Language test runners do not execute supertests as ordinary test cases. Their parameters represent arbitrary values, and assume defines a logical scope rather than skipping one runtime example.

Keep ordinary tests for fast examples, integration behavior, and fixtures. Use supertests for important claims that should survive inputs and changes you did not anticipate.