reftests: Allow minor differences to be tolerated
authorSimon McVittie <smcv@debian.org>
Sat, 13 Feb 2021 16:19:10 +0000 (16:19 +0000)
committerSimon McVittie <smcv@debian.org>
Wed, 8 Nov 2023 16:30:21 +0000 (16:30 +0000)
commit22bb2f7d342419226cf9dd3aefcd096d445aaeac
treead94c09d52b4db006072226aea819a57c3d5195d
parent43e534ceef2fb05d82b29e2fe5f07be0e54f3ab1
reftests: Allow minor differences to be tolerated

Based on an earlier patch by Michael Biebl, with additional inspiration
from librsvg's reftests.

Each .ui or .node reftest can have an accompanying .keyfile file
like this:

    [reftest]
    tolerated-diff-level=20
    tolerated-diff-pixels=1000

If the image differs, but the number of pixels that differ is no more
than tolerated-diff-pixels and the differences are no more than
tolerated-diff-level, then we treat it as a success with warnings, save
the .diff.png for analysis, and use g_test_incomplete() to record the
test-case as "TODO".

Signed-off-by: Simon McVittie <smcv@debian.org>
Forwarded: https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/3195
Applied-upstream: no, upstream want reftests to be a strict pass/fail with identical results required

Gbp-Pq: Name reftests-Allow-minor-differences-to-be-tolerated.patch
testsuite/reftests/gtk-reftest.c