reftests: Allow minor differences to be tolerated
authorSimon McVittie <smcv@debian.org>
Sat, 13 Feb 2021 16:19:10 +0000 (16:19 +0000)
committerJeremy Bícha <jbicha@ubuntu.com>
Sat, 5 Aug 2023 13:56:12 +0000 (14:56 +0100)
commit45d5ef0199d784e5d72cd50f23dd02d78a212b3c
treea33036213635b7bf8221916814b213cd0e2561ae
parent8a616dc04a71bc081c46b2ba3ebd1e2df7faf4cf
reftests: Allow minor differences to be tolerated

Based on an earlier patch by Michael Biebl, as used in Debian's GTK 3
packaging, with additional inspiration from librsvg's reftests.

Each .ui or .node reftest can have an accompanying .keyfile file
like this:

    [reftest]
    tolerated-diff-level=20
    tolerated-diff-pixels=1000

If the image differs, but the number of pixels that differ is no more
than tolerated-diff-pixels and the differences are no more than
tolerated-diff-level, then we treat it as a success with warnings, save
the .diff.png for analysis, and use g_test_incomplete() to record the
test-case as "TODO".

Signed-off-by: Simon McVittie <smcv@debian.org>
Forwarded: https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/3195
Applied-upstream: no, upstream want reftests to be a strict pass/fail with identical results required

Gbp-Pq: Topic debian
Gbp-Pq: Name reftests-Allow-minor-differences-to-be-tolerated.patch
testsuite/gsk/compare-render.c
testsuite/reftests/gtk-reftest.c
testsuite/reftests/image-compare.c