reftests: Allow minor differences to be tolerated
authorSimon McVittie <smcv@debian.org>
Sat, 13 Feb 2021 16:19:10 +0000 (16:19 +0000)
committerAmin Bandali <bandali@ubuntu.com>
Wed, 4 Oct 2023 20:11:12 +0000 (21:11 +0100)
commit518b49d1bed58b2a6164235927624a56c393e39c
treeb00c48779c525630c2c2e6052981cab477d13028
parentef13932b0c9291e18da3cdaa1f85b1083804907b
reftests: Allow minor differences to be tolerated

Based on an earlier patch by Michael Biebl, as used in Debian's GTK 3
packaging, with additional inspiration from librsvg's reftests.

Each .ui or .node reftest can have an accompanying .keyfile file
like this:

    [reftest]
    tolerated-diff-level=20
    tolerated-diff-pixels=1000

If the image differs, but the number of pixels that differ is no more
than tolerated-diff-pixels and the differences are no more than
tolerated-diff-level, then we treat it as a success with warnings, save
the .diff.png for analysis, and use g_test_incomplete() to record the
test-case as "TODO".

Signed-off-by: Simon McVittie <smcv@debian.org>
Forwarded: https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/3195
Applied-upstream: no, upstream want reftests to be a strict pass/fail with identical results required

Gbp-Pq: Topic debian
Gbp-Pq: Name reftests-Allow-minor-differences-to-be-tolerated.patch
testsuite/gsk/compare-render.c
testsuite/reftests/gtk-reftest.c
testsuite/reftests/image-compare.c