reftests: Allow minor differences to be tolerated
authorSimon McVittie <smcv@debian.org>
Sat, 13 Feb 2021 16:19:10 +0000 (16:19 +0000)
committerRaspbian forward porter <root@raspbian.org>
Wed, 22 Nov 2023 04:54:09 +0000 (04:54 +0000)
commit27e2d402f9c7823ccf5dcee90bec94db9c2f6fc7
tree1cbaa66b8efdfa00572ba21fccd433d06dcb3f94
parent762741c0db8e4be466184427c6e3bda947c04295
reftests: Allow minor differences to be tolerated

Based on an earlier patch by Michael Biebl, with additional inspiration
from librsvg's reftests.

Each .ui or .node reftest can have an accompanying .keyfile file
like this:

    [reftest]
    tolerated-diff-level=20
    tolerated-diff-pixels=1000

If the image differs, but the number of pixels that differ is no more
than tolerated-diff-pixels and the differences are no more than
tolerated-diff-level, then we treat it as a success with warnings, save
the .diff.png for analysis, and use g_test_incomplete() to record the
test-case as "TODO".

Signed-off-by: Simon McVittie <smcv@debian.org>
Forwarded: https://gitlab.gnome.org/GNOME/gtk/-/merge_requests/3195
Applied-upstream: no, upstream want reftests to be a strict pass/fail with identical results required

Gbp-Pq: Name reftests-Allow-minor-differences-to-be-tolerated.patch
testsuite/reftests/gtk-reftest.c