fetcher: Avoid too large queues for metadata processing
authorColin Walters <walters@verbum.org>
Mon, 14 Nov 2022 19:06:05 +0000 (14:06 -0500)
committerColin Walters <walters@verbum.org>
Mon, 14 Nov 2022 19:06:05 +0000 (14:06 -0500)
commit97e9bc7fd4a73c5c63e5390b3935cf986432ab6e
tree1a4f72b0ae902f1e2522418eb8a6b1457d62ab0e
parent06ff6a0448bd8c072d6d6f15a054bb9beed185df
fetcher: Avoid too large queues for metadata processing

We added backoff/queueing for fetching via HTTP, but we have
another queue in the metadata scanning which can also grow
up to the number of outstanding objects, which can be large.

Capping the scanning operation when we have hit our operation
limit will avoid potentially large amounts of allocations in the
case of e.g. a slow network.

Closes: https://github.com/ostreedev/ostree/issues/2732
src/libostree/ostree-repo-pull.c