gl: Do less work to maintain caches

gl: Do less work to maintain caches

Remember which atlases were removed, and only
check those when looking for icons or glyphs
to remove. For most frames, we don't have to
check at all since no atlases were removed.