Auto merge of #4520 - goffrie:master, r=alexcrichton

Don't visit the same unit multiple times.

This fixes some accidentally-exponential behaviour we were seeing in our
fairly large workspace. It brings a no-op cargo run from about 10
seconds to 0.3 seconds on my machine.

Also changes an association list to a hashmap because that was showing
up in profiles too.