.Google's Gary Illyes shows Googlebot gathers web links as opposed to "following" them, negating formal paperwork.Googlebot accumulates web links rather than observing them in real-time.Gary Illyes discussed this information in Google.com's 'Browse Off The Document' podcast.Google's paperwork negates the real crawler habits.