List of Google common crawlers

Separate controls for search discovery, possible model-training use and visits triggered by a user. The available controls and their effects differ by provider.