AI crawler access is not one switch
Separate controls for search discovery, possible model-training use and visits triggered by a user. The available controls and their effects differ by provider.
RFC 9309: Robots Exclusion Protocol
IETFrfc-editor.org