# As a condition of accessing this website, you agree to abide by the # following content signals: # (a) If a content-signal = yes, you may collect content for the # corresponding use. # (b) If a content-signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a content signal for a # corresponding use, the website operator neither grants nor restricts # permission via content signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search # does not include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of # content for generative AI search answers). # ai-train: training or fine-tuning AI models. # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # Search engines and programs that read a page to answer someone's question may read every public page. # Training AI models on the site is reserved (docs/content-security-plan.md, 5.7). The same is said three # ways below, because crawlers read different ones: the content signal, the shared training token, and # the names of the crawlers that collect for training. scripts/seo.test.mjs keeps them in step. User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=no # Settings screen: browser-local settings only. Nothing to index. Disallow: /settings Disallow: /ko/settings Disallow: /en/settings Disallow: /api/ # The data gateway answers people who are using the explorer, not crawlers. Disallow: /g/ # Last in its group: a reader that follows the first matching line, not the longest, then still keeps out of the paths above. Allow: / # The shared token for AI training (draft-kelley-aipref-training-preference-signal). A crawler keeps # following its own group; this group only says what the pages may be used for. User-agent: __usage_ai-training Disallow: / # Crawlers that collect pages to train AI models are asked to stay out of the whole site (operator # decision 2026-11-12). Search engines and the AI search and on-request fetchers (OAI-SearchBot, # ChatGPT-User, Claude-SearchBot, Claude-User and the like) are not named here, so the rules of the # first group still apply to them. The names are AI_TRAINING_CRAWLERS in src/lib/seo.ts. User-agent: GPTBot User-agent: ClaudeBot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: CCBot User-agent: Bytespider Disallow: / # Google-Extended covers both Gemini training and the Gemini apps reading the site to answer, and the two # cannot be set apart. From 2027 Google reads the shared token above as a rule for Google-Extended unless # Google-Extended is named, so it is named and allowed here: being in Gemini's answers was chosen over # staying out of its training (operator decision 2026-11-12). A named group replaces the first group for # that name, so the paths are repeated. User-agent: Google-Extended Disallow: /settings Disallow: /ko/settings Disallow: /en/settings Disallow: /api/ Disallow: /g/ Allow: / Sitemap: https://bibliadatum.org/sitemap-index.xml