{"author":{"id":1,"name":"ansateza","url":"https://llmpages.org/author/ansateza/"},"canonical_url":"https://llmpages.org/for-ai-crawlers/","categories":null,"content":"For AI Crawlers\n\nFor AI Crawlers: Efficient Content Access via TCT Protocol\n\nReduce bandwidth by 83%, tokens by 86%. Efficient, verifiable, standardized content delivery.\n\nQuick Start\n\nDiscovery: Use https://llmpages.org/llm-sitemap.json to discover all available content.\n\nPer-Article JSON: Access machine-readable content at {canonical-url}/llm/\n\nExample:\n\n\nC-URL: https://example.com/article/\n\nM-URL: https://example.com/article/llm/\n\n\nProtocol Benefits\n\nMeasured results from 970+ production URLs:\n\n\n83% bandwidth savings – 103 KB → 17.7 KB average\n\n86% token reduction – 13,900 → 1,960 tokens\n\n90%+ skip rate – Unchanged content returns 304 Not Modified\n\nTemplate-invariant – Same content hash regardless of theme\n\nVerifiable – SHA-256 content fingerprints\n\n\nHTTP Headers\n\nRequired Headers in M-URL Response:\n\n\nLink: <canonical-url>; rel=\"canonical\" – Bidirectional handshake\n\nETag: \"sha256-abc123...\" – Content fingerprint\n\nCache-Control: max-age=0, must-revalidate – Conditional request discipline\n\nVary: Accept – Content negotiation\n\nContent-Type: application/json; charset=utf-8\n\n\nConditional Requests\n\nTo save 90%+ bandwidth:\n\n\nFirst request: Store the ETag value\n\nSubsequent requests: Send If-None-Match: \"sha256-abc123...\"\n\nIf unchanged: Receive 304 Not Modified (zero body)\n\nIf changed: Receive 200 OK with new content and new ETag\n\n\nClient Libraries\n\nPython:\n\npip install collab-tunnel\n\nGitHub: github.com/antunjurkovic-collab/collab-tunnel-python\n\nProduction Sites\n\nTest against these verified implementations:\n\n\nwellbeing-support.com – 400 URLs, 10/10 compliance\n\nomacedonii.com – 500 URLs, 10/10 compliance\n\nbestdemotivationalposters.com – 500 URLs, 9/10 compliance\n\n\nValidation\n\nVerify TCT compliance: llmpages.org/validator\n\nSupport\n\n\nEmail: antunjurkovic@gmail.com\n\nGitHub: github.com/antunjurkovic-collab\n\nDocumentation: Integration Guide","content_media_type":"text/plain; charset=utf-8","excerpt":"For AI Crawlers: Efficient Content Access via TCT Protocol Reduce bandwidth by 83%, tokens by 86%. Efficient, verifiable, standardized content delivery. Quick Start Discovery: Use https://llmpages.org/llm-sitemap.json to discover all available content. Per-Article JSON: Access machine-readable content at {canonical-url}/llm/ Example: C-URL: https://example.com/article/ M-URL: https://example.com/article/llm/ Protocol Benefits Measured results from 970+ production URLs: 83% bandwidth savings &#8211;","headings":[{"anchor":null,"level":2,"text":"Quick Start"},{"anchor":null,"level":2,"text":"Protocol Benefits"},{"anchor":null,"level":2,"text":"HTTP Headers"},{"anchor":null,"level":2,"text":"Conditional Requests"},{"anchor":null,"level":2,"text":"Client Libraries"},{"anchor":null,"level":2,"text":"Production Sites"},{"anchor":null,"level":2,"text":"Validation"},{"anchor":null,"level":2,"text":"Support"}],"image":null,"images":null,"lastModified":"2025-11-07T16:30:11+00:00","llm_url":"https://llmpages.org/for-ai-crawlers/llm/","post_id":122,"post_type":"page","profile":"https://www.ietf.org/archive/id/draft-jurkovikj-collab-tunnel-03.html#tct-m-url-profile","published":"2025-10-18T18:39:12+00:00","slug":"for-ai-crawlers","tags":null,"title":"For AI Crawlers","word_count":205}