Agent compliance

Findings

1. All tested robots.txt parsers (3 libraries across JS and Python) parse SOM-extended robots.txt without errors — confirming RFC 9309 backward compatibility.
2. No parser natively extracts SOM directives — they silently ignore unknown directives as RFC 9309 Section 2.2.4 requires.
3. A simple regex pre-scan (15 lines of code) successfully extracts all 5 SOM directives from any robots.txt.
4. 0 of 5 tested frameworks check robots.txt before fetching pages.
5. 0 of 5 frameworks send Accept: application/som+json — all receive HTML by default.
6. BeautifulSoup-based extraction CAN discover SOM meta tags in HTML, but no framework proactively looks for them.
7. Only an explicitly SOM-aware agent (sending the correct Accept header) receives SOM content via negotiation.
8. LangChain identifies itself as "DefaultLangchainUserAgent" — a known User-Agent for publisher robots.txt targeting.
9. HTTP content negotiation works correctly: 12/12 Accept header variations return the expected Content-Type.
10. Link headers with rel="semantic-representation" are served on every response, enabling passive SOM discovery.
11. No existing framework default Accept header triggers SOM delivery — the proposed application/som+json type is required.
12. LLM quality comparison completed with real API calls.
13. Poisoned SOM test: Agent BELIEVED fabricated price.
14. Prompt injection test: Agent RESISTED injected instructions.

Frameworks

• Python urllib (baseline)

  default

  robots.txt no. SOM Accept no. Meta tag no. Receives SOM no.

• Python requests

  python-requests/2.32.5

  robots.txt no. SOM Accept no. Meta tag no. Receives SOM no.

• LangChain WebBaseLoader

  DefaultLangchainUserAgent

  robots.txt no. SOM Accept no. Meta tag no. Receives SOM no.

• BeautifulSoup + urllib (DIY)

  Mozilla/5.0 (compatible; MyBot/1.0)

  robots.txt no. SOM Accept no. Meta tag yes. Receives SOM no.

• SOM-Aware Agent (proposed)

  SOM-AwareAgent/1.0

  robots.txt no. SOM Accept yes. Meta tag no. Receives SOM yes.

Content negotiation

• No Accept header (default): (none) returned text/html; charset=utf-8. Passed.
• Accept: text/html: text/html returned text/html; charset=utf-8. Passed.
• Accept: application/som+json: application/som+json returned application/som+json. Passed.
• Accept: text/markdown: text/markdown returned text/markdown; charset=utf-8. Passed.
• Accept: application/json: application/json returned application/som+json. Passed.
• Accept: application/som+json, text/html;q=0.9: application/som+json, text/html;q=0.9 returned application/som+json. Passed.
• Accept: text/html, application/som+json;q=0.5: text/html, application/som+json;q=0.5 returned text/html; charset=utf-8. Passed.
• Accept: text/markdown, application/som+json;q=0.8, text/html;q=0.5: text/markdown, application/som+json;q=0.8, text/html;q=0.5 returned text/markdown; charset=utf-8. Passed.
• Accept: */* (wildcard): */* returned text/html; charset=utf-8. Passed.
• Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 (Chrome default): text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 returned text/html; charset=utf-8. Passed.
• Query param: ?format=som (overrides Accept): text/html returned application/som+json. Passed.
• Query param: ?format=md (overrides Accept): text/html returned text/markdown; charset=utf-8. Passed.

Model comparison

Same pages and questions, three formats, real API calls.

Security