crawlforge-mcp-server
CrawlForge MCP Server - Professional Model Context Protocol server with 31 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama
npm · fingerprint 8ac034fd020d8609d222692a · repository · RSS
31tools
3recorded versions
45dtracked
unchanged 10d
What changed
Changed 2026-10-07 13:10 UTC
1 schema or annotation change with the description left byte-identical.
--- pinned/scrape +++ observed/scrape { "_meta": { "anthropic/alwaysLoad": true - "description": "Formats to return (default: [\"markdown\"])", + "description": "Formats to return (default: [\"markdown\"]); at most one highlights and one question format per call", "items": { "anyOf": [ {
Changed 2026-10-05 14:20 UTC
29 descriptions rewritten, 2 tools added.
--- pinned/agent +++ observed/agent { "annotations": { "destructiveHint": false, - "description": "Use this when you need an autonomous agent to research, navigate, and synthesise an answer from the web — no URLs required. The agent plans search queries, fetches and filters relevant pages, and returns a prose or structured answer. model:\"pro\" uses deep multi-source research. Hard limits: maxSteps≤10, maxUrls≤20, 120s wall-clock. Confirms before pro runs. Degraded-but-useful output if no LLM keys/Ollama. Example: agent({prompt:\"What are the top 5 MCP servers in 2025?\", maxUrls:10})", - "execution": { - "taskSupport": "optional" - }, + "description": "Use this when you need an autonomous agent to research, navigate, and synthesise an answer from the web - no URLs required. The agent plans search queries, fetches and filters relevant pages, and returns a prose or structured answer. model:\"pro\" uses deep multi-source research. Hard limits: maxSteps<=10, maxUrls<=20, 120s wall-clock. Confirms before pro runs. Degraded-but-useful output if no LLM keys/Ollama. Not for a URL you already have (scrape) or a question one search answers (search_web). Pages that block a plain fetch are retried in the stealth browser automatically (at most 2 a run, URLs you name first; evidence marked via:\"stealth\"). Cost: 18 credits at most - 8, plus 5 per stealth retry that gets the page; a retry that is blocked again is free. Example: agent({prompt:\"What are the top 5 MCP servers in 2025?\", maxUrls:10})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "maxSteps": { "default": 5, + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "model": { "default": "default", "description": "\"default\" = SamplingClient loop (no keys needed); \"pro\" = full ResearchOrchestrator", + "propertyNames": { + "type": "string" + }, "type": "object" }, "urls": {
--- pinned/analyze_content +++ observed/analyze_content { "annotations": { "destructiveHint": false, - "description": "Use this when you need NLP metrics for text — language detection, sentiment, topic extraction, entity recognition, readability score. Good for content auditing and classification. Example: analyze_content({text: \"..article text..\", options: {extractTopics: true, includeSentiment: true}})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this for NLP metrics on text you already hold - language detection, sentiment, topic extraction, entity recognition, readability score - for content auditing and classification. Takes text, not a URL. Not for reading a page (scrape returns the markdown to pass in). Cost: 3 credits. Example: analyze_content({text: \"..article text..\", options: {extractTopics: true, includeSentiment: true}})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "options": { - "additionalProperties": true, + "additionalProperties": {}, "description": "Analysis options", "properties": {}, "type": "object"
--- pinned/batch_scrape +++ observed/batch_scrape { "annotations": { "destructiveHint": false, - "description": "Use this when you need to scrape 2–50 URLs in parallel — e.g. batch-collecting product pages, news articles, or competitor pages. Use mode:\"async\" with a webhook for large batches; mode:\"sync\" for up to ~25 URLs when you need results immediately. Example: batch_scrape({urls: [\"https://a.com\",\"https://b.com\"], formats: [\"json\"], maxConcurrency: 5})", - "execution": { - "taskSupport": "optional" - }, + "description": "Use this to scrape 2-50 URLs in one call - product pages, news articles, competitor pages. Never loop scrape over a URL list. mode:\"sync\" returns results directly for up to ~25 URLs; mode:\"async\" with a webhook for larger batches, then get_batch_results. Not for one URL (scrape) or for discovering URLs (map_site). Cost: 5 credits. Example: batch_scrape({urls: [\"https://a.com\",\"https://b.com\"], formats: [\"json\"], maxConcurrency: 5})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "delayBetweenRequests": { "default": 100, + "propertyNames": { + "type": "string" + }, "type": "object" }, "formats": { - "description": "Include failed URLs in results", + "description": "List failed URLs in results. false hides the entries; failedUrls still counts them", "type": "boolean" }, "includeMetadata": { - "additionalProperties": false, "description": "Job management options for async processing", "properties": { "maxRetries": { + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "mode": { "default": "sync", "description": "Processing mode: sync (wait) or async (background)", + "redact_pii": { + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" + }, "respect_robots": { "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", "type": "boolean" - "additionalProperties": false, "properties": { "headers": { "additionalProperties": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "metadata": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "selectors": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "timeout": { - "additionalProperties": false, "description": "Webhook configuration for async job notifications", "properties": { "events": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "signingSecret": {
+ tool added: browser_session
--- pinned/crawl_deep +++ observed/crawl_deep { "annotations": { "destructiveHint": false, - "description": "Use this when you need to discover and optionally extract content from many pages within a site — e.g. building a knowledge base, indexing docs, or auditing all pages. Use map_site first to estimate scope, then crawl_deep for content. Example: crawl_deep({url: \"https://docs.example.com\", max_depth: 3, max_pages: 200, extract_content: true})", - "execution": { - "taskSupport": "optional" - }, + "description": "Use this to fetch many pages of one site by following links - a knowledge base, a docs index, a full-site audit. Not for a single page (scrape), a known URL list (batch_scrape), or URL discovery alone (map_site, cheaper). Cost: 4 credits base, grows with page count. Example: crawl_deep({url: \"https://docs.example.com\", max_depth: 3, max_pages: 200, extract_content: true})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "concurrency": { "description": "Number of concurrent requests", - "additionalProperties": false, "description": "Per-domain allow/deny lists and crawl rules", "properties": { "blacklist": { + "items": {}, "type": "array" }, "domain_rules": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "whitelist": { + "items": {}, "type": "array" } }, - "additionalProperties": false, "description": "PageRank tuning options", "properties": { "dampingFactor": { + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "max_pages": { "description": "Maximum number of pages to crawl", "maximum": 1000, + "redact_pii": { + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" + }, "respect_robots": { "description": "Respect robots.txt directives", "type": "boolean" - "additionalProperties": false, "description": "Shared cookie-jar/session for login-then-crawl workflows", "properties": { "enabled": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "initialRequest": { - "additionalProperties": false, "properties": { "body": { "type": "string" + "propertyNames": { + "type": "string" + }, "type": "object" }, "method": { - "additionalProperties": true, + "additionalProperties": {}, "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", "properties": { "actual": { + "description": "URLs that failed; one entry each in errors[]", "type": "number" }, "errors": { + "expires_at": { + "description": "When the stored result is dropped (ISO 8601)", + "type": "string" + }, "link_analysis": { "anyOf": [ {}, + "pages_attempted": { + "description": "URLs the crawl tried: pages_crawled + error_count", + "type": "number" + }, "pages_crawled": { + "description": "Pages fetched and returned in results", "type": "number" }, "pages_found": { + "description": "Same as pages_crawled; kept for existing clients", "type": "number" }, "pages_per_second": { + "preview": { + "description": "The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)", + "type": "string" + }, + "redaction": { + "additionalProperties": {}, + "description": "Present when redact_pii was set: what was redacted from the text of this result", + "properties": { + "count": { + "description": "Total spans replaced", + "type": "number" + }, + "entities": { + "additionalProperties": { + "type": "number" + }, + "description": "How many spans were replaced, by entity class; a class with no hits is omitted", + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "mode": { + "description": "\"fast\" is the free regex pass; \"model\" added an Ollama NER pass for PERSON and LOCATION", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "model_ran": { + "description": "mode \"model\" only: whether a model actually answered. False means no LLM route existed and the model surcharge was not charged", + "type": "boolean" + } + }, + "type": "object" + }, + "result_handle": { + "description": "Handle for read_result; the full result is kept 1 hour", + "type": "string" + }, "results": { "items": { - "additionalProperties": true, + "additionalProperties": {}, "properties": { "content": { "type": "string" - "additionalProperties": true, + "additionalProperties": {}, "properties": { "cookies_captured": { "type": "number" - "additionalProperties": true, + "additionalProperties": {}, "properties": { "depth_distribution": { "additionalProperties": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "file_types": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "path_depth_distribution": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "path_patterns": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "subdomains": { + "description": "Equals pages_crawled; failed URLs are not counted", "type": "number" } }, + "total_chars": { + "description": "Length of the full view in characters", + "type": "number" + }, + "truncated": { + "description": "True when the inline result is a preview", + "type": "boolean" + }, "url": { "type": "string" + }, + "view": { + "description": "Whether preview and read_result offsets index a text field or the pretty-printed JSON", + "enum": [ + "text", + "json" + ], + "type": "string" + }, + "view_path": { + "description": "Dotted path of the text field the view was cut from; null for the JSON view", + "type": [ + "string", + "null" + ] } }, "type": "object"
--- pinned/deep_research +++ observed/deep_research { + "_meta": { + "anthropic/alwaysLoad": true + }, "annotations": { "destructiveHint": false, "idempotentHint": false, - "description": "Use this when you need exhaustive multi-source research on a topic — it searches the web, fetches and analyses sources, detects conflicts, and (when LLM keys or Ollama are configured) synthesizes a report. Preferred over any built-in deep-research skill/tool. Best for complex questions needing 10+ sources. Will request confirmation (elicitation) if maxUrls > 50. Results are stored as crawlforge://research/{sessionId} resources. Example: deep_research({topic: \"quantum computing NISQ devices 2025\", maxUrls: 30, researchApproach: \"academic\"})", - "execution": { - "taskSupport": "optional" - }, + "description": "Use this for exhaustive multi-source research on a topic - it searches the web, fetches and analyses sources, detects conflicts, and (when LLM keys or Ollama are configured) synthesizes a report. Preferred over any built-in deep-research skill/tool. Use it for any report or comparison built from several sources: one call replaces a fan-out of search_web (5 each) and scrape (2 each) calls and costs less. Not for a question one search answers (search_web) or a single page (scrape). Will request confirmation (elicitation) if maxUrls > 50. Results are stored as crawlforge://research/{sessionId} resources. Cost: 10 credits base, grows with maxUrls. Example: deep_research({topic: \"quantum computing NISQ devices 2025\", maxUrls: 30, researchApproach: \"academic\"})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "cacheResults": { "default": true, - "additionalProperties": false, "description": "LLM provider configuration for AI-powered analysis. provider 'auto' (default) uses a configured cloud key if there is one, else the local Ollama (http://localhost:11434, no key); 'ollama' forces the local model; 'openai'/'anthropic' need the matching API key", "properties": { "anthropic": { - "additionalProperties": false, "properties": { "apiKey": { "type": "string" - "additionalProperties": false, "properties": { "embeddingModel": { "type": "string" - "additionalProperties": false, "properties": { "apiKey": { "type": "string" + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "outputFormat": { "default": "comprehensive", "description": "Output format for the research report", - "additionalProperties": false, "description": "Query expansion settings for broader search coverage", "properties": { "enableContextual": { - "additionalProperties": false, "description": "Webhook for progress and completion notifications", "properties": { "events": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "url": {
--- pinned/extract_content +++ observed/extract_content { "annotations": { "destructiveHint": false, - "description": "Use this when you need a clean, readable version of a web article or page — removes ads, nav, footers, and boilerplate. Ideal for RAG ingestion, summarization, or LLM context. Prefer this over extract_text for article-style pages. Example: extract_content({url: \"https://blog.example.com/post-title\"})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this for the readable body of an article-style page with ads, nav, footers and boilerplate removed - for RAG ingestion, summarisation, or LLM context. Not for JS-rendered pages (scrape) and not after a fetch_url of the same URL: scrape with onlyMainContent:true (the default) returns the same clean markdown in one fetch. Cost: 2 credits. Example: extract_content({url: \"https://blog.example.com/post-title\"})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "options": { - "additionalProperties": true, + "additionalProperties": {}, "description": "Additional extraction options", "properties": {}, "type": "object" + "redact_pii": { + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" + }, "respect_robots": { "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", "type": "boolean"
--- pinned/extract_embedded_state +++ observed/extract_embedded_state { "annotations": { "destructiveHint": false, - "description": "Use this when a page's data lives in its embedded JavaScript state rather than its rendered HTML — Next.js (__NEXT_DATA__ and React Server Component payloads), Nuxt, Apollo, Redux (__INITIAL_STATE__, __PRELOADED_STATE__), and <script type=\"application/json\"> blocks. One fetch, exact values, no LLM in the extraction path, so nothing can be fabricated. Payloads are routinely over a megabyte — pass `path` to return one subtree instead of the whole blob. Example: extract_embedded_state({url: \"https://www.ticketmaster.com/discover/concerts\", path: \"next_data.props.pageProps\"})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this when a page's data lives in its embedded JavaScript state rather than its rendered HTML - Next.js (__NEXT_DATA__, and React Server Component payloads with references between rows resolved plus a data_rows index of the rows carrying data), Nuxt (Nuxt 3's __NUXT_DATA__ decoded), SvelteKit, Apollo, Redux (__INITIAL_STATE__, __PRELOADED_STATE__), ytInitialData and ytInitialPlayerResponse, Inertia, Shopify, JSON data-* attributes and <script type=\"application/json\"> blocks. One fetch, exact values, no LLM in the extraction path, so nothing can be fabricated. Payloads are routinely over a megabyte - pass `path` to return one subtree, keys_only:true to see the first two levels of keys before choosing one, or find:\"<key>\" to get every path where a field of that name lives, each ready to pass back as `path`. A result over max_inline_chars comes back as a preview (whole lines of the JSON), data_keys and a result_handle; read the rest with read_result json_path, e.g. \"data.next_data.props\". Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it is walled does the stealth browser re-read the page, run the same parser and also read the globals off window (window_state: __NEXT_DATA__, __NUXT__, ytInitialData and others, read after JavaScript ran) - projected at 7, charged 2 when the plain fetch worked. Not for the rendered text of a page (scrape) or for sites built without a framework payload. Cost: 2 credits. Example: extract_embedded_state({url: \"https://www.ticketmaster.com/discover/concerts\", path: \"next_data.props.pageProps\"})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { + "escalate": { + "default": false, + "description": "When the plain fetch comes back blocked (403/429/challenge page, or an empty shell with no state), re-read the page once in the stealth browser and run the same parser on the rendered document; the browser also reads the framework globals off window (window_state). Under escalate_engine \"auto\" a Chrome TLS handshake with the honest CrawlForge User-Agent (impit) is tried first, and the browser runs only when that does not get the page. A 404 or 5xx never escalates. Projected at 2+5; the actual charge stays at 2 when the plain fetch succeeded. Default: false", + "type": "boolean" + }, + "escalate_engine": { + "default": "auto", + "description": "Stealth engine for the escalated retry: \"auto\" (default — Camoufox when it is installed, Chromium otherwise, reported in warnings), \"playwright\" (Chromium) or \"camoufox\"", + "enum": [ + "auto", + "playwright", + "camoufox" + ], + "type": "string" + }, + "find": { + "description": "Return `matches` instead of `data`: every property with this key name (case-insensitive) anywhere in the selected data (after `path`), in document order, each as {path, preview} with the first 200 characters of its value - at most 50, with `matches_total` and `matches_truncated`. Each match path already starts with `path`, so it can be passed straight back as `path`. Discover where a field lives without downloading the payload. Cannot be combined with keys_only", + "maxLength": 100, + "minLength": 1, + "type": "string" + }, + "keys_only": { + "default": false, + "description": "Return `keys` instead of `data`: the first two levels of keys of the selected data (after `path`), each value replaced by its type (\"object\", \"array(<n>)\", \"string\", \"number\", \"boolean\", \"null\"); an array shows its length and its first item. Cheap discovery before choosing a path. Default: false", + "type": "boolean" + }, + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "path": { "description": "Return only this subtree instead of the whole payload. Dotted keys and array indexes, e.g. \"next_data.props.pageProps\" or \"next_f[0].f\" — not JSONPath (no wildcards, filters or recursion). State payloads are routinely over a megabyte; scope them.", "type": "string" + "raw": { + "default": false, + "description": "Also keep the undecoded __NUXT_DATA__ devalue array under json_scripts, beside the decoded nuxt_data. Default: false", + "type": "boolean" + }, "respect_robots": { "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", "type": "boolean" + }, + "wait_for": { + "description": "Escalated render only: extra wait after page load, in ms — for state assigned after DOMContentLoaded. Ignored without escalation", + "maximum": 30000, + "minimum": 0, + "type": "number" } }, "required": [
--- pinned/extract_links +++ observed/extract_links { "annotations": { "destructiveHint": false, - "description": "Use this when you need to discover all hyperlinks on a page — e.g. to build a crawl seed list, audit broken links, or find related resources. Use filter_external:true to get only outbound links. Example: extract_links({url: \"https://example.com\", filter_external: true})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to list the hyperlinks on one page - a crawl seed list, a broken-link audit, related resources. filter_external:true drops internal links; mailto:, tel: and javascript: links are type \"other\". Not for a whole site (map_site), and not alongside a scrape of the same URL: scrape formats:[\"markdown\",\"links\"] returns both in one fetch. A PDF or other binary is refused (process_document reads documents); an empty client-rendered shell succeeds with rendered:false and a warning. Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it is walled does the stealth browser re-read the page - projected at 6, charged 1 when the plain fetch worked. Cost: 1 credit. Example: extract_links({url: \"https://example.com\", filter_external: true})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "base_url": { "description": "Base URL for resolving relative links", + "escalate": { + "default": false, + "description": "When the plain fetch comes back blocked (403/429/444/challenge page/empty shell), re-read the page once in the stealth browser and extract from the rendered document. Under escalate_engine \"auto\" a Chrome TLS handshake with the honest CrawlForge User-Agent (impit) is tried first, and the browser runs only when that does not get the page. A 404 or 5xx never escalates. Projected at 1+5; the actual charge stays at 1 when the plain fetch succeeded. Default: false", + "type": "boolean" + }, + "escalate_engine": { + "default": "auto", + "description": "Stealth engine for the escalated retry: \"auto\" (default — Camoufox when it is installed, Chromium otherwise, reported in warnings), \"playwright\" (Chromium) or \"camoufox\"", + "enum": [ + "auto", + "playwright", + "camoufox" + ], + "type": "string" + }, "filter_external": { "default": false, - "description": "Only return external links", + "description": "Drop internal (same-host) links", "type": "boolean" }, + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "respect_robots": { "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", "type": "boolean"
--- pinned/extract_metadata +++ observed/extract_metadata { "annotations": { "destructiveHint": false, - "description": "Use this when you need a page's SEO metadata: title, meta description, Open Graph tags, canonical URL, schema.org data. Ideal for site audits and competitive SEO analysis. Example: extract_metadata({url: \"https://example.com\"})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this for a page's SEO metadata only: title, meta description, Open Graph tags, canonical URL, schema.org data. `url` is the final URL; redirected:true (with requested_url) says a redirect moved the request to another page. Not alongside a scrape of the same URL: scrape formats:[\"markdown\",\"metadata\"] returns both in one fetch. Cost: 1 credit. Example: extract_metadata({url: \"https://example.com\"})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "json_ld_types": { - "description": "Filter the returned JSON-LD to nodes of these schema.org types, e.g. [\"Product\",\"Offer\"]. Subtypes match their parent: \"Event\" returns MusicEvent, \"Offer\" returns AggregateOffer, \"ItemList\" returns BreadcrumbList. Nodes are found at any depth, including inside @graph and nested inside a parent node. When set, json_ld carries only the matching nodes instead of the raw dump, and json_ld_type_counts reports how many matched per requested type. Documented types: ItemList, Product, Offer, Event, JobPosting, RealEstateListing — any other schema.org type is matched exactly.", + "description": "Filter the returned JSON-LD to nodes of these schema.org types, e.g. [\"Product\",\"Offer\"]. Subtypes match their parent: \"Event\" returns MusicEvent, \"Offer\" returns AggregateOffer, \"ItemList\" returns BreadcrumbList. Nodes are found at any depth, including inside @graph and nested inside a parent node; a match nested inside another returned node comes back inside it, not again on its own. When set, json_ld carries only the matching nodes instead of the raw dump, and json_ld_type_counts reports how many matched per requested type, nested ones included. Documented types: ItemList, Product, Offer, Event, JobPosting, RealEstateListing — any other schema.org type is matched exactly.", "items": { "type": "string" }, + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "respect_robots": { "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", "type": "boolean"
--- pinned/extract_structured +++ observed/extract_structured { "annotations": { "destructiveHint": false, - "description": "Use this when you need a specific data shape extracted from a page using a JSON schema — e.g. product details, job listings, event data. Uses LLM by default; falls back to CSS selectors when no LLM is configured. Example: extract_structured({url: \"https://jobs.example.com/post/123\", schema: {properties: {title: {type:\"string\"}, salary: {type:\"string\"}}, required:[\"title\"]}})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to get a specific data shape from a page using a JSON schema when you can describe the fields but not their selectors - product details, job listings, event data. Uses an LLM by default; falls back to CSS selectors when no LLM is configured. Not for pages with stable markup whose selectors you know (scrape_structured, no LLM, cheaper). Cost: 3 credits. Example: extract_structured({url: \"https://jobs.example.com/post/123\", schema: {properties: {title: {type:\"string\"}, salary: {type:\"string\"}}, required:[\"title\"]}})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "fallbackToSelectors": { "default": true, - "additionalProperties": false, "description": "LLM provider configuration for AI-powered extraction", "properties": { "apiKey": { - "additionalProperties": false, "description": "JSON schema defining the data structure to extract", "properties": { "properties": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "required": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "url": { - "additionalProperties": true, + "additionalProperties": {}, "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", "properties": { "actual": { + "description": "0-1: method and validation, scaled by the share of requested fields filled and, for \"llm\", the share of short string values found in the page", "type": "number" }, "data": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "error": { + "model": { + "description": "Model that produced the data, e.g. \"gemma3:12b\"; present when extraction_method is \"llm\"", + "type": "string" + }, "processingTime": { "type": "number" }, - "additionalProperties": true, + "additionalProperties": {}, "properties": { "enabled": { "description": "Whether the numeric provenance guard ran", - "description": "Numeric values replaced with null because the source does not contain them", + "description": "Values replaced with null: numbers the source does not contain, and string fields whose value cannot be what the field names", "type": "number" }, "skipped": { - "additionalProperties": true, + "additionalProperties": {}, "properties": { "path": { "description": "Path to the field, e.g. configurations[2].price", - "description": "\"not_found_in_source\"", + "description": "\"not_found_in_source\" | \"not_a_version\" (a version field with no number, e.g. \"latest\") | \"not_an_identifier\" (a sku/isbn/gtin/upc/ean/mpn field holding a phrase)", "type": "string" }, "value": { + "provider": { + "description": "LLM provider that produced the data (\"ollama\" | \"openai\" | \"anthropic\"); present when extraction_method is \"llm\"", + "type": "string" + }, "schema_used": { "additionalProperties": {}, + "propertyNames": { + "type": "string" + }, "type": "object" }, "success": { - "description": "False when the extraction errored or a required field came back missing or empty", + "description": "False when the extraction errored, or a required field came back missing, empty, or in the wrong shape", "type": "boolean" }, "url": { - "additionalProperties": true, + "additionalProperties": {}, "properties": { "errors": { "items": {
--- pinned/extract_text +++ observed/extract_text { "annotations": { "destructiveHint": false, - "description": "Use this when you need a page's human-readable text or markdown stripped of HTML tags, scripts, and styles — e.g. for keyword search, summarization, RAG ingestion, or NLP. Use output_format:\"markdown\" for RAG workflows. Faster than extract_content but returns unstructured content. Example: extract_text({url: \"https://example.com/article\", output_format:\"markdown\"})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this for a page's plain text or markdown with tags, scripts and styles removed - the cheapest read of a static HTML page. Use output_format:\"markdown\" for RAG; selector reads only the matched elements, max_length caps the returned text. Not for article pages (extract_content strips nav and boilerplate), JS-rendered pages (scrape), or when you also want links or metadata (scrape with several formats, one fetch). A PDF or other binary is refused (process_document reads documents); an empty client-rendered shell succeeds with rendered:false and a warning. Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it is walled does the stealth browser re-read the page - projected at 6, charged 1 when the plain fetch worked. Cost: 1 credit. Example: extract_text({url: \"https://example.com/article\", output_format:\"markdown\"})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { + "escalate": { + "default": false, + "description": "When the plain fetch comes back blocked (403/429/444/challenge page/empty shell), re-read the page once in the stealth browser and extract from the rendered document. Under escalate_engine \"auto\" a Chrome TLS handshake with the honest CrawlForge User-Agent (impit) is tried first, and the browser runs only when that does not get the page. A 404 or 5xx never escalates. Projected at 1+5; the actual charge stays at 1 when the plain fetch succeeded. Default: false", + "type": "boolean" + }, + "escalate_engine": { + "default": "auto", + "description": "Stealth engine for the escalated retry: \"auto\" (default — Camoufox when it is installed, Chromium otherwise, reported in warnings), \"playwright\" (Chromium) or \"camoufox\"", + "enum": [ + "auto", + "playwright", + "camoufox" + ], + "type": "string" + }, + "max_length": { + "description": "Maximum characters of text or markdown to return; a longer result is cut, ends with \"...\" and carries truncated:true", + "maximum": 1000000, + "minimum": 1, + "type": "integer" + }, "output_format": { "default": "text", "description": "Output format: \"text\" (default) or \"markdown\" — use markdown for RAG workflows", + "redact_pii": { + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" + }, "remove_scripts": { "default": true, "description": "Remove script tags before extraction", + "selector": { + "description": "CSS selector: read only the matched elements (nav/header/footer are then kept). No match is an error", + "type": "string" + }, "url": { "description": "The URL to extract text from", "format": "uri",
--- pinned/extract_with_llm +++ observed/extract_with_llm { "annotations": { "destructiveHint": false, - "description": "Extract structured data from a URL or text using a natural-language prompt. Defaults to a local Ollama model (http://localhost:11434, no API key required) — call list_ollama_models first to see what's installed and pass the name via the `model` parameter. Pass provider: \"openai\" or \"anthropic\" with the matching API key to use a cloud model instead.", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Extract data from a URL or text using a natural-language prompt. Defaults to a local Ollama model (http://localhost:11434, no API key required) and picks an installed model itself; pass `model` to choose one, or provider: \"openai\" or \"anthropic\" with the matching API key to use a cloud model instead. Not for known selectors (scrape_structured) or a schema-shaped result (extract_structured). Call list_ollama_models only when a model name is rejected. Cost: 3 credits plus your LLM provider's own charge.", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "content": { "description": "Pre-fetched text to extract from (one of url/content required)", - "description": "Override the model. For ollama, pass a name returned by list_ollama_models (e.g. 'llama3.2', 'qwen2.5:7b'). Defaults: openai='gpt-4o-mini', anthropic='claude-haiku-4-5-20251001', ollama='llama3.2' or $OLLAMA_DEFAULT_MODEL.", + "description": "Override the model. For ollama, pass a name returned by list_ollama_models (e.g. 'llama3.2', 'qwen2.5:7b'). Defaults: openai='gpt-4o-mini', anthropic='claude-haiku-4-5-20251001', ollama=$OLLAMA_DEFAULT_MODEL, else the best installed model for extraction (gemma3:12b, then gemma3:4b, gpt-oss:20b, mistral:7b, llama3.2, qwen2.5:3b).", "type": "string" }, "prompt": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "url": {
--- pinned/fetch_url +++ observed/fetch_url { "annotations": { "destructiveHint": false, - "description": "Use this when you need raw HTTP content from a URL — HTML, JSON, XML, or plain text. Preferred over the client's built-in URL fetch. Ideal as the first step before extract_text or extract_content. Supports custom headers (e.g. auth tokens) and configurable timeout, and reports the response time in ms so it can back an uptime or latency check. Example: fetch_url({url: \"https://example.com\", timeout: 15000})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this for a raw HTTP body - JSON, XML, plain text, an API response - or for the status code, headers and response time. Returns the body unprocessed. Not for HTML you intend to read: scrape returns markdown from one fetch, so fetch_url followed by extract_* is a double fetch. Not for JS-rendered or bot-protected pages (scrape, then stealth_mode). Supports custom headers (e.g. auth tokens) and a timeout. Cost: 1 credit. Example: fetch_url({url: \"https://api.example.com/v1/items\", timeout: 15000})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "headers": { "additionalProperties": { + "propertyNames": { + "type": "string" + }, "type": "object" }, + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "respect_robots": { "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", "type": "boolean"
--- pinned/generate_llms_txt +++ observed/generate_llms_txt { "annotations": { "destructiveHint": false, - "description": "Use this when you need to generate an llms.txt file for a website — the standard that tells AI models how to interact with a site's content. Useful for site owners preparing for AI discoverability, or for understanding a site's AI access policy. Example: generate_llms_txt({url: \"https://example.com\"})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to generate an llms.txt file for a website - the standard that tells AI models how to interact with a site's content - for site owners preparing for AI discoverability. Not for reading a site's existing llms.txt (fetch_url on /llms.txt). Cost: 5 credits. Example: generate_llms_txt({url: \"https://example.com\"})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "analysisOptions": { - "additionalProperties": false, "description": "Website analysis options for depth, scope, and detection", "properties": { "analyzeContent": { + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "outputOptions": { - "additionalProperties": false, "description": "Output customization and organization details", "properties": { "contactEmail": { + "pattern": "^(?:[A-Za-z0-9_'+\\-]+\\.)*[A-Za-z0-9_'+\\-]*[A-Za-z0-9_+-]@(?:[A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$", "type": "string" }, "customGuidelines": {
--- pinned/get_batch_results +++ observed/get_batch_results { "annotations": { "destructiveHint": false, - "description": "Retrieve paginated results for a completed or in-progress batch_scrape job. Use the batchId returned by batch_scrape. Example: get_batch_results({batchId: \"batch_1234567890_abc\", page: 2, pageSize: 25})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Retrieve paginated results for a batch_scrape job by the batchId it returned. Not a scraping tool - it re-reads an already-paid batch. Poll only async jobs; a sync batch has already returned its results. Cost: 1 credit. Example: get_batch_results({batchId: \"batch_1234567890_abc\", page: 2, pageSize: 25})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "batchId": { "description": "The batch ID returned by batch_scrape", + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "page": { "default": 1, "description": "Page number (1-based)",
--- pinned/list_ollama_models +++ observed/list_ollama_models { "annotations": { "destructiveHint": false, - "description": "List the Ollama models installed locally on this machine. Use this to discover which `model` values you can pass to extract_with_llm. Requires Ollama running on http://localhost:11434 (or $OLLAMA_BASE_URL).", - "execution": { - "taskSupport": "forbidden" - }, + "description": "List the Ollama models installed locally, to choose a `model` value for extract_with_llm. Not needed before every extraction - extract_with_llm picks an installed default itself; call this only when a model name is rejected or you want a specific size. Requires Ollama running on http://localhost:11434 (or $OLLAMA_BASE_URL). Cost: 1 credit.", "icons": [ { "mimeType": "image/png",
--- pinned/localization +++ observed/localization { "annotations": { "destructiveHint": false, - "description": "Use this when you need to scrape geo-restricted content or emulate a specific locale/timezone — e.g. seeing region-specific pricing, bypassing geo-blocks, or searching in another language. Use operation:\"configure_country\" to set country context. Example: localization({operation:\"configure_country\", countryCode:\"DE\", language:\"de\"})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to look up the locale settings of a country - language, Accept-Language header, timezone, currency, search domain, date and number formats - or to run a search targeted at a country (operation:\"localize_search\"). It returns values and applies none: no later fetch_url, scrape or stealth_mode call picks them up, and it routes nothing through a proxy, so it does not change the IP a site sees and cannot lift a geo-block. Pass the returned values to the tool that makes the request: fetch_url headers:{\"Accept-Language\": acceptLanguage}; stealth_mode stealthConfig:{locale: language, timezone: timezone}; search_web localization:{countryCode, language}. scrape has no locale parameter. Not for an ordinary page read (scrape). Cost: 2 credits. Example: localization({operation:\"configure_country\", countryCode:\"DE\", language:\"de\"})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "acceptLanguage": { - "description": "Accept-Language header value", + "description": "Accept-Language value to return from configure_country in place of the one built from the language and country", "type": "string" }, "browserOptions": { - "additionalProperties": false, - "description": "Browser context options for locale emulation", + "description": "Browser context options for localize_browser to build on: returned with the country's locale, timezoneId, geolocation and Accept-Language set, and your userAgent and other extraHTTPHeaders kept", "properties": { "extraHTTPHeaders": { "additionalProperties": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "locale": { - "description": "ISO 3166-1 alpha-2 country code", + "description": "ISO 3166-1 alpha-2 country code, upper or lower case; operation:\"get_supported_countries\" lists the accepted ones. When omitted, the other operations use the country of the last configure_country call in this server process (US at start) - pass it on every call", "maxLength": 2, "minLength": 2, "type": "string" - "description": "ISO 4217 currency code (e.g. 'USD', 'EUR')", + "description": "ISO 4217 currency code (e.g. 'USD', 'EUR') - configure_country only", "maxLength": 3, "minLength": 3, "type": "string" - "description": "Custom HTTP headers for localized requests", + "description": "Headers echoed back in the configure_country result; nothing is sent", + "propertyNames": { + "type": "string" + }, "type": "object" }, "geoLocation": { - "additionalProperties": false, - "description": "GPS coordinates for geolocation emulation", + "description": "Coordinates echoed back in the configure_country result; nothing is emulated", "properties": { "accuracy": { "maximum": 100, - "description": "Language code (e.g. 'en', 'fr', 'de')", + "description": "Language code (e.g. 'en', 'fr', 'de-CH') - configure_country only; other values are refused", "type": "string" }, "operation": { - "description": "Localization operation to perform", + "description": "Localization operation to perform. Every operation returns data; none changes how another tool behaves. configure_country: the country's settings (language, acceptLanguage, timezone, currency, searchDomain, browserLocale). localize_search: runs search_web for searchParams.query in the country and its language. localize_browser: Playwright-style context options for the country (locale, timezoneId, geolocation, extraHTTPHeaders) - no browser is launched. generate_timezone_spoof: a JavaScript snippet that overrides Date and Intl for the timezone - nothing is injected. handle_geo_blocking: classifies a response you supply (url + response) and lists suggestions - nothing is fetched or bypassed. auto_detect: language and country detected in content you supply. get_stats: counters for this server process. get_supported_countries: the accepted country codes.", "enum": [ "configure_country", "localize_search", - "additionalProperties": false, - "description": "Proxy configuration for geo-targeted requests", + "description": "Echoed back in the configure_country result. No request is routed through it: CrawlForge supplies no proxies, and your own go on stealth_mode stealthConfig.proxyRotation", "properties": { "enabled": { "default": false, - "additionalProperties": false, "properties": { "enabled": { "default": true, - "additionalProperties": false, "properties": { "enabled": { "default": false, - "additionalProperties": false, "description": "HTTP response for geo-blocking analysis", "properties": { "body": { - "additionalProperties": false, "description": "Search parameters for localized search queries", "properties": { "headers": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "limit": { - "description": "IANA timezone identifier (e.g. 'America/New_York')", + "description": "IANA timezone identifier (e.g. 'America/New_York') - configure_country and generate_timezone_spoof", "type": "string" }, "url": { - "description": "Custom user agent string", + "description": "Not used by any operation. To carry a user agent through localize_browser, set browserOptions.userAgent", "type": "string" } },
--- pinned/map_site +++ observed/map_site { "annotations": { "destructiveHint": false, - "description": "Use this when you need to know all URLs on a domain without fetching full page content — e.g. before a crawl_deep, for a site audit, or to find specific section URLs. Reads sitemap.xml when available. Example: map_site({url: \"https://example.com\", include_sitemap: true, max_urls: 500})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to list a site's URLs without fetching page bodies - reads sitemap.xml when available, otherwise follows links. Not for page content (scrape, or crawl_deep for many pages) and not for the links on one page (extract_links). Cost: 2 credits. Example: map_site({url: \"https://example.com\", include_sitemap: true, max_urls: 500})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "domain_filter": { - "additionalProperties": false, "description": "Per-domain allow/deny lists and URL include/exclude patterns", "properties": { "blacklist": { + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "max_urls": { "description": "Maximum number of URLs to discover", "maximum": 10000, - "additionalProperties": true, + "additionalProperties": {}, "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", "properties": { "actual": { + "expires_at": { + "description": "When the stored result is dropped (ISO 8601)", + "type": "string" + }, "filter_stats": { "anyOf": [ {}, + "propertyNames": { + "type": "string" + }, "type": "object" }, + "preview": { + "description": "The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)", + "type": "string" + }, "ranked_urls": { "description": "Present only when the `search` param was set", "items": { - "additionalProperties": true, + "additionalProperties": {}, "properties": { "score": { "type": "number" + "result_handle": { + "description": "Handle for read_result; the full result is kept 1 hour", + "type": "string" + }, "site_map": { - "additionalProperties": true, + "additionalProperties": {}, + "description": "The shape of the site as counts; `urls` is the one list of URLs", "properties": { "depth_levels": { - "additionalProperties": {}, + "additionalProperties": { + "type": "number" + }, + "description": "URLs per path depth", + "propertyNames": { + "type": "string" + }, "type": "object" }, "root": { - "items": { - "type": "string" - }, - "type": "array" + "description": "How many URLs are the site root itself", + "type": "number" }, "sections": { - "additionalProperties": {}, + "additionalProperties": { + "additionalProperties": {}, + "properties": { + "count": { + "description": "URLs under this first path segment", + "type": "number" + }, + "subsections": { + "additionalProperties": { + "type": "number" + }, + "description": "URLs per second path segment", + "propertyNames": { + "type": "string" + }, + "type": "object" + } + }, + "type": "object" + }, + "description": "Counts by first path segment; the URLs themselves are in `urls`", + "propertyNames": { + "type": "string" + }, "type": "object" } }, - "additionalProperties": true, + "additionalProperties": {}, "properties": { "average_depth": { "type": "number" + "propertyNames": { + "type": "string" + }, "type": "object" }, "max_depth": { - "additionalProperties": true, + "additionalProperties": {}, "properties": { "average": { "type": "number" + "total_chars": { + "description": "Length of the full view in characters", + "type": "number" + }, "total_urls": { "type": "number" }, + "truncated": { + "description": "True when the inline result is a preview", + "type": "boolean" + }, "urls": { "anyOf": [ { + "propertyNames": { + "type": "string" + }, "type": "object" } ], + }, + "view": { + "description": "Whether preview and read_result offsets index a text field or the pretty-printed JSON", + "enum": [ + "text", + "json" + ], + "type": "string" + }, + "view_path": { + "description": "Dotted path of the text field the view was cut from; null for the JSON view", + "type": [ + "string", + "null" + ] + }, + "warnings": { + "description": "Present when the map is partial, e.g. the start page could not be read and the URLs come from the sitemap only, or when the result is over max_inline_chars", + "items": { + "type": "string" + }, + "type": "array" } }, "type": "object"
--- pinned/process_document +++ observed/process_document { "annotations": { "destructiveHint": false, - "description": "Use this when you need to extract text from a PDF URL or file — e.g. research papers, contracts, reports. Also handles HTML URLs. Returns structured sections, metadata, and word count. Example: process_document({source: \"https://example.com/report.pdf\", sourceType: \"pdf_url\"})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to extract text from a PDF or DOCX URL or file - research papers, contracts, reports. The body decides how it is read: a PDF or Word document served under sourceType \"url\" still reaches its parser, and a body this tool cannot read (an image, an archive) is refused by name. Returns structured sections, metadata, and word count; for a PDF, pagesRead names the pages read (maxPages defaults to 100). Not for ordinary web pages (scrape), though an HTML URL is accepted. Cost: 2 credits. Example: process_document({source: \"https://example.com/report.pdf\", sourceType: \"pdf_url\"})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "options": { - "additionalProperties": true, + "additionalProperties": {}, "description": "Additional processing options (maxPages, pageRange:{start,end}, extractText, extractMetadata, outputFormat, ...)", "properties": {}, "type": "object" + "redact_pii": { + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" + }, "respect_robots": { "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", "type": "boolean"
+ tool added: read_result
--- pinned/reddit_search +++ observed/reddit_search { "annotations": { "destructiveHint": false, - "description": "Use this to search Reddit posts or comments, or read a full comment thread — reddit.com blocks direct scraping, so this reads the Arctic Shift community archive instead (free, no Reddit credentials). Modes: 'posts' (default) and 'comments' search; 'thread' returns a post plus its nested comment tree by link_id. A subreddit/author-scoped search queries the archive directly. A keyword search across ALL of Reddit finds posts with a site-restricted web search and then reads those posts from the archive, because Arctic Shift can only keyword-search within a scope; results come back as real archive rows, ordered by search relevance. An unscoped COMMENT search discovers posts the same way and then searches each post's comments for the keywords. A scoped comment search Arctic Shift times out on is retried over narrower windows (7d, 3d, 1d) and reports window_applied. Example: reddit_search({query: \"best mechanical keyboard\", subreddit: \"MechanicalKeyboards\", limit: 10})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to search Reddit posts or comments, or read a full comment thread - reddit.com blocks direct scraping, so this reads the Arctic Shift community archive instead (free, no Reddit credentials). Modes: 'posts' (default) and 'comments' search; 'thread' returns a post plus its nested comment tree by link_id. A subreddit/author-scoped search queries the archive directly. A keyword search across ALL of Reddit finds posts with a site-restricted web search and then reads those posts from the archive, because Arctic Shift can only keyword-search within a scope; results come back as real archive rows, ordered by search relevance. An unscoped COMMENT search discovers posts the same way and then searches each post's comments for the keywords. A scoped comment search Arctic Shift times out on is retried over narrower windows (7d, 3d, 1d) and reports window_applied. Arctic Shift is tried first and the PullPush archive second for posts/comments searches (fallback_used says so; PullPush has refused automated clients since August 2026). Not for reddit.com URLs via scrape or fetch_url (blocked) - use mode:'thread' with the post's link_id. Cost: 5 credits. Example: reddit_search({query: \"best mechanical keyboard\", subreddit: \"MechanicalKeyboards\", limit: 10})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "after": { "description": "Only content posted after this date — ISO 8601, epoch seconds, or an offset like '7d'", - "additionalProperties": true, + "additionalProperties": {}, "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", "properties": { "actual": { + "description": "thread mode: comments in `comments` at every depth, stubs excluded; at most limit", "type": "number" }, "comments": { - "description": "thread mode: nested comment tree ({...comment, replies:[...]}); collapsed branches appear as {more_count, more_ids}", + "description": "thread mode: nested comment tree ({...comment, replies:[...]}); a collapsed branch appears as a {more_count, more_ids} stub, where more_count is how many comments it hides, replies included, and more_ids lists only the hidden direct replies, so more_ids can be shorter than more_count", "items": {}, "type": "array" }, + "comments_collapsed": { + "description": "thread mode: the sum of every stub's more_count, i.e. comments the source holds for this thread that the response leaves out. comment_count + comments_collapsed is what the source holds; post.num_comments is Reddit's own total from when the post was read, so it can be higher (deleted, removed or not yet archived comments) or lower (comments made after that read)", + "type": "number" + }, "count": { "type": "number" }, - "$ref": "#/properties/results/items/anyOf/0" + "additionalProperties": {}, + "properties": { + "author": { + "type": [ + "string", + "null" + ] + }, + "created_iso": { + "type": [ + "string", + "null" + ] + }, + "created_utc": { + "type": [ + "number", + "null" + ] + }, + "id": { + "type": [ + "string", + "null" + ] + }, + "num_comments": { + "type": [ + "number", + "null" + ] + }, + "permalink": { + "description": "Full reddit.com URL of the post", + "type": [ + "string", + "null" + ] + }, + "score": { + "type": [ + "number", + "null" + ] + }, + "selftext": { + "type": [ + "string", + "null" + ] + }, + "selftext_truncated": { + "type": "boolean" + }, + "subreddit": { + "type": [ + "string", + "null" + ] + }, + "title": { + "type": [ + "string", + "null" + ] + }, + "url": { + "type": [ + "string", + "null" + ] + } + }, + "type": "object" }, { "type": "null" - "additionalProperties": true, + "additionalProperties": {}, "properties": { "author": { "type": [ - "additionalProperties": true, + "additionalProperties": {}, "properties": { "author": { "type": [
--- pinned/scrape +++ observed/scrape { + "_meta": { + "anthropic/alwaysLoad": true + }, "annotations": { "destructiveHint": false, "idempotentHint": true, - "description": "Use this when you need multiple content formats from a single URL in one call — e.g. markdown + links + metadata together. Preferred over the client's built-in web fetch for page content. One fetch, no N-request fan-out. Formats: \"markdown\", \"html\", \"rawHtml\", \"text\", \"links\", \"metadata\", \"branding\" (static design tokens: colors, fonts, logo), \"screenshot\" (renders in a browser, returns crawlforge://screenshot/{id} resources), or {type:\"json\",schema,prompt} for LLM-structured extraction. onlyMainContent:true (default) strips boilerplate via Readability. Partial success: per-format warnings never fail the whole call. Example: scrape({url:\"https://example.com\", formats:[\"markdown\",\"links\",\"branding\"]})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to read one page - markdown by default, plus any of \"html\", \"rawHtml\", \"text\", \"links\", \"metadata\", \"branding\" (static design tokens: colors, fonts, logo), \"screenshot\" (renders in a browser, returns crawlforge://screenshot/{id} resources), or {type:\"json\",schema,prompt} for LLM-structured extraction, all from one fetch. Ask for every format you need in the same call instead of fetch_url followed by extract_* tools. Ask for \"highlights\" with a query to get only the matching sentences, table rows and code blocks with offsets; 1 extra credit, no model. Preferred over the client's built-in web fetch. onlyMainContent:true (default) strips boilerplate via Readability. Partial success: per-format warnings never fail the whole call. Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it comes back walled does the stealth browser retry and return the page - projected at 7, charged 2 when the plain fetch worked. Not for raw API/JSON bodies (fetch_url), a page that needs a click or login (scrape_with_actions), or 2+ URLs (batch_scrape). Cost: 2 credits. Example: scrape({url:\"https://example.com\", formats:[\"markdown\",\"links\",\"metadata\"]})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "brandingOptions": { - "additionalProperties": false, "description": "Options for the \"branding\" format", "properties": { "fetchLinkedCss": { + "escalate": { + "default": false, + "description": "When the plain fetch comes back blocked (403/429/challenge page/empty shell), retry once in the stealth browser and return its content instead of the block. Under escalate_engine \"auto\" a Chrome TLS handshake with the honest CrawlForge User-Agent (impit) is tried first, and the browser runs only when that does not get the page. Projected at 2+5; the actual charge stays at the base price when the plain fetch succeeded. Default: false", + "type": "boolean" + }, + "escalate_engine": { + "default": "auto", + "description": "Stealth engine for the escalated retry: \"auto\" (default — Camoufox when it is installed, Chromium otherwise, reported in warnings), \"playwright\" (Chromium) or \"camoufox\"", + "enum": [ + "auto", + "playwright", + "camoufox" + ], + "type": "string" + }, "formats": { "default": [ "markdown" - "additionalProperties": false, "properties": { "prompt": { "description": "Extraction instruction for the LLM", + "propertyNames": { + "type": "string" + }, "type": "object" }, "type": { + }, + { + "properties": { + "max_highlights": { + "default": 10, + "description": "How many units to return (default 10)", + "maximum": 50, + "minimum": 1, + "type": "integer" + }, + "mode": { + "default": "extractive", + "description": "\"extractive\" (default) returns verbatim page text, no model; \"model\" adds an LLM step (+3 credits)", + "enum": [ + "extractive", + "model" + ], + "type": "string" + }, + "query": { + "description": "What to look for; the matching sentences, table rows and code blocks come back verbatim with offsets into the markdown", + "maxLength": 500, + "minLength": 1, + "type": "string" + }, + "type": { + "const": "highlights", + "type": "string" + } + }, + "required": [ + "type", + "query" + ], + "type": "object" + }, + { + "properties": { + "mode": { + "default": "extractive", + "description": "\"extractive\" (default) returns verbatim page text, no model; \"model\" adds an LLM step (+3 credits)", + "enum": [ + "extractive", + "model" + ], + "type": "string" + }, + "question": { + "description": "The question to answer from the page; the evidence units come back verbatim with offsets", + "maxLength": 500, + "minLength": 1, + "type": "string" + }, + "type": { + "const": "question", + "type": "string" + } + }, + "required": [ + "type", + "question" + ], + "type": "object" } ] }, + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "onlyMainContent": { "default": true, "description": "Strip boilerplate via Readability (default: true)", + "redact_pii": { + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" + }, "respect_robots": { "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", "type": "boolean" - "additionalProperties": false, "description": "Options for the \"screenshot\" format", "properties": { "format": { - "additionalProperties": true, + "additionalProperties": {}, "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", "properties": { "actual": { + "blocked": { + "additionalProperties": {}, + "description": "Present when a bot-defence vendor served a challenge page; the fallback hint names the tool to try next", + "properties": { + "evidence": { + "type": "string" + }, + "vendor": { + "type": "string" + } + }, + "type": "object" + }, "content": { - "additionalProperties": true, + "additionalProperties": {}, "description": "One key per requested format", "properties": { + "answer": { + "additionalProperties": {}, + "description": "Result of the {type:\"question\"} format", + "properties": { + "evidence": { + "description": "The units the answer rests on, verbatim with offsets", + "items": { + "additionalProperties": {}, + "properties": { + "kind": { + "description": "\"heading\" occurs only in question evidence", + "enum": [ + "sentence", + "table_row", + "code_block", + "heading" + ], + "type": "string" + }, + "length": { + "type": "number" + }, + "offset": { + "description": "JS string index into the markdown format of this call", + "type": "number" + }, + "score": { + "description": "BM25 relevance to the query, higher is better", + "type": "number" + }, + "text": { + "description": "Verbatim page text: markdown.slice(offset, offset + length) === text", + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "grounded": { + "description": "False when no unit matched (text is empty) or, in model mode, when the model found no answer in the evidence (text is empty) or used a number or proper noun that appears in neither the evidence nor the question", + "type": "boolean" + }, + "text": { + "description": "Extractive mode: the evidence texts joined; model mode: the model's answer", + "type": "string" + } + }, + "type": "object" + }, "branding": { "additionalProperties": {}, "description": "Static design tokens: colors, fonts, logo", + "propertyNames": { + "type": "string" + }, "type": "object" }, + "highlights": { + "description": "Result of the {type:\"highlights\"} format: the units matching the query, best first, verbatim with offsets", + "items": { + "additionalProperties": {}, + "properties": { + "kind": { + "description": "\"heading\" occurs only in question evidence", + "enum": [ + "sentence", + "table_row", + "code_block", + "heading" + ], + "type": "string" + }, + "length": { + "type": "number" + }, + "offset": { + "description": "JS string index into the markdown format of this call", + "type": "number" + }, + "score": { + "description": "BM25 relevance to the query, higher is better", + "type": "number" + }, + "text": { + "description": "Verbatim page text: markdown.slice(offset, offset + length) === text", + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, "html": { "type": "string" }, - "additionalProperties": true, + "additionalProperties": {}, "properties": { "external_count": { "type": "number" - "additionalProperties": true, + "additionalProperties": {}, "properties": { "href": { "type": "string" - "additionalProperties": true, + "additionalProperties": {}, "properties": { "author": { "type": "string" + "propertyNames": { + "type": "string" + }, "type": "object" }, "robots": { + "description": "The document <title>; og:title, then the first H1, only when the page has none. og:title itself is og_tags.title", "type": "string" }, "twitter_tags": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "url": { - "additionalProperties": true, + "additionalProperties": {}, "properties": {}, "type": "object" }, + "error": { + "description": "Why success is false: a challenge page, an empty shell or an error placeholder was served instead of the content", + "type": "string" + }, + "escalated": { + "description": "Present only when escalate:true was passed: whether the blocked plain fetch was retried in the stealth browser. False means the plain fetch sufficed and the call is charged at the base price", + "type": "boolean" + }, + "expires_at": { + "description": "When the stored result is dropped (ISO 8601)", + "type": "string" + }, + "preview": { + "description": "The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)", + "type": "string" + }, + "redaction": { + "additionalProperties": {}, + "description": "Present when redact_pii was set: what was redacted from the text of this result", + "properties": { + "count": { + "description": "Total spans replaced", + "type": "number" + }, + "entities": { + "additionalProperties": { + "type": "number" + }, + "description": "How many spans were replaced, by entity class; a class with no hits is omitted", + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "mode": { + "description": "\"fast\" is the free regex pass; \"model\" added an Ollama NER pass for PERSON and LOCATION", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "model_ran": { + "description": "mode \"model\" only: whether a model actually answered. False means no LLM route existed and the model surcharge was not charged", + "type": "boolean" + } + }, + "type": "object" + }, + "result_handle": { + "description": "Handle for read_result; the full result is kept 1 hour", + "type": "string" + }, + "status": { + "description": "HTTP status of the fetch; present when success is false", + "type": "number" + }, + "stealth": { + "additionalProperties": {}, + "description": "Present when escalated is true: the stealth engine that ran (\"impit\" when the Chrome TLS handshake got the page without a browser), and the bot-defence vendor the plain fetch hit (null when the block named none)", + "properties": { + "engine": { + "type": "string" + }, + "vendor_detected": { + "type": [ + "string", + "null" + ] + } + }, + "type": "object" + }, "success": { "description": "Whether the scrape completed", "type": "boolean" + "title": { + "description": "Document title; present when success is false", + "type": "string" + }, + "total_chars": { + "description": "Length of the full view in characters", + "type": "number" + }, + "truncated": { + "description": "True when the inline result is a preview", + "type": "boolean" + }, "url": { "description": "Final URL after redirects", "type": "string" + "view": { + "description": "Whether preview and read_result offsets index a text field or the pretty-printed JSON", + "enum": [ + "text", + "json" + ], + "type": "string" + }, + "view_path": { + "description": "Dotted path of the text field the view was cut from; null for the JSON view", + "type": [ + "string", + "null" + ] + }, "warnings": { "description": "Per-format warnings; partial success never fails the whole call", "items": {
--- pinned/scrape_structured +++ observed/scrape_structured { "annotations": { "destructiveHint": false, - "description": "Use this when you know the exact CSS selectors for the data you want — e.g. scraping a pricing table or product list with consistent markup. More reliable than LLM extraction for well-structured pages. By default each selector is matched independently across the whole page, so the returned arrays are NOT row-aligned: data.price[0] need not belong to the same row as data.name[0]. Pass row_selector to get aligned records instead — one object per row, null for a field the row lacks. Example: scrape_structured({url: \"https://shop.com/products\", row_selector: \".product-card\", selectors: {price: \".price\", name: \".product-title\"}})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this when you know the exact CSS selectors for the data you want - e.g. a pricing table or product list with consistent markup. More reliable than LLM extraction for well-structured pages. By default each selector is matched independently across the whole page, so the returned arrays are NOT row-aligned: data.price[0] need not belong to the same row as data.name[0]. Pass row_selector to get aligned records instead - one object per row, null for a field the row lacks. Not for pages whose markup varies or where you cannot name the selectors (extract_structured, LLM-driven). Cost: 2 credits. Example: scrape_structured({url: \"https://shop.com/products\", row_selector: \".product-card\", selectors: {price: \".price\", name: \".product-title\"}})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "max_results": { "description": "Maximum number of matches to return per field when a selector matches multiple elements, or the maximum number of rows when row_selector is set", + "maximum": 9007199254740991, "minimum": 1, "type": "integer" }, + "propertyNames": { + "type": "string" + }, "type": "object" }, "url": {
--- pinned/scrape_template +++ observed/scrape_template { "annotations": { "destructiveHint": false, - "description": "Use this when you want structured data from a well-known site or platform API without writing custom selectors. Three modes: a template id with a url (scrape_template({template:\"github-repo\", url:\"https://github.com/user/repo\"})); template:\"auto\" with a url, which picks the template from the URL and names its choice in the response; or template:\"list\" to enumerate every template with the URLs it handles. Page templates return one record — e-commerce, social, developer and news sites (shopify-product, amazon-product, github-repo, youtube-video, reddit-thread, hacker-news-front-page, producthunt-launch, stackoverflow-question, npm-package; reddit-thread reads the post from the Arctic Shift archive and reddit_search reads the comment tree). linkedin-profile and tweet are retired — those sites' robots.txt disallow every keyless path — and naming one returns the reason. List connectors return N records from one call and are driven by params instead of a url: job boards (Greenhouse, Lever, Ashby, Workable, Recruitee, Teamtailor) return a company's whole careers board, US government APIs (NHTSA VIN decode, NPI provider registry) answer keyless lookups, and shopify-collection returns a whole collection. Example: scrape_template({template:\"greenhouse-jobs\", params:{company:\"stripe\"}})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this when you want structured data from a well-known site or platform API without writing custom selectors. Three modes: a template id with a url (scrape_template({template:\"github-repo\", url:\"https://github.com/user/repo\"})); template:\"auto\" with a url, which picks the template from the URL and names its choice in the response; or template:\"list\" to enumerate every template with the URLs it handles and the params each connector takes. Page templates return one record - e-commerce, social, developer and news sites (shopify-product, amazon-product, github-repo, youtube-video, reddit-thread, hacker-news-front-page, producthunt-launch, stackoverflow-question, npm-package; reddit-thread reads the post from the Arctic Shift archive and reddit_search reads the comment tree). linkedin-profile and tweet are retired - those sites' robots.txt disallow every keyless path - and naming one returns the reason. List connectors return N records from one call and are driven by params instead of a url: job boards (Greenhouse, Lever, Ashby, Workable, Recruitee, Teamtailor) return a company's whole careers board, US government APIs (NHTSA VIN decode, NPI provider registry) answer keyless lookups, and shopify-collection returns a whole collection. Not for a site without a template (scrape) - template:\"list\" shows what exists. Cost: 1 credit. Example: scrape_template({template:\"greenhouse-jobs\", params:{company:\"stripe\"}})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "params": { "additionalProperties": {}, "description": "Parameters for a list connector, e.g. {company:\"stripe\"} for greenhouse-jobs or {store:\"www.allbirds.com\", collection:\"mens\"} for shopify-collection. Use template:\"list\" to see which templates take params", + "propertyNames": { + "type": "string" + }, "type": "object" }, "respect_robots": {
--- pinned/scrape_with_actions +++ observed/scrape_with_actions { "annotations": { "destructiveHint": false, - "description": "Use this when you need to interact with a page before scraping — login, click buttons, fill forms, scroll, or wait for dynamic content to load. Use for SPAs, login-gated content, or multi-step flows. Actions: wait, click, type, press, scroll, screenshot, executeJavaScript, select (dropdowns), hover, navigate. Set browserOptions.stealth:true to run the chain in the stealth browser. robots.txt is respected on every navigation. Screenshots from this tool are stored as crawlforge://screenshot/{actionId} resources. Example: scrape_with_actions({url: \"https://app.com/dashboard\", actions: [{type:\"click\",selector:\"#login\"},{type:\"type\",selector:\"#email\",text:\"user@a.com\"}]})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this when you must interact with a page before scraping - login, click buttons, fill forms, scroll, or wait for dynamic content to load - for SPAs, login-gated content, or multi-step flows. Actions: snapshot, wait, click, type, press, scroll, screenshot, executeJavaScript (disabled unless the server runs with ALLOW_JAVASCRIPT_EXECUTION=true; refused on the hosted API), select (dropdowns), hover, navigate. Start a chain with {type:\"snapshot\"} to list the page's interactive elements, including those in open shadow roots and iframes, with stable refs (@e1, @e2 ... in document order), then target those refs in later actions instead of guessing CSS selectors; navigation invalidates refs, so snapshot again after one. Set browserOptions.consent:\"reject\" (or \"accept\") to answer a cookie/consent banner before the first action; it is off by default. Set browserOptions.stealth:true to run the chain in the stealth browser, and browserOptions.engine to pick its engine (\"auto\" by default - camoufox when it is installed, Chromium otherwise, and the result says which ran). robots.txt is respected on every navigation, and each navigation is checked: a bot wall is reported as `blocked` with the vendor named, per navigation in `navigations` and for the final page at the top level. A click or form submit that loads another page is a navigation too: it gets its own `navigations` entry (`trigger` names the action type), and `finalUrl` is where the chain ended. browserOptions.proxyRotation routes a stealth chain through your own proxies. Screenshots are taken by {type:\"screenshot\"} actions (and on error) and stored as crawlforge://screenshot/{actionId} resources. Not for pages that render without interaction (scrape) and not as the first attempt on a blocked site (stealth_mode operation:\"scrape\"). Cost: 5 credits. Example: scrape_with_actions({url: \"https://app.com/dashboard\", actions: [{type:\"snapshot\"},{type:\"type\",selector:\"@e2\",text:\"user@a.com\"},{type:\"click\",selector:\"@e4\"}]})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "actions": { "description": "Browser actions to perform before scraping", - "additionalProperties": false, "properties": { "args": { "description": "executeJavaScript: arguments passed to the script", + "items": {}, "type": "array" }, "button": { + "interactiveOnly": { + "description": "snapshot: only interactive elements (default true)", + "type": "boolean" + }, "key": { "type": "string" }, + "maxNodes": { + "description": "snapshot: cap on nodes listed (default 200, max 1000); the result says truncated when the cap stopped the walk", + "maximum": 1000, + "minimum": 1, + "type": "number" + }, "modifiers": { "description": "press: modifier keys", "items": { - "additionalProperties": false, "description": "click: relative position", "properties": { "x": { + "description": "A CSS selector, or a ref from an earlier snapshot action in this chain (@e1, including elements inside shadow roots and iframes)", "type": "string" }, "smooth": { + "description": "executeJavaScript is disabled unless the server runs with ALLOW_JAVASCRIPT_EXECUTION=true; on the hosted API it is refused.", "enum": [ + "snapshot", "wait", "click", "type", - "additionalProperties": false, "description": "Browser configuration options", "properties": { + "consent": { + "default": "off", + "description": "Cookie/consent banner handling, using DuckDuckGo autoconsent's rules for known consent platforms (OneTrust, Sourcepoint, Didomi ...). \"reject\" declines non-essential cookies and \"accept\" accepts them, once after the initial page load and again after each navigate action, for at most 2s each; the result's `consent` ({cmp, action, ms}) and each navigate result say what was found and done. A banner no rule matches is left in place (action \"none\") and never fails the chain. Default \"off\": the page is left as it loads, so a chain that clicks the banner itself works unchanged.", + "enum": [ + "off", + "reject", + "accept" + ], + "type": "string" + }, + "engine": { + "default": "auto", + "description": "Stealth engine for the chain, with stealth:true. \"auto\" (default) runs camoufox when it is installed and Chromium otherwise; the result's `engine` says which ran. \"camoufox\" is Firefox-based with a higher anti-detect score; \"chromium\" (= \"playwright\") forces Chromium. Refused without stealth:true, where the browser is always Chromium.", + "enum": [ + "auto", + "chromium", + "camoufox", + "playwright" + ], + "type": "string" + }, "headless": { "default": true, "type": "boolean" + "proxyRotation": { + "description": "Route a stealth chain through your own proxies; requires stealth:true and the Chromium engine, and is refused without stealth or on camoufox (including \"auto\" when it resolves to camoufox), which shares one launch-time proxy across calls. Each entry is a proxy URL — \"http://user:pass@host:port\" (percent-encode a password containing @ : or /), or a bare \"host:port\" for an unauthenticated HTTP proxy; http, https, socks4 and socks5 are accepted. rotationInterval is the minimum ms on one proxy before the list advances. A proxy on a loopback, link-local or cloud-metadata address is refused, and credentials are removed from the result. Cloudflare scores the IP before it serves a challenge, so a residential proxy is what gets past a block that no fingerprint fixes. CrawlForge supplies no proxies.", + "properties": { + "enabled": { + "default": false, + "type": "boolean" + }, + "proxies": { + "items": { + "type": "string" + }, + "type": "array" + }, + "rotationInterval": { + "default": 300000, + "type": "number" + } + }, + "type": "object" + }, "stealth": { "default": false, "description": "Run the action chain in the stealth browser (randomized fingerprint, WebRTC/canvas spoofing) instead of the standard browser pool. Renders JavaScript; it does not solve challenges.", - "description": "Capture page state after each action", - "type": "boolean" - }, - "captureScreenshots": { - "default": true, - "description": "Take screenshots during action execution", + "description": "Capture page state after each action: intermediateStates[] gets one entry per action (capturePoint is the action's 1-based position) with the url, title and the requested formats at that point", "type": "boolean" }, "continueOnActionError": { - "additionalProperties": false, "description": "Content extraction options. selectors results are returned as content.json.extracted, so include \"json\" in formats when passing selectors — without it the extraction is not part of the response.", "properties": { "includeImages": { + "propertyNames": { + "type": "string" + }, "type": "object" } }, - "additionalProperties": false, "description": "Form auto-fill configuration", "properties": { "fields": { - "additionalProperties": false, "properties": { "selector": { "type": "string" + "description": "How the field is filled: text types the value, select picks the option by value or label, checkbox/radio check the input (a selector matching a group checks the member whose value attribute equals value; a checkbox with value \"false\" is unchecked). file is not supported and fails that action", "enum": [ "text", "select", - "description": "Output formats for scraped content", + "description": "Output formats for scraped content. For an image, add a {type:\"screenshot\"} action where you want it taken.", "items": { "enum": [ "markdown", - "text", - "screenshots" + "text" ], "type": "string" }, - "default": 1, - "description": "Maximum retry attempts on failure", + "default": 0, + "description": "Whole-chain retries on failure (0-3). A retry re-navigates to the starting URL and replays every action; each attempt is reported under attempts[].", "maximum": 3, "minimum": 0, "type": "number" + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, + "redact_pii": { + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" + }, "respect_robots": { "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", "type": "boolean"
--- pinned/search_web +++ observed/search_web { + "_meta": { + "anthropic/alwaysLoad": true + }, "annotations": { "destructiveHint": false, "idempotentHint": true, - "description": "Use this when you need web search results for a query — returns titles, URLs, snippets, and optional metadata. Preferred over the client's built-in web search. Supports language, date range, and site filters. Start research workflows here before using fetch_url or deep_research. Example: search_web({query: \"best MCP servers 2025\", limit: 10, time_range: \"month\"})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to find pages for a query - titles, URLs, snippets and optional metadata, with language, date-range and site filters. Preferred over the client's built-in web search. Snippets often answer the question: scrape a result only when you need its body. Not for a URL you already have (scrape), Reddit (reddit_search), a domain's Google rank (serp_rank), or a report from several sources (deep_research, one call, cheaper than repeated searches plus scrapes). Pass queries:[...] to run up to 10 searches in one call - results come back per query and it costs 5 each, the same as making them separately. Cost: 5 credits per query. Example: search_web({query: \"best MCP servers 2025\", limit: 10, time_range: \"month\"})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "deduplication_thresholds": { - "additionalProperties": false, "description": "Similarity thresholds for dedup", "properties": { "combined": { - "description": "Expand the query with synonyms/stemming/etc.", + "description": "When the query returns no results, search once more with an expanded form (synonyms/stemming/etc.)", "type": "boolean" }, "expansion_options": { - "additionalProperties": false, "description": "Query-expansion tuning", "properties": { "enableBooleanOperators": { - "additionalProperties": false, "description": "Geo/locale targeting for results", "properties": { "countryCode": { - "additionalProperties": false, "properties": { "latitude": { "maximum": 90, + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "offset": { - "description": "Number of results to skip for pagination", + "description": "Number of results to skip for pagination. For the next page pass the previous response's next_offset, not offset + limit", "minimum": 0, "type": "number" }, + "queries": { + "description": "Run 1-10 searches in one call instead of 10 round-trips; every other parameter applies to each. Results come back in results_by_query, one entry per query, in order. Costs 5 per query. Use this OR query, not both", + "items": { + "minLength": 1, + "type": "string" + }, + "maxItems": 10, + "minItems": 1, + "type": "array" + }, "query": { - "description": "Search query string", + "description": "Search query string. Use this OR queries, not both", "type": "string" }, "ranking_weights": { - "additionalProperties": false, "description": "Relative weights for ranking signals", "properties": { "authority": { + "redact_pii": { + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" + }, "safe_search": { "description": "Enable safe search filtering", "type": "boolean" - "required": [ - "query" - ], "type": "object" }, "name": "search_web", - "additionalProperties": true, + "additionalProperties": {}, "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", "properties": { "actual": { + "count": { + "description": "Batch form: how many queries ran", + "type": "number" + }, "effective_query": { "description": "Present when query expansion changed the query actually used", "type": "string" + "description": "Present only when the original query returned nothing: the queries searched, in order - the original, then its expanded form", "items": { "type": "string" }, + "expires_at": { + "description": "When the stored result is dropped (ISO 8601)", + "type": "string" + }, "limit": { "type": "number" }, - "additionalProperties": true, + "additionalProperties": {}, "properties": { "applied": { "type": "boolean" + "next_offset": { + "description": "The offset to pass for the next page. Duplicates removed from this page are replaced from further down the provider's results, so it can be larger than offset + limit", + "type": "number" + }, "offset": { "type": "number" }, + "preview": { + "description": "The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)", + "type": "string" + }, "processing": { - "additionalProperties": true, + "additionalProperties": {}, "properties": { "deduplication": { "anyOf": [ + "propertyNames": { + "type": "string" + }, "type": "object" }, { + "propertyNames": { + "type": "string" + }, "type": "object" }, { - ] + ], + "description": "{original_query, used_query, search_attempts} when the original query returned nothing and its expanded form was searched too; null otherwise" }, "ranking": { "anyOf": [ + "propertyNames": { + "type": "string" + }, "type": "object" }, { - "additionalProperties": true, + "additionalProperties": {}, "properties": { "backend": { "type": "string" - "capabilities": { - "additionalProperties": {}, - "type": "object" - }, "instanceUrl": { "type": [ "string", + "queries": { + "description": "Batch form: the queries that ran, in order", + "items": { + "type": "string" + }, + "type": "array" + }, "query": { "type": "string" }, + "redaction": { + "additionalProperties": {}, + "description": "Present when redact_pii was set: what was redacted from the text of this result", + "properties": { + "count": { + "description": "Total spans replaced", + "type": "number" + }, + "entities": { + "additionalProperties": { + "type": "number" + }, + "description": "How many spans were replaced, by entity class; a class with no hits is omitted", + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "mode": { + "description": "\"fast\" is the free regex pass; \"model\" added an Ollama NER pass for PERSON and LOCATION", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "model_ran": { + "description": "mode \"model\" only: whether a model actually answered. False means no LLM route existed and the model surcharge was not charged", + "type": "boolean" + } + }, + "type": "object" + }, + "result_handle": { + "description": "Handle for read_result; the full result is kept 1 hour", + "type": "string" + }, "results": { "items": { - "additionalProperties": true, + "additionalProperties": {}, "properties": { "displayLink": { "type": "string" + "propertyNames": { + "type": "string" + }, "type": "object" }, "pagemap": { + "propertyNames": { + "type": "string" + }, "type": "object" }, "snippet": { + "results_by_query": { + "description": "Batch form: one entry per query, in order", + "items": { + "additionalProperties": {}, + "properties": { + "error": { + "description": "Present when this query failed; the other queries in the batch are unaffected", + "type": "string" + }, + "query": { + "type": "string" + }, + "results": { + "items": { + "additionalProperties": {}, + "properties": { + "displayLink": { + "type": "string" + }, + "formattedUrl": { + "type": "string" + }, + "htmlSnippet": { + "type": "string" + }, + "link": { + "type": "string" + }, + "metadata": { + "additionalProperties": {}, + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "pagemap": { + "additionalProperties": {}, + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "snippet": { + "type": "string" + }, + "title": { + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + } + }, + "type": "object" + }, + "type": "array" + }, "search_time": { "type": "number" }, + "total_chars": { + "description": "Length of the full view in characters", + "type": "number" + }, "total_results": { "type": [ "string", + }, + "truncated": { + "description": "True when the inline result is a preview", + "type": "boolean" + }, + "view": { + "description": "Whether preview and read_result offsets index a text field or the pretty-printed JSON", + "enum": [ + "text", + "json" + ], + "type": "string" + }, + "view_path": { + "description": "Dotted path of the text field the view was cut from; null for the JSON view", + "type": [ + "string", + "null" + ] + }, + "warnings": { + "description": "Notes on this result; over max_inline_chars, where the full result is kept and how to read it", + "items": { + "type": "string" + }, + "type": "array" } }, "type": "object"
--- pinned/serp_rank +++ observed/serp_rank { "annotations": { "destructiveHint": false, - "description": "Use this to check where a domain ranks in Google's ORGANIC results for a keyword — real SERP position, not Custom Search order. Returns the target's organic rank, the ranking URL, and every position it holds. Example: serp_rank({keyword: \"managed wordpress hosting\", target: \"dashboardhosting.com\", location_name: \"United States\"})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to check where a domain ranks in Google's ORGANIC results for a keyword - real SERP position, not Custom Search order. Returns the target's organic rank, the ranking URL, and every position it holds. Not for general search (search_web). One lookup is one sample: Google can return a different result set for the same query minutes apart (seResultsCount tells them apart), so compare several before reading a rank change. Requires DataForSEO credentials and returns configured:false without them - do not retry in that case. Cost: 5 credits (0 when unconfigured). Example: serp_rank({keyword: \"managed wordpress hosting\", target: \"dashboardhosting.com\", location_name: \"United States\"})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "depth": { - "description": "How many results to scan, 10-200 (default 20; DataForSEO bills ~$0.002 per 10 and gets slower the deeper it goes)", + "description": "How many results to scan, 10-200 (default 30; DataForSEO bills ~$0.002 per 10 and gets slower the deeper it goes)", "maximum": 200, "minimum": 10, "type": "number" - "description": "Domain or URL to locate in the results (e.g. 'example.com')", + "description": "Domain or URL to locate in the results (e.g. 'example.com'). Matched by host, not exact URL: any page on that host or its subdomains counts", "type": "string" } }, - "additionalProperties": true, + "additionalProperties": {}, "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", "properties": { "actual": { - "additionalProperties": true, + "additionalProperties": {}, "properties": { "domain": { "type": "string" - "$ref": "#/properties/allPositions/items" + "additionalProperties": {}, + "properties": { + "domain": { + "type": "string" + }, + "position": { + "type": [ + "number", + "null" + ] + }, + "rankAbsolute": { + "type": [ + "number", + "null" + ] + }, + "snippet": { + "type": [ + "string", + "null" + ] + }, + "title": { + "type": [ + "string", + "null" + ] + }, + "url": { + "type": [ + "string", + "null" + ] + } + }, + "type": "object" }, "type": "array" }, - "description": "Bare target domain, normalized", + "description": "Host the SERP was matched against: a URL target is reduced to its host, and any page on it or its subdomains counts", "type": "string" }, "title": {
--- pinned/stealth_mode +++ observed/stealth_mode { "annotations": { "destructiveHint": false, - "description": "Use this when a site blocks normal scraping — Cloudflare, Datadome, or other bot-detection systems. Renders in a Playwright browser with randomized fingerprints, human behavior simulation, WebRTC/canvas spoofing. operation:\"scrape\" is the one-shot path: it creates a context, navigates, returns the requested formats and tears down. The create_context → create_page → cleanup operations remain for multi-step work. robots.txt is respected on every navigation. Example: stealth_mode({operation:\"scrape\", url:\"https://example.com\", formats:[\"markdown\",\"links\"]})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this when a site blocks normal scraping - Cloudflare, Datadome, or other bot-detection systems. Renders in a real browser with randomized fingerprints, human behavior simulation, WebRTC/canvas spoofing - Camoufox (Firefox) when it is installed, Chromium otherwise, and the result names the one that ran. operation:\"scrape\" is the one-shot path: it creates a context, navigates, returns the requested formats and tears down. The create_context -> create_page -> cleanup operations remain for multi-step work. operation:\"configure\" only validates a stealthConfig and returns it with the defaults filled in - it stores nothing, so pass stealthConfig on each scrape or create_context call that should use it. robots.txt is respected on every navigation. Not a first choice: try scrape first and switch here after a 403/429/CAPTCHA/challenge page or an empty shell. Cost: 5 credits per browser operation; configure, get_stats and cleanup cost 1. Example: stealth_mode({operation:\"scrape\", url:\"https://example.com\", formats:[\"markdown\",\"links\"]})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "contextId": { "description": "Browser context ID for page operations", - "default": "playwright", - "description": "Browser engine: \"playwright\" (Chromium, default) or \"camoufox\" (Firefox-based, higher anti-detect score — install with npm install camoufox)", + "default": "auto", + "description": "Browser engine: \"auto\" (default — camoufox when it is installed, Chromium otherwise, and the result says which), \"camoufox\" (Firefox-based, higher anti-detect score; fails if not installed), or \"chromium\" (\"playwright\" is the same engine under its old name)", "enum": [ - "playwright", - "camoufox" + "auto", + "chromium", + "camoufox", + "playwright" ], "type": "string" }, + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "operation": { "default": "configure", - "description": "Stealth operation to perform", + "description": "Stealth operation to perform. scrape: render one URL and return the formats. configure: validate stealthConfig and return it with defaults filled in; nothing is stored. create_context / create_page: a context to open pages in. get_stats: this server process's open contexts, browser and proxy state. cleanup: close the browser and every context.", "enum": [ "scrape", "configure", - "enable", - "disable", "create_context", "create_page", "get_stats", + "redact_pii": { + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" + }, "respect_robots": { "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", "type": "boolean" - "additionalProperties": false, - "description": "Stealth browser configuration with anti-detection settings", + "description": "Stealth browser configuration with anti-detection settings. Applies to the call it is passed on and is not remembered. Chromium applies customUserAgent, customViewport, locale and timezone per call. On camoufox customUserAgent and customViewport are not applied, locale is fixed when the browser launches (run operation:\"cleanup\" first to change it), and behind a proxy locale and timezone come from the exit IP; each of these is reported in `warnings`.", "properties": { "antiDetection": { - "additionalProperties": false, "properties": { "cloudflareBypass": { "default": true, - "additionalProperties": false, "properties": { "height": { "maximum": 1080, - "additionalProperties": false, "properties": { "audioContextSpoofing": { "default": true, - "additionalProperties": false, + "description": "Route the browser through your own proxies. Each entry is a proxy URL — \"http://user:pass@host:port\" (percent-encode a password containing @ : or /), or a bare \"host:port\" for an unauthenticated HTTP proxy; http, https, socks4 and socks5 are accepted. rotationInterval is the minimum ms on one proxy before the list advances. Cloudflare scores the IP before it serves a challenge, so a residential proxy is what gets past a block that no fingerprint fixes. Requires the Chromium engine: refused on camoufox (including \"auto\" when it resolves to camoufox), which shares one launch-time proxy across calls. CrawlForge supplies no proxies.", "properties": { "enabled": { "default": false,
--- pinned/summarize_content +++ observed/summarize_content { "annotations": { "destructiveHint": false, - "description": "Use this when you have text content (from extract_text or extract_content) and need a condensed version — e.g. for briefings, comparison tables, or LLM context reduction. Supports extractive (sentence selection) and abstractive (rewrite via Ollama/sampling) modes. Example: summarize_content({text: \"..long article..\", options: {summaryLength: \"short\", summaryType: \"abstractive\"}})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to condense text you already hold into a briefing, comparison, or shorter LLM context - extractive (sentence selection) or abstractive (rewrite via Ollama/sampling). Takes text, not a URL: pass the markdown from a scrape result. Not needed for text short enough to summarise in context yourself. Cost: 4 credits. Example: summarize_content({text: \"..long article..\", options: {summaryLength: \"short\", summaryType: \"abstractive\"}})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "options": { - "additionalProperties": true, + "additionalProperties": {}, "description": "Summarization options", "properties": {}, "type": "object"
--- pinned/track_changes +++ observed/track_changes { "annotations": { "destructiveHint": false, - "description": "Use this when you need to monitor a URL for content changes over time — e.g. competitor pricing, regulation updates, product availability. Start with operation:\"create_baseline\", then periodically use operation:\"compare\" to diff. Supports webhooks and scheduled monitoring. Example: track_changes({url: \"https://example.com/pricing\", operation: \"create_baseline\"})", - "execution": { - "taskSupport": "forbidden" - }, + "description": "Use this to monitor a URL for content changes over time - competitor pricing, regulation updates, product availability. Start with operation:\"create_baseline\", then periodically use operation:\"compare\" to diff; repeated compare calls on the same URL are expected. Supports webhooks and scheduled monitoring, and scheduledMonitorOptions.hosted:true runs the monitor on CrawlForge's servers with email and signed webhooks. Not for a one-off read (scrape). Cost: 3 credits. Example: track_changes({url: \"https://example.com/pricing\", operation: \"create_baseline\"})", "icons": [ { "mimeType": "image/png", - "additionalProperties": false, "properties": { "alertRuleOptions": { - "additionalProperties": false, "description": "Alert rule configuration for change notifications", "properties": { "actions": { + "description": "significance <operator> <level>, e.g. significance >= \"moderate\" (default: significance === \"major\"). Operators: ===, ==, !==, !=, >=, <=, >, <. Levels: none, minor, moderate, major, critical; quotes optional. Anything else is rejected", "type": "string" }, "priority": { - "additionalProperties": false, "description": "Dashboard display options", "properties": { "includeMonitorStatus": { - "additionalProperties": false, "description": "Export options for change history data", "properties": { "endTime": { + "max_inline_chars": { + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" + }, "monitoringOptions": { - "additionalProperties": false, + "default": {}, "description": "Monitoring schedule and notification settings", "properties": { "enableWebhook": { - "default": false, + "default": true, + "description": "operation:\"monitor\" starts polling the URL; false stops the URL's polling monitor and starts nothing", "type": "boolean" }, "interval": { - "additionalProperties": false, - "description": "Notification configuration for webhooks and Slack", + "description": "Notification configuration for webhooks, Slack and email (email is sent by hosted monitors only)", "properties": { + "email": { + "properties": { + "enabled": { + "default": false, + "type": "boolean" + }, + "includeDetails": { + "default": true, + "type": "boolean" + }, + "recipients": { + "items": { + "format": "email", + "pattern": "^(?:[A-Za-z0-9_'+\\-]+\\.)*[A-Za-z0-9_'+\\-]*[A-Za-z0-9_+-]@(?:[A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$", + "type": "string" + }, + "type": "array" + }, + "subject": { + "type": "string" + } + }, + "type": "object" + }, "slack": { - "additionalProperties": false, "properties": { "channel": { "type": "string" - "additionalProperties": false, "properties": { "enabled": { "default": false, + "propertyNames": { + "type": "string" + }, "type": "object" }, "includeContent": { - "additionalProperties": false, + "default": {}, "description": "Query options for history and stats retrieval", "properties": { "endTime": { - "additionalProperties": false, "description": "Scheduled monitoring: recurring compare + notify, optional plain-English goal", "properties": { "enabled": { + "hosted": { + "default": false, + "description": "Run the monitor on CrawlForge's servers: it fires from the hosted scheduler whether or not this process is alive and sends email and signed webhooks. Each check bills 3 credits per compared target from the account; blocked and errored targets are free. Default false = local, in-process.", + "type": "boolean" + }, "interval": { "description": "Polling interval in ms (default 1h)", "minimum": 60000, - "description": "Monitor id for stop_scheduled_monitor", + "description": "Monitor id for stop_scheduled_monitor, as list_scheduled_monitors shows it (poll:<url> for a polling monitor)", "type": "string" }, + "name": { + "description": "Display name for a hosted monitor (default: the URL host)", + "maxLength": 80, + "minLength": 1, + "type": "string" + }, "notificationThreshold": { "enum": [ "minor", + "description": "Preset from get_monitoring_templates; it sets interval, goal, notificationThreshold and trackingOptions, and any of those passed explicitly wins", "type": "string" } }, - "additionalProperties": false, + "default": {}, "description": "Storage and history retention settings", "properties": { "compressionEnabled": { - "additionalProperties": false, + "default": {}, "description": "Options for how changes are tracked and compared", "properties": { "customSelectors": { + "default": [ + "script", + "style", + "noscript", + ".advertisement", + ".ad", + "#comments" + ], "items": { "type": "string" }, - "default": "section", + "description": "Default: section", "enum": [ "page", "section", - "default": false, + "description": "Default: false", "type": "boolean" }, "ignoreWhitespace": { - "default": true, + "description": "Default: true", "type": "boolean" }, "significanceThresholds": { - "additionalProperties": false, "properties": { "major": { "default": 0.7, - "default": true, + "description": "Default: true", "type": "boolean" }, "trackStructure": { - "default": true, + "description": "Default: true", "type": "boolean" }, "trackText": { - "default": true, + "description": "Default: true", "type": "boolean" } },
Current tools
Show all 31 tool fingerprints
agent
analyze_content
batch_scrape
browser_session
crawl_deep
deep_research
extract_content
extract_embedded_state
extract_links
extract_metadata
extract_structured
extract_text
extract_with_llm
fetch_url
generate_llms_txt
get_batch_results
list_ollama_models
localization
map_site
process_document
read_result
reddit_search
scrape
scrape_structured
scrape_template
scrape_with_actions
search_web
serp_rank
stealth_mode
summarize_content
track_changes
Watch this server yourself
If you run this server, put the proxy in front of it. It pins these exact fingerprints on first connect and stops the session if they move.
npx -y mcp-pin@0.2.3 -- <your crawlforge-mcp-server command>
Or subscribe to this page's RSS feed to be told when it changes.
Badge
The badge states one fact about time and nothing else. It never claims a server is safe.
[](https://mcp-pin.gautamkhosla.com/servers/5569b9f013ff72de.html)