AI data extraction uses large language models and AI-native APIs to pull structured, schema-conformant data from any website, without writing site-specific CSS selectors or XPath expressions. You send a URL and a plain-English objective to the API, and the model returns clean, structured output regardless of the page's DOM layout. The pattern has three steps: discover URLs with the Search API, convert pages to clean markdown with the Extract API, and enforce a JSON schema with the Task API. That pipeline runs on thousands of URLs with predictable cost, at $1 per 1,000 URLs for extraction. The sections below walk through each step in enough technical depth to take this approach from a single test request to a production pipeline handling hundreds of thousands of URLs per month.
_Traditional web scraping_[_Traditional web scraping_](/articles/what-is-web-scraping) means writing CSS selectors or XPath expressions that target specific elements in a page's DOM. A selector like `div.product-price > span.value` works until the site redesigns its layout, at which point the selector returns nothing or the wrong value without warning. Maintaining a scraper across dozens of sources means updating selectors every time a site changes its markup, which happens more often than most teams anticipate.
_AI data extraction_ takes a different approach. You send a URL and a plain-English objective to an API. The model reads the page's content, identifies the information matching your objective, and returns it in a structured format. There are no selectors to maintain. A price is still a price whether the site puts it in a `<span>, a <div class="pricing-block">`, or a JavaScript-rendered component. Research published in Scientific Reports confirms that AI-driven extraction outperforms conventional rule-based crawlers[outperforms conventional rule-based crawlers](https://www.nature.com/articles/s41598-025-25616-x) by 35% in accuracy and 40% in processing efficiency across benchmark datasets.
The token problem makes this distinction concrete. Raw HTML averages 5 to 10 times more tokens than the same content as clean markdown. Passing raw HTML directly to an LLM means paying to process navigation menus, footer links, cookie banners, and advertising markup that contain no useful information. Parallel's Extract API strips all of that before the model processes the page, returning dense markdown that carries the signal without the noise. Researchers testing LLM-based extraction pipelines achieved F1 scores above 99% for text and table extraction[F1 scores above 99% for text and table extraction](https://www.sciencedirect.com/science/article/pii/S2666546826000674) from unstructured sources.
**AI-native output formats** include clean markdown, compressed excerpts, and structured JSON. Each format reduces LLM token consumption compared to raw HTML, and each fits a different stage of a data pipeline. Markdown works as input to downstream tasks; excerpts give you the most relevant passages from a long page; structured JSON slots into a database or data warehouse without transformation.
The choice between traditional scraping and AI extraction depends on your use case:
| Factor | Traditional scraping | AI extraction |
|---|---|---|
| Maintenance burden | High (breaks on layout changes) | Low (objectives describe intent, not location) |
| Layout-change resilience | Fragile | Resilient |
| Output format | Raw HTML or custom parser output | Clean markdown, excerpts, or structured JSON |
| LLM token cost | High — 5-10x more tokens than needed | Low — pre-cleaned, token-efficient output |
| Time to production | Hours to days per site | Minutes for any site |
Traditional scraping still works well for high-volume pipelines on stable, well-documented sites with consistent markup. **AI data extraction tools** built on API-first infrastructure win for heterogeneous sources, novel layouts, and any pipeline where an agent needs to reason about the output.
An _objective_ is a plain-English instruction that tells the extraction model what information you want from a page. You don't describe where the information lives in the DOM. You describe what you need.
A strong objective: `"Extract the company name, founding year, CEO name, total funding raised, and headquarters city from this page."`
A weak objective: `"Get info from this page."`
The difference in output quality is significant. Vague objectives produce vague output with inconsistent field names and missing values. Specific objectives produce structured output where every field maps to a clear intent. Write your objective as if you're briefing a researcher: name the exact fields you want, in the terms that matter to your use case.
The model uses the objective to focus its attention on relevant content. Instead of parsing the full DOM, it reads the clean markdown that the Extract API[Extract API](https://docs.parallel.ai/extract/extract-quickstart) produces and extracts only the fields that match the stated goal. This is why the objective-driven model is more resilient than selectors: it reasons about _what_ information is present, not _where_ it appears in the markup. Researchers studying AI-assisted data extraction[AI-assisted data extraction](https://pmc.ncbi.nlm.nih.gov/articles/PMC12593456/) confirmed that combining LLM extraction with human verification produces results comparable to traditional double-extraction workflows at a fraction of the time.
Contrast this with prompting a raw LLM with an HTML dump. Feeding a full HTML page to an LLM wastes tokens on navigation, ads, and boilerplate markup. The model spends its context window on content that carries no signal. Parallel's Extract API removes that waste before the model ever processes the page. The result is faster responses, lower token costs, and more accurate extraction.
The connection between objectives and schemas matters here. The objective tells the model _what_ to find. The schema tells it _how to format_ the output. You can run the Extract API with just an objective and get clean markdown back. To get structured JSON with consistent field names and types, you add a schema at the Task API layer. The two parameters work together: objective drives content selection, schema drives output structure.
Here's a minimal Extract API request that shows the `url` and `objective` parameters:
123456789101112131415import requests
response = requests.post(
"https://api.parallel.ai/v1beta/extract",
headers={"x-api-key": "YOUR_API_KEY"},
json={
"urls": ["https://example-company.com/about"],
"objective": "Extract the company name, founding year, CEO name, and headquarters city.",
"excerpts": True
}
)
# Returns clean markdown excerpts — token-efficient input for downstream tasks
data = response.json()
print(data["results"][0]["excerpts"])``` import requests response = requests.post( "https://api.parallel.ai/v1beta/extract", headers={"x-api-key": "YOUR_API_KEY"}, json={ "urls": ["https://example-company.com/about"], "objective": "Extract the company name, founding year, CEO name, and headquarters city.", "excerpts": True }) # Returns clean markdown excerpts — token-efficient input for downstream tasksdata = response.json()print(data["results"][0]["excerpts"])``` The response contains clean markdown excerpts scoped to your objective. The model discards navigation, footers, and irrelevant copy. What remains feeds directly into a schema-enforcement step without any manual cleaning.
Clean markdown is useful. Structured JSON with consistent field names, typed values, and per-field confidence scores is what production pipelines actually need.
Downstream consumers (databases, data warehouses, LLM pipelines) require consistent field names and types across every record. When you extract data from 10,000 company profile pages without a schema, you get 10,000 slightly different JSON objects. Some have `company_name`, others have `name` or `companyName`. Some have `funding` as a string, others as a number. Schema definition solves this at the source. The JSON schema[JSON schema](https://json-schema.org/) specification provides the standard format that the Task API[Task API](https://docs.parallel.ai/task-api/task-quickstart) uses to enforce output structure.
The Task API accepts a JSON schema that specifies exactly which fields to return, their types, and their structure. You define the schema once. Every record the API returns conforms to it. Researchers running large-scale benchmarks on LLM-based extraction measured a 93% schema validation rate[93% schema validation rate](https://arxiv.org/html/2602.15189v1) across tens of thousands of real-world web pages, confirming that schema-constrained extraction works at production scale.
Here's a realistic schema for B2B company profile enrichment, pulling data from sources like Crunchbase[Crunchbase](https://www.crunchbase.com/) or LinkedIn company pages[LinkedIn company pages](https://www.linkedin.com/):
123456789101112{
"type": "object",
"properties": {
"company_name": { "type": "string" },
"industry": { "type": "string" },
"employee_count": { "type": "number" },
"headquarters_city": { "type": "string" },
"funding_stage": { "type": "string" },
"website": { "type": "string" }
},
"required": ["company_name", "industry", "headquarters_city"]
}``` { "type": "object", "properties": { "company_name": { "type": "string" }, "industry": { "type": "string" }, "employee_count": { "type": "number" }, "headquarters_city": { "type": "string" }, "funding_stage": { "type": "string" }, "website": { "type": "string" } }, "required": ["company_name", "industry", "headquarters_city"]}``` The schema supports strings, numbers, booleans, arrays, and nested objects. For a funding data use case, you'd add fields like `total_raised`, `last_round_type`, `last_round_date`, and an `investors` array. Required fields tell the API which fields must appear in every record. Optional fields return `null` when the source page doesn't contain them.
The Task API returns two additional signals alongside the extracted values: **confidence levels** and **citations**. Confidence is a per-field score that tells you how certain the model is about each extracted value. A `company_name` with confidence 0.97 came from clear, unambiguous text. A `funding_stage` with confidence 0.61 came from inferred or ambiguous content. You use confidence thresholds as a quality gate before records enter your database. Filter anything below 0.75 for human review rather than letting uncertain data propagate downstream.
Citations are source URLs attached to each field. When a Task API response tells you a company raised $12M in a Series A, it also tells you which URL that claim came from. This supports auditability in regulated industries and gives compliance teams a traceable chain from extracted claim to source. This is especially valuable for data enrichment[data enrichment](/articles/what-is-data-enrichment) workflows and sales enrichment[sales enrichment](/articles/ai-web-enrichment-for-sales) pipelines where data provenance matters.
Here's a Task API call that passes the schema and an objective, then prints confidence scores and citations:
12345678910111213141516171819202122232425262728293031import requests
schema = {
"type": "object",
"properties": {
"company_name": {"type": "string"},
"industry": {"type": "string"},
"employee_count": {"type": "number"},
"headquarters_city": {"type": "string"},
"funding_stage": {"type": "string"},
"website": {"type": "string"}
},
"required": ["company_name", "industry", "headquarters_city"]
}
response = requests.post(
"https://api.parallel.ai/v1beta/task_runs",
headers={"x-api-key": "YOUR_API_KEY"},
json={
"objective": "Extract company profile data including industry, employee count, headquarters, funding stage, and website.",
"output_schema": schema,
"urls": ["https://example-company.com/about"],
"processor": "base"
}
)
result = response.json()
for field, value in result["output"].items():
confidence = result["confidence"].get(field, "N/A")
citations = result["citations"].get(field, [])
print(f"{field}: {value} (confidence: {confidence}, sources: {citations})")``` import requests schema = { "type": "object", "properties": { "company_name": {"type": "string"}, "industry": {"type": "string"}, "employee_count": {"type": "number"}, "headquarters_city": {"type": "string"}, "funding_stage": {"type": "string"}, "website": {"type": "string"} }, "required": ["company_name", "industry", "headquarters_city"]} response = requests.post( "https://api.parallel.ai/v1beta/task_runs", headers={"x-api-key": "YOUR_API_KEY"}, json={ "objective": "Extract company profile data including industry, employee count, headquarters, funding stage, and website.", "output_schema": schema, "urls": ["https://example-company.com/about"], "processor": "base" }) result = response.json()for field, value in result["output"].items(): confidence = result["confidence"].get(field, "N/A") citations = result["citations"].get(field, []) print(f"{field}: {value} (confidence: {confidence}, sources: {citations})")``` When a source page doesn't contain a requested field, the Task API returns `null` for that field rather than omitting it or hallucinating a value. This keeps your schema consistent across records — every object has the same keys regardless of what the source page contains. Required fields that the model cannot find return `null` with a low confidence score, flagging the record for review rather than dropping it without notice.
A single Extract API call runs in 1 to 20 seconds synchronously. A for-loop iterating 1,000 URLs in sequence would take between 17 minutes and 5.5 hours. Production pipelines need a different architecture.
**Cost modeling** for a product price-monitoring pipeline across 500,000 e-commerce URLs per month:
| URLs per month | Extract API cost | Task API cost (base tier) |
|---|---|---|
| 1,000 | $1.00 | $10.00 |
| 10,000 | $10.00 | $100.00 |
| 100,000 | $100.00 | $1,000.00 |
| 1,000,000 | $1,000.00 | $10,000.00 |
Extract API pricing is $0.001 per URL, flat. Task API pricing scales by processor tier[pricing scales by processor tier](/pricing): lite at $5 per 1,000, base at $10 per 1,000, core at $25 per 1,000. For product price monitoring, the Extract API alone handles most use cases since you need clean markdown output rather than deep research[deep research](/articles/what-is-deep-research) synthesis.
Use Extract for high-volume, simpler structured output: product catalogs, directory listings, pricing pages, and any case where you need clean markdown at scale. Use Task when you need deep research, multi-source synthesis, per-field confidence scores, and citations, typically at lower volume and higher accuracy requirements.
The batch processing pattern for 1,000 URLs uses async workers[async workers](https://docs.python.org/3/library/asyncio.html) rather than a sequential loop:
1234567891011121314151617181920212223242526import asyncio
import aiohttp
API_KEY = "YOUR_API_KEY"
ENDPOINT = "https://api.parallel.ai/v1beta/extract"
async def extract_url(session, url, objective):
async with session.post(
ENDPOINT,
headers={"x-api-key": API_KEY},
json={"urls": [url], "objective": objective, "excerpts": True}
) as resp:
return await resp.json()
async def batch_extract(urls, objective, concurrency=20):
semaphore = asyncio.Semaphore(concurrency)
async with aiohttp.ClientSession() as session:
async def bounded_extract(url):
async with semaphore:
return await extract_url(session, url, objective)
results = await asyncio.gather(*[bounded_extract(u) for u in urls])
return results
# Process 1,000 URLs with 20 concurrent workers
urls = [...] # your list of 1,000 URLs
results = asyncio.run(batch_extract(urls, "Extract product name, price, and availability."))``` import asyncioimport aiohttp API_KEY = "YOUR_API_KEY"ENDPOINT = "https://api.parallel.ai/v1beta/extract" async def extract_url(session, url, objective): async with session.post( ENDPOINT, headers={"x-api-key": API_KEY}, json={"urls": [url], "objective": objective, "excerpts": True} ) as resp: return await resp.json() async def batch_extract(urls, objective, concurrency=20): semaphore = asyncio.Semaphore(concurrency) async with aiohttp.ClientSession() as session: async def bounded_extract(url): async with semaphore: return await extract_url(session, url, objective) results = await asyncio.gather(*[bounded_extract(u) for u in urls]) return results # Process 1,000 URLs with 20 concurrent workersurls = [...] # your list of 1,000 URLsresults = asyncio.run(batch_extract(urls, "Extract product name, price, and availability."))``` This pattern runs 20 concurrent requests, respecting rate limits while processing the full batch in 1 to 20 minutes depending on page complexity.
For failed extractions, implement exponential backoff on transient errors (HTTP 429, 503) and flag persistent failures for review rather than retrying indefinitely. A record that fails three times has a source problem, not a transient API problem.
Monitoring extraction quality at scale requires a sampling strategy. Check 1% of records against ground truth on a weekly basis. Use confidence score thresholds as a proxy quality gate, rejecting records where any required field falls below your defined threshold. Set the threshold based on your tolerance for downstream errors. Most production pipelines use 0.75 to 0.85.
AI agents[AI agents](/articles/what-is-an-ai-agent) need fresh, structured, web-grounded facts. Static datasets go stale. Raw HTML is too token-expensive to pass directly to an LLM. A 10-page website as raw HTML can exceed 50,000 tokens. The same content as clean markdown often fits in under 5,000 tokens. That 10x reduction matters when you're paying per token and managing context window limits across thousands of agent calls.
The **three-API pipeline** solves the agent data problem in three steps:
Consider a competitive intelligence agent that runs weekly. It uses the Search API to discover new competitor pages, the Extract API to pull clean content from each page, and the Task API to store normalized JSON records with citations. The agent never handles raw HTML, never blows its context window on navigation markup, and every claim in its output traces to a source URL.
Token-efficient output changes what's possible in an agent loop. Instead of summarizing a single page per LLM call to stay within context limits, you can process five or ten pages per call when the input is clean markdown. That changes the economics and the latency of agent pipelines that depend on web data.
Task API's JSON output eliminates the post-processing layer that most agent architectures require. Instead of extracting free-form text and running a second LLM call to structure it, you get structured output in a single API call. The output conforms to the schema you defined, with every field typed and every claim cited.
Here's an illustrative three-step agent data fetch (integration pattern, not production-ready code):
1234567891011121314151617181920import requests
API_KEY = "YOUR_API_KEY"
HEADERS = {"x-api-key": API_KEY}
# Step 1: Discover relevant URLs
search_resp = requests.post("https://api.parallel.ai/v1beta/search", headers=HEADERS,
json={"objective": "Find competitor pricing pages for B2B project management software"})
top_urls = [r["url"] for r in search_resp.json()["results"][:3]]
# Step 2: Extract clean markdown from each URL
extract_resp = requests.post("https://api.parallel.ai/v1beta/extract", headers=HEADERS,
json={"urls": top_urls, "objective": "Extract pricing tiers, prices, and included features", "excerpts": True})
markdown_content = extract_resp.json()["results"][0]["excerpts"]
# Step 3: Structure the output into a schema with citations
task_resp = requests.post("https://api.parallel.ai/v1beta/task_runs", headers=HEADERS,
json={"objective": "Structure competitive pricing data", "output_schema": PRICING_SCHEMA,
"content": markdown_content, "processor": "base"})
structured_data = task_resp.json()["output"] # schema-conformant JSON with citations``` import requests API_KEY = "YOUR_API_KEY"HEADERS = {"x-api-key": API_KEY} # Step 1: Discover relevant URLssearch_resp = requests.post("https://api.parallel.ai/v1beta/search", headers=HEADERS, json={"objective": "Find competitor pricing pages for B2B project management software"})top_urls = [r["url"] for r in search_resp.json()["results"][:3]] # Step 2: Extract clean markdown from each URLextract_resp = requests.post("https://api.parallel.ai/v1beta/extract", headers=HEADERS, json={"urls": top_urls, "objective": "Extract pricing tiers, prices, and included features", "excerpts": True})markdown_content = extract_resp.json()["results"][0]["excerpts"] # Step 3: Structure the output into a schema with citationstask_resp = requests.post("https://api.parallel.ai/v1beta/task_runs", headers=HEADERS, json={"objective": "Structure competitive pricing data", "output_schema": PRICING_SCHEMA, "content": markdown_content, "processor": "base"})structured_data = task_resp.json()["output"] # schema-conformant JSON with citations``` The APIs are REST-based and work with any agent framework. LangChain, LlamaIndex, and custom OpenAI function-calling loops all support HTTP calls. You define the schema to match your agent's tool call response schema, and the Task API output passes through without transformation.
Citations give agents a built-in grounding mechanism. Every claim in the structured output links to a source URL. Agents that include citations in their outputs hallucinate less because every assertion has a verifiable source attached to it, not just a confidence score.
Most AI extraction tools publish no compliance documentation. Parallel holds **SOC 2 Type 2** certification, meaning an independent auditor has reviewed and validated its security controls. For procurement teams in finance, healthcare, and enterprise SaaS, SOC 2 Type 2 is the baseline requirement for any external API handling potentially sensitive data. Full compliance documentation is available at the Trust Center[Trust Center](https://trust.parallel.ai/).
Parallel does not store request payloads or response data after serving the response. **Zero data retention** means the URLs you send and the data you extract don't persist on Parallel's infrastructure. For GDPR compliance and data-sensitive use cases, this eliminates a class of data residency and retention concerns. Your data passes through the API and leaves.
**Data provenance** comes built into the Task API. Every response includes citations linking extracted claims to source URLs. Compliance teams that need to trace AI-generated data back to its origin get that chain from the API response itself, without building a separate audit layer.
The Task API's asynchronous architecture means a slow or unresponsive source page doesn't block your pipeline. Requests run independently, and you poll for results or receive them via webhook. The pipeline continues processing other URLs while a slow source resolves.
One point to address with your legal team before you scale: if your extraction pipeline pulls personal data — names, email addresses, phone numbers, or physical addresses — from public web sources, GDPR and CCPA implications apply[GDPR and CCPA implications apply](https://ico.org.uk/about-the-ico/what-we-do/our-work-on-artificial-intelligence/response-to-the-consultation-series-on-generative-ai/the-lawful-basis-for-web-scraping-to-train-generative-ai-models/) regardless of which tool you use. The extraction API processes whatever is on the page. Your team owns the compliance decision about what data to collect and how to store it.
AI data extraction uses large language models to identify and pull structured information from web pages based on a natural language objective, without site-specific selectors. The API returns data as structured JSON, clean markdown, or both, depending on your configuration.
Web scraping parses a page's DOM using CSS selectors or XPath and breaks when layouts change. AI extraction reads page content as text and applies model reasoning, so it works across layout variations without code changes.
The best tools expose a REST API, accept natural language objectives, return structured JSON with a defined schema, and publish per-URL pricing. Parallel's Extract API and Task API meet all four criteria.
Send a POST request to the extraction API with the target URL and a plain-English objective describing the fields you want. The API returns clean markdown or structured JSON depending on your schema configuration. The Parallel Extract API docs[Parallel Extract API docs](https://docs.parallel.ai/home) are the starting point.
Parallel's Extract API costs $1 per 1,000 URLs. A pipeline processing 100,000 URLs per month costs $100 in extraction fees. Task API pricing scales with research depth, starting at $5 per 1,000 runs for the lite tier.
Yes. Parallel's Extract API handles JavaScript-rendered pages. You pass the URL and objective, and the API returns clean content regardless of whether the page requires JavaScript execution. You don't manage a headless browser.
You have the pattern: discover URLs with Search, extract clean content with Extract, and structure your output with Task. The full API reference walks you through every parameter. Start Building[Start Building](https://docs.parallel.ai/home)
By Parallel
April 17, 2026
