- [Data residency vs data sovereignty for AI agents](https://parallel.ai/articles/data-residency-vs-data-sovereignty)
Tags:Industry terms
October 16, 2026
# AI data extraction: how to extract structured data from websites at scale
AI data extraction lets you describe the data you want in plain English and get schema-conformant JSON back, at a cost per URL you can forecast. This guide covers how objective-driven extraction differs from traditional scraping, how to define a JSON schema that holds up, the three-API pipeline for discovery, cleaning, and structured output, what scaling to hundreds of thousands of URLs costs, and the enterprise requirements to check.
Tags:Industry Terms
## Key takeaways
- - **AI data extraction** replaces fragile CSS/XPath selectors with natural language objectives that survive layout changes across any site.
- - Defining a **JSON schema** before you call the API is the single biggest factor in output consistency and downstream usability.
- - A three-API pipeline, Search, Extract, and Task, covers URL discovery, content cleaning, and structured output at production scale.
- - Cost per URL is predictable: $1 per 1,000 URLs for clean markdown extraction, with Task API pricing scaling by research depth from $5 per 1,000 runs.
- - **SOC 2 Type 2** certification and zero data retention on Enterprise plans make this pipeline viable for enterprise and regulated-industry use cases.
## What AI data extraction is (and how it differs from traditional scraping)
_Traditional web scraping_[_Traditional web scraping_] means writing CSS selectors or XPath expressions that target specific elements in a page's DOM. A selector like `div.product-price > span.value` works until the site redesigns its layout, at which point the selector returns nothing or the wrong value without warning. Maintaining a scraper across dozens of sources means updating selectors every time a site changes its markup, which happens more often than most teams anticipate.
_AI data extraction_ takes a different approach. You send a URL and a plain-English objective to an API. The model reads the page's content, identifies the information matching your objective, and returns it in a structured format. There are no selectors to maintain. A price is still a price whether the site puts it in a `<span>, a <div class="pricing-block">`, or a JavaScript-rendered component. Research published in Scientific Reports confirms that AI-driven extraction outperforms conventional rule-based crawlers[outperforms conventional rule-based crawlers] by 35% in accuracy and 40% in processing efficiency across benchmark datasets.
Raw HTML averages 5 to 10 times more tokens than the same content as clean markdown. Passing raw HTML directly to an LLM means paying to process navigation menus, footer links, cookie banners, and advertising markup that contain no useful information. Parallel's Extract API strips all of that before the model processes the page, returning dense markdown. Researchers testing LLM-based extraction pipelines achieved F1 scores above 99% for text and table extraction[F1 scores above 99% for text and table extraction] from unstructured sources.
**AI-native output formats** include clean markdown, compressed excerpts, and structured JSON. Each format reduces LLM token consumption compared to raw HTML, and each fits a different stage of a data pipeline. Markdown works as input to downstream tasks; excerpts give you the most relevant passages from a long page; structured JSON slots into a database or data warehouse without transformation.
The choice between traditional scraping and AI extraction depends on your use case:
| Factor | Traditional scraping | AI extraction |
|---|---|---|
| Maintenance burden | High (breaks on layout changes) | Low (objectives describe intent, not location) |
| Layout-change resilience | Fragile | Resilient |
| Output format | Raw HTML or custom parser output | Clean markdown, excerpts, or structured JSON |
| LLM token cost | High: 5-10x more tokens than needed | Low (pre-cleaned, token-efficient output) |
| Time to production | Hours to days per site | Minutes for any site |
Traditional scraping still works well for high-volume pipelines on stable, well-documented sites with consistent markup. **AI data extraction tools** built on API-first infrastructure win for heterogeneous sources, novel layouts, and any pipeline where an agent needs to reason about the output.
## How objective-driven extraction works
An _objective_ is a plain-English instruction that tells the extraction model what information you want from a page. It names the information you need rather than where it lives in the DOM.
A strong objective: `"Extract the company name, founding year, CEO name, total funding raised, and headquarters city from this page."`
A weak objective: `"Get info from this page."`
Vague objectives produce vague output with inconsistent field names and missing values. Specific objectives produce structured output where every field maps to a clear intent. Write your objective as if you're briefing a researcher: name the exact fields you want, in the terms that matter to your use case.
The model uses the objective to focus its attention on relevant content. Instead of parsing the full DOM, it reads the clean markdown that the Extract API[Extract API] produces and extracts only the fields that match the stated goal. This is why the objective-driven model is more resilient than selectors: it reasons about _what_ information is present, not _where_ it appears in the markup. Researchers studying AI-assisted data extraction[AI-assisted data extraction] confirmed that combining LLM extraction with human verification produces results comparable to traditional double-extraction workflows at a fraction of the time.
Feeding a full HTML page to an LLM wastes tokens on navigation, ads, and boilerplate markup. The model spends its context window on content that carries no signal. Parallel's Extract API removes that waste before the model ever processes the page, which means faster responses, lower token costs, and more accurate extraction.
The objective tells the model _what_ to find. The schema tells it _how to format_ the output. You can run the Extract API with an objective and get clean markdown back. To get structured JSON with consistent field names and types, you add a schema at the Task API layer.
Here's a minimal Extract API request that shows the `url` and `objective` parameters:
1234567891011121314import requests
response = requests.post(
"https://api.parallel.ai/v1/extract",
headers={"x-api-key": "YOUR_API_KEY"},
json={
"urls": ["https://example-company.com/about"],
"objective": "Extract the company name, founding year, CEO name, and headquarters city."
}
)
# Returns clean markdown excerpts — token-efficient input for downstream tasks
data = response.json()
print(data["results"][0]["excerpts"])``` import requests response = requests.post( "https://api.parallel.ai/v1/extract", headers={"x-api-key": "YOUR_API_KEY"}, json={ "urls": ["https://example-company.com/about"], "objective": "Extract the company name, founding year, CEO name, and headquarters city." }) # Returns clean markdown excerpts — token-efficient input for downstream tasksdata = response.json()print(data["results"][0]["excerpts"])``` The response contains clean markdown excerpts scoped to your objective. The model discards navigation, footers, and irrelevant copy. What remains feeds directly into a schema-enforcement step without any manual cleaning.
## Getting structured JSON output with schema definition
Production pipelines usually need more than clean markdown: structured JSON with consistent field names, typed values, and per-field confidence levels.
Downstream consumers (databases, data warehouses, LLM pipelines) require consistent field names and types across every record. When you extract data from 10,000 company profile pages without a schema, you get 10,000 slightly different JSON objects. Some have `company_name`, others have `name` or `companyName`. Some have `funding` as a string, others as a number. A schema fixes this at the source. The JSON schema[JSON schema] specification provides the standard format that the Task API[Task API] uses to enforce output structure.
The Task API accepts a JSON schema that specifies exactly which fields to return, their types, and their structure. You define the schema once. Every record the API returns conforms to it. Researchers running large-scale benchmarks on LLM-based extraction measured a 93% schema validation rate[93% schema validation rate] across tens of thousands of real-world web pages.
Here's a realistic schema for B2B company profile enrichment, pulling data from sources like Crunchbase[Crunchbase] or LinkedIn company pages[LinkedIn company pages]:
123456789101112{
"type": "object",
"properties": {
"company_name": { "type": "string" },
"industry": { "type": "string" },
"employee_count": { "type": "number" },
"headquarters_city": { "type": "string" },
"funding_stage": { "type": "string" },
"website": { "type": "string" }
},
"required": ["company_name", "industry", "headquarters_city"]
}``` { "type": "object", "properties": { "company_name": { "type": "string" }, "industry": { "type": "string" }, "employee_count": { "type": "number" }, "headquarters_city": { "type": "string" }, "funding_stage": { "type": "string" }, "website": { "type": "string" } }, "required": ["company_name", "industry", "headquarters_city"]}``` The schema supports strings, numbers, booleans, arrays, and nested objects. For a funding data use case, you'd add fields like `total_raised`, `last_round_type`, `last_round_date`, and an `investors` array. Required fields tell the API which fields must appear in every record. Optional fields return `null` when the source page doesn't contain them.
The Task API returns two additional signals alongside the extracted values: **confidence levels** and **citations**. Confidence is a per-field rating of low, medium, or high that tells you how certain the model is about each extracted value. A `company_name` rated high came from clear, unambiguous text. A `funding_stage` rated low came from inferred or ambiguous content. You use confidence as a quality gate before records enter your database: send low-confidence fields to human review rather than letting uncertain data propagate downstream.
Citations are source URLs attached to each field. When a Task API response tells you a company raised $12M in a Series A, it also tells you which URL that claim came from. This supports auditability in regulated industries and gives compliance teams a traceable chain from extracted claim to source. Provenance also matters in data enrichment[data enrichment] workflows and sales enrichment[sales enrichment] pipelines, where you need to know which source each value came from.
Here's a Task API call that passes the schema and an objective, then prints confidence scores and citations:
12345678910111213141516171819202122232425262728293031323334353637import requests
schema = {
"type": "object",
"properties": {
"company_name": {"type": "string"},
"industry": {"type": "string"},
"employee_count": {"type": "number"},
"headquarters_city": {"type": "string"},
"funding_stage": {"type": "string"},
"website": {"type": "string"}
},
"required": ["company_name", "industry", "headquarters_city"]
}
response = requests.post(
"https://api.parallel.ai/v1/tasks/runs",
headers={"x-api-key": "YOUR_API_KEY"},
json={
"input": "Extract company profile data from https://example-company.com/about, including industry, employee count, headquarters, funding stage, and website.",
"processor": "base",
"task_spec": {"output_schema": {"type": "json", "json_schema": schema}}
}
)
run_id = response.json()["run_id"]
# Once the run completes, fetch the result
result = requests.get(
f"https://api.parallel.ai/v1/tasks/runs/{run_id}/result",
headers={"x-api-key": "YOUR_API_KEY"}
).json()
for field, value in result["output"]["content"].items():
print(f"{field}: {value}")
# result["output"]["basis"] carries citations, reasoning, and a
# calibrated confidence score for every field``` import requests schema = { "type": "object", "properties": { "company_name": {"type": "string"}, "industry": {"type": "string"}, "employee_count": {"type": "number"}, "headquarters_city": {"type": "string"}, "funding_stage": {"type": "string"}, "website": {"type": "string"} }, "required": ["company_name", "industry", "headquarters_city"]} response = requests.post( "https://api.parallel.ai/v1/tasks/runs", headers={"x-api-key": "YOUR_API_KEY"}, json={ "input": "Extract company profile data from https://example-company.com/about, including industry, employee count, headquarters, funding stage, and website.", "processor": "base", "task_spec": {"output_schema": {"type": "json", "json_schema": schema}} }) run_id = response.json()["run_id"] # Once the run completes, fetch the resultresult = requests.get( f"https://api.parallel.ai/v1/tasks/runs/{run_id}/result", headers={"x-api-key": "YOUR_API_KEY"}).json() for field, value in result["output"]["content"].items(): print(f"{field}: {value}")# result["output"]["basis"] carries citations, reasoning, and a# calibrated confidence score for every field``` When a source page doesn't contain a requested field, the Task API returns `null` for that field rather than omitting it or hallucinating a value. This keeps your schema consistent across records. Every object has the same keys regardless of what the source page contains. Required fields that the model cannot find return `null` with low confidence, flagging the record for review rather than dropping it without notice.
## Scaling AI extraction in production
A single Extract API call runs in 1 to 20 seconds synchronously. A for-loop iterating 1,000 URLs in sequence would take between 17 minutes and 5.5 hours, so production pipelines run requests concurrently.
**Cost modeling** for a product price-monitoring pipeline across 500,000 e-commerce URLs per month:
| URLs per month | Extract API cost | Task API cost (base tier) |
|---|---|---|
| 1,000 | $1.00 | $10.00 |
| 10,000 | $10.00 | $100.00 |
| 100,000 | $100.00 | $1,000.00 |
| 1,000,000 | $1,000.00 | $10,000.00 |
Extract API pricing is $0.001 per URL, flat. Task API pricing scales by processor tier[pricing scales by processor tier]: lite at $5 per 1,000, base at $10 per 1,000, core at $25 per 1,000. For product price monitoring, the Extract API alone handles most use cases since you need clean markdown output rather than deep research[deep research] synthesis.
Use Extract for high-volume, simpler structured output: product catalogs, directory listings, pricing pages, and any case where you need clean markdown at scale. Use Task when you need deep research, multi-source synthesis, per-field confidence levels, and citations, typically at lower volume and higher accuracy requirements.
The batch processing pattern for 1,000 URLs uses async workers[async workers] rather than a sequential loop:
1234567891011121314151617181920212223242526import asyncio
import aiohttp
API_KEY = "YOUR_API_KEY"
ENDPOINT = "https://api.parallel.ai/v1/extract"
async def extract_url(session, url, objective):
async with session.post(
ENDPOINT,
headers={"x-api-key": API_KEY},
json={"urls": [url], "objective": objective}
) as resp:
return await resp.json()
async def batch_extract(urls, objective, concurrency=20):
semaphore = asyncio.Semaphore(concurrency)
async with aiohttp.ClientSession() as session:
async def bounded_extract(url):
async with semaphore:
return await extract_url(session, url, objective)
results = await asyncio.gather(*[bounded_extract(u) for u in urls])
return results
# Process 1,000 URLs with 20 concurrent workers
urls = [...] # your list of 1,000 URLs
results = asyncio.run(batch_extract(urls, "Extract product name, price, and availability."))``` import asyncioimport aiohttp API_KEY = "YOUR_API_KEY"ENDPOINT = "https://api.parallel.ai/v1/extract" async def extract_url(session, url, objective): async with session.post( ENDPOINT, headers={"x-api-key": API_KEY}, json={"urls": [url], "objective": objective} ) as resp: return await resp.json() async def batch_extract(urls, objective, concurrency=20): semaphore = asyncio.Semaphore(concurrency) async with aiohttp.ClientSession() as session: async def bounded_extract(url): async with semaphore: return await extract_url(session, url, objective) results = await asyncio.gather(*[bounded_extract(u) for u in urls]) return results # Process 1,000 URLs with 20 concurrent workersurls = [...] # your list of 1,000 URLsresults = asyncio.run(batch_extract(urls, "Extract product name, price, and availability."))``` This pattern runs 20 concurrent requests, respecting rate limits while processing the full batch in 1 to 20 minutes depending on page complexity.
For failed extractions, implement exponential backoff on transient errors (HTTP 429, 503) and flag persistent failures for review rather than retrying indefinitely. A record that fails three times usually has a problem at the source.
Monitoring extraction quality at scale requires a sampling strategy. Check 1% of records against ground truth on a weekly basis. Use confidence levels as a proxy quality gate, rejecting records where any required field comes back low. Set the gate based on your tolerance for downstream errors; stricter pipelines also route medium-confidence fields to review.
## Using AI extraction in agent pipelines
AI agents[AI agents] need fresh, structured, web-grounded facts. Static datasets go stale. Raw HTML is too token-expensive to pass directly to an LLM. A 10-page website as raw HTML can exceed 50,000 tokens. The same content as clean markdown often fits in under 5,000 tokens. That 10x reduction matters when you're paying per token and managing context window limits across thousands of agent calls.
The **three-API pipeline** solves the agent data problem in three steps:
- **Search API**[**Search API**] takes a natural language query and returns ranked URLs with titles and compressed excerpts. You get the right pages without manually curating a URL list.
- **Extract API** converts each URL to clean, token-efficient markdown scoped to your objective. You pass this to the next step rather than raw HTML.
- **Task API** structures the markdown into a JSON schema with per-field confidence levels and citations. The output slots directly into your agent's structured memory or tool call response schema.
A competitive intelligence agent that runs weekly can use the Search API to discover new competitor pages, the Extract API to pull clean content from each page, and the Task API to store normalized JSON records with citations, so each claim traces to a source URL. Raw HTML and navigation markup never reach the agent context.
Instead of summarizing a single page per LLM call to stay within context limits, you can process five or ten pages per call when the input is clean markdown, which lowers both the cost and the latency of each agent step.
Task API's JSON output eliminates the post-processing layer that most agent architectures require. Instead of extracting free-form text and running a second LLM call to structure it, you get structured output in a single API call. The output conforms to the schema you defined, with every field typed and every claim cited.
Here's an illustrative three-step agent data fetch (integration pattern, not production-ready code):
123456789101112131415161718192021import requests
API_KEY = "YOUR_API_KEY"
HEADERS = {"x-api-key": API_KEY}
# Step 1: Discover relevant URLs
search_resp = requests.post("https://api.parallel.ai/v1/search", headers=HEADERS,
json={"objective": "Find competitor pricing pages for B2B project management software",
"search_queries": ["B2B project management pricing"]})
top_urls = [r["url"] for r in search_resp.json()["results"][:3]]
# Step 2: Extract clean markdown from each URL
extract_resp = requests.post("https://api.parallel.ai/v1/extract", headers=HEADERS,
json={"urls": top_urls, "objective": "Extract pricing tiers, prices, and included features"})
markdown_content = extract_resp.json()["results"][0]["excerpts"]
# Step 3: Structure the output into a schema with citations
task_resp = requests.post("https://api.parallel.ai/v1/tasks/runs", headers=HEADERS,
json={"input": "\n".join(markdown_content), "processor": "base",
"task_spec": {"output_schema": PRICING_SCHEMA}})
structured_data = task_resp.json() # poll the run, then fetch its result``` import requests API_KEY = "YOUR_API_KEY"HEADERS = {"x-api-key": API_KEY} # Step 1: Discover relevant URLssearch_resp = requests.post("https://api.parallel.ai/v1/search", headers=HEADERS, json={"objective": "Find competitor pricing pages for B2B project management software", "search_queries": ["B2B project management pricing"]})top_urls = [r["url"] for r in search_resp.json()["results"][:3]] # Step 2: Extract clean markdown from each URLextract_resp = requests.post("https://api.parallel.ai/v1/extract", headers=HEADERS, json={"urls": top_urls, "objective": "Extract pricing tiers, prices, and included features"})markdown_content = extract_resp.json()["results"][0]["excerpts"] # Step 3: Structure the output into a schema with citationstask_resp = requests.post("https://api.parallel.ai/v1/tasks/runs", headers=HEADERS, json={"input": "\n".join(markdown_content), "processor": "base", "task_spec": {"output_schema": PRICING_SCHEMA}})structured_data = task_resp.json() # poll the run, then fetch its result``` The APIs are REST-based and work with any agent framework. LangChain, LlamaIndex, and custom OpenAI function-calling loops all support HTTP calls. You define the schema to match your agent's tool call response schema, and the Task API output passes through without transformation.
Citations give agents a grounding mechanism: every claim in the structured output links to a source URL, so a person or a downstream check can verify each assertion against its source instead of relying on a confidence rating alone.
## Enterprise considerations: compliance, governance, and reliability
Check each AI extraction vendor's compliance documentation before procurement. Parallel holds **SOC 2 Type 2** certification, meaning an independent auditor has reviewed and validated its security controls. For procurement teams in finance, healthcare, and enterprise SaaS, SOC 2 Type 2 is the baseline requirement for any external API handling potentially sensitive data. Full compliance documentation is available at the Trust Center[Trust Center].
On Enterprise plans, Parallel doesn't store request payloads or response data after serving the response. **Zero data retention** means the URLs you send and the data you extract don't persist on Parallel's infrastructure. For GDPR compliance and data-sensitive use cases, this settles most retention questions.
**Data provenance** comes built into the Task API. Every response includes citations linking extracted claims to source URLs. Compliance teams that need to trace AI-generated data back to its origin get that chain from the API response itself, without building a separate audit layer.
The Task API's asynchronous architecture means a slow or unresponsive source page doesn't block your pipeline. Requests run independently, and you poll for results or receive them via webhook. The pipeline continues processing other URLs while a slow source resolves.
One point to address with your legal team before you scale: if your extraction pipeline pulls personal data (names, email addresses, phone numbers, or physical addresses) from public web sources, GDPR and CCPA implications apply[GDPR and CCPA implications apply] regardless of which tool you use. The extraction API processes whatever is on the page. Your team owns the compliance decision about what data to collect and how to store it.
## Frequently asked questions about AI data extraction
### What is AI data extraction?
AI data extraction uses large language models to identify and pull structured information from web pages based on a natural language objective, without site-specific selectors. The API returns data as structured JSON, clean markdown, or both, depending on your configuration.
### How is AI data extraction different from web scraping?
Web scraping parses a page's DOM using CSS selectors or XPath and breaks when layouts change. AI extraction reads page content as text and applies model reasoning, so it works across layout variations without code changes.
### What are the best AI data extraction tools for developers?
The best tools expose a REST API, accept natural language objectives, return structured JSON with a defined schema, and publish per-URL pricing. Parallel's Extract API and Task API meet all four criteria.
### How do I extract structured data from a website using an API?
Send a POST request to the extraction API with the target URL and a plain-English objective describing the fields you want. The API returns clean markdown or structured JSON depending on your schema configuration. The Parallel Extract API docs[Parallel Extract API docs] are the starting point.
### How much does AI data extraction cost at scale?
Parallel's Extract API costs $1 per 1,000 URLs. A pipeline processing 100,000 URLs per month costs $100 in extraction fees. Task API pricing scales with research depth, starting at $5 per 1,000 runs for the lite tier.
### Can AI data extraction handle dynamic JavaScript-rendered pages?
Yes. Parallel's Extract API handles JavaScript-rendered pages. You pass the URL and objective, and the API returns clean content regardless of whether the page requires JavaScript execution. You don't manage a headless browser.
The pattern is to discover URLs with Search, extract clean content with Extract, and structure your output with Task. The full API reference walks you through every parameter. Start Building[Start Building]