Spider
App · 10 actions
Scrape, crawl, search and transform public web content with Spider Cloud.
Actions
| Action | What it does |
|---|---|
| Collect links on the Unlimited plan | Fetch publicly accessible web content with your Spider account. |
| Collect website links | Fetch publicly accessible web content with your Spider account. |
| Convert HTML | Convert existing HTML to the chosen content format. |
| Crawl a website | Fetch publicly accessible web content with your Spider account. |
| Crawl a website on the Unlimited plan | Fetch publicly accessible web content with your Spider account. |
| Get credit balance | Read the current Spider credit balance. |
| Scrape a page | Fetch publicly accessible web content with your Spider account. |
| Scrape a page on the Unlimited plan | Fetch publicly accessible web content with your Spider account. |
| Search the web | Search public web pages by keyword. |
| Take a screenshot | Capture a publicly accessible page as an image. |
Collect links on the Unlimited plan
Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.
Inputs
- Url
urltextRequired - Url.
- Return format
return_formattext - Return format.One of:
raw,markdown,commonmark,text,xml,empty. - Request
requesttext - Request.One of:
smart,http,browser. - Metadata
metadatatrue/false - Metadata.
Outputs
Response.
datalist- Data.
Example output
{
"data": []
}Collect website links
Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.
Inputs
- Url
urltextRequired - Url.
- Return format
return_formattext - Return format.One of:
raw,markdown,commonmark,text,xml,empty. - Request
requesttext - Request.One of:
smart,http,browser. - Metadata
metadatatrue/false - Metadata.
Outputs
Response.
datalist- Data.
Example output
{
"data": []
}Convert HTML
Convert existing HTML to the chosen content format. Uses Spider credits.
Inputs
- Data
datalistRequired - Data.
- Return format
return_formattext - Return format.One of:
raw,markdown,commonmark,text,xml,empty. - Readability
readabilitytrue/false - Readability.
- Clean
cleantrue/false - Clean.
- Clean full
clean_fulltrue/false - Clean full.
Outputs
Response.
dataobject- Data.
Example output
{
"data": {
"content": [],
"error": "example",
"status": 1
}
}Crawl a website
Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.
Inputs
- Url
urltextRequired - Url.
- Limit
limitnumber - Limit.
- Return format
return_formattext - Return format.One of:
raw,markdown,commonmark,text,xml,empty. - Request
requesttext - Request.One of:
smart,http,browser. - Depth
depthnumber - Depth.
- Subdomains
subdomainstrue/false - Subdomains.
- Metadata
metadatatrue/false - Metadata.
Outputs
Response.
datalist- Data.
Example output
{
"data": []
}Crawl a website on the Unlimited plan
Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.
Inputs
- Url
urltextRequired - Url.
- Limit
limitnumber - Limit.
- Return format
return_formattext - Return format.One of:
raw,markdown,commonmark,text,xml,empty. - Request
requesttext - Request.One of:
smart,http,browser. - Depth
depthnumber - Depth.
- Subdomains
subdomainstrue/false - Subdomains.
- Metadata
metadatatrue/false - Metadata.
Outputs
Response.
datalist- Data.
Example output
{
"data": []
}Get credit balance
Read the current Spider credit balance.
Inputs
None.
Outputs
Response.
dataobject- Data.
Example output
{
"data": {
"data": {
"created_at": "example",
"credits": 1.0,
"id": "example",
"updated_at": "example",
"user_id": "example"
}
}
}Scrape a page
Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.
Inputs
- Url
urltextRequired - Url.
- Return format
return_formattext - Return format.One of:
raw,markdown,commonmark,text,xml,empty. - Request
requesttext - Request.One of:
smart,http,browser. - Metadata
metadatatrue/false - Metadata.
Outputs
Response.
datalist- Data.
Example output
{
"data": []
}Scrape a page on the Unlimited plan
Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.
Inputs
- Url
urltextRequired - Url.
- Return format
return_formattext - Return format.One of:
raw,markdown,commonmark,text,xml,empty. - Request
requesttext - Request.One of:
smart,http,browser. - Metadata
metadatatrue/false - Metadata.
Outputs
Response.
datalist- Data.
Example output
{
"data": []
}Search the web
Search public web pages by keyword. Uses Spider credits.
Inputs
- Search
searchtextRequired - Search.
- Search limit
search_limitnumber - Search limit.From 1 to 100.
Outputs
Response.
dataobject- Data.
Example output
{
"data": {
"content": []
}
}Take a screenshot
Capture a publicly accessible page as an image. Uses Spider credits.
Inputs
- Url
urltextRequired - Url.
Outputs
Response.
datalist- Data.
Example output
{
"data": []
}