Spider

App · 10 actions

Scrape, crawl, search and transform public web content with Spider Cloud.

Actions

ActionWhat it does
Collect links on the Unlimited planFetch publicly accessible web content with your Spider account.
Collect website linksFetch publicly accessible web content with your Spider account.
Convert HTMLConvert existing HTML to the chosen content format.
Crawl a websiteFetch publicly accessible web content with your Spider account.
Crawl a website on the Unlimited planFetch publicly accessible web content with your Spider account.
Get credit balanceRead the current Spider credit balance.
Scrape a pageFetch publicly accessible web content with your Spider account.
Scrape a page on the Unlimited planFetch publicly accessible web content with your Spider account.
Search the webSearch public web pages by keyword.
Take a screenshotCapture a publicly accessible page as an image.

Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.

Inputs

UrlurltextRequired
Url.
Return formatreturn_formattext
Return format.One of: raw, markdown, commonmark, text, xml, empty.
Requestrequesttext
Request.One of: smart, http, browser.
Metadatametadatatrue/false
Metadata.

Outputs

Response.

datalist
Data.
Example output
{
  "data": []
}

Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.

Inputs

UrlurltextRequired
Url.
Return formatreturn_formattext
Return format.One of: raw, markdown, commonmark, text, xml, empty.
Requestrequesttext
Request.One of: smart, http, browser.
Metadatametadatatrue/false
Metadata.

Outputs

Response.

datalist
Data.
Example output
{
  "data": []
}

Convert HTML

Convert existing HTML to the chosen content format. Uses Spider credits.

Inputs

DatadatalistRequired
Data.
Return formatreturn_formattext
Return format.One of: raw, markdown, commonmark, text, xml, empty.
Readabilityreadabilitytrue/false
Readability.
Cleancleantrue/false
Clean.
Clean fullclean_fulltrue/false
Clean full.

Outputs

Response.

dataobject
Data.
Example output
{
  "data": {
    "content": [],
    "error": "example",
    "status": 1
  }
}

Crawl a website

Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.

Inputs

UrlurltextRequired
Url.
Limitlimitnumber
Limit.
Return formatreturn_formattext
Return format.One of: raw, markdown, commonmark, text, xml, empty.
Requestrequesttext
Request.One of: smart, http, browser.
Depthdepthnumber
Depth.
Subdomainssubdomainstrue/false
Subdomains.
Metadatametadatatrue/false
Metadata.

Outputs

Response.

datalist
Data.
Example output
{
  "data": []
}

Crawl a website on the Unlimited plan

Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.

Inputs

UrlurltextRequired
Url.
Limitlimitnumber
Limit.
Return formatreturn_formattext
Return format.One of: raw, markdown, commonmark, text, xml, empty.
Requestrequesttext
Request.One of: smart, http, browser.
Depthdepthnumber
Depth.
Subdomainssubdomainstrue/false
Subdomains.
Metadatametadatatrue/false
Metadata.

Outputs

Response.

datalist
Data.
Example output
{
  "data": []
}

Get credit balance

Read the current Spider credit balance.

Inputs

None.

Outputs

Response.

dataobject
Data.
Example output
{
  "data": {
    "data": {
      "created_at": "example",
      "credits": 1.0,
      "id": "example",
      "updated_at": "example",
      "user_id": "example"
    }
  }
}

Scrape a page

Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.

Inputs

UrlurltextRequired
Url.
Return formatreturn_formattext
Return format.One of: raw, markdown, commonmark, text, xml, empty.
Requestrequesttext
Request.One of: smart, http, browser.
Metadatametadatatrue/false
Metadata.

Outputs

Response.

datalist
Data.
Example output
{
  "data": []
}

Scrape a page on the Unlimited plan

Fetch publicly accessible web content with your Spider account. Uses Spider credits or plan allowance.

Inputs

UrlurltextRequired
Url.
Return formatreturn_formattext
Return format.One of: raw, markdown, commonmark, text, xml, empty.
Requestrequesttext
Request.One of: smart, http, browser.
Metadatametadatatrue/false
Metadata.

Outputs

Response.

datalist
Data.
Example output
{
  "data": []
}

Search the web

Search public web pages by keyword. Uses Spider credits.

Inputs

SearchsearchtextRequired
Search.
Search limitsearch_limitnumber
Search limit.From 1 to 100.

Outputs

Response.

dataobject
Data.
Example output
{
  "data": {
    "content": []
  }
}

Take a screenshot

Capture a publicly accessible page as an image. Uses Spider credits.

Inputs

UrlurltextRequired
Url.

Outputs

Response.

datalist
Data.
Example output
{
  "data": []
}

On this page