Network Commands

http_request

Make an HTTP request.

Parameter	Type	Required	Default	Description
url	string	✅	—	Request URL. Supports ${variable}.
method	string	❌	GET	HTTP method: GET, POST, PUT, DELETE, PATCH
headers	object	❌	—	Request headers as key-value pairs
params	object	❌	—	URL query parameters
body	any	❌	—	Request body (for POST/PUT/PATCH)
timeout	number	❌	30000	Request timeout (ms)

Example:

url: https://api.example.com/data
method: POST
headers:
  Authorization: Bearer ${api_key}
  Content-Type: application/json
body:
  query: ${search_term}
output_variable: api_response

crawl_links

Crawl a website by following links.

Parameter	Type	Required	Default	Description
link_type	string	❌	internal	Link type: internal, external, both
max_depth	number	❌	2	Maximum crawl depth
max_urls	number	❌	100	Maximum URLs to visit
concurrency	number	❌	1	Number of parallel crawls
allowed_extensions	array	❌	["html", "htm", "php", ...]	File extensions to follow
randomize	boolean	❌	false	Randomize visit order
delay	number	❌	1000	Delay between visits (ms)
wait_until	string	❌	load	Page load state to wait for
do	array	❌	—	Commands to run on each crawled page

infinite_scroll

Scroll until the bottom is reached or a max count is hit.

Parameter	Type	Required	Default	Description
selector	string	❌	—	Container element to scroll (omit for window)
max_scrolls	number	❌	0 (infinite)	Maximum scroll count
scroll_delay	number	❌	1500	Delay between scrolls (ms)
do	array	❌	—	Commands to run after each scroll (e.g., extract_data)

Example:

infinite_scroll:
  max_scrolls: 10
  scroll_delay: 2000
  do:
    - extract_data: products