AI Vision Analysis

AI Vision Analysis is RTILA X’s ability to see a web page the way a human does. Instead of blindly guessing at HTML structure, RTILA takes a screenshot, paints numbered boxes over every visible element, and sends that annotated image to an AI model. The AI then identifies what’s on the page and generates precise selectors for scraping and automation.

Think of it as giving the AI a pair of eyes. Before Vision Analysis, automation tools could only “read” raw HTML code. With Vision Analysis, the AI looks at the page like you would — seeing buttons, forms, product grids, navigation menus — and figures out exactly what to click and where.

---

How Vision Analysis Works

The Vision Analysis pipeline follows a five-step process:

Screenshot → Annotate → AI Analyze → Generate Selectors → Build Config

Step-by-Step Breakdown

1. Screenshot Capture — RTILA scrolls through the entire page and captures a full-page screenshot. For very tall pages, it stitches multiple viewport screenshots together into one seamless image.

2. Element Annotation — Before taking the screenshot, RTILA paints numbered red bounding boxes over every visible element (buttons, links, text blocks, images, inputs). Each box gets a unique number (1, 2, 3…).

3. AI Analysis — The annotated screenshot is sent to your configured AI model (local or cloud) along with a system prompt that tells the AI what to look for.

4. Selector Generation — The AI identifies the elements you need and references them by their box numbers (e.g., <<BOX:5>>). RTILA then resolves these box numbers into real CSS selectors.

5. Configuration Output — The resolved selectors are assembled into a complete project configuration with datasets, commands, and extraction rules.

The Full Pipeline in Action

Here’s what happens behind the scenes when you trigger Vision Analysis:

User clicks "Auto-Extract Data with AI"
        │
        ▼
┌─────────────────────────────────┐
│  1. Pause recording (if active) │
│  2. Start navigation monitor    │
│  3. Scroll through entire page  │
│  4. Annotate visible elements   │
│  5. Capture full-page image     │
│  6. Stitch screenshots together │
│  7. Clean up annotations        │
└─────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────┐
│  8. Send annotated image to AI  │
│  9. AI returns dataset config   │
│  10. Resolve <<BOX:N>> tokens   │
│  11. Validate selectors         │
│  12. Send nav report for cmds   │
│  13. AI returns command config  │
│  14. Assemble final config      │
└─────────────────────────────────┘
        │
        ▼
  Datasets & Commands generated!

---

What the AI Detects

When the AI analyzes a page, it identifies the following:

Detection	Description	Example
Page Type	The general category of the page	E-commerce, blog, social media, form, search results
Layout Structure	How the page is organized	Header, navigation, sidebar, content grid, footer
Interactive Elements	Buttons, links, inputs, dropdowns, checkboxes	Login button, search box, “Add to Cart”
Data Structures	Repeating patterns of data	Product cards, article listings, table rows
Forms	Login forms, search forms, multi-step forms	Login form with username/password fields
Pagination	How to navigate between pages	Numbered pages, “Next” button, infinite scroll
Network APIs	Background API calls the page makes	/api/products, /api/search
Sitemaps	Links to sitemap files found in robots.txt	https://example.com/sitemap.xml
Shadow DOM	Whether the page uses Shadow DOM components	Web components with encapsulated DOM
Scrollable Containers	Elements that scroll independently	Product lists with their own scrollbar

Detection Report Structure

The analysis produces a structured report like this:

{
  "analysis": {
    "features": {
      "has_forms": true,
      "has_iframes": false,
      "has_shadow_dom": false,
      "has_pagination": true,
      "has_captcha": false,
      "has_infinite_scroll": false,
      "is_blocked_by_waf": false,
      "scrollable_containers": []
    },
    "counts": {
      "form_elements": 1,
      "input_elements": 3,
      "button_elements": 5,
      "iframe_elements": 0,
      "link_elements": 47,
      "image_elements": 23,
      "table_elements": 0
    },
    "content": {
      "main_content_selector": "main",
      "schema_dot_org": [],
      "meta": {
        "description": "...",
        "keywords": "..."
      }
    }
  }
}

---

When Vision Analysis Is Used

Vision Analysis is triggered in three scenarios:

1. AI Assistant Conversations

When you ask the AI Assistant to build an automation:

“Scrape product names and prices from https://example.com/products”

The AI uses Vision Analysis to see the page before generating commands. This ensures the selectors it generates actually match the page structure.

2. Visual Builder Wand (✨)

In the Visual Builder’s Data Extraction tab, click the wand icon (✨) to trigger AI Vision Analysis on the current page:

1. RTILA captures and annotates the current page.
2. The AI identifies data patterns and generates datasets.
3. The AI also generates navigation and extraction commands.
4. Everything is inserted into your project automatically.

3. Headless Capture (Right-Click)

While recording or browsing, right-click anywhere on the page and select “Auto-Extract Data with AI”:

1. RTILA performs a full headless capture of the page.
2. The annotated screenshot is sent to the AI.
3. Datasets and commands are generated in the background.
4. Results are stored in the project automatically.

Tip: Headless capture works even when the side panel is closed. The background service worker handles the entire capture process independently.

---

Element Annotation Process

The annotation process is the key to Vision Analysis. Here’s how it works:

What Gets Annotated

Element Type	Box Color	Example
Interactive (buttons, links, inputs)	Red border + yellow number	Buttons, links, form fields
Text (headings, paragraphs, spans)	Red border + yellow number	Titles, descriptions, labels
Images (img, svg)	Red border + yellow number	Product images, icons

The Annotation Map

Each annotated element gets an entry in the element map:

{
  "5": {
    "selector": "css=div.product-card > h2.title",
    "type": "text",
    "text": "Wireless Headphones",
    "tagName": "h2",
    "safeClasses": ["product-card", "title"]
  },
  "12": {
    "selector": "css=button.add-to-cart",
    "type": "interactable",
    "text": "Add to Cart",
    "tagName": "button",
    "safeClasses": ["add-to-cart"]
  }
}

Selector Generation Logic

The annotation system generates selectors using this priority:

1. Stable attributes first — data-id, data-product-id, aria-label, role, name
2. ID — If the element has a non-dynamic ID
3. Safe classes — Classes that aren’t auto-generated (filters out css-, sc-, Mui, etc.)
4. Position — :nth-of-type() as a last resort
5. XPath fallback — For complex nested structures

Dynamic Element Detection

The system automatically detects and avoids unstable selectors:

Pattern	Example	Why It’s Skipped
Pure numbers	12345	Dynamic IDs
Random strings	aB3kF9x2	Auto-generated hashes
UUIDs	f47ac10b-58cc-4372-a567-0e02b2c3d479	Session-based IDs
Long hex strings	a1b2c3d4e5f6	Build-time hashes
Short classes	ab	Too generic
No vowels	xyz	Likely auto-generated

---

Vision Context Report Format

When the AI completes its analysis, it returns a structured response:

Dataset Generation Response

{
  "datasets": {
    "products": {
      "item_selector": "<<BOX:3>>",
      "properties": [
        {
          "name": "title",
          "selector": "<<BOX:5>>",
          "type": "text"
        },
        {
          "name": "price",
          "selector": "<<BOX:7>>",
          "type": "text"
        }
      ]
    }
  }
}

Command Generation Response

{
  "commands": [
    {
      "command": "goto",
      "params": { "url": "<<START_URL>>" }
    },
    {
      "command": "extract_data",
      "params": { "dataset": "<<DATASET:products>>" }
    }
  ]
}

Token Resolution

Before the configuration is used, all tokens are resolved:

Token	Resolved To	Example
<<BOX:N>>	CSS selector from element map	css=div.product-card > h2
<<START_URL>>	The current page URL	https://example.com/products
<<DATASET:name>>	The primary dataset name	products

---

WAF Detection

The Vision Analysis includes automatic WAF (Web Application Firewall) detection. This tells you when a website is actively blocking automated access.

The Detection Flag

{
  "analysis": {
    "features": {
      "is_blocked_by_waf": true
    }
  }
}

What Triggers WAF Detection

The system checks for these indicators:

Indicator	Example
Page title contains “access denied”	“Access Denied - Example.com”
Page title contains “attention required”	“Attention Required! Cloudflare”
Page title contains “just a moment”	“Just a moment… (Cloudflare)”
Page text mentions Cloudflare + Ray ID	“Cloudflare Ray ID: 12345”
Page text mentions DataDome	“Protected by DataDome”
Page text asks for human verification	“Please verify you are a human”
Page asks to enable JavaScript/cookies	“Enable JavaScript and cookies to continue”
HTTP 403 or 429 responses from network	Blocked or rate-limited

What Happens When WAF Is Detected

When is_blocked_by_waf is true:

1. The AI informs you that the site has bot protection.
2. It recommends configuring proxies in your stealth settings.
3. It suggests enabling CAPTCHA solving if applicable.
4. It may recommend adjusting your humanoid behavior sensitivity.

Important: WAF detection is informational. It tells you the site is protected, but doesn’t prevent RTILA from attempting to scrape. You’ll need to configure appropriate stealth measures (proxies, CAPTCHA solving, humanoid behavior) to bypass the protection.

---

Configuring Vision Analysis

Vision Analysis uses your configured AI provider:

Provider	Vision Support	Notes
OpenRouter	✅ Full support	Any vision-capable model works
Local LLM (GGUF)	⚠️ Model-dependent	Only models with vision capabilities work
Local LLM (API)	⚠️ Model-dependent	Depends on the API server’s model

Vision-Specific Settings

Vision Analysis respects your global AI settings:

• Model — The model used for vision analysis must support image input
• Temperature — Set to 0.1 for deterministic selector generation
• Max Tokens — Set to 4000 to accommodate full dataset configurations

Tips for Best Results

Tip	Why It Matters
Use a vision-capable model	Not all models can process images
Ensure the page is fully loaded	Incomplete pages produce poor results
Scroll through the page first	Loads lazy-loaded content for better detection
Use on pages with clear structure	Product grids, tables, and lists work best
Retry on complex SPAs	Single-page apps may need multiple attempts