Datasets

A Dataset defines what data RTILA X extracts from a web page and how that data is structured. Think of a dataset as a template: you tell RTILA where to look on the page, what information to grab from each item, and how to process that information before saving it.

Every scraping project needs at least one dataset. Without a dataset, RTILA can navigate pages and click buttons, but it won’t know what data to collect.

---

Dataset Structure

Every dataset is made up of five core components:

Component	Description
Name	A unique identifier for the dataset. Used in commands like extract_data and when viewing results.
Item Selector	A CSS or XPath selector that identifies the repeating container element on the page. Each match becomes one row of data.
Properties	The individual fields (columns) to extract from each item. For example: title, price, image URL.
Required Fields	A list of property names that must have a value. If a required field is empty, a warning is shown.
Parent Dataset	An optional link to another dataset. Used for inheriting data from a parent extraction (advanced use).

---

Creating Datasets

There are three ways to create a dataset in RTILA X:

Method 1: Ask the AI Assistant

The fastest way. Simply tell the AI what you want to extract:

1. Open the AI Assistant from the sidebar.
2. Type a request like: “Extract all product names, prices, and image URLs from https://example.com/products”
3. The AI analyzes the page, identifies the repeating structure, and creates the dataset automatically.
4. The dataset appears in your project, ready to use.

Method 2: Build Manually in the Visual Builder

For full control over every selector and property:

1. Open your project in the Visual Builder (click Builder in the ribbon).
2. Switch to the Data Extraction tab.
3. Click the + button to add a new dataset.
4. A default dataset named “Default” is created with one property.
5. Click the element picker (pointer icon) next to the property’s selector field.
6. Click elements on the page to build your selectors visually.
7. Add more properties by clicking + in the properties area.

Method 3: AI Visual Extraction (Wand Icon)

Let AI look at the page and figure out the data structure for you:

1. In the Data Extraction tab, click the wand icon (✨).
2. RTILA takes a full-page screenshot and sends it to the AI.
3. The AI identifies repeating patterns, data structures, and interactive elements.
4. It generates one or more datasets with properties automatically.
5. You can review and adjust the generated selectors.

Tip: AI Visual Extraction works best on pages with clear, repeating structures like product grids, article lists, or table rows.

---

Item Selector

The Item Selector is the most important part of a dataset. It tells RTILA which element on the page repeats to form your data rows.

How It Works

Imagine a product listing page with 20 product cards. Each card has the same HTML structure:

<div class="product-card">
  <h2>Product Name</h2>
  <span class="price">$29.99</span>
  <img src="/images/product.jpg" />
</div>

The Item Selector would be: css=.product-card

RTILA finds all 20 elements matching .product-card and creates 20 rows of data.

Selector Format

Selectors use the format type=value:

Type	Example	Use Case
css=	css=.product-card	Most common. Uses standard CSS selectors.
xpath=	xpath=//div[@class='product']	For complex queries CSS can’t handle.
text=	text=Add to Cart	Match elements by their exact text.
variable=	variable=my_var	Use a runtime variable’s value as the selector.

Choosing a Good Item Selector

Strategy	Example	When to Use
Class name	css=.product-item	When items share a common class
ID-based	css=#results-list > li	When there’s a unique container
Attribute	css=[data-product-id]	When items have data attributes
Tag + class	css=article.post	Combining tag and class for specificity
XPath	xpath=//div[contains(@class,'card')]	When CSS isn’t specific enough

Warning: Avoid selectors that match only one element. If your item selector matches just one element, you’ll only get one row of data. The item selector should match the repeating container, not a single item inside it.

Configuring the Item Selector

1. In the Data Extraction tab, click the gear icon (⚙️) next to your dataset name.
2. The Dataset Settings modal opens.
3. In the Selector for Each Item’s Container field, enter or pick your selector.
4. Use the element picker to visually select the container on the page.
5. Click Save.

---

Properties

Properties define the individual columns of data extracted from each item. Each property has the following fields:

Field	Description
Name	The column name in your output data (e.g., “title”, “price”, “image_url”).
Selector	The selector to find this specific piece of data within each item. Relative to the item selector.
Type	What kind of data to extract (text, attribute, count, etc.). See below.
Attribute	(Only for attribute type) Which HTML attribute to read (e.g., href, src).
Property	(Only for property type) Which JavaScript DOM property to read (e.g., value, checked).
Download	If checked and the value is a URL, RTILA downloads the file automatically.
Transformations	A chain of processing steps applied to the value after extraction.

Property Types

RTILA X supports 9 property types:

Type	What It Extracts	Example Use
text	The visible text content of the element	Product names, descriptions
html	The inner HTML including tags	Rich text content
attribute	A specific HTML attribute value	Links (href), images (src)
property	A JavaScript DOM property	Input values, checkbox states
count	The number of elements matching the selector	Number of reviews, ratings
page_url	The current page URL	Tracking which page data came from
selector_path	The CSS path of the matched element	Debugging selector issues
list	A nested list of sub-items	Reviews within a product, tags
index	The row number (1-based)	Adding row numbers to output

Property Selector Behavior

Property selectors are relative to the item selector. If your item selector is css=.product-card and your property selector is css=h2, RTILA looks for an h2 element inside each .product-card.

You can also use css=self as a property selector to reference the item element itself.

The Download Option

When a property extracts a URL (like an image src), you can enable the Download option:

1. Select the property.
2. Check the Download file if value is a URL checkbox.
3. During execution, RTILA downloads the file and saves it locally.
4. The output value becomes the local file path instead of the URL.

Tip: Downloaded files are saved to RTILA’s output directory. The file path is stored in the extracted data so you can find it later.

---

Transformations

Transformations process extracted values after extraction. You can chain multiple transformations together, and they run in order.

Transformation	What It Does	Example
trim	Removes whitespace from start and end	" $29.99 " → "$29.99"
prefix	Adds text to the beginning	"29.99" → "$29.99"
suffix	Adds text to the end	"29.99" → "29.99 USD"
replace	Finds and replaces text	"$29.99" → "29.99"
regex	Extracts using a regular expression	"$29.99" → "29.99"
extract_regex	Extracts all regex matches as an array	Extract all emails from text
join	Joins an array into a string	["a","b"] → "a, b"
cast	Converts to integer or float	"29.99" → 29.99
json_parse	Parses a JSON string into an object	'{"a":1}' → {a: 1}
json_path	Extracts a value from JSON using a path	data.items[0].name
script	Runs custom JavaScript	return value.toUpperCase();

Adding Transformations

1. Select a property in the Data Extraction tab.
2. Click Edit Transformations.
3. Choose a transformation type from the dropdown.
4. Click Add.
5. Configure the transformation parameters.
6. Add more transformations as needed.
7. Drag to reorder transformations.
8. Click Save & Close.

Transformation Examples

Remove currency symbol and convert to number:

1. replace — Find: $, Replace with: "" (empty)
2. cast — To: float

Extract price from mixed text:

1. regex — Pattern: (\d+\.\d{2}), Group: 1
2. cast — To: float

Build full URL from relative path:

1. prefix — Value: https://example.com

---

Dataset Previews

The Data Extraction tab shows a live preview table at the bottom. This table shows exactly what data will be extracted based on your current selectors.

How the Preview Works

1. RTILA looks at the current page in the Visual Builder browser.
2. It finds all elements matching your Item Selector.
3. For each item, it extracts all Properties.
4. The results appear in the preview table immediately.

Refreshing the Preview

Click the refresh icon (🔄) floating button in the bottom-right corner to re-run the extraction on the current page state. This is useful after:

• Navigating to a different page
• Changing selectors
• Adding or removing properties

Interpreting Preview Results

Preview Shows	What It Means
Data rows with values	Your selectors are working correctly
Empty cells	The selector didn’t find data for that property
“(Empty Required)” in red	A required field has no value
“(Scripted value)”	The value comes from a script transformation
“(Variable value)”	The value comes from a runtime variable
“No matching data found”	Your item selector doesn’t match any elements

Tip: If the preview shows “No matching data found,” use the element picker to re-select your item selector. The page structure may have changed.

---

Required Fields

Required fields ensure data quality. When you mark a property as required, RTILA checks that the field has a value for every extracted row.

Setting Required Fields

1. Click the gear icon next to your dataset name.
2. In the Required Fields field, enter property names separated by commas.
3. Example: title,price
4. Click Save.

What Happens with Missing Required Fields

• The preview table shows (Empty Required) in red text for empty required cells.
• A warning banner appears above the preview table listing all missing required fields.
• The data is still extracted and saved — required fields generate warnings, not errors.

Note: Required fields do not prevent extraction. They serve as a visual indicator that some data may be incomplete. If you need to filter out incomplete rows, use the filter_condition parameter in the extract_data command.

---

Nested Datasets (List Properties)

Some pages have nested repeating structures. For example, a product page might have multiple reviews, each with a reviewer name, rating, and comment.

How List Properties Work

1. Add a property with type list.
2. Set its selector to the repeating sub-element (e.g., css=.review).
3. Add sub-properties inside the list property.
4. Each sub-property has its own selector, relative to the list item.

Example: Product with Reviews

Dataset: products
  Item Selector: css=.product-card
  Properties:
    - title (text, css=h2)
    - price (text, css=.price)
    - reviews (list, css=.review)
        - reviewer_name (text, css=.reviewer)
        - rating (attribute, css=.stars, attribute=data-rating)
        - comment (text, css=.comment)

This produces data like:

title	price	reviews
Product A	$29.99	[{reviewer_name: “John”, rating: “5”, comment: “Great!”}, …]

---

Multiple Datasets Per Project

A single project can have multiple datasets. This is useful when a page has different types of repeating data.

Example Use Cases

• E-commerce page: One dataset for products, another for categories
• News site: One dataset for headlines, another for author information
• Search results: One dataset for results, another for pagination info

Managing Multiple Datasets

• Datasets appear as tabs in the Data Extraction tab.
• Click a dataset tab to switch between them.
• Click the clone icon on an active dataset to duplicate it.
• Click the trash icon to delete a dataset.
• Each dataset is extracted independently.

How Extraction Works with Multiple Datasets

When your project runs, each extract_data command specifies which dataset to use:

Command 1: extract_data → dataset: "products"
Command 2: extract_data → dataset: "reviews"

Each dataset’s results are stored separately and can be exported independently.

---

How Extraction Works Step by Step

Here’s exactly what happens when RTILA extracts data:

1. Navigate to the page — RTILA loads the target URL in the browser.
2. Wait for content — RTILA waits for the page to finish loading.
3. Find items — RTILA searches the page for all elements matching the Item Selector.
4. For each item found:
   • RTILA looks inside the item for each Property’s selector.
   • It extracts the value based on the property type (text, attribute, etc.).
   • It applies any Transformations in order.
   • It checks Required Fields and flags missing values.
5. Build the result — All extracted items are collected into an array of objects.
6. Save the data — The results are stored in the project’s data collection.
7. Deduplication — RTILA checks if the same data was already extracted (using checkpoints) and skips duplicates.

Extraction and Checkpoints

RTILA uses a checkpoint system to avoid extracting the same data twice:

• Each extracted item is hashed and stored in memory.
• On subsequent runs, RTILA compares new items against previously extracted ones.
• Duplicates are skipped automatically.
• If you clear your database but not your checkpoint, you may see a warning about previously extracted data.

Tip: If you’ve cleared your database and want to re-extract all data, also clear your project’s checkpoint by starting a fresh run (not a resume).

---

Dataset Settings Modal

The Dataset Settings modal lets you configure advanced dataset options:

Setting	Description
Selector for Each Item’s Container	The item selector (see above)
Parent Dataset	Link to another dataset for data inheritance
Required Fields	Comma-separated list of property names that must have values

Parent Dataset (Advanced)

The Parent Dataset option allows one dataset to inherit data from another. This is useful for complex pages where data is spread across different sections.

When a dataset has a parent:

• The child dataset can access the parent’s extracted data.
• Properties from the parent are merged with the child’s properties.
• This enables multi-level data extraction workflows.

Note: Parent datasets are an advanced feature. Most scraping tasks only need a single dataset with a well-crafted item selector.

---

Tips and Best Practices

Tip	Why It Matters
Start with the item selector	Get this right first — everything else depends on it
Use the element picker	More reliable than typing selectors by hand
Test with the preview	Always check the preview before running the full project
Use specific selectors	css=.product-card h2 is better than css=h2
Avoid dynamic classes	Classes like css-1a2b3c change frequently
Add transformations for cleanup	Trim whitespace, remove symbols, convert types
Use required fields for quality	Flag incomplete data early
Name properties clearly	Use descriptive names like product_title not text1
Test on multiple pages	Make sure selectors work across different pages of the same site