Queue System

The Queue System is the backbone of all project execution in RTILA X. Every time you run a project, schedule an automation, or trigger a trigger chain, a queue record is created. The system then processes these records one at a time, managing the lifecycle from “pending” through “running” to a final state like “completed” or “failed.”

Think of the queue as a to-do list for RTILA’s automation engine. You add tasks to the list, and the engine works through them in order, reporting back on each one’s progress.

---

How the Queue Works

The queue system follows a simple lifecycle:

1. A queue record is created with status pending. This happens when you click “Run,” when a scheduler triggers a project, or when a trigger chain is queued.
2. The executor picks up the record. A background process (called the “job executor”) subscribes to real-time changes on the queue collection. When it sees a new pending record, it starts processing it.
3. The status changes to running. The executor launches the automation engine (the rtila-engine sidecar process) with the appropriate arguments.
4. The automation runs. The engine opens a browser, executes your commands, extracts data, and saves results.
5. The record reaches a final status. Depending on the outcome, the status becomes completed, failed, stopped, or killed.

The Job Executor

The job executor is a Rust-based background process that runs inside the Tauri shell. It:

• Subscribes to the PocketBase queue collection using Server-Sent Events (SSE).
• Listens for create events (new pending jobs) and update events (status changes like “stopped”).
• Launches the appropriate sidecar process (rtila-engine for project runs, rtila-relay for trigger chain runs).
• Monitors the process output and updates the queue record when the process exits.
• Handles stopping requests by killing the running process.

Reconnection and Resilience

The job executor includes automatic reconnection logic:

• If the SSE connection drops, it reconnects with exponential backoff (starting at 2 seconds, doubling up to 60 seconds).
• If the database is temporarily unreachable, it retries until the connection is restored.
• On startup, it cleans up any stale “running” jobs left over from a previous crash.

---

Queue Statuses

Every queue record has a status that tells you what’s happening with that job:

Status	Meaning	What Happens Next
pending	The job is waiting to be picked up by the executor.	The executor will start it shortly.
running	The automation engine is actively processing this job.	You can monitor progress in the Live Monitor.
completed	The job finished successfully.	Results are saved. Triggers may fire.
failed	The job encountered an error and could not finish.	Check the error message. You can retry.
stopped	You manually stopped the job.	The session is marked as stopped.
killed	The job was forcefully terminated (e.g., by the stop handler).	Similar to stopped, but more abrupt.

Status Transitions

pending → running → completed
                  → failed
                  → stopped (user-initiated)
                  → killed (force-terminated)

A job can only move forward through these states. Once a job reaches a final state (completed, failed, stopped, or killed), it cannot be re-run. You would need to create a new queue record by running the project again.

---

Queue Record Fields

Each queue record contains the following fields:

Field	Type	Description
id	String	Unique identifier for this queue record.
type	String	The type of job: project_run, relay_run, or trigger_chain_run.
status	String	Current status (see table above).
args	Array	Command-line arguments passed to the automation engine.
pid	Number	The process ID of the running engine process (set when running).
project	Relation	The project being executed (for project_run type).
session	Relation	The session created for this run (set after the job starts).
chat	Relation	The AI chat that triggered this run (if applicable).
owner	Relation	The user profile that owns this job.
scheduler	Relation	The scheduler that triggered this run (if scheduled).
checkpoint	JSON	Checkpoint data for resume capability.
variables	JSON	Runtime variables passed to the project.
trigger_chain	Relation	The trigger chain being executed (for trigger_chain_run type).
trigger_chain_run	Relation	The trigger chain run record (for trigger_chain_run type).
retry_attempt	Number	The current retry attempt number (for trigger chain retries).
correlation_id	String	A unique correlation ID for tracing this job across logs and systems.
exit_code	Number	The exit code of the engine process (set after completion).
error	String	Error message if the job failed.
created	Date	When the queue record was created.

The type Field

The type field determines how the executor processes the job:

Type	Sidecar Used	Description
project_run	rtila-engine	Runs a full automation project (browser, commands, extraction).
relay_run	rtila-relay	Executes post-execution triggers for a completed project.
trigger_chain_run	rtila-relay	Executes a trigger chain (a sequence of post-execution steps).

The correlation_id Field

Every queue record gets a unique correlation_id (a UUID) when it’s created. This ID is:

• Passed to the automation engine as a command-line argument.
• Included in all log messages generated during the run.
• Useful for tracing a specific run across multiple systems and log files.

You won’t typically need to interact with this field directly, but it’s invaluable when debugging issues or contacting support.

---

Real-Time Monitoring

RTILA X monitors the queue in real time using PocketBase subscriptions. This means the UI updates instantly when a job’s status changes — no manual refreshing needed.

How It Works

1. When you open the Automation Projects page, the project store initializes a queue listener.
2. This listener subscribes to all changes on the queue collection.
3. When a new pending or running job appears, it’s added to the runningJobs list in the store.
4. When a job’s status changes to a final state, it’s removed from the list.
5. The UI reacts to these changes: the “Run” button becomes a “Monitor” button, the running project count in the sidebar updates, and the Live Monitor can be opened.

The Running Jobs List

The project store maintains a list of all currently active jobs (those with status pending or running). This list is used to:

• Show a spinning icon next to running projects in the project table.
• Display the running project count in the sidebar.
• Enable the “Monitor” button instead of “Run” for active projects.
• Allow you to open the Live Monitor for any running project.

Polling Reconciler

In addition to real-time subscriptions, RTILA X includes a polling reconciler that runs every 5 seconds. This is a safety net that:

• Queries the actual queue state from the database.
• Removes any jobs from the local list that are no longer active.
• Adds any active jobs that might have been missed by the SSE subscription.

This ensures the UI always reflects the true state of the queue, even if a real-time event was missed.

---

Stopping a Running Project

You can stop a running project at any time. Here’s what happens:

From the UI

1. Select the running project in the project list.
2. Click Run (which opens the Live Monitor since the project is running).
3. In the Live Monitor, click Stop Project.
4. A confirmation dialog appears.
5. Click Stop Project to confirm.

What Happens Behind the Scenes

1. The UI sends an update to the queue record, changing its status to stopped.
2. The job executor detects this status change via its SSE subscription.
3. The executor identifies the running process using the pid field.
4. On Windows, it runs taskkill /PID <pid> /F /T to forcefully terminate the process tree.
5. On macOS/Linux, it runs kill -9 <pid> to terminate the process.
6. The session associated with the run is updated to stopped status.
7. The queue record’s status is updated to killed.
8. The project is removed from the running jobs list.

Notifications

If the stopped project was triggered by a scheduler with notifications enabled, a notification is sent:

“Project ‘[Project Name]’ was stopped by the user.”

---

Relay Runs (Post-Execution Triggers)

When a project run completes, RTILA X can automatically execute post-execution triggers — actions that send your scraped data to external destinations like databases, cloud storage, or email.

How Relay Runs Work

1. A project run completes with status completed.
2. The automation engine checks if the project has any triggers configured.
3. If triggers exist, the engine emits a relay-engaged event.
4. The Tauri shell catches this event and creates a new queue record with type relay_run.
5. The executor picks up this relay run and launches the rtila-relay sidecar.
6. The relay processes each trigger in sequence, sending data to the configured destinations.

Relay Run Queue Record

A relay run queue record has:

Field	Value
type	relay_run
project	The project that was just executed
session	The session of the completed run
owner	The user who owns the project
args	Arguments including the project ID, session ID, and API token

Monitoring Relay Runs

Relay runs appear in the running jobs list just like project runs. You’ll see them in the sidebar’s running project count. However, they don’t have a Live Monitor — they run in the background and complete automatically.

Relay Completion Notification

When a relay run completes, you’ll see a notification:

“Mission accomplished! Routing output payloads for ‘[Project Name]’ to your designated webhooks.”

This is the “RTILA Relay Engaged” toast that appears briefly when triggers are being processed.

---

Stale Job Cleanup

RTILA X includes an automatic cleanup mechanism to handle jobs that get stuck in a “running” state — for example, if the application crashes while a project is running.

The Cleanup Cron Job

A cron job called failStaleRunningSessions runs every hour. It:

1. Finds all sessions with status running that started more than 24 hours ago.
2. Marks each stale session as failed.
3. Sets the session’s endTime to the current time.
4. Finds the corresponding queue record and marks it as failed too.

Why This Is Needed

If RTILA X crashes or is force-closed while a project is running:

• The browser process might still be running (or might have been killed by the OS).
• The queue record remains in running status because no one updated it.
• The session also remains in running status.

Without cleanup, these stale records would:

• Show a permanently “running” project in the UI.
• Prevent you from running the project again (the system thinks it’s already running).
• Consume a slot in the running jobs count.

The hourly cleanup ensures these orphaned records are properly finalized.

Manual Cleanup

If you need to clean up a stuck job before the hourly cron runs:

1. Go to Project Runs in the sidebar.
2. Find the stuck session (it will show “running” status with no end time).
3. You can delete the session, which will also clean up the associated queue record.

Warning: Only delete sessions you’re certain are stuck. If a project is genuinely still running, deleting its session could cause issues.

---

Queue and the Free License

If you’re on the Free Community License, the queue system enforces your license limits:

• You can only have 1 project.
• The queue will still process runs for that project normally.
• If you try to create a second project, you’ll be prompted to upgrade.

The queue system itself doesn’t limit the number of concurrent runs — that’s determined by your project’s max_concurrent_workers setting.

---

Tips and Best Practices

Tip	Why It Matters
Don’t close RTILA X while a project is running.	The queue record will be left in “running” state until the hourly cleanup.
Check the running jobs count in the sidebar.	It tells you how many projects are actively executing.
Use the Live Monitor for long-running projects.	You can see real-time logs and stop the project if needed.
If a project seems stuck, check the Project Runs page.	You can see if the session is truly running or if it’s stale.
The correlation_id helps with debugging.	If you contact support, provide the correlation_id from the queue record.
Relay runs are automatic.	You don’t need to manually trigger them — they fire after project completion.
Stopped jobs can’t be resumed.	If you stop a project, you’ll need to run it again from the beginning (or from a checkpoint).