Cloud Orchestration Triggers

Cloud orchestration triggers let you kick off data pipeline runs in enterprise data platforms after your automation finishes scraping. Instead of manually triggering a pipeline in Azure Data Factory, Dataiku, or Apache Airflow every time your scrape completes, RTILA X does it for you automatically.

Think of it this way: your automation collects the raw data, and then these triggers tell your data platform to process, transform, and load that data into dashboards, reports, or data warehouses. It’s the bridge between “data collected” and “data ready for analysis.”

When to Use Cloud Orchestration Triggers

Scenario	Trigger to Use	Why
Your company uses Azure Data Factory for ETL pipelines	Azure Data Factory	Trigger ADF pipelines after each scrape completes
Your team uses Dataiku DSS for data science	Dataiku DSS	Run Dataiku scenarios automatically after data is collected
Your team uses Apache Airflow for workflow orchestration	Apache Airflow	Trigger Airflow DAGs to process scraped data
You need to schedule downstream transformations	Any of the above	Automate the entire data pipeline end-to-end
You want data to flow into BI dashboards automatically	Any of the above	Connect scraping → transformation → visualization

Azure Data Factory

Azure Data Factory (ADF) is Microsoft’s cloud-based ETL (Extract, Transform, Load) service. It lets you build data pipelines that move and transform data between different sources and destinations. If your organization uses Azure, ADF is likely your primary data pipeline tool.

Description

The Azure Data Factory trigger connects to your Azure Data Factory instance and either triggers a pipeline run or checks the status of an existing run. This means after your scraping automation finishes, RTILA X can automatically tell ADF to start processing the collected data.

Parameters

Parameter	Type	Required	Default	Description
subscription_id	string	✅ Yes	—	Your Azure subscription ID. You can find this in the Azure portal under Subscriptions or by running az account show in the Azure CLI. It looks like a GUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
resource_group	string	✅ Yes	—	The name of the Azure resource group that contains your Data Factory. Found in the Azure portal under Resource Groups.
factory_name	string	✅ Yes	—	The name of your Azure Data Factory instance. This is the name you gave it when you created it in Azure.
pipeline_name	string	✅ Yes	—	The name of the pipeline you want to trigger. This must match exactly the pipeline name in ADF.
operation	string	✅ Yes	—	What you want to do. Options: trigger_run (start a pipeline) or get_run_status (check if a pipeline finished).
run_id	string	❌ No	—	The unique ID of a specific pipeline run. Only used with get_run_status. You get this ID from the response of a trigger_run operation.
parameters	object	❌ No	—	Parameters to pass to the ADF pipeline. These are key-value pairs that the pipeline can use. Example: {"input_file": "products.csv"}. Supports ${variable} substitution.
tenant_id	string	❌ No	—	Your Azure Active Directory (Azure AD) tenant ID. Required for authentication. Found in the Azure portal under Azure Active Directory → Properties.
client_id	string	❌ No	—	The client (application) ID of your Azure AD service principal. This is the app registration you created for RTILA to authenticate with Azure.
client_secret	string	❌ No	—	The client secret for your service principal. This is like a password for your app registration.

Operations

Operation	What It Does	When to Use
trigger_run	Starts a new pipeline run in Azure Data Factory	After your scrape completes and you want ADF to process the data
get_run_status	Checks whether a specific pipeline run has finished	Use in a second trigger to verify the pipeline completed

Example: Trigger a Pipeline After Scraping

{
  "subscription_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "resource_group": "my-data-rg",
  "factory_name": "my-data-factory",
  "pipeline_name": "ProcessScrapedProducts",
  "operation": "trigger_run",
  "parameters": {
    "input_path": "scraped-data/products/${date}.csv"
  },
  "tenant_id": "12345678-1234-1234-1234-123456789012",
  "client_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "client_secret": "your-client-secret-here"
}

Example: Check Pipeline Status

{
  "subscription_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "resource_group": "my-data-rg",
  "factory_name": "my-data-factory",
  "pipeline_name": "ProcessScrapedProducts",
  "operation": "get_run_status",
  "run_id": "${adf_run_id}",
  "tenant_id": "12345678-1234-1234-1234-123456789012",
  "client_id": "abcdef12-3456-7890-abcd-ef1234567890",
  "client_secret": "your-client-secret-here"
}

Setting Up Azure Authentication

To use the Azure Data Factory trigger, you need an Azure AD service principal:

1. Go to the Azure portal → Azure Active Directory → App registrations.
2. Click New registration and give it a name (e.g., “RTILA X Integration”).
3. Copy the Application (client) ID — this is your client_id.
4. Copy the Directory (tenant) ID — this is your tenant_id.
5. Go to Certificates & secrets → New client secret.
6. Copy the Value of the secret — this is your client_secret.
7. Go to your Data Factory → Access control (IAM).
8. Add a role assignment: Data Factory Contributor for your service principal.

Tip: Store your client_secret as a project variable so it’s not visible in plain text in your trigger configuration.

Output

The trigger stores the Azure Data Factory response in the output variable:

Operation	Output Contains
trigger_run	The pipeline run ID, which you can use with get_run_status
get_run_status	The run status: Queued, InProgress, Succeeded, or Failed

---

Dataiku DSS

Dataiku DSS (Data Science Studio) is a collaborative data science platform used by data teams to design, build, and deploy data pipelines and machine learning models. If your data team uses Dataiku, this trigger lets you automatically run Dataiku scenarios after your scraping completes.

Description

The Dataiku DSS trigger connects to your Dataiku server and can run scenarios, check scenario status, or list datasets in a project. A “scenario” in Dataiku is an automated workflow that processes data — similar to a pipeline in other tools.

Parameters

Parameter	Type	Required	Default	Description
host	string	✅ Yes	—	The URL of your Dataiku server. Example: https://dataiku.company.com. Do not include a trailing slash.
api_key	string	✅ Yes	—	Your Dataiku API key. Generated in Dataiku under Administration → My Account → API Keys.
project_key	string	✅ Yes	—	The Dataiku project key where your scenario lives. This is a short code like SCRAPING or PRD_ANALYTICS. Found in the Dataiku project URL.
operation	string	✅ Yes	—	What you want to do. Options: run_scenario, get_scenario_status, list_datasets.
scenario_id	string	❌ No	—	The ID of the scenario to run or check. Required for run_scenario and get_scenario_status. Found in the Dataiku scenario URL.
wait_for_completion	boolean	❌ No	false	If true, RTILA waits for the scenario to finish before moving to the next trigger. If false, RTILA triggers the scenario and moves on immediately.

Operations

Operation	What It Does	When to Use
run_scenario	Triggers a Dataiku scenario to run	After scraping, to process the collected data
get_scenario_status	Checks if a scenario has finished running	Verify processing completed before proceeding
list_datasets	Lists all datasets in the Dataiku project	Discover what datasets are available

Example: Run a Scenario After Scraping

{
  "host": "https://dataiku.company.com",
  "api_key": "abc123def456ghi789jkl012mno345",
  "project_key": "WEB_SCRAPING",
  "operation": "run_scenario",
  "scenario_id": "ProcessDailyScrape",
  "wait_for_completion": true
}

Example: List Available Datasets

{
  "host": "https://dataiku.company.com",
  "api_key": "abc123def456ghi789jkl012mno345",
  "project_key": "WEB_SCRAPING",
  "operation": "list_datasets"
}

Finding Your Dataiku Settings

Setting	Where to Find It
Server URL	Your browser’s address bar when logged into Dataiku (e.g., https://dataiku.company.com)
API Key	Dataiku → Click your username → My Account → API Keys → Generate new key
Project Key	Open a project in Dataiku → Look at the URL: https://dataiku.company.com/projects/WEB_SCRAPING/ — the project key is WEB_SCRAPING
Scenario ID	Open a scenario → Look at the URL or the scenario settings

Output

Operation	Output Contains
run_scenario	The scenario run details, including run ID and initial status
get_scenario_status	The current status: RUNNING, SUCCESS, FAILED, etc.
list_datasets	A list of all dataset names in the project

---

Apache Airflow

Apache Airflow is the most popular open-source workflow orchestration tool. It lets you define data pipelines as code (called DAGs — Directed Acyclic Graphs) and schedule them to run automatically. If your team uses Airflow, this trigger lets you kick off DAG runs after your scraping completes.

Description

The Apache Airflow trigger connects to your Airflow webserver via the REST API (v2) and can trigger DAG runs, check run status, or list available DAGs. This lets you automate the handoff from scraping to data processing.

Parameters

Parameter	Type	Required	Default	Description
base_url	string	✅ Yes	—	The URL of your Airflow webserver. Example: https://airflow.company.com or http://localhost:8080.
username	string	✅ Yes	—	Your Airflow username. Created in Airflow under Admin → Users.
password	string	✅ Yes	—	Your Airflow password.
dag_id	string	✅ Yes	—	The ID of the DAG you want to trigger. This must match exactly the DAG ID defined in your Airflow DAG file.
operation	string	✅ Yes	—	What you want to do. Options: trigger_dag, get_dag_run_status, list_dags.
dag_run_id	string	❌ No	—	The unique ID of a specific DAG run. Only used with get_dag_run_status.
conf	object	❌ No	—	Configuration to pass to the DAG. These are key-value pairs the DAG can access. Supports ${variable} substitution.
execution_date	string	❌ No	—	The execution date in ISO 8601 format (e.g., 2025-01-15T00:00:00Z). If not provided, Airflow uses the current time.

Operations

Operation	What It Does	When to Use
trigger_dag	Triggers a new DAG run in Airflow	After scraping, to process the collected data
get_dag_run_status	Checks if a specific DAG run has finished	Verify processing completed
list_dags	Lists all available DAGs	Discover what workflows are available

Example: Trigger a DAG After Scraping

{
  "base_url": "https://airflow.company.com",
  "username": "rtila_user",
  "password": "airflow-password",
  "dag_id": "process_scraped_products",
  "operation": "trigger_dag",
  "conf": {
    "input_file": "products_${date}.csv",
    "source": "rtila_scraper"
  }
}

Example: Check DAG Run Status

{
  "base_url": "https://airflow.company.com",
  "username": "rtila_user",
  "password": "airflow-password",
  "dag_id": "process_scraped_products",
  "operation": "get_dag_run_status",
  "dag_run_id": "${airflow_run_id}"
}

Example: List All Available DAGs

{
  "base_url": "https://airflow.company.com",
  "username": "rtila_user",
  "password": "airflow-password",
  "dag_id": "",
  "operation": "list_dags"
}

Finding Your Airflow Settings

Setting	Where to Find It
Webserver URL	Your browser’s address bar when logged into Airflow (e.g., https://airflow.company.com)
Username	Created by your Airflow admin, or in Admin → Users
Password	Set when the user was created
DAG ID	Open Airflow → Look at the DAG list → The ID is shown in the first column

Output

Operation	Output Contains
trigger_dag	The DAG run details, including the run ID
get_dag_run_status	The run state: queued, running, success, or failed
list_dags	A list of all DAG IDs

---

Choosing the Right Orchestration Trigger

Scenario	Recommended Trigger
Your company uses Microsoft Azure	Azure Data Factory
Your data team uses Dataiku	Dataiku DSS
Your team uses open-source orchestration	Apache Airflow
You need to pass parameters to the pipeline	Any of them (all support parameters/conf)
You need to wait for the pipeline to finish	Dataiku (wait_for_completion: true)

Common Patterns

Pattern 1: Scrape → Transform → Load

Project Run → Extract Data → Export to File → Azure Data Factory (trigger_run)

Your automation scrapes data, exports it to a file, then triggers an ADF pipeline that transforms and loads the data into a warehouse.

Pattern 2: Scrape → Process → Verify

Project Run → Extract Data → Dataiku (run_scenario) → Dataiku (get_scenario_status)

Trigger a Dataiku scenario, then check that it completed successfully.

Pattern 3: Scrape → Airflow → Notify

Project Run → Extract Data → Airflow (trigger_dag) → Slack Webhook (notify team)

Trigger an Airflow DAG and send a Slack notification to your team.

Tips and Best Practices

Tip	Why It Matters
Test authentication before running	Verify your Azure/ADF/Airflow credentials work before adding them to production automations
Use ${variable} substitution in parameters	Makes your pipeline triggers dynamic and reusable
Use wait_for_completion when order matters	If the next trigger depends on the pipeline finishing, wait for it
Store credentials as project variables	Avoids hardcoding secrets in your trigger configuration
Check pipeline status after triggering	Use get_run_status in a second trigger to verify success
Use descriptive DAG/pipeline/scenario names	Makes it easier to identify which pipeline was triggered

Error Handling

Error	Cause	Solution
401 Unauthorized	Wrong credentials	Check username/password or client_id/client_secret
403 Forbidden	Insufficient permissions	Grant the necessary roles/permissions to your service account
404 Not Found	Wrong pipeline/DAG/scenario name	Verify the name matches exactly what’s in the platform
Connection timeout	Server unreachable	Check the URL and your network connection
Pipeline/DAG not found	Wrong ID	Verify the pipeline/DAG/scenario ID in the platform’s UI