Cloud Storage Triggers

Cloud storage triggers upload your scraped data files to cloud storage services. Instead of saving files to your local computer, these triggers automatically upload them to Amazon S3, Google Cloud Storage, or Azure Blob Storage.

Cloud storage triggers are essential for teams that need to share data, store large datasets, or integrate with cloud-based analytics tools.

AWS S3

The AWS S3 trigger uploads files to an Amazon S3 bucket. Amazon S3 is the most popular cloud storage service, used by millions of businesses worldwide.

Description

The AWS S3 trigger uploads a file from your local RTILA output directory to an Amazon S3 bucket. You can also download files from S3 to your local machine.

Parameters

Parameter	Type	Required	Default	Description
aws_access_key_id	string	✅ Yes	“”	Your AWS access key ID. Found in your AWS IAM console under Users → Security credentials.
aws_secret_access_key	string	✅ Yes	“”	Your AWS secret access key. Shown only once when you create the key.
region_name	string	✅ Yes	“”	The AWS region where your bucket is located. Example: `us-east-1`, `eu-west-1`, `ap-southeast-1`.
bucket_name	string	✅ Yes	“”	The name of your S3 bucket. Must be globally unique across all AWS accounts.
operation	string	✅ Yes	“upload”	The operation to perform. Options: `upload` (send file to S3) or `download` (get file from S3).
source_path	string	✅ Yes	“”	The local file path to upload (for upload) or the destination path (for download). Relative to the RTILA output directory.
destination_key	string	✅ Yes	“”	The S3 object key (path within the bucket). Example: `data/products/2024/report.csv`.

Example: Upload a File to S3

{
  "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
  "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "region_name": "us-east-1",
  "bucket_name": "my-scraping-data",
  "operation": "upload",
  "source_path": "output/products_report.csv",
  "destination_key": "scraped-data/products/${date}/report.csv"
}

Example: Download a File from S3

{
  "aws_access_key_id": "AKIAIOSFODNN7EXAMPLE",
  "aws_secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "region_name": "us-east-1",
  "bucket_name": "my-scraping-data",
  "operation": "download",
  "source_path": "downloads/config.json",
  "destination_key": "config/settings.json"
}

Tips

Create your S3 bucket before running the trigger. The trigger will fail if the bucket doesn’t exist.
Use descriptive destination keys with dates to organize your files (e.g., data/2024/01/report.csv).
Store your AWS credentials as project variables to avoid hardcoding them.
Use IAM roles instead of access keys when possible for better security.
S3 bucket names must be globally unique. If your chosen name is taken, choose a different one.

Google Cloud Storage

The Google Cloud Storage trigger uploads files to a Google Cloud Storage bucket. Google Cloud Storage is Google’s cloud storage service, integrated with the Google Cloud Platform.

Description

The Google Cloud Storage trigger uploads a file from your local RTILA output directory to a Google Cloud Storage bucket. You can also download files from GCS to your local machine.

Parameters

Parameter	Type	Required	Default	Description
gcp_credentials_json	string	✅ Yes	“”	Your Google Cloud service account JSON credentials. This is the full JSON string from your service account key file.
bucket_name	string	✅ Yes	“”	The name of your GCS bucket.
blob_name	string	✅ Yes	“”	The object name (path within the bucket). Example: `data/products/report.csv`.
operation	string	✅ Yes	“upload”	The operation to perform. Options: `upload` or `download`.
source_path	string	✅ Yes	“”	The local file path to upload (for upload) or the destination path (for download).

Example: Upload a File to GCS

{
  "gcp_credentials_json": "{\"type\": \"service_account\", \"project_id\": \"my-project\", ...}",
  "bucket_name": "my-scraping-data",
  "blob_name": "scraped-data/products/${date}/report.csv",
  "operation": "upload",
  "source_path": "output/products_report.csv"
}

Tips

Create a service account in the Google Cloud Console with storage permissions.
Download the service account JSON key file and paste its contents into the gcp_credentials_json field.
Store your GCP credentials as a project variable to avoid hardcoding them.
GCS bucket names must be globally unique.
Use the blob_name parameter to organize files within your bucket using folder-like paths.

Azure Blob Storage

The Azure Blob Storage trigger uploads files to an Azure Blob Storage container. Azure Blob Storage is Microsoft’s cloud storage service, part of the Azure cloud platform.

Description

The Azure Blob Storage trigger uploads a file from your local RTILA output directory to an Azure Blob Storage container. You can also download files from Azure Blob Storage to your local machine.

Parameters

Parameter	Type	Required	Default	Description
connection_string	string	✅ Yes	“”	Your Azure Storage connection string. Found in the Azure portal under Storage account → Access keys.
container_name	string	✅ Yes	“”	The name of the blob container.
blob_name	string	✅ Yes	“”	The blob name (path within the container). Example: `data/products/report.csv`.
operation	string	✅ Yes	“upload”	The operation to perform. Options: `upload` or `download`.
source_path	string	✅ Yes	“”	The local file path to upload (for upload) or the destination path (for download).

Example: Upload a File to Azure Blob Storage

{
  "connection_string": "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=xxxx;EndpointSuffix=core.windows.net",
  "container_name": "scraping-data",
  "blob_name": "products/${date}/report.csv",
  "operation": "upload",
  "source_path": "output/products_report.csv"
}

Tips

Find your connection string in the Azure portal: Storage account → Access keys → Connection string.
Create the container before running the trigger if it doesn’t exist.
Use the blob_name parameter to organize files within your container.
Store your connection string as a project variable to avoid hardcoding it.
Azure Blob Storage supports both block blobs (for files) and page blobs (for virtual disks). RTILA uses block blobs.

Choosing the Right Cloud Storage Trigger

Scenario	Recommended Trigger
Already using AWS	AWS S3
Already using Google Cloud	Google Cloud Storage
Already using Microsoft Azure	Azure Blob Storage
Need the most popular option	AWS S3
Need integration with Google services	Google Cloud Storage
Need integration with Microsoft services	Azure Blob Storage

Common Cloud Storage Patterns

Pattern 1: Upload After Each Run

Configure a cloud storage trigger to upload your scraped data after every run:

Project Run → Extract Data → Export to File → Upload to S3

Pattern 2: Convert Then Upload

Convert data to a specific format before uploading:

Project Run → Extract Data → Convert to CSV → Upload to GCS

Pattern 3: Compress Then Upload

Compress large files before uploading to save storage space:

Project Run → Extract Data → Export to File → Compress to ZIP → Upload to Azure Blob

Error Handling

Error	Cause	Solution
Invalid credentials	Wrong access key or secret	Verify your credentials are correct
Bucket not found	Bucket doesn’t exist	Create the bucket before running
Permission denied	Insufficient permissions	Check your IAM/ACL permissions
File not found	Source file doesn’t exist	Verify the source path is correct
Connection timeout	Network issue	Check your internet connection
Region mismatch	Wrong region specified	Verify the region name matches your bucket’s region

Security Best Practices

Practice	Why It Matters
Use IAM roles instead of access keys	IAM roles are more secure and don’t require storing credentials
Use environment variables for credentials	Avoid hardcoding credentials in your project configuration
Use least-privilege permissions	Grant only the minimum permissions needed (e.g., write-only for upload-only triggers)
Enable bucket versioning	Protect against accidental overwrites
Enable server-side encryption	Encrypt data at rest for compliance
Use lifecycle policies	Automatically delete old files to save storage costs