Hightouch needs to store some data to power your syncs and to power features like the sync debugger. You have full control over where this data is stored.
If you prefer more direct control or have specific security concerns, you can configure Hightouch to store this data in your own Amazon S3, Google Cloud Storage, or Azure Blob Storage account. If you prefer convenience, Hightouch can store this data in our own secure, encrypted infrastructure.
Depending on your privacy and compliance needs, you can configure Hightouch to store all data-at-rest within your Virtual Private Cloud (VPC), or in a secure, encrypted bucket.
To prevent making excessive API requests and send only necessary updates to your destinations, Hightouch uses a process called change data capture (CDC) or diffing.
In this process, Hightouch stores query results and execution plans after each sync run. When the next sync run occurs, Hightouch uses these diff files to determine incremental changes that should be sent downstream.
By default, Hightouch stores the diff files in a secure encrypted bucket hosted by Hightouch. Business tier accounts can use the Lightning sync engine to compute and stores diffs directly within their data warehouse.
In addition to storing previous query results, Hightouch stores row-level log metadata including successes and failures, operations performed, and API request and response payloads. This data powers the in-app debugger and can be stored either in your VPC or Hightouch's encrypted bucket.
If you're on a Free, Starter, or Pro plan Hightouch stores data-at-rest in a secure, encrypted, Hightouch-managed bucket. For workspaces running in a Hightouch AWS region, this is an Amazon S3 bucket. For workspaces running in a Hightouch Google Cloud region, this is a Google Cloud Storage bucket. No matter your region, all data in Hightouch managed buckets is encrypted at rest. If you require data-at-rest to live entirely in your VPC, see self-hosted storage.
Data automatically expires from Hightouch-managed buckets after 30 days. If change data capture is done in Hightouch-managed buckets, syncs that have not run in over 30 days will require a Full Resync or Reset CDC sync since Hightouch depends on diffing files to detect changes in the data model.
Business tier customers can configure Hightouch to store all customer data-at-rest within their own external storage bucket or blob. Hightouch integrates with these cloud storage providers:
Amazon S3
Google Cloud Storage (GCS)
Microsoft Azure Blob Storage
If you choose to self-host your storage, Hightouch only processes data-in-transit. You can select any supported storage provider to store your data, regardless of your Hightouch region.
When hosting your own storage, Hightouch places full control over object lifecycle, security, and expiration into your hands.
We don't expire objects automatically or change your object encryption settings.
Ensure that you've configured object expiration, encryption, and access control settings according to your needs.
Setting up self-hosted storage disrupts the change data capture process for active syncs.
To reset it, after you've configured self-hosted storage, you need to trigger a full resync or reset cdc sync for all existing syncs that previously ran with Hightouch-managed storage.
Make sure that all your syncs satisfy the full resync prerequisites before setting up self-hosted storage.
Don't hesitate to if you have any doubts or concerns.
Once you've run a sync after setting up a custom storage bucket, you can't
make further changes to your storage configuration, including disabling it.
This is because changing your storage configuration is disruptive to Hightouch
syncs. If you need to make such a change, please
.
Hightouch supports authenticating with AWS using Cross-account roles (via STS AssumeRole), or with an Access Key ID / Secret Access Key that you provide. We strongly encourage you to use Cross-account roles, as it doesn't require Hightouch to hold any of your secrets.
In Hightouch, on the Storage tab of the Settings page, select Amazon S3 as the Cloud provider.
Select your AWS region, enter your Bucket name, and select the AWS credentials you previously set up.
Click Save.
Once you save your settings, your new syncs automatically start using your bucket. Run a few syncs and visit your S3 bucket to check files are saving there.
Don't hesitate to if you have any questions.
In the Google Cloud console, create a new bucket. We recommend the name <company>-hightouch-bucket. Copy the bucket name and save it for later.
Configure your bucket object lifecycle, to enhance security and cut down on costs.
Hightouch needs the following IAM permissions to store and retrieve items from your bucket:
Permission
Details
storage.objects.list
Grants access to view objects and their metadata, excluding ACLs. Can also list the objects in a bucket.
storage.objects.create
Grants permission to create, replace, and delete objects; list objects in a bucket; read object metadata when listing (excluding IAM policies); and read bucket metadata, excluding IAM policies.
storage.objects.get
Grants access to view objects and their metadata, excluding ACLs. Can also list the objects in a bucket.
In Hightouch, on the Storage tab of the Settings page, select Google Cloud Storage as the Cloud provider.
Enter the Project ID and Bucket name and select the Google Cloud credentials you previously set up.
Click Save.
Once you save your settings, your new syncs automatically start using your bucket. Run a few syncs and visit your Google Cloud bucket to check files are saving there.
Don't hesitate to if you have any questions.
In your Azure portal, create a container. We recommend the name <company>-hightouch-container. Copy the container name and save it for later.
Configure your storage lifecycle, to enhance security and cut down on costs.
The easiest way to grant Hightouch access is to grant the app the Storage Blob Contributor role for the storage account.
Alternatively, you may grant only the Storage Blob Delegator role at the account level, and Storage Blob Contributor
for the storage container.
If you want to create a custom role with more granular permissions, Hightouch needs the following IAM permissions to store and retrieve items from your Blob Storage Container:
Once you save your settings, your new syncs automatically start using your container. Run a few syncs and visit your Blob Storage container to check files are saving there.
Don't hesitate to if you have any questions.
Your Azure storage configuration may fail to save for a few reasons:
invalid_client: Check that you've provided your client secret value, not client secret ID. See the configuration docs for more information.
This request is not authorized to perform this operation: Check that Hightouch has adequate access to your storage account. If you've configured your storage account's Network access to be from selected virtual networks and IP addresses, be sure to add Hightouch's IP addresses to your network rules.
Ready to get started?
Jump right in or a book a demo. Your first destination is always free.