Materialize

Materialize is a streaming database powered by Timely and Differential Dataflow, purpose-built for low-latency applications. It lets you ask complex questions about your data using SQL, and incrementally maintains the results of these SQL queries as the underlying data changes.

View Materialize's documentation.

Overview

Hightouch lets you pull data stored in your Materialize database and push it to downstream destinations. Most of the setup occurs in the Hightouch UI, but you need access to your Materialize instance for information like your host, port, database name, cluster, and credentials.

You may need to allowlist Hightouch's IP addresses to let our systems connect to your warehouse. Reference our networking docs to determine which IP addresses you need to allowlist.

Connection configuration

To get started, go to the Sources overview page and click the Add source button. Select Materialize and follow the steps below.

Configure your source

Enter the following required fields into Hightouch:

Host: The hostname or IP address of your Materialize server.
Port: The port number of your Materialize server. The default port number is 6875, but yours may be different.
Database: This specifies the database to use when Hightouch executes queries in Materialize.

Provide credentials

Enter the following fields into Hightouch:

User: This can be your personal Materialize login or a dedicated user for Hightouch. At minimum, this user must have read access to the data you wish to sync.
Password: The password for the user specified above.

Test your connection

When setting up a source for the first time, Hightouch validates the following:

Network connectivity
Materialize credentials
Permission to list schemas and tables
Permission to write to hightouch_planner schema
Permission to write to hightouch_audit schema

All configurations must pass the first three, while those with the Lightning engine must pass all of them.

Some sources may initially fail connection tests due to timeouts. Once a connection is established, subsequent API requests should happen more quickly, so it's best to retry tests if they first fail. You can do this by clicking Test again.

If you've retried the tests and verified your credentials are correct but the tests are still failing, don't hesitate to .

Next steps

Once your source configuration has passed the necessary validation, your source setup is complete. Next, you can set up models to define which data you want to pull from Materialize.

The Materialize source supports these modeling methods:

writing a query in the SQL editor
using the visual table selector
leveraging existing dbt models
leveraging existing Looker Looks
leveraging existing Sigma workbooks

You may also want to consider storing sync logs in Materialize. Like using the Lightning sync engine versus the standard one, this feature lets you use Materialize instead of Hightouch infrastructure. Rather than performance gains, it makes your sync log data available for more complex analysis. Refer to the warehouse sync logs docs to learn more.

You must enable the Lightning sync engine to store sync logs in your warehouse.

Tips and troubleshooting

If you encounter an error or question not listed below and need assistance, don't hesitate to . We're here to help.

Canceling statement due to statement timeout

This error occurs when the required execution time of your Materialize query exceeds the timeout limit for the database. To address the error, increase the timeout by executing the following query:

set statement_timeout = '300 s'; -- 300 seconds, 5 minutes

Be sure to adjust the time to be as long as it takes for your query to execute.