Share
Amazon Redshift (Edge) Setup Guide

Amazon Redshift is a fully managed, reliable data warehouse service in the cloud that offers large-scale storage and analysis of data set and performs large-scale database migrations. It is a part of the larger cloud-computing platform Amazon Web Services (AWS).

Hevo can load data from any of your Pipelines into an Amazon Redshift data warehouse. You can set up the Redshift Destination on the fly, as part of the Pipeline creation process, or independently. The ingested data is first staged in Hevo’s S3 bucket before it is batched and loaded to the Amazon Redshift Destination.

If you are new to AWS and Redshift, you can follow the steps listed below to create an AWS account and after that, create an Amazon Redshift database to which the Hevo Pipeline will load the data. Alternatively, you can create users and assign them the required permissions to set up and manage databases within Amazon Redshift. Read AWS Identity and Access Management for more details.


Modifying Amazon Redshift Destination Configuration

You can modify some settings of your Amazon Redshift Destination after its creation. However, any configuration changes will affect all the Pipelines using that Destination

To modify the configuration of your Amazon Redshift Destination in Edge:

  1. In the detailed view of your Destination, do one of the following:

    • Click the More () icon to access the Destination Actions menu, and then click Edit Destination.

      Access Edit Destination

    • In the Destination Configuration section, click EDIT.

      Access Edit Destination

  2. On the <Your Destination Name> editing page:

    Edit Destination Configuration

    • You can specify a new name for your Destination, not exceeding 255 characters.

    • In the Authentication section, you can click Change to update the password for your database user.

  3. Click TEST & SAVE to check the connection to your Amazon Redshift Destination and then save the modified configuration.


Data Type Evolution in Redshift Destinations

Hevo has a standardized data system that defines unified internal data types, referred to as Hevo data types. During the data ingestion phase, the Source data types are mapped to the Hevo data types, which are then transformed into the Destination-specific data types during the data loading phase. A mapping is then generated to evolve the schema of the Destination tables.

The following image illustrates the data type hierarchy applied to Amazon Redshift Destination tables:

Redshift Data Type Hierarchy

Data Type Mapping

The following table shows the mapping between Hevo data types and Redshift data types:

Hevo Data Type Redshift Data Type
-  ARRAY
-  JSON
-  VARCHAR
-  XML
VARCHAR
BOOLEAN BOOLEAN
BYTEARRAY VARBYTE
-  BYTE
-  SHORT
SMALLINT
DATE DATE
-  DATETIME
-  TIMESTAMP
TIMESTAMP
DATETIME_TZ TIMESTAMPTZ
DECIMAL -  DECIMAL
-  VARCHAR
INTEGER INTEGER
LONG BIGINT
FLOAT REAL
DOUBLE DOUBLE PRECISION
TIME TIME
TIME_TZ TIMETZ

Handling the Decimal data type

For Amazon Redshift Destinations, Hevo maps DECIMAL data values with a fixed precision (P) and scale (S) to the DECIMAL (NUMERIC) or VARCHAR data types. This mapping is decided based on the number of significant digits (P) in the numeric value and the number of digits in the numeric value to the right of the decimal point (S). Refer to the table below to understand the mapping:

Precision and Scale of the Decimal Data Value Redshift Data Type
Precision: >0 and <= 38
Scale: > 0 and <= 37
NUMERIC

For precision and scale values other than those mentioned in the table above, Hevo maps the DECIMAL data type to a VARCHAR data type.

Read Numeric Types to know more about the data types, their range, and the literal representation Amazon Redshift uses to represent various numeric values.

Handling of Unsupported Data Types

Hevo does not allow direct mapping of a Source data type to any of the Amazon Redshift data types not listed in the table above. Hence, if the Source object is mapped to an existing Redshift table with columns of unsupported data types, it may become inconsistent. To prevent any inconsistencies during schema evolution, Hevo maps all unsupported data types to the VARCHAR data type in Redshift.


Destination Considerations

  • Hevo converts the table and column names to lowercase during loading. Therefore, if you want to query your Destination irrespective of the case, ensure that the warehouse parameter enable_case_sensitive_identifier remains set to its default value, FALSE. If this parameter is set to TRUE, you need to use the table and column names in lowercase when querying the Destination.

  • Hevo stages the ingested data in an Amazon S3 bucket, from where it is loaded to the Destination tables using the COPY command. Hence, if you have enabled enhanced VPC routing, ensure that your VPC is configured correctly. Enhanced VPC routing affects the way your Amazon Redshift cluster accesses other resources in your AWS network, such as the S3 bucket, specifically for the COPY and UNLOAD commands. Read Enhanced VPC Routing in Amazon Redshift.

  • Hevo uses the Amazon Redshift COPY command to load data into the Destination tables; however, it has the following limitations:

    • If a Source object has a row size exceeding 4 MB, Hevo cannot load the object, as the COPY command supports a maximum row size of 4 MB. For example, an object having a VARBYTE row with 6 MB of data cannot be loaded, even though the VARBYTE data type supports up to 16 MB. To avoid such a scenario, ensure that each row in your Source objects contains less than 4 MB of data.

    • The command does not support Not a Number (NaN) values in FLOAT and DOUBLE columns, so Hevo loads NaN values in these columns as NULL.

    • Amazon Redshift supports a maximum length of 65,535 bytes for VARCHAR field values. To handle this limitation, Hevo utilizes the TRUNCATECOLUMNS parameter in the COPY command when loading data into Destination tables. This ensures that if any VARCHAR field value exceeds the limit, Redshift automatically truncates the excess bytes. For example, if a Source object has an ARRAY, JSON, XML, or VARCHAR field value of 70,000 bytes, Redshift stores only the first 65,535 bytes and truncates the remaining data.

  • If a Source object has a floating-point field value that exceeds Amazon Redshift’s supported range, Hevo skips the record when loading data into Destination tables. These skipped records are logged as loading failures for their respective objects. You can check the object to see the failure reason, and from there, you can download the failed Events for further analysis.


Limitations

  • Hevo replicates a maximum of 1600 columns to each Amazon Redshift table, of which five are Hevo-reserved metadata columns used during data replication. Therefore, your Pipeline can replicate up to 1595 (1600-5) columns for each table.
Last updated on Jun 19, 2025

Tell us what went wrong

Skip to the section