Prerequisites

• An active AWS account and an S3 bucket from which data is to be ingested exist.

• You are logged as an IAM user with permission to:

  • Create an IAM policy.

  • Create an IAM user (to generate the access credentials).

• The access credentials are available to authenticate Hevo on your AWS account.

• Your S3 bucket policy provides the Hevo account ID access to the bucket and the objects within it.

---

Create an IAM Policy

Create an IAM policy with the ListBucket and GetObject permissions. These permissions are required for Hevo to access data from your S3 bucket.

To do this:

1. Log in to the AWS IAM Console.

2. In the left navigation pane, under Access management, click Policies.

   

3. In the Policies page, click Create policy.

   

4. In the Specify permissions page, click JSON, and in the Policy editor section, paste the following JSON statements:

   

   Note: Replace the placeholder values in the commands below with your own. For example, <your_bucket_name> with Hevo-S3-bucket.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:ListBucket"
               ],
               "Resource": [
                   "arn:aws:s3:::<your_bucket_name>",
                   "arn:aws:s3:::<your_bucket_name>/*"
               ]
           }
       ]
   }
   

   The JSON statements allow Hevo to access and ingest data from the bucket name you specify.

   

5. At the bottom of the page, click Next.

   

6. In the Review and create page, specify the Policy name, and at the bottom of the page, click Create policy.

   

---

Obtain Amazon S3 Credentials

You must generate access credentials to access and ingest your S3 data.

Your access credentials include the access key and the secret access key. To generate these, you need to create an IAM user for Hevo and assign the policy you created in Step 1 above to it.

Note: The secret key is associated with an access key and is visible only once. Therefore, you must make sure to save the details or download the key file for later use.

1. Create an IAM user and assign the IAM policy

1. Log in to the AWS IAM Console.

2. In the left navigation pane, under Access management, click Users.

   

3. In the Users page, click Create user.

   

4. In the Specify user details page, specify the User name and click Next.

   

5. In the Set permissions page, Permissions options section, click Attach policies directly.

   

6. In the Permissions policies section, search and select the check box corresponding to the policy that you created in Step 1 above and at the bottom of the page, click Next.

   

7. At the bottom of the Review and create page, click Create user.

   

2. Generate the access keys

1. In the Users page of your IAM console, click the user that you created above.

   

2. In the <User name> page, select the Security credentials tab.

   

3. In the Access keys section, click Create access key.

   

4. In the Access key best practices & alternatives page, select Command Line Interface (CLI).

   

5. At the bottom of the page, select the I understand the above…. check box and click Next.

   

6. (Optional) Specify a description for the access key.

7. Click Create access key.

   

8. In the Retrieve access keys page, Access key section, click the copy icon in the Access key and Secret access key fields and save the keys securely like any other password.

   Optionally, click Download .csv file to save the keys on your local machine.

   Note: Once you leave this page, you cannot view these keys again.

   

Use these keys while configuring your Pipeline.

---

Create S3 Bucket Policy

Create a policy for your S3 bucket and Hevo access to the bucket and its objects. The Principal key in the policy provides this access to the Hevo account ID.

Note: The following steps must be performed by a Root user or a user with administrative access. In AWS, permissions and roles are managed through the IAM page.

1. Log in to your AWS account and open the Amazon S3 console.

2. In the left navigation pane of your Amazon S3 dashboard, click Buckets.

   

3. In the General purpose buckets page, click the S3 bucket you want to configure as your Source.

   

4. In the <Bucket name> page, under the Permissions tab, navigate to the Bucket policy section and click Edit.

   

5. In the Policy editor section, paste the following JSON statements:

   Note: Replace the placeholder values in the commands below with your own. For example, <your_bucket_name> with hevo-s3-bucket.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Principal": { "AWS": "arn:aws:iam::891695506716:root" },
               "Action": [
                   "s3:GetObject",
                   "s3:ListBucket"
               ],
               "Resource": [
                   "arn:aws:s3:::<your_bucket_name>",
                   "arn:aws:s3:::<your_bucket_name>/*"
               ]
           }
       ]
   }
   

   The JSON statements allow the Hevo account ID access to the specified bucket and its objects.

   

6. At the bottom of the page, click Save changes.

   

---