Configure cross-account entry of Amazon Redshift clusters in Amazon SageMaker Studio utilizing VPC peering

With cloud computing, as compute energy and information grew to become extra out there, machine studying (ML) is now making an influence throughout each trade and is a core a part of each enterprise and trade.

Amazon SageMaker Studio is the primary absolutely built-in ML improvement surroundings (IDE) with a web-based visible interface. You’ll be able to carry out all ML improvement steps and have full entry, management, and visibility into every step required to construct, practice, and deploy fashions.

Amazon Redshift is a totally managed, quick, safe, and scalable cloud information warehouse. Organizations usually need to use SageMaker Studio to get predictions from information saved in a knowledge warehouse corresponding to Amazon Redshift.

As described within the AWS Well-Architected Framework, separating workloads throughout accounts permits your group to set frequent guardrails whereas isolating environments. This may be significantly helpful for sure safety necessities, in addition to to simplify value controls and monitoring between initiatives and groups. Organizations with a multi-account structure usually have Amazon Redshift and SageMaker Studio in two separate AWS accounts. Additionally, Amazon Redshift and SageMaker Studio are usually configured in VPCs with personal subnets to enhance safety and scale back the danger of unauthorized entry as a greatest follow.

Amazon Redshift natively supports cross-account information sharing when RA3 node varieties are used. In case you’re utilizing some other Amazon Redshift node varieties, corresponding to DS2 or DC2, you need to use VPC peering to determine a cross-account connection between Amazon Redshift and SageMaker Studio.

On this submit, we stroll by means of step-by-step directions to determine a cross-account connection to any Amazon Redshift node kind (RA3, DC2, DS2) by connecting the Amazon Redshift cluster situated in a single AWS account to SageMaker Studio in one other AWS account in the identical Area utilizing VPC peering.

Answer overview

We begin with two AWS accounts: a producer account with the Amazon Redshift information warehouse, and a shopper account for Amazon SageMaker ML use circumstances that has SageMaker Studio arrange. The next is a high-level overview of the workflow:

Arrange SageMaker Studio with VPCOnly mode within the shopper account. This prevents SageMaker from offering web entry to your studio notebooks. All SageMaker Studio visitors is thru the desired VPC and subnets.
Replace your SageMaker Studio area to activate SourceIdentity to propagate the consumer profile title.
Create an AWS Identity and Access Management (IAM) function within the Amazon Redshift producer account that the SageMaker Studio IAM function will assume to entry Amazon Redshift.
Replace the SageMaker IAM execution function within the SageMaker Studio shopper account that SageMaker Studio will use to imagine the function within the producer Amazon Redshift account.
Arrange a peering connection between VPCs within the Amazon Redshift producer account and SageMaker Studio shopper account.
Question Amazon Redshift in SageMaker Studio within the shopper account.

The next diagram illustrates our answer structure.

Conditions

The steps on this submit assume that Amazon Redshift is launched in a personal subnet within the Amazon Redshift producer account. Launching Amazon Redshift in a personal subnet offers an extra layer of safety and isolation in comparison with launching it in a public subnet as a result of the personal subnet isn’t immediately accessible from the web and safer from exterior assaults.

To obtain public libraries, you will need to create a VPC and a personal and public subnet within the SageMaker shopper account. Then launch a NAT gateway within the public subnet and add an web gateway for SageMaker Studio within the personal subnet to entry the web. For directions on methods to set up a connection to a personal subnet, seek advice from How do I set up a NAT gateway for a private subnet in Amazon VPC?

Arrange SageMaker Studio with VPCOnly mode within the shopper account

To create SageMaker Studio with VPCOnly mode, full the next steps:

On the SageMaker console, select Studio within the navigation pane.
Launch SageMaker Studio, select Commonplace setup, and select Configure.

In case you’re already utilizing AWS IAM Identity Center (successor to AWS Single Sign-On) for accessing your AWS accounts, you need to use it for authentication. In any other case, you need to use IAM for authentication and use your current federated roles.

Within the Basic settings part, choose Create a brand new function.
Within the Create an IAM function part, optionally specify your Amazon Simple Storage Service (Amazon S3) buckets by deciding on Any, Particular, or None, then select Create function.

This creates a SageMaker execution function, corresponding to AmazonSageMaker-ExecutionRole-00000000.

Underneath Community and Storage Part, select your VPC, subnet (personal subnet), and safety group that you just created as a prerequisite.
Choose VPC Solely, then select Subsequent.

Replace your SageMaker Studio area to activate SourceIdentity to propagate the consumer profile title

SageMaker Studio is built-in with AWS CloudTrail to allow directors to watch and audit consumer exercise and API calls from SageMaker Studio notebooks. You’ll be able to configure SageMaker Studio to report the consumer identification (particularly, the user profile name) to watch and audit consumer exercise and API calls from SageMaker Studio notebooks in CloudTrail occasions.

To log particular consumer exercise amongst a number of consumer profiles, we really helpful that you just activate SourceIdentity to propagate the SageMaker Studio area with the consumer profile title. This lets you persist the consumer data into the session so you may attribute actions to a particular consumer. This attribute can be persevered over while you chain roles, so you may get fine-grained visibility into their actions within the producer account. As of the time this submit was written, you may solely configure this utilizing the AWS Command Line Interface (AWS CLI) or any command line device.

To replace this configuration, all apps within the area should be within the Stopped or Deleted state.

Use the next code to allow the propagation of the consumer profile title because the SourceIdentity:

update-domain
--domain-id <worth>
[--default-user-settings <value>]
[--domain-settings-for-update "ExecutionRoleIdentityConfig=USER_PROFILE_NAME"]

This requires that you just add sts:SetSourceIdentity within the belief relationship on your execution function.

Create an IAM function within the Amazon Redshift producer account that SageMaker Studio should assume to entry Amazon Redshift

To create a task that SageMaker will assume to entry Amazon Redshift, full the next steps:

Open the IAM console within the Amazon Redshift producer account.

Select Roles within the navigation pane, then select Create function.

On the Choose trusted entity web page, choose Customized belief coverage.
Enter the next customized belief coverage into the editor and supply your SageMaker shopper account ID and the SageMaker execution function that you just created:

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<<SageMaker-Consumer-Account-ID>>:role/service-role/AmazonSageMaker-ExecutionRole-XXXXXX"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:SetSourceIdentity"
           ]
            
        }
    ]
}

Select Subsequent.
On the Add required permissions web page, select Create coverage.
Add the next pattern coverage and make obligatory edits based mostly in your configuration.

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Sid": "GetRedshiftCredentials",
            "Effect": "Allow",
            "Action": "redshift:GetClusterCredentials",
            "Resource": [
                "arn:aws:redshift:<<redshift-region-name>>:<<REDSHIFT-PRODUCER-ACCOUNT-ID>>:dbname:<<redshift-cluster-name>>/<<redshift-db-name>>",
                "arn:aws:redshift:<<redshift-region-name>>:<<REDSHIFT-PRODUCER-ACCOUNT-ID>>:dbuser:<<redshift-cluster-name>>/${redshift:DbUser}",
                "arn:aws:redshift:<<redshift-region-name>>:<<REDSHIFT-PRODUCER-ACCOUNT-ID>>:cluster:<<redshift-cluster-name>>"
            ],
            "Situation": {
                "StringEquals": {
                    "redshift:DbUser": "${aws:SourceIdentity}"
                }
            }
        },
        {
            "Sid": "DynamicUserCreation",
            "Impact": "Permit",
            "Motion": "redshift:CreateClusterUser",
            "Useful resource": "arn:aws:redshift:<<redshift-region-name>>:<<REDSHIFT-PRODUCER-ACCOUNT-ID>>:dbuser:<<redshift-cluster-name>>/${redshift:DbUser}",
            "Situation": {
                "StringEquals": {
                    "redshift:DbUser": "${aws:SourceIdentity}"
                }
            }
        },
        {
            "Impact": "Permit",
            "Motion": "redshift:JoinGroup",
            "Useful resource": "arn:aws:redshift:<<redshift-region-name>>:<<REDSHIFT-PRODUCER-ACCOUNT-ID>>:dbgroup:<<redshift-cluster-name>>/*"
        },
        {
            "Sid": "DataAPIPermissions",
            "Impact": "Permit",
            "Motion": [
                "redshift-data:ExecuteStatement",
                "redshift-data:CancelStatement",
                "redshift-data:ListStatements",
                "redshift-data:GetStatementResult",
                "redshift-data:DescribeStatement",
                "redshift-data:ListDatabases",
                "redshift-data:ListSchemas",
                "redshift-data:ListTables",
                "redshift-data:DescribeTable"
            ],
            "Useful resource": "*"
        },
        {
            "Sid": "ReadPermissions",
            "Impact": "Permit",
            "Motion": [
                "redshift:Describe*",
                "redshift:ViewQueriesInConsole"
            ],
            "Useful resource": "*"
        }
    ]
}

Save the coverage by including a reputation, corresponding to RedshiftROAPIUserAccess.

The SourceIdentity attribute is used to tie the identification of the unique SageMaker Studio consumer to the Amazon Redshift database consumer. The actions by the consumer within the producer account can then be monitored utilizing CloudTrail and Amazon Redshift database audit logs.

On the Title, evaluate, and create web page, enter a task title, evaluate the settings, and select Create function.

Replace the IAM function within the SageMaker shopper account that SageMaker Studio assumes within the Amazon Redshift producer account

To replace the SageMaker execution function for it to imagine the function that we simply created, full the next steps:

Open the IAM console within the SageMaker shopper account.
Select Roles within the navigation pane, then select the SageMaker execution function that we created (AmazonSageMaker-ExecutionRole-*).
Within the Permissions coverage part, on the Add permissions menu, select Create inline coverage.

Within the editor, on the JSON tab, enter the next coverage, the place <StudioRedshiftRoleARN> is the ARN of the function you created within the Amazon Redshift producer account:

{
    "Model": "2012-10-17",
    "Assertion": {
        "Impact": "Permit",
        "Motion": "sts:AssumeRole",
        "Useful resource": "<StudioRedshiftRoleARN>"
    }
}

You may get the ARN of the function created within the Amazon Redshift producer account on the IAM console, as proven within the following screenshot.

Select Assessment coverage.
For Title, enter a reputation on your coverage.
Select Create coverage.

Your permission insurance policies ought to look just like the next screenshot.

Arrange a peering connection between the VPCs within the Amazon Redshift producer account and SageMaker Studio shopper account

To ascertain communication between the SageMaker Studio VPC and Amazon Redshift VPC, the 2 VPCs must be peered utilizing VPC peering. Full the next steps to determine a connection:

In both the Amazon Redshift or SageMaker account, open the Amazon VPC console.
Within the navigation pane, select Peering connections, then select Create peering connection.
For Title, enter a reputation on your connection.
Underneath Choose an area VPC to look with, select an area VPC.
Underneath Choose one other VPC to look with, specify one other VPC in the identical Area and one other account.
Select Create peering connection.

Assessment the VPC peering connection and select Settle for request to activate.

After the VPC peering connection is efficiently established, you create routes on each the SageMaker and Amazon Redshift VPCs to finish connectivity between them.

Within the SageMaker account, open the Amazon VPC console.
Select Route tables within the navigation pane, then select the VPC that’s related to SageMaker and edit the routes.
Add CIDR for the vacation spot Amazon Redshift VPC and the goal because the peering connection.
Moreover, add a NAT gateway.
Select Save modifications.

Within the Amazon Redshift account, open the Amazon VPC console.
Select Route tables within the navigation pane, then select the VPC that’s related to Amazon Redshift and edit the routes.
Add CIDR for the vacation spot SageMaker VPC and the goal because the peering connection.
Moreover, add an web gateway.
Select Save modifications.

You’ll be able to hook up with SageMaker Studio out of your VPC by means of an interface endpoint in your VPC as an alternative of connecting over the web. Whenever you use a VPC interface endpoint, communication between your VPC and the SageMaker API or runtime is carried out completely and securely inside the AWS community.

To create a VPC endpoint, within the SageMaker account, open the VPC console.
Select Endpoints within the navigation pane, then select Create endpoint.
Specify the SageMaker VPC, the respective subnets and applicable safety teams to permit inbound and outbound NFS visitors on your SageMaker notebooks area, and select Create VPC endpoint.

Question Amazon Redshift in SageMaker Studio within the shopper account

After all of the networking has been efficiently established, comply with the steps on this part to connect with the Amazon Redshift cluster within the SageMaker Studio shopper account utilizing the AWS SDK for pandas library:

In SageMaker Studio, create a brand new pocket book.
If the AWS SDK for pandas bundle isn’t put in you may set up it utilizing the next:

!pip set up awswrangler #AWS SDK for pandas

This set up isn’t persistent and will likely be misplaced if the KernelGateway App is deleted. Customized packages might be added as a part of a Lifecycle Configuration.

Enter the next code within the first cell and run the code. Change RoleArn and region_name values based mostly in your account settings:

import boto3
import awswrangler as wr
import pandas as pd
from datetime import datetime
import json
sts_client = boto3.shopper('sts')

# Name the assume_role methodology of the STSConnection object and move the function
# ARN and a task session title.
assumed_role_object=sts_client.assume_role(
    RoleArn="arn:aws:iam::<<REDSHIFT-PRODUCER-ACCOUNT-ID>>:function/<<redshift-account-role>>",
    RoleSessionName="RedshiftSession"
)
credentials=assumed_role_object['Credentials']

# Use the non permanent credentials that AssumeRole returns to make a 
# connection to Amazon S3  
redshift_session=boto3.Session(
    aws_access_key_id=credentials['AccessKeyId'],
    aws_secret_access_key=credentials['SecretAccessKey'],
    aws_session_token=credentials['SessionToken'],
    region_name="<<redshift-region-name>>",
)

Enter the next code in a brand new cell and run the code to get the present SageMaker consumer profile title:

def get_userprofile_name():
    metadata_file_path="/decide/ml/metadata/resource-metadata.json"
    with open(metadata_file_path, 'r') as logs:
        metadata = json.load(logs)
    return metadata.get("UserProfileName")

Enter the next code in a brand new cell and run the code:

con_redshift = wr.redshift.connect_temp(
    cluster_identifier="<<redshift-cluster-name>>",
    database="<<redshift-db-name>>",
    consumer=get_userprofile_name(),
    auto_create=True,
    db_groups=[<<list-redshift-user-group>>],
    boto3_session = redshift_session
)

To efficiently question Amazon Redshift, your database administrator must assign the newly created consumer with the required learn permissions inside the Amazon Redshift cluster within the producer account.

Enter the next code in a brand new cell, replace the question to match your Amazon Redshift desk, and run the cell. This could return the information efficiently for additional information processing and evaluation.

df = wr.redshift.read_sql_query(
    sql="SELECT * FROM customers",
    con=con_redshift
)

Now you can begin constructing your information transformations and evaluation based mostly on your corporation necessities.

Clear up

To scrub up any sources to keep away from incurring recurring prices, delete the SageMaker VPC endpoints, Amazon Redshift cluster, and SageMaker Studio apps, customers, and area. Additionally delete any S3 buckets and objects you created.

Conclusion

On this submit, we confirmed methods to set up a cross-account connection between personal Amazon Redshift and SageMaker Studio VPCs in several accounts utilizing VPC peering and entry Amazon Redshift information in SageMaker Studio utilizing IAM function chaining, whereas additionally logging the consumer identification when the consumer accessed Amazon Redshift from SageMaker Studio. With this answer, you remove the necessity to manually transfer information between accounts to entry information. We additionally walked by means of methods to entry the Amazon Redshift cluster utilizing the AWS SDK for pandas library in SageMaker Studio and put together the info on your ML use circumstances.

To be taught extra about Amazon Redshift and SageMaker, seek advice from the Amazon Redshift Database Developer Guide and Amazon SageMaker Documentation.

In regards to the Authors

Supriya Puragundla is a Senior Options Architect at AWS. She helps key buyer accounts on their AI and ML journey. She is captivated with data-driven AI and the world of depth in machine studying.

Marc Karp is a Machine Studying Architect with the Amazon SageMaker group. He focuses on serving to clients design, deploy, and handle ML workloads at scale. In his spare time, he enjoys touring and exploring new locations.