Motivation

This document describes how to expose a resource inside AWS infrastructure with VPC Endpoint Service (VPCe Service or PrivateLink). It allows configuring access to the resource from the Astro environment, based on the Custom Endpoint solution.

The following example is for EC2 instance with launched OpenSSH server on port 22.

Consideration about Custom VPCe Service

Benefits of VPC Endpoint Services

1. Enhanced Security: VPC Endpoint Services allow you to keep your services private within your VPC, reducing exposure to the public internet.
2. Reduced Data Transfer Costs: Traffic between your VPC and the endpoint service stays within the AWS network, potentially lowering data transfer costs.
3. Improved Performance: By avoiding the public internet, you may experience lower latency and higher throughput.
4. Simplified Network Architecture: Eliminates the need for internet gateways, NAT devices, or VPN connections for accessing AWS services.
5. Granular Access Control: You can control which principals (AWS accounts, IAM users, and IAM roles) can access your endpoint service.
6. Private DNS Support: You can associate a private DNS name with your endpoint service for easier access.

Limitations of VPC Endpoint Services

1. Regional Availability: VPC Endpoint Services are region-specific, which may limit global accessibility.
2. DNS Complexity: If using private DNS, there can be added complexity in DNS configuration and management.
3. Scalability Considerations: Each VPC Endpoint Service is associated with a specific Network Load Balancer, which has its own scalability limits.
4. Cost Implications: While potentially reducing data transfer costs, there are charges associated with VPC Endpoint usage.

Setup

Prerequisites (EC2 instance is just an example)

1. Working target resource (e.g. EC2 instance).
   
2. Security group with Astro network ranges (172.20.0.0/19 in our example) and all ports (port 22 for OpenSSH) you want to use for receiving traffic.
   
3. Security group for your target (e.g. EC2 instance) that allows all traffic from the NLB Security group created in Step 2:
   

Create Target group

1. Get the IP address (e.g 172.31.23.2) for the resource you want to connect to (it might be multiple IP addresses for the same service) and the ports you have available to use for access to service\application.
2. Go to Target groups menu in AWS EC2 section.
3. Set target type as "IP addresses" and the name of target group, specify TCP port where traffic has to be forwarded and VPC for balancer creation.
   
4. In the next menu, select VPC with targets and add all the IP addresses you need. Click Include as a pending below, then finish creation.
   
5. After you see all targets as Healthy, this step for creating the target group can be counted as completed. Repeated this process for each of the services you want to access over PrivateLink.

Create NLB

1. Go to Load balancer menu and start new NLB creation.
2. Specify name of the NLB, select Internal and IPv4 in Basic configuration.
   
3. In the Network mapping section, select your VPC and Availability zones with the subnets, respectively. Multiple AZ selection is recommended for the redundancy.
   Important. VPC Endpoint in the remote account has to match AZ ID of the subnets what you select in NLB network mapping. AWS illustration is below:
   
4. In Listeners and routing section, set TCP port where you want to receive a traffic. This is the same port you use in the NLB Security group. In Default action, set the Target group you created before. Note that one Listener can forward traffic to one target group. If you want to use the same NLB for the multiple services, you must add listeners accordingly.
   
5. Complete NLB creation and wait until new resource provisioning is finished (takes several minutes).
6. Go to the Attributes tab for the created NLB and make sure that Cross-zone load balancing attribute is enabled.

Create VPC Endpoint Service

1. Open Endpoint Service menu and start creation process.
2. Set the name and Network type. In list of balancers select one created before.
   
3. In Additional settings you can select if incoming Endpoint connections have to be accepted manually or automatically.
4. Complete VPC Endpoint Service creation and go to the Allow principals tab in your newly created resources.
5. Add the following principal as allowed arn:aws:iam::<Astro AWS account ID>:role/astronomer-remote-management.
   
6. Get Service name (e.g. com.amazonaws.vpce.us-east-2.vpce-svc-0aa6ce6f77b2fe73c) and let the Astro team know for the next steps.

Potential questions and recommendations

Multiple targets behind single NLB

A single pair of VPCe Service and NLB can be used for the multiple target groups:

• In the case of multiple target groups without overlapping ports, you can add as many Listeners and targets as you need.
• If you need to add multiple target groups for the same port (e.g. multiple targets for port 443), use unique Listener ports.

The main rule about traffic separation is illustrated in the following AWS diagram, where TCP\TLS are Listeners examples and circles are Target groups.

Matching for listening and forwarding ports

In the example case, we added an EC2 instance for access to OpenSSH (port 22). We set this port for the target group and for the listener but could it be done differently? The Target group port must always match the port at your resource that can accept connections. Otherwise, the health check will not pass, and traffic will not be forwarded to the target at all. Technically, you can use one port for health check and another for actual traffic, but it adds unnecessary complexity to the configuration.

For the listener, you have full flexibility and, any port can be defined. One thing that requires attention, all ports from the Listeners list have to be allowed in the Security group assigned on NLB.
If you have a single target group or multiple target groups without overlapping ports — use the same port for Listener and Target group. If you need to add multiple target groups for the same port (e.g. multiple targets for port 443), use unique Listener ports.

About Cross Region Endpoints

Cross region endpoints are not possible. The only way to configure cross region endpoints is to create an intermediate VPC in the same region as Astro, and put the NLB in there:

(original diagram)