AWS Guide

This guide covers using Isolate with Amazon Web Services resources. Isolate integrates with AWS services to provide secure, audited access to RDS PostgreSQL databases and EKS clusters through SSM tunneling.

Overview

Isolate on AWS leverages native Amazon services for authentication, networking, and infrastructure provisioning. All database instances run in private subnets with no public IP addresses, accessible only through SSM Session Manager tunnels for enhanced security.

AWS Integration Features

  • RDS PostgreSQL: Databases provisioned from S3-stored pg_dump backups
  • EKS Cluster Access: Audited kubectl access with session recording
  • SSM Session Manager: Secure tunneling without bastion hosts or SSH keys
  • Private Subnets: All resources isolated from public internet
  • S3 + Athena Audit: SQL queries and K8s commands stored in S3, queryable via Athena
  • Multi-Account Federation: Discover and access resources across AWS accounts

RDS Instances

PostgreSQL on RDS

Database instances are created as Amazon RDS PostgreSQL instances from pg_dump backups stored in S3. The provisioning process involves:

  1. RDS Instance Creation: Pulumi creates an RDS PostgreSQL instance in a private subnet
  2. Backup Download: ECS Fargate task downloads pg_dump files from S3
  3. Database Restore: pg_restore runs against the RDS endpoint
  4. User Setup: Database users and permissions are configured

Provisioning typically takes 5-15 minutes (RDS instances take longer to start than CloudSQL). The instance progresses through: pending → provisioning → restoring → ready

S3 Backups

Production database backups are stored as pg_dump files in S3 buckets. When creating an instance, you can select from available backup sources organized by:

  • Environment: Production, staging, development
  • Service: Individual application services
  • Timestamp: Backup creation date and time
  • Databases: Specific databases within each backup

Limitations: No API SQL Import

Unlike CloudSQL, Amazon RDS does not support direct SQL file import via API. Instead, Isolate uses a networked approach:

Important Limitation

RDS does not have an equivalent to gcloud sql import. All data restoration happens through pg_restore over the network from ECS Fargate tasks. This means backup files must be accessible via network connection during provisioning.

For very large databases (>100GB), this approach may be slower than CloudSQL's direct import. Consider using RDS snapshots for large database provisioning instead of pg_dump files.

EKS Cluster Access

K8s Proxy via SSM

Kubernetes access goes through an auditing proxy running on an EC2 instance in a private subnet. The connection flow uses SSM Session Manager for secure tunneling:

kubectl → SSM Tunnel → K8s Proxy (EC2) → EKS Cluster

To connect to an EKS cluster:

  1. Ensure AWS CLI and Session Manager plugin are installed
  2. Run isolate k8s clusters to list available clusters
  3. Run isolate k8s connect <cluster-name>
  4. Export the generated kubeconfig: export KUBECONFIG=~/.isolate/kubeconfig-<cluster-name>.yaml
  5. Use kubectl: kubectl get pods

Session Recording

All kubectl exec sessions are recorded as asciinema v2 files and stored in S3. View recordings in the web UI at /k8s/recordings. Sessions automatically expire after 1 hour and require reconnection.

Connectivity

SSM Session Manager Tunneling

AWS Systems Manager Session Manager provides secure, auditable access to EC2 instances without SSH keys, bastion hosts, or public IPs. Isolate uses SSM for port forwarding:

aws ssm start-session --target i-0123456789abcdef0 \ --document-name AWS-StartPortForwardingSession \ --parameters '{"portNumber":["5432"],"localPortNumber":["5432"]}'

Prerequisites for SSM tunneling:

  • AWS CLI v2 installed and configured
  • Session Manager plugin: brew install session-manager-plugin (macOS)
  • Valid AWS credentials with SSM permissions
  • Target EC2 instance has SSM Agent and appropriate IAM role

Direct psql Connection

After the CLI establishes an SSM tunnel, connect with any PostgreSQL client:

# Via Isolate's auditing proxy (recommended) isolate connect <instance-id> psql -h localhost -p 5432 -U isolate -d mydb # Direct RDS connection (bypasses audit logging) psql "sslmode=require host=<rds-endpoint> user=isolate dbname=mydb"

Note: Direct RDS connections bypass Isolate's audit logging. Always use isolate connect for audited database access.

Configuration

Environment Variables

Isolate on AWS requires several environment variables for proper operation:

Required Variables

AWS_REGION
AWS region where resources are provisioned (e.g., us-east-1)
AWS_ACCOUNT_ID
AWS account ID for resource identification and IAM policies
BACKUP_BUCKET
S3 bucket name containing pg_dump backup files
DYNAMODB_TABLE
DynamoDB table name for storing instance and session metadata
RECORDINGS_BUCKET
S3 bucket for storing kubectl exec session recordings
AUDIT_BUCKET
S3 bucket for storing SQL and K8s audit logs (Parquet files)

IAM Roles and Policies

Isolate components require specific IAM roles:

  • ECS Task Role: RDS access, DynamoDB, S3 for backups and audit logs
  • Proxy EC2 Role: DynamoDB read, S3 write for recordings, EKS access
  • Lambda TTL Cleanup Role: RDS delete, DynamoDB read/write
  • Federation Agent Role: RDS/EKS describe permissions (in remote accounts)

Troubleshooting

Instance Provisioning Issues

If RDS instances fail to provision or get stuck:

  1. Check RDS Console: Verify instance status and any error messages
  2. Review Subnet Groups: Ensure DB subnet group spans multiple AZs
  3. Verify Security Groups: Check if ECS tasks can reach RDS on port 5432
  4. Check Quotas: RDS has limits on instances per region
  5. Review Logs: Check ECS task logs for restore failures

SSM Session Manager Issues

Common SSM connectivity problems and solutions:

  • "Target not connected": EC2 instance may not have SSM agent or proper IAM role
  • Permission denied: Check if your AWS credentials have ssm:StartSession permission
  • Plugin not found: Install Session Manager plugin for AWS CLI
  • VPC endpoints missing: Private subnets need SSM VPC endpoints or NAT Gateway
# Check if EC2 instance is SSM-managed aws ssm describe-instance-information \ --filters Key=InstanceIds,Values=i-0123456789abcdef0 # Verify Session Manager plugin session-manager-plugin # Test SSM connectivity aws ssm start-session --target i-0123456789abcdef0

Database Connection Problems

RDS-specific connectivity issues:

  • Timeout errors: Check security group rules between ECS and RDS
  • SSL connection errors: RDS enforces SSL; use sslmode=require
  • Database not available: Check if RDS instance is in "available" state
  • Authentication failed: Verify credentials from instance details page

EKS Access Issues

Kubernetes-specific troubleshooting:

  • kubectl commands hang: K8s proxy session may have expired (1-hour limit)
  • Permission denied: Check if your AWS user/role has EKS cluster access
  • Cluster unreachable: Verify EKS cluster endpoint accessibility from proxy VM
  • Authentication failed: EKS uses AWS IAM for authentication

Multi-Account Federation

Issues with cross-account resource discovery:

  • Agent not syncing: Check if agent can reach main account's API endpoint
  • Permission denied: Verify agent's IAM role has RDS/EKS describe permissions
  • Network issues: Ensure agent VM has internet access or VPC endpoints
  • Token expired: Federation tokens may need regeneration

Performance Optimization

For better performance on AWS:

  • RDS Instance Classes: Use memory-optimized instances (r6g, r5) for large datasets
  • EBS Storage: Use gp3 volumes with higher IOPS for faster backup restoration
  • Network Bandwidth: Larger instance types have better network performance
  • Regional Placement: Keep RDS, ECS, and S3 backups in the same region