Troubleshooting RDS Endpoint Connectivity Issues with AWS Client VPN
By Ramu Punuru
10mins Read
Recently, I encountered an issue where I was unable to connect to an RDS endpoint located in a private subnet using the AWS Client VPN. This connection had been stable and reliable previously, with no recent changes to the configuration. Here’s a detailed account of the problem and how it was resolved.
Initial Findings
Initial Findings
- When I connected to our backup ISP, I was able to connect to the AWS Client VPN without any issues. However, connections to the RDS endpoint using the PGAdmin client were failing with a timeout error.
- Running nslookup command on our RDS endpoint resolved to an IPv6 address, even though our RDS was configured to use only IPv4, please see the screenshot below.
Troubleshooting Steps
Step 1: Disable IPv6
My first attempt to resolve the issue was to disable IPv6 on my local network using the below commands.
$sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1 (to disable IPv6 on all interfaces)
$sudo sysctl -w net.ipv6.conf.all.disable_ipv6=0 (to re-enable IPv6)
I hoped this would force the system to use IPv4 exclusively and allow the connections to go through. Unfortunately, this did not resolve the problem.
Step 2: Update AWS Client VPN Settings
Next, I reviewed the AWS Client VPN settings and added the DNS server configuration, which is an optional step during the setup process. This change forced the nslookup command to resolve the RDS endpoint to its private IPv4 address
Note:
- DNS Resolver service is built into each Availability Zone in an AWS Region. The Route 53 Resolver is located at 169.254.169.253 (IPv4), fd00:ec2::253 (IPv6), and at the primary private IPV4 CIDR range provisioned to your VPC plus two.
- For example, if you have a VPC with an IPv4 CIDR of 10.0.0.0/16 and an IPv6 CIDR of fd00:ec2::253, you can reach the Route 53 Resolver at 169.254.169.253 (IPv4), fd00:ec2::253 (IPv6), or 10.0.0.2 (IPv4).
Outcome
After updating the DNS configuration in the VPN settings, I disconnected and reconnected to the VPN. This time, the connections to the RDS were successful.
Further Testing
To ensure the problem was fully resolved, I switched to a different ISP for testing. Unfortunately, I encountered the same issue: I could not connect to the RDS endpoint using the AWS Client VPN, and the connection was timing out.
I followed the same troubleshooting steps as before, but this time the nslookup command resolved the endpoint to both IPv4 and IPv6 addresses.
After updating the DNS configuration in the VPN settings, I disconnected and reconnected to the VPN. This time, the connections to the RDS were successful.
Further Testing
To ensure the problem was fully resolved, I switched to a different ISP for testing. Unfortunately, I encountered the same issue: I could not connect to the RDS endpoint using the AWS Client VPN, and the connection was timing out.
I followed the same troubleshooting steps as before, but this time the nslookup command resolved the endpoint to both IPv4 and IPv6 addresses.
Further Troubleshooting
Configuration Review
I reviewed the entire configuration again but couldn’t find any issues. This was confusing because the setup worked perfectly with a different ISP.
Network Trace Analysis
Using Microsoft Network Monitor, I performed a network trace and noticed that the client was closing the connection. I also analysed the RDS logs and discovered it was an SSL handshake issue.
Log for reference:
2024-05-30 08:09:55 UTC:10.40.41.126(54814):[unknown]@[unknown]:[14706]:LOG: could not accept SSL connection: EOF detected
Workaround
While the root cause of the SSL issue remained unclear, I decided to disable the SSL option in the PGAdmin tool and try again.
Configuration Review
I reviewed the entire configuration again but couldn’t find any issues. This was confusing because the setup worked perfectly with a different ISP.
Network Trace Analysis
Using Microsoft Network Monitor, I performed a network trace and noticed that the client was closing the connection. I also analysed the RDS logs and discovered it was an SSL handshake issue.
Log for reference:
2024-05-30 08:09:55 UTC:10.40.41.126(54814):[unknown]@[unknown]:[14706]:LOG: could not accept SSL connection: EOF detected
Workaround
While the root cause of the SSL issue remained unclear, I decided to disable the SSL option in the PGAdmin tool and try again.
Success!
This worked like a charm. Disabling the SSL option allowed me to connect to the RDS endpoint successfully. Disabling SSL poses no risk, as the default encryption in the PGAdmin tool ensures secure connections.
Conclusion
Network issues can be challenging, especially when they involve multiple ISPs and varying DNS resolutions. Here are the key takeaways from this experience:
1. DNS Configuration: Ensure that your VPN settings include the proper DNS server configuration to avoid unintended DNS resolution issues.
2. ISP Differences: Be aware that different ISPs can introduce unique challenges that might not be immediately apparent.
3. SSL Handshake Issues: If you come across SSL handshake problems, temporarily disabling SSL (if allowed by security policies) can serve as a potential workaround. It is important to verify whether the tools in use provide default encryption to ensure secure connections even when SSL is disabled.
If you encounter similar problems, I hope this guide helps you navigate through the troubleshooting process.
This worked like a charm. Disabling the SSL option allowed me to connect to the RDS endpoint successfully. Disabling SSL poses no risk, as the default encryption in the PGAdmin tool ensures secure connections.
Conclusion
Network issues can be challenging, especially when they involve multiple ISPs and varying DNS resolutions. Here are the key takeaways from this experience:
1. DNS Configuration: Ensure that your VPN settings include the proper DNS server configuration to avoid unintended DNS resolution issues.
2. ISP Differences: Be aware that different ISPs can introduce unique challenges that might not be immediately apparent.
3. SSL Handshake Issues: If you come across SSL handshake problems, temporarily disabling SSL (if allowed by security policies) can serve as a potential workaround. It is important to verify whether the tools in use provide default encryption to ensure secure connections even when SSL is disabled.
If you encounter similar problems, I hope this guide helps you navigate through the troubleshooting process.