AWS GWLB对访问ALB流量做安全检测

B站视频：​ ​​https://www.bilibili.com/video/BV1Rd4y1F7Fc/?spm_id_from=333.999.0.0​​

欢迎关注我的微信公众号：自刘地

AWS GWLB对访问ALB流量做安全检测_Application Load Bal


一、架构图

上一篇介绍了GWLB结合​​Network Load Balancer​​的场景，这里介绍一下GWLB结合​​Application Load Balancer​​的场景，主要区别还是在于路由表的设计。另外因为ALB可以关联AWS WAF，所以在代码里面顺带为ALB关联了一个WAF策略，策略只允许来自中国地区的IP访问，来自其他国家地区的请求会被拒绝。

这个架构图高清图片我放到百度网盘了（博客不能添加百度网盘链接，请移步到公众号查看）

AWS GWLB对访问ALB流量做安全检测_AWS_02

实验环境一共有两个VPC，左边的是业务VPC，里面有两个APP模拟HTTP的业务，EC2放在私有子网里面，在不同的AZ。

有一个面向互联网的Application Load Balancer，向公网发布了这个HTTP的服务，这样互联网上的用户，就可以通过ALB访问后面的业务。我们要做的是把这个流量，引导到防火墙上去，做安全检测。这里使用Linux的iptables来模拟防火墙。

另外，APP可以通过NAT GW上网，这个APP主动访问互联网的流量，也需要送到防火墙上去，做安全检测。所以，APP有两种流量都需要送到防火墙上去做安全检测，下面看一下APP的两种流量路径。

一、来自于互联网对ALB访问的流量路径。

  1. 首先，互联网上的用户，对ALB的公有DNS发起请求，这个DNS请求，会解析到ALB的两个公网IP地址，然后流量通过ISP路由，到达AWS的IGW，然后抵达ALB。
  2. 流量到达ALB之后，需要转发给APP，但是去往本VPC CIDR段的流量指向了Endpoint，所以ALB会依据路由表将流量发送到Endpoint上，这里假设解析ALB的主IP地址是Public Subnet1，所以流量会给到Endpoint1。
  3. Endpoint1收到流量之后，会通过Private Link，把流量发送到GWLB上。
  4. GWLB会通过GENEVE封装报文，把流量发送到防火墙。
  5. 防火墙做完安全检测之后，又会把流量送回给GWLB，然后通过Private Link，送到GWLB Endpoint1。
  6. 到达GWLB Endpoint1之后，匹配到local路由，流量最终发送到APP1上。

继续看一下回包流程。

  1. APP1收到了来自于ALB的流量，APP1看到的源IP地址是ALB弹性接口的私有IP地址，APP1关联的路由表匹配到10.120.0.0/16，将流量发送的Endpoint1。
  2. 接下来又是一样的流程，流量会经过Private Link到防火墙绕一圈再回来。
  3. 流量回到GWLB Endpoint1上之后，会匹配到默认路由，将流量发送到NAT GW1。
  4. 流量到NAT GW1之后，匹配到默认路由，流量通过IGW发送回客户端。

以上就是完整的从互联网对ALB发起请求和回包的流程。

二、APP1主动访问互联网的流量路径

  1. APP1对互联网地址发起请求，APP1查询子网关联的路由表，匹配到默认路由，流量送到Endpoint1。
  2. 流量到GWLB Endpoint1之后，通过Private Link把流量送到防火墙检测，然后再发回来。
  3. Endpoint1收到流量之后，会匹配到默认路由，将流量发送到NAT GW1。
  4. 流量到NAT GW1之后，匹配到默认路由，流量通过IGW发送到目的地。

继续看回包流程。

  1. 互联网上的主机收到报文以后，源地址是NAT GW1的公网IP地址，报文通过ISP路由，流量到达NAT GW1。
  2. 到达NAT GW1之后查询转换表项，准备将流量发送到APP1的私有IP地址，NAT GW1去往本VPC CIDR段的流量指向了Endpoint1，所以流量发送到Endpoint1。
  3. 流量到GWLB Endpoint1之后，通过Private Link把流量送到防火墙检测，然后再发回来。
  4. Endpoint1收到流量之后，匹配到local路由，将流量发送到APP1。

二、创建实验环境

堆栈大概需要8分钟左右的时间创建完成。

上传堆栈文件。

AWS GWLB对访问ALB流量做安全检测_CloudFormation_03

编辑堆栈名称，修改实例密钥。

AWS GWLB对访问ALB流量做安全检测_Application Load Bal_04

允许创建IAM资源。

AWS GWLB对访问ALB流量做安全检测_CloudFormation_05

Parameters:
  EC2InstanceAmiId:
    Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
    Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2'
  Environment:
    Type: String
    AllowedValues:
      - dev
      - prod
    Default: dev
  MyKeyPair:
    Description: Amazon EC2 Key Pair
    Type: AWS::EC2::KeyPair::KeyName
    Default: Global_Tokyo_KeyPair
  WebServerPort:
    Description: Apache Http Server Port
    Type: String
    Default: 8443
    AllowedValues:
      - 8443
      - 8888
      - 8088

Resources:

  BastionSsmRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /

  BastionSsmPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: PrivatelianceInstanceAccess
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - ssm:DescribeAssociation
              - ssm:GetDeployablePatchSnapshotForInstance
              - ssm:GetDocument
              - ssm:DescribeDocument
              - ssm:GetManifest
              - ssm:GetParameter
              - ssm:GetParameters
              - ssm:ListAssociations
              - ssm:ListInstanceAssociations
              - ssm:PutInventory
              - ssm:PutComplianceItems
              - ssm:PutConfigurePackageResult
              - ssm:UpdateAssociationStatus
              - ssm:UpdateInstanceAssociationStatus
              - ssm:UpdateInstanceInformation
            Resource: "*"
          - Effect: Allow
            Action:
              - ssmmessages:CreateControlChannel
              - ssmmessages:CreateDataChannel
              - ssmmessages:OpenControlChannel
              - ssmmessages:OpenDataChannel
            Resource: "*"
          - Effect: Allow
            Action:
              - ec2messages:AcknowledgeMessage
              - ec2messages:DeleteMessage
              - ec2messages:FailMessage
              - ec2messages:GetEndpoint
              - ec2messages:GetMessages
              - ec2messages:SendReply
            Resource: "*"
      Roles:
        - !Ref BastionSsmRole

  BastionSsmProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref BastionSsmRole

#=========================================SecVpc========================================#
# 创建SecVpc
  SecVpc:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.100.10.0/16
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'
      Tags:
       - Key: Name
         Value: !Sub ${AWS::StackName}-SecVpc

# 创建IGW并且关联到VPC
  SecVpcIGW:
    Type: "AWS::EC2::InternetGateway"
    Properties:
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpcIGW

  SecVpcAttachIgw:
    Type: "AWS::EC2::VPCGatewayAttachment"
    Properties:
      VpcId: !Ref SecVpc
      InternetGatewayId: !Ref SecVpcIGW

#---------------------------SecVpc创建6个子网-------------------------------------#

# SecVpc AZ1内创建公有子网
  SecVpcAz1PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.10.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ1-Public-Subnet

# SecVpc AZ2内创建公有子网
  SecVpcAz2PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.20.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ2-Public-Subnet

# SecVpc AZ1内创建私有子网
  SecVpcAz1PrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.30.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ1-Private-Subnet

# SecVpc AZ2内创建私有子网
  SecVpcAz2PrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.40.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ2-Private-Subnet


# SecVpc AZ1内创建TGW子网
  SecVpcAz1TgwSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.50.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ1-TGW-Subnet

# SecVpc AZ2内创建TGW子网
  SecVpcAz2TgwSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref SecVpc
      CidrBlock: 10.100.60.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-SecVpc-AZ2-TGW-Subnet

#---------------------------SecVpc创建路由表-------------------------------------#

# 公有子网路由表及关联
  SecVpcAz1PublicRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ1-Public-RouteTable

  SecVpcAz1PublicRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz1PublicRouteTable
      SubnetId: !Ref SecVpcAz1PublicSubnet

  SecVpcAz2PublicRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ2-Public-RouteTable

  SecVpcAz2PublicRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz2PublicRouteTable
      SubnetId: !Ref SecVpcAz2PublicSubnet

# Private子网路由表及关联
  SecVpcAz1PrivateRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ1-Private-RouteTable

  SecVpcAz1PrivateRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz1PrivateRouteTable
      SubnetId: !Ref SecVpcAz1PrivateSubnet

  SecVpcAz2PrivateRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ2-Private-RouteTable

  SecVpcAz2PrivateRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz2PrivateRouteTable
      SubnetId: !Ref SecVpcAz2PrivateSubnet


# Tgw路由表及关联
  SecVpcAz1TgwRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ1-Tgw-RouteTable

  SecVpcAz1TgwRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz1TgwRouteTable
      SubnetId: !Ref SecVpcAz1TgwSubnet

  SecVpcAz2TgwRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref SecVpc
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ2-Tgw-RouteTable

  SecVpcAz2TgwRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref SecVpcAz2TgwRouteTable
      SubnetId: !Ref SecVpcAz2TgwSubnet

#---------------------------NAT Gateway------------------------------------#

# AZ1 NAT GW
  SecVpcAz1NatGatewayEIP:
     Type: AWS::EC2::EIP
     Properties:
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ1-NatGateway-EIP

  SecVpcAz1NatGateway:
     Type: AWS::EC2::NatGateway
     Properties:
        AllocationId: !GetAtt SecVpcAz1NatGatewayEIP.AllocationId
        SubnetId: !Ref SecVpcAz1PublicSubnet
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ1-NatGateway

# AZ2 NAT GW
  SecVpcAz2NatGatewayEIP:
     Type: AWS::EC2::EIP
     Properties:
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ2-NatGateway-EIP

  SecVpcAz2NatGateway:
     Type: AWS::EC2::NatGateway
     Properties:
        AllocationId: !GetAtt SecVpcAz2NatGatewayEIP.AllocationId
        SubnetId: !Ref SecVpcAz2PublicSubnet
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-AZ2-NatGateway

  SecVpcAz1PrivateSubnetToInternetRoute:
     DependsOn: SecVpcAz1NatGateway
     Type: AWS::EC2::Route
     Properties:
        RouteTableId: !Ref SecVpcAz1PrivateRouteTable
        DestinationCidrBlock: '0.0.0.0/0'
        NatGatewayId: !Ref SecVpcAz1NatGateway

  SecVpcAz2PrivateSubnetToInternetRoute:
     DependsOn: SecVpcAz2NatGateway
     Type: AWS::EC2::Route
     Properties:
        RouteTableId: !Ref SecVpcAz2PrivateRouteTable
        DestinationCidrBlock: '0.0.0.0/0'
        NatGatewayId: !Ref SecVpcAz2NatGateway

#---------------------------添加路由------------------------------------#

# 公有子网添加默认路由去往IGW
  SecVpcAz1PublicSubnetToInternetRoute:
    Type: "AWS::EC2::Route"
    DependsOn: SecVpcIGW
    Properties:
     RouteTableId: !Ref SecVpcAz1PublicRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     GatewayId: !Ref SecVpcIGW

  SecVpcAz2PublicSubnetToInternetRoute:
    Type: "AWS::EC2::Route"
    DependsOn: SecVpcIGW
    Properties:
     RouteTableId: !Ref SecVpcAz2PublicRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     GatewayId: !Ref SecVpcIGW

#---------------------------SecVpc创建安全组------------------------------------#
  SecVpcSg:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: SG to test ping
      VpcId: !Ref SecVpc
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 22
        ToPort: 22
        CidrIp: 0.0.0.0/0
      - IpProtocol: icmp
        FromPort: -1
        ToPort: -1
        CidrIp: 0.0.0.0/0
      - IpProtocol: -1
        FromPort: -1
        ToPort: -1
        CidrIp: 10.100.0.0/16
      - IpProtocol: tcp
        FromPort: 3389
        ToPort: 3389
        CidrIp: 0.0.0.0/0
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpcSg


#---------------------------SecVpc创建EC2实例------------------------------------#

#--------------------------IAM Instance Role and Profile------------------------------------#
  ApplianceRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${AWS::StackName}-appliance-role"
      ManagedPolicyArns:
        - "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /

  AppliancePolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: AppServer
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - ec2:DescribeNetworkInterfaces
            Resource: '*'
      Roles:
        - !Ref ApplianceRole

  ApplianceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref ApplianceRole

#---------------------------SecVpc创建GWLB------------------------------------#

# Gateway Load Balancer (GWLB), Target Group, Listener
  Gwlb:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      LoadBalancerAttributes:
        - Key: load_balancing.cross_zone.enabled
          Value: true
      Name: gwlb1
      Type: gateway
      Subnets:
        - !Ref SecVpcAz1PrivateSubnet
        - !Ref SecVpcAz2PrivateSubnet
      Tags:
      - Key: Name
        Value: !Sub "${AWS::StackName}-gwlb-1"

  # Target Group:
  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: tg1
      Port: 6081
      Protocol: GENEVE
      TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
        Value: "20"
      VpcId: !Ref SecVpc
      HealthCheckPort: 80
      HealthCheckProtocol: HTTP
      TargetType: instance
      Targets:
        - Id: !Ref Appliance1
        - Id: !Ref Appliance2
      Tags:
      - Key: Name
        Value: !Sub "${AWS::StackName}-tg-1"

  # Listener:
  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
      - Type: forward
        TargetGroupArn: !Ref TargetGroup
      LoadBalancerArn: !Ref Gwlb

#---------------------------SecVpc创建EC2实例------------------------------------#


# EC2 Instances (Appliances acting as target for GWLB):
  Appliance1:
    DependsOn: [Gwlb, SecVpcAz1PrivateSubnetToInternetRoute]
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t2.micro
      IamInstanceProfile: !Ref ApplianceProfile
      SecurityGroupIds:
        - !Ref SecVpcSg
      SubnetId: !Ref SecVpcAz1PrivateSubnet
      Tags:
        - Key: Name
          Value: !Sub "${AWS::StackName}-appliance-1"
      UserData:
        Fn::Base64: |
          #!/bin/bash -ex

          # Install packages:
          yum update -y;
          yum install jq -y;
          yum install httpd -y;
          yum install htop -y;
          sudo yum install iptables-services -y;

          # Enable IP Forwarding:
          echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/00-defaults.conf;
          sysctl -p /etc/sysctl.d/00-defaults.conf;

          # Configure hostname:
          hostnamectl set-hostname gwlb-target-1;

          # Configure SSH client alive interval for ssh session timeout:
          echo 'ClientAliveInterval 60' | sudo tee --append /etc/ssh/sshd_config;
          service sshd restart;

          # Set dark background for vim:
          touch /home/ec2-user/.vimrc;
          echo "set background=dark" >> /home/ec2-user/.vimrc;

          # Define variables:
          curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document > /home/ec2-user/iid;
          export instance_interface=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/);
          export instance_vpcid=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/$instance_interface/vpc-id);
          export instance_az=$(cat /home/ec2-user/iid |grep 'availability' | awk -F': ' '{print $2}' | awk -F',' '{print $1}');
          export instance_ip=$(cat /home/ec2-user/iid |grep 'privateIp' | awk -F': ' '{print $2}' | awk -F',' '{print $1}' | awk -F'"' '{print$2}');
          export instance_region=$(cat /home/ec2-user/iid |grep 'region' | awk -F': ' '{print $2}' | awk -F',' '{print $1}' | awk -F'"' '{print$2}');
          export local_az_gwlb_ip=$(aws --region $instance_region ec2 describe-network-interfaces --filters Name=vpc-id,Values=$instance_vpcid | jq ' .NetworkInterfaces[] | select(.AvailabilityZone=='$instance_az') | select(.InterfaceType=="gateway_load_balancer") |.PrivateIpAddress' -r);
          export remote_az_gwlb_ip=$(aws --region $instance_region ec2 describe-network-interfaces --filters Name=vpc-id,Values=$instance_vpcid | jq ' .NetworkInterfaces[] | select(.AvailabilityZone!='$instance_az') | select(.InterfaceType=="gateway_load_balancer") |.PrivateIpAddress' -r);


          # Start http and configure index.html:
          systemctl enable httpd;
          systemctl start httpd;
          touch /var/www/html/index.html;
          echo "<html>" >> /var/www/html/index.html
          echo "  <head>" >> /var/www/html/index.html
          echo "    <title>Gateway Load Balancer POC</title>" >> /var/www/html/index.html
          echo "    <meta http-equiv='Content-Type' content='text/html; charset=ISO-8859-1'>" >> /var/www/html/index.html
          echo "  </head>" >> /var/www/html/index.html
          echo "  <body>" >> /var/www/html/index.html
          echo "    <h1>Welcome to Gateway Load Balancer POC:</h1>" >> /var/www/html/index.html
          echo "    <h2>This is appliance running in $instance_az. Happy testing!</h2>" >> /var/www/html/index.html
          echo "  </body>" >> /var/www/html/index.html
          echo "</html>" >> /var/www/html/index.html

          # Start and configure iptables:
          systemctl enable iptables;
          systemctl start iptables;

          # Configuration below allows allows all traffic:
          # Set the default policies for each of the built-in chains to ACCEPT:
          iptables -P INPUT ACCEPT;
          iptables -P FORWARD ACCEPT;
          iptables -P OUTPUT ACCEPT;

          # Flush the nat and mangle tables, flush all chains (-F), and delete all non-default chains (-X):
          iptables -t nat -F;
          iptables -t mangle -F;
          iptables -F;
          iptables -X;

          # Configure nat table to hairpin traffic back to GWLB:
          iptables -t nat -A PREROUTING -p udp -s $local_az_gwlb_ip -d $instance_ip -i eth0 -j DNAT --to-destination $local_az_gwlb_ip:6081;
          iptables -t nat -A POSTROUTING -p udp --dport 6081 -s $local_az_gwlb_ip -d $local_az_gwlb_ip -o eth0 -j MASQUERADE;

          iptables -t nat -A PREROUTING -p udp -s $remote_az_gwlb_ip -d $instance_ip -i eth0 -j DNAT --to-destination $remote_az_gwlb_ip:6081;
          iptables -t nat -A POSTROUTING -p udp --dport 6081 -s $remote_az_gwlb_ip -d $remote_az_gwlb_ip -o eth0 -j MASQUERADE;

          # Save iptables:
          service iptables save;

  Appliance2:
    DependsOn: [ Gwlb, SecVpcAz2PrivateSubnetToInternetRoute ]
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t2.micro
      IamInstanceProfile: !Ref ApplianceProfile
      SecurityGroupIds:
        - !Ref SecVpcSg
      SubnetId: !Ref SecVpcAz2PrivateSubnet
      Tags:
        - Key: Name
          Value: !Sub "${AWS::StackName}-appliance-2"
      UserData:
        Fn::Base64: |
          #!/bin/bash -ex

          # Install packages:
          yum update -y;
          yum install jq -y;
          yum install httpd -y;
          yum install htop -y;
          sudo yum install iptables-services -y;

          # Enable IP Forwarding:
          echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/00-defaults.conf;
          sysctl -p /etc/sysctl.d/00-defaults.conf;

          # Configure hostname:
          hostnamectl set-hostname gwlb-target-2;

          # Configure SSH client alive interval for ssh session timeout:
          echo 'ClientAliveInterval 60' | sudo tee --append /etc/ssh/sshd_config;
          service sshd restart;

          # Set dark background for vim:
          touch /home/ec2-user/.vimrc;
          echo "set background=dark" >> /home/ec2-user/.vimrc;

          # Define variables:
          curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document > /home/ec2-user/iid;
          export instance_interface=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/);
          export instance_vpcid=$(curl --silent http://169.254.169.254/latest/meta-data/network/interfaces/macs/$instance_interface/vpc-id);
          export instance_az=$(cat /home/ec2-user/iid |grep 'availability' | awk -F': ' '{print $2}' | awk -F',' '{print $1}');
          export instance_ip=$(cat /home/ec2-user/iid |grep 'privateIp' | awk -F': ' '{print $2}' | awk -F',' '{print $1}' | awk -F'"' '{print$2}');
          export instance_region=$(cat /home/ec2-user/iid |grep 'region' | awk -F': ' '{print $2}' | awk -F',' '{print $1}' | awk -F'"' '{print$2}');
          export local_az_gwlb_ip=$(aws --region $instance_region ec2 describe-network-interfaces --filters Name=vpc-id,Values=$instance_vpcid | jq ' .NetworkInterfaces[] | select(.AvailabilityZone=='$instance_az') | select(.InterfaceType=="gateway_load_balancer") |.PrivateIpAddress' -r);
          export remote_az_gwlb_ip=$(aws --region $instance_region ec2 describe-network-interfaces --filters Name=vpc-id,Values=$instance_vpcid | jq ' .NetworkInterfaces[] | select(.AvailabilityZone!='$instance_az') | select(.InterfaceType=="gateway_load_balancer") |.PrivateIpAddress' -r);


          # Start http and configure index.html:
          systemctl enable httpd;
          systemctl start httpd;
          touch /var/www/html/index.html;
          echo "<html>" >> /var/www/html/index.html
          echo "  <head>" >> /var/www/html/index.html
          echo "    <title>Gateway Load Balancer POC</title>" >> /var/www/html/index.html
          echo "    <meta http-equiv='Content-Type' content='text/html; charset=ISO-8859-1'>" >> /var/www/html/index.html
          echo "  </head>" >> /var/www/html/index.html
          echo "  <body>" >> /var/www/html/index.html
          echo "    <h1>Welcome to Gateway Load Balancer POC:</h1>" >> /var/www/html/index.html
          echo "    <h2>This is appliance running in $instance_az. Happy testing!</h2>" >> /var/www/html/index.html
          echo "  </body>" >> /var/www/html/index.html
          echo "</html>" >> /var/www/html/index.html

          # Start and configure iptables:
          systemctl enable iptables;
          systemctl start iptables;

          # Configuration below allows allows all traffic:
          # Set the default policies for each of the built-in chains to ACCEPT:
          iptables -P INPUT ACCEPT;
          iptables -P FORWARD ACCEPT;
          iptables -P OUTPUT ACCEPT;

          # Flush the nat and mangle tables, flush all chains (-F), and delete all non-default chains (-X):
          iptables -t nat -F;
          iptables -t mangle -F;
          iptables -F;
          iptables -X;

          # Configure nat table to hairpin traffic back to GWLB:
          iptables -t nat -A PREROUTING -p udp -s $local_az_gwlb_ip -d $instance_ip -i eth0 -j DNAT --to-destination $local_az_gwlb_ip:6081;
          iptables -t nat -A POSTROUTING -p udp --dport 6081 -s $local_az_gwlb_ip -d $local_az_gwlb_ip -o eth0 -j MASQUERADE;

          iptables -t nat -A PREROUTING -p udp -s $remote_az_gwlb_ip -d $instance_ip -i eth0 -j DNAT --to-destination $remote_az_gwlb_ip:6081;
          iptables -t nat -A POSTROUTING -p udp --dport 6081 -s $remote_az_gwlb_ip -d $remote_az_gwlb_ip -o eth0 -j MASQUERADE;

          # Save iptables:
          service iptables save;


  SecVpcBastionLinux:
    Type: AWS::EC2::Instance
    Properties:
      IamInstanceProfile: !Ref BastionSsmProfile
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t3.nano
      NetworkInterfaces:
        - AssociatePublicIpAddress: true
          DeviceIndex: 0
          GroupSet:
            - Ref: SecVpcSg
          SubnetId: !Ref SecVpcAz1PublicSubnet
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-SecVpc-Bastion-Linux

# Create VPC Endpoint Service:

  VpcEndpointService:
    Type: AWS::EC2::VPCEndpointService
    Properties:
      GatewayLoadBalancerArns:
        - !Ref Gwlb
      AcceptanceRequired: false

# Create Lambda Custom Resource to retrieve VPC Endpoint Service Name:

  VpceServiceLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: arn:aws:logs:*:*:*
              - Effect: Allow
                Action:
                  - ec2:DescribeVpcEndpointServiceConfigurations
                  - ec2:DescribeVpcEndpointServicePermissions
                  - ec2:DescribeVpcEndpointServices
                Resource: "*"


  VpceServiceLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
        LogGroupName: !Sub /aws/lambda/${AWS::StackName}-service
        RetentionInDays: 1

  VpceServiceName:
    Type: AWS::Lambda::Function
    DependsOn: VpceServiceLogGroup
    Properties:
      FunctionName: !Sub ${AWS::StackName}-service
      Handler: "index.handler"
      Role: !GetAtt VpceServiceLambdaExecutionRole.Arn
      Code:
        ZipFile: |
          import json
          import logging
          import time

          import boto3
          import cfnresponse
          from botocore.exceptions import ClientError

          try:
              ec2 = boto3.client('ec2')
          except ClientError as e:
              logger.error(f"ERROR: failed to connect to EC2 client: {e}")
              sys.exit(1)

          def handler(event, context):
              logger = logging.getLogger()
              logger.setLevel(logging.INFO)
              logger.info('Received event: {}'.format(json.dumps(event)))

              responseData = {}
              responseStatus = cfnresponse.FAILED

              try:
                  serviceid = event["ResourceProperties"]["VpceServiceId"]
              except Exception as e:
                  logger.info('Attribute retrival failure: {}'.format(e))

              try:
                  if event["RequestType"] == "Delete":
                      responseStatus = cfnresponse.SUCCESS
                      cfnresponse.send(event, context, responseStatus, responseData)
              except Exception:
                  logger.exception("Signaling failure to CloudFormation.")
                  cfnresponse.send(event, context, cfnresponse.FAILED, {})

              if event["RequestType"] == "Create":
                  logger.info("Retrieving VPC Endpoint Service Name:")
                  try:
                      response = ec2.describe_vpc_endpoint_service_configurations(
                          Filters=[
                              {
                                  'Name': 'service-id',
                                  'Values': [serviceid]
                              }
                          ]
                      )
                  except Exception as e:
                      logger.info('ec2.describe_vpc_endpoint_service_configurations failure: {}'.format(e))

                  service_name = response['ServiceConfigurations'][0]['ServiceName']

                  time.sleep(120)

                  responseData['ServiceName'] = service_name
                  responseStatus = cfnresponse.SUCCESS
                  cfnresponse.send(event, context, responseStatus, responseData)
      Runtime: python3.7
      Timeout: 150

  RetrieveVpceServiceName:
    Type: Custom::RetrieveAttributes
    Properties:
      ServiceToken: !GetAtt VpceServiceName.Arn
      VpceServiceId: !Ref VpcEndpointService

#=========================================VpcB========================================#
# 创建VpcB
  VpcB:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: 10.120.10.0/16
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'
      Tags:
       - Key: Name
         Value: !Sub ${AWS::StackName}-VpcB

# 创建IGW并且关联到VPC
  VpcBIGW:
    Type: "AWS::EC2::InternetGateway"
    Properties:
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcBIGW

  VpcBAttachIgw:
    Type: "AWS::EC2::VPCGatewayAttachment"
    Properties:
      VpcId: !Ref VpcB
      InternetGatewayId: !Ref VpcBIGW

#---------------------------VpcB创建8个子网-------------------------------------#

# VpcB AZ1内创建公有子网
  VpcBAz1PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcB
      CidrBlock: 10.120.10.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcB-AZ1-Public-Subnet

# VpcB AZ2内创建公有子网
  VpcBAz2PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcB
      CidrBlock: 10.120.20.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcB-AZ2-Public-Subnet

# VpcB AZ1内创建APP子网
  VpcBAz1AppSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcB
      CidrBlock: 10.120.30.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcB-AZ1-App-Subnet

# VpcB AZ2内创建APP子网
  VpcBAz2AppSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcB
      CidrBlock: 10.120.40.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcB-AZ2-App-Subnet

# VpcB AZ1内创建GWLB子网
  VpcBAz1GwlbeSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcB
      CidrBlock: 10.120.50.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcB-AZ1-GWLBe-Subnet

# VpcB AZ2内创建GWLB子网
  VpcBAz2GwlbeSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcB
      CidrBlock: 10.120.60.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcB-AZ2-GWLBe-Subnet

# VpcB AZ1内创建TGW子网
  VpcBAz1TgwSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcB
      CidrBlock: 10.120.70.0/24
      AvailabilityZone:
        Fn::Select:
          - 0
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcB-AZ1-TGW-Subnet

# VpcB AZ2内创建TGW子网
  VpcBAz2TgwSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VpcB
      CidrBlock: 10.120.80.0/24
      AvailabilityZone:
        Fn::Select:
          - 1
          - Fn::GetAZs: ""
      Tags:
      - Key: Name
        Value: !Sub ${AWS::StackName}-VpcB-AZ2-TGW-Subnet

#---------------------------VpcB创建路由表-------------------------------------#

# 公有子网路由表及关联
  VpcBAz1PublicRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcB
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ1-Public-RouteTable

  VpcBAz1PublicRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcBAz1PublicRouteTable
      SubnetId: !Ref VpcBAz1PublicSubnet

  VpcBAz2PublicRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcB
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ2-Public-RouteTable

  VpcBAz2PublicRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcBAz2PublicRouteTable
      SubnetId: !Ref VpcBAz2PublicSubnet

# App子网路由表及关联
  VpcBAz1AppRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcB
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ1-App-RouteTable

  VpcBAz1AppRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcBAz1AppRouteTable
      SubnetId: !Ref VpcBAz1AppSubnet

  VpcBAz2AppRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcB
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ2-App-RouteTable

  VpcBAz2AppRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcBAz2AppRouteTable
      SubnetId: !Ref VpcBAz2AppSubnet

# GWLBe子网路由表及关联
  VpcBAz1GwlbeRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcB
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ1-GWLBe-RouteTable

  VpcBAz1GwlbeRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcBAz1GwlbeRouteTable
      SubnetId: !Ref VpcBAz1GwlbeSubnet

  VpcBAz2GwlbeRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcB
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ2-GWLBe-RouteTable

  VpcBAz2GwlbeRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcBAz2GwlbeRouteTable
      SubnetId: !Ref VpcBAz2GwlbeSubnet

# Tgw路由表及关联
  VpcBAz1TgwRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcB
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ1-Tgw-RouteTable

  VpcBAz1TgwRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcBAz1TgwRouteTable
      SubnetId: !Ref VpcBAz1TgwSubnet

  VpcBAz2TgwRouteTable:
    Type: "AWS::EC2::RouteTable"
    Properties:
      VpcId: !Ref VpcB
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ2-Tgw-RouteTable

  VpcBAz2TgwRouteTableAssociation:
    Type: "AWS::EC2::SubnetRouteTableAssociation"
    Properties:
      RouteTableId: !Ref VpcBAz2TgwRouteTable
      SubnetId: !Ref VpcBAz2TgwSubnet

#---------------------------NAT Gateway------------------------------------#

# AZ1 NAT GW
  VpcBAz1NatGatewayEIP:
     Type: AWS::EC2::EIP
     Properties:
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ1-NatGateway-EIP

  VpcBAz1NatGateway:
     Type: AWS::EC2::NatGateway
     Properties:
        AllocationId: !GetAtt VpcBAz1NatGatewayEIP.AllocationId
        SubnetId: !Ref VpcBAz1PublicSubnet
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ1-NatGateway

# AZ2 NAT GW
  VpcBAz2NatGatewayEIP:
     Type: AWS::EC2::EIP
     Properties:
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ2-NatGateway-EIP

  VpcBAz2NatGateway:
     Type: AWS::EC2::NatGateway
     Properties:
        AllocationId: !GetAtt VpcBAz2NatGatewayEIP.AllocationId
        SubnetId: !Ref VpcBAz2PublicSubnet
        Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-AZ2-NatGateway

# VpcB 创建Endpoint
  VpcBEndpoint1:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !Ref VpcB
      ServiceName: !GetAtt RetrieveVpceServiceName.ServiceName
      VpcEndpointType: GatewayLoadBalancer
      SubnetIds:
        - !Ref VpcBAz1GwlbeSubnet

  VpcBEndpoint2:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      VpcId: !Ref VpcB
      ServiceName: !GetAtt RetrieveVpceServiceName.ServiceName
      VpcEndpointType: GatewayLoadBalancer
      SubnetIds:
        - !Ref VpcBAz2GwlbeSubnet

#---------------------------添加路由------------------------------------#

# 公有子网添加默认路由去往IGW
  VpcBAz1PublicSubnetToInternetRoute:
    Type: "AWS::EC2::Route"
    DependsOn: VpcBIGW
    Properties:
     RouteTableId: !Ref VpcBAz1PublicRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     GatewayId: !Ref VpcBIGW

  VpcBAz2PublicSubnetToInternetRoute:
    Type: "AWS::EC2::Route"
    DependsOn: VpcBIGW
    Properties:
     RouteTableId: !Ref VpcBAz2PublicRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     GatewayId: !Ref VpcBIGW

# App子网添加默认路由去往GWLBe
  VpcBAz1AppSubnetRoute1:
    Type: "AWS::EC2::Route"
    DependsOn: VpcBEndpoint2
    Properties:
     RouteTableId: !Ref VpcBAz1AppRouteTable
     DestinationCidrBlock: 10.120.20.0/24
     VpcEndpointId: !Ref VpcBEndpoint2

  VpcBAz1AppSubnetRoute2:
    Type: "AWS::EC2::Route"
    DependsOn: VpcBEndpoint1
    Properties:
     RouteTableId: !Ref VpcBAz1AppRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     VpcEndpointId: !Ref VpcBEndpoint1

  VpcBAz2AppSubnetRoute1:
    Type: "AWS::EC2::Route"
    DependsOn: VpcBEndpoint1
    Properties:
     RouteTableId: !Ref VpcBAz2AppRouteTable
     DestinationCidrBlock: 10.120.10.0/24
     VpcEndpointId: !Ref VpcBEndpoint1

  VpcBAz2AppSubnetRoute2:
    Type: "AWS::EC2::Route"
    DependsOn: VpcBEndpoint2
    Properties:
     RouteTableId: !Ref VpcBAz2AppRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     VpcEndpointId: !Ref VpcBEndpoint2


# Gwlbe子网添加默认路由去往NAT GW
  VpcBAz1GwlbeSubnetRoute1:
    Type: "AWS::EC2::Route"
    DependsOn: VpcBAz1NatGateway
    Properties:
     RouteTableId: !Ref VpcBAz1GwlbeRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     NatGatewayId: !Ref VpcBAz1NatGateway

  VpcBAz2GwlbeSubnetRoute1:
    Type: "AWS::EC2::Route"
    DependsOn: VpcBAz2NatGateway
    Properties:
     RouteTableId: !Ref VpcBAz2GwlbeRouteTable
     DestinationCidrBlock: 0.0.0.0/0
     NatGatewayId: !Ref VpcBAz2NatGateway

#---------------------------VpcB创建安全组------------------------------------#
  VpcBBastionVpcSg:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: SG to test ping
      VpcId: !Ref VpcB
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: 22
        ToPort: 22
        CidrIp: 0.0.0.0/0
      - IpProtocol: icmp
        FromPort: -1
        ToPort: -1
        CidrIp: 0.0.0.0/0
      - IpProtocol: -1
        FromPort: -1
        ToPort: -1
        CidrIp: 10.120.0.0/16
      - IpProtocol: tcp
        FromPort: 3389
        ToPort: 3389
        CidrIp: 0.0.0.0/0
      - IpProtocol: tcp
        FromPort: !Ref WebServerPort
        ToPort: !Ref WebServerPort
        CidrIp: 0.0.0.0/0
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcBSg


#---------------------------VpcB创建EC2实例------------------------------------#

  VpcBBastionLinux:
    Type: AWS::EC2::Instance
    Properties:
      IamInstanceProfile: !Ref BastionSsmProfile
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t3.nano
      NetworkInterfaces:
        - AssociatePublicIpAddress: true
          DeviceIndex: 0
          GroupSet:
            - Ref: VpcBBastionVpcSg
          SubnetId: !Ref VpcBAz1PublicSubnet
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-Bastion-Linux

  VpcBApp1Linux:
    Type: AWS::EC2::Instance
    DependsOn: [ VpcBEndpoint1, VpcBAz2AppSubnetRoute1 ]
    Properties:
      IamInstanceProfile: !Ref BastionSsmProfile
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t3.nano
      NetworkInterfaces:
        - DeviceIndex: 0
          GroupSet:
            - Ref: VpcBBastionVpcSg
          SubnetId: !Ref VpcBAz1AppSubnet
      UserData:
        Fn::Base64:
          !Sub |
          #!/bin/bash
          yum update -y
          yum install -y httpd
          sed -i.bak 's/Listen 80/Listen ${WebServerPort}/g' /etc/httpd/conf/httpd.conf
          echo "<h2>Hello World from $(hostname -f)</h2>" > /var/www/html/index.html
          systemctl start httpd.service
          systemctl enable httpd.service
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-App1-Linux

  VpcBApp2Linux:
    Type: AWS::EC2::Instance
    DependsOn: [VpcBEndpoint2, VpcBAz2AppSubnetRoute2]
    Properties:
      IamInstanceProfile: !Ref BastionSsmProfile
      ImageId: !Ref EC2InstanceAmiId
      KeyName: !Ref MyKeyPair
      InstanceType: t3.nano
      NetworkInterfaces:
        - DeviceIndex: 0
          GroupSet:
            - Ref: VpcBBastionVpcSg
          SubnetId: !Ref VpcBAz2AppSubnet
      UserData:
        Fn::Base64:
          !Sub |
          #!/bin/bash
          yum update -y
          yum install -y httpd
          sed -i.bak 's/Listen 80/Listen ${WebServerPort}/g' /etc/httpd/conf/httpd.conf
          echo "<h2>Hello World from $(hostname -f)</h2>" > /var/www/html/index.html
          systemctl start httpd.service
          systemctl enable httpd.service
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-VpcB-App2-Linux

# -----------创建ALB-----------
  VpcBEc2TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckIntervalSeconds: 30
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 15
      HealthyThresholdCount: 5
      Matcher:
        HttpCode: '200'
      Name: VpcBEc2TargetGroup
      Port: !Ref WebServerPort
      Protocol: HTTP
      TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
        Value: '20'
      Targets:
      - Id: !Ref VpcBApp1Linux
        Port: !Ref WebServerPort
      - Id: !Ref VpcBApp2Linux
        Port: !Ref WebServerPort
      UnhealthyThresholdCount: 3
      VpcId: !Ref VpcB
      Tags:
        - Key: Name
          Value: VpcB-Target-Group

  VpcBAlbListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref VpcBEc2TargetGroup
      LoadBalancerArn: !Ref VpcBApplicationLoadBalancer
      Port: !Ref WebServerPort
      Protocol: HTTP

  VpcBApplicationLoadBalancer:
    DependsOn: VpcBIGW
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: !Sub ${AWS::StackName}-VpcB-ALB
      Type: application
      Scheme: internet-facing
      Subnets:
      - !Ref VpcBAz1PublicSubnet
      - !Ref VpcBAz2PublicSubnet
      SecurityGroups:
      - !GetAtt VpcBBastionVpcSg.GroupId

# -----------创建WAF-----------

  WafTestPermitCnIpWebACL:
    Type: 'AWS::WAFv2::WebACL'
    Properties:
      Name: waf-test-web-acl
      Scope: REGIONAL
      Description: WebACL permit all from china ip address.
      DefaultAction:
        Block: {}
      VisibilityConfig:
        SampledRequestsEnabled: true
        CloudWatchMetricsEnabled: true
        MetricName: permit-cn
      Rules:
        - Name: waf-test-permit-cn-ipaddress
          Priority: 0
          Statement:
            GeoMatchStatement:
              CountryCodes:
              - CN
          Action:
            Allow: {}
          VisibilityConfig:
            SampledRequestsEnabled: true
            CloudWatchMetricsEnabled: true
            MetricName: permit-cn
      Tags:
        - Key: Name
          Value: waf-test-permit-cn-webacl

  WafTestPermitCnIpWebACLAssociation:
    Type: 'AWS::WAFv2::WebACLAssociation'
    Properties:
      WebACLArn: !GetAtt WafTestPermitCnIpWebACL.Arn
      ResourceArn: !Ref VpcBApplicationLoadBalancer

# -------------自定义资源修改路由表------------------------
# Create Lambda Custom Resource to edit existing route in an RTB
  ReplaceRouteLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: arn:aws:logs:*:*:*
              - Effect: Allow
                Action:
                  - ec2:ReplaceRoute
                Resource: "*"

  ReplaceRouteLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
        LogGroupName: !Sub /aws/lambda/${AWS::StackName}ReplaceRoute
        RetentionInDays: 1

  ReplaceRoute:
    Type: AWS::Lambda::Function
    DependsOn: ReplaceRouteLogGroup
    Properties:
      FunctionName: !Sub ${AWS::StackName}ReplaceRoute
      Handler: "index.handler"
      Role: !GetAtt ReplaceRouteLambdaExecutionRole.Arn
      Code:
        ZipFile: |
          import json
          import logging
          import time
          import boto3
          import cfnresponse
          from botocore.exceptions import ClientError
          try:
              ec2 = boto3.client('ec2')
          except ClientError as e:
              logger.error(f"ERROR: failed to connect to EC2 client: {e}")
              sys.exit(1)
          def handler(event, context):
              logger = logging.getLogger()
              logger.setLevel(logging.INFO)
              logger.info('Received event: {}'.format(json.dumps(event)))
              responseData = {}
              responseStatus = cfnresponse.FAILED
              try:
                  DestCidr = event["ResourceProperties"]["DestCidr"]
                  VpceId = event["ResourceProperties"]["VpceId"]
                  RtbId = event["ResourceProperties"]["RtbId"]

              except Exception as e:
                  logger.info('Attribute retrival failure: {}'.format(e))
              try:
                  if event["RequestType"] == "Delete":
                      response = ec2.replace_route(
                          LocalTarget = True,
                          DestinationCidrBlock = DestCidr,
                          RouteTableId = RtbId
                      )
                      responseStatus = cfnresponse.SUCCESS
                      cfnresponse.send(event, context, responseStatus, responseData)
              except Exception:
                  logger.exception("Signaling failure to CloudFormation.")
                  cfnresponse.send(event, context, cfnresponse.FAILED, {})
              if event["RequestType"] == "Create":
                  logger.info(f"Replacing target to {VpceId} for {DestCidr} for {RtbId}")
                  try:
                      response = ec2.replace_route(
                          DestinationCidrBlock = DestCidr,
                          VpcEndpointId = VpceId,
                          RouteTableId = RtbId
                      )
                  except Exception as e:
                      logger.info('ec2.describe_vpc_endpoint_service_configurations failure: {}'.format(e))

                  responseStatus = cfnresponse.SUCCESS
                  cfnresponse.send(event, context, responseStatus, responseData)
      Runtime: python3.7
      Timeout: 150

# 修改路由表
  VpcBAz1PublicRtb1EditLocalRoute:
    Type: Custom::ReplaceRoute
    DependsOn: VpcBEndpoint1
    Properties:
      ServiceToken: !GetAtt ReplaceRoute.Arn
      DestCidr: 10.120.0.0/16
      VpceId: !Ref VpcBEndpoint1
      RtbId: !Ref VpcBAz1PublicRouteTable

  VpcBAz2PublicRtb1EditLocalRoute:
    Type: Custom::ReplaceRoute
    DependsOn: VpcBEndpoint2
    Properties:
      ServiceToken: !GetAtt ReplaceRoute.Arn
      DestCidr: 10.120.0.0/16
      VpceId: !Ref VpcBEndpoint2
      RtbId: !Ref VpcBAz2PublicRouteTable

  VpcBAz1AppRtb1EditLocalRoute:
    Type: Custom::ReplaceRoute
    DependsOn: VpcBEndpoint1
    Properties:
      ServiceToken: !GetAtt ReplaceRoute.Arn
      DestCidr: 10.120.0.0/16
      VpceId: !Ref VpcBEndpoint1
      RtbId: !Ref VpcBAz1AppRouteTable

  VpcBAz2AppRtb1EditLocalRoute:
    Type: Custom::ReplaceRoute
    DependsOn: VpcBEndpoint2
    Properties:
      ServiceToken: !GetAtt ReplaceRoute.Arn
      DestCidr: 10.120.0.0/16
      VpceId: !Ref VpcBEndpoint2
      RtbId: !Ref VpcBAz1AppRouteTable

三、测试

​3.1 访问ALB测试

复制ALB的DNS名称。

AWS GWLB对访问ALB流量做安全检测_AWS_06

访问ALB的DNS名称测试业务连通性。

AWS GWLB对访问ALB流量做安全检测_AWS_07

​3.2 APP抓包测试

连接App2

AWS GWLB对访问ALB流量做安全检测_AWS_08

通过Session Manger连接。

AWS GWLB对访问ALB流量做安全检测_AWS_09

在App上抓取TCP 8443的报文，ALB会终结HTTP会话，所以在APP上看到的是来自ALB的请求地址。

另外可以通过​​curl ifconfig.me​​查看App访问公网的IP地址，是NAT GW2的公网IP地址。

sudo -i
tcpdump -n port 8443

AWS GWLB对访问ALB流量做安全检测_AWS_10

​3.3 appliance抓包测试

在App1上ping公网地址。

AWS GWLB对访问ALB流量做安全检测_Gateway Load Balance_11

在appliance上抓取​​GENEVE​​的报文，并使用​​grep​​对ICMP报文进行过滤。

root@gwlb-target-1 ~]# tcpdump -n -i eth0 port 6081 | grep ICMP

AWS GWLB对访问ALB流量做安全检测_AWS_12

​3.4 防火墙高可用测试

在App1上发起对公网地址的访问，同时在appliance1和appliance2上开启抓包，确认流量目前通过appliance1转发。

AWS GWLB对访问ALB流量做安全检测_CloudFormation_13

Stop正在转发流量的appliance实例。

AWS GWLB对访问ALB流量做安全检测_Application Load Bal_14

等待大概1分钟左右（依据是否为新建会话，还有协议切换时间不同），流量切换到另外一台防火墙。因为GWLB开启了​​Cross-zone load balancing​​，所以流量可以跨AZ，通过appliance2转发流量。

AWS GWLB对访问ALB流量做安全检测_Gateway Load Balance_15

​3.5 AWS WAF 策略测试

查看创建的Web ACLs。

AWS GWLB对访问ALB流量做安全检测_CloudFormation_16

规则只允许来自中国区的IP地址，来自其他区域的IP请求会被拒绝。

AWS GWLB对访问ALB流量做安全检测_AWS_17

查看策略详情。

AWS GWLB对访问ALB流量做安全检测_Application Load Bal_18

开启代理服务器之后，从日本发起对ALB的访问请求，访问会被拒绝。

AWS GWLB对访问ALB流量做安全检测_Application Load Bal_19

查看Web ACLs的日志记录，来自日本的请求被拒绝​​JP​​，来自中国区的流量允许访问。

AWS GWLB对访问ALB流量做安全检测_AWS_20