IPSEC VPN - Invalid ESP packet detected (HMAC validation failed)
The ESP packet invalid error is due to an encryption key mismatch after a VPN tunnel has been established. When an IPSec VPN tunnel is up, but traffic is not able to pass through the tunnel, Wireshark (or an equivalent program) can be used to determine whether there is an encryption mismatch. A mismatch could occur for many reasons, one of the most common is the instability of an ISP link (ADSL, Cable), or it could effectively be any device in the physical connection.
The following information is required to troubleshoot the problem.
1) Take a packet sniffer trace on both FortiGates.
2) Run the "diag vpn tunnel list” command a few times on both FortiGates when generating traffic that will pass through the tunnel.
In the following example, the error message was seen on the recipient FortiGate:
date=2010-12-28 time=18:19:35 devname=Kosad_VPN device_id=FG300B3910600118 log_id=0101037132 type=event subtype=ipsec pri=critical vd="root" msg="IPsec ESP" action="error" rem_ip=180.87.33.2 loc_ip=121.133.8.18 rem_port=32528 loc_port=4500 out_intf="port2" cookies="88d40f65d555ccaf/05464e20e4afc835" user="N/A" group="N/A" xauth_user="N/A" xauth_group="N/A" vpn_tunnel="fortinet_0" status=esp_error error_num=Invalid ESP packet detected (HMAC validation failed). spi=c32b09f7 seq=00000012 |
This is the output of the command “diag vpn tunnel list” on the FortiGate.
inet ver=1 serial=2 192.168.1.205:4500->121.133.8.18:4500 lgwy=dyn tun=intf mode=auto bound_if=4 proxyid_num=1 child_num=0 refcnt=7 ilast=0 olast=0 stat: rxp=41 txp=56 rxb=4920 txb=3360 dpd: mode=active on=1 idle=5000ms retry=3 count=0 seqno=696 natt: mode=keepalive draft=32 interval=10 remote_port=4500 proxyid=P2_60C_Fortinet proto=0 sa=1 ref=2 auto_negotiate=0 serial=1 src: 0:182.40.101.0/255.255.255.0:0 dst: 0:100.100.100.0/255.255.255.0:0 SA: ref=3 options=0000000d type=00 soft=0 mtu=1428 expire=1106 replaywin=0 seqno=15 life: type=01 bytes=0/0 timeout=1777/1800 dec: spi=29a26eb6 esp=3des key=24 bf25e69df90257f64c55dda4069f01834cd0382fe4866ff2 ah=sha1 key=20 38b2600170585d2dfa646caed5bc86d920aed7ff enc: spi=c32b09f7 esp=3des key=24 0abd3c70032123c3369a6f225a385d30f0b2fb1cd9687ec8 ah=sha1 key=20 214d8e717306dffceec3760464b6e8edb436c6 |
Disable Replay Detection in Phase 2 settings of IPsec VPN.
Picture from Logo