|ESP-AES-128-Pad||15 (variable, worst case)|
|AH||24 (including 12-byte ICV)|
Note: AES-128 is known as AES-CBC in Wireshark.
|IPsec Mode||Overhead Elements||Maximum Bytes Overhead|
|ESP-AES-128||ESP-SP + ESP-Sequence + ESP-IV-AES-128 + ESP-AES-128-Pad + ESP-Pad-Length + ESP-Next-Header||4 + 4 + 16 + 15 + 1 + 1 = 41|
|ESP-AES-128 + ESP-HMAC-SHA1-96||ESP-AES-128 + ESP-HMAC-SHA1-96||41 + 12 = 53|
|VPN Mode||Overhead Elements||Maximum Bytes Overhead|
|Transport: IP + ESP-AES-128||IP + ESP-AES-128||20 + 41 = 61|
|Transport: IP + ESP-AES-128 + ESP-HMAC-SHA1-96||IP + ESP-AES-128 + ESP-HMAC-SHA1-96||20 + 41 + 12 = 73|
|Transport: IP + NAT-T + ESP-AES-128 + ESP-HMAC-SHA1-96||IP + UDP + ESP-AES-128 + ESP-HMAC-SHA1-96||20 + 8 + 41 + 12 = 81|
|Tunnel: IP + NAT-T + ESP-AES-128||IP + UDP + ESP-AES-128 + IP||20 + 8 + 41 + 20 = 89|
|Tunnel: IP + NAT-T + ESP-AES-128 + ESP-HMAC-SHA1-96||IP + UDP + ESP-AES-128 + ESP-HMAC-SHA1-96 + IP||20 + 8 + 41 + 12 + 20 = 101|
In the Trusted User -> Edge Router VPN case, we use an IPsec tunnel with a maximum of 89 bytes of overhead. Our interfaces are Ethernet so the MTUs are set for 1500. Even though 1500 - 89 = 1411, larger MTUs do work in this configuration. This is because the padding and alignment to the 1500 MTU Ethernet payloads does not result in the worst-case padding situation, so an overhead of 89 bytes is overly pessimistic in this specific case.
To calculate the proper MTU for a "IP + UDP + ESP-AES-128 + IP" tunnel given a known host interface MTU, follow this process:
There should be a much simpler formula to calculate all this instead of such a lengthy explanation.
The presence of ESP-HMAC in IPsec packets is set via /ip ipsec proposal set # auth-algorithms. MD5, NULL and SHA1 are the available options.
The cipher used for ESP looks to be set in two places: /ip ipsec proposal set # enc-algorithms and /ip ipsec peer set # enc-algorithm. The first is used during ISAKMP negotiation, and the 2nd...? Need to experiment more to see which overrides in RouterOS.