When terminating multiple dial-up IPsec tunnels on a single public IP, and you have IKE phase-1 configured for each site, explicit Local and Peer IDs are required. Without them, the hub firewall cannot differentiate incoming connections and decide which IKE Phase 1 the tunnel lands on.

Setting these IDs ensures the FortiGate correctly matches the inbound IKE_SA_INIT payload to the specific Phase 1 configuration. Standard practice is to configure the Local ID on the remote sites and the corresponding Peer ID on the DC firewall.

If there is only one dial-up Phase 1 on the DC's public IP, you can safely skip setting IKE IDs. The FortiGate only has one configuration to match against anyway, so it knows exactly where to land the tunnel. Just keep in mind that skipping IDs is a bit of a trap if you ever need to add a second dial-up tunnel to that IP later.

When building the BGP overlay, the hub and spoke need to dynamically exchange IPs to route to each other. You can use set exchange-interface-ip enable to just swap the tunnel interface IPs, but the cleaner approach is peering BGP over loopbacks.

To get loopback peering working across a dial-up setup, use set exchange-ip-addr4 <loopback-ip>. This explicitly hands over the loopback address during the IKE exchange so BGP can establish properly.

Example real-world use case: large multi-spoke overlay where each spoke needs its IPsec tunnel interface IP advertised back so the hub can route. Without it, you'd have to statically configure every tunnel IP.

• VPN troubleshooting: FD46611
• HA reference: FD49264
• Yuriskinfo debug cheat sheet: github.com/yuriskinfo/cheat-sheets

The FIB acts as the gatekeeper for SD-WAN rules. You cannot apply an SD-WAN rule if the routing table doesn't already agree the traffic belongs in the SD-WAN zone.

Order of evaluation:

1. Regular Policy Routes (PBR) — always checked first. If a manual PBR matches with a valid gateway, traffic routes immediately and both SD-WAN rules and the standard table are bypassed.
2. FIB lookup — if no PBR match, FortiOS does a standard routing-table lookup before applying SD-WAN rules.
   • If the best route points to a non-SD-WAN interface, SD-WAN rules are ignored entirely. This protects LAN-to-LAN and out-of-band management traffic from being hijacked.
   • If the best route points to an SD-WAN member (e.g. default route via the sd-wan zone), the SD-WAN engine unlocks.
3. SD-WAN rules — evaluated top-down. Application signatures and SLA health pick the best physical member.
   • Caveat: the chosen member must also have a valid route to the destination, otherwise FortiGate skips it and evaluates the next.
4. Implicit SD-WAN rule — fallback at the bottom. Hands control back to the FIB and load-balances across members using ECMP.

This is why "I added an SD-WAN rule and nothing happened" almost always traces back to step 2: the routing table doesn't see the SD-WAN zone as the best path for that destination.

When to disable: connecting to legacy gear (older switches, OOB consoles) that only supports older MAC algorithms. Note this weakens management plane security; only do it if you must.

config system global
    set strong-crypto disable
    set ssh-mac-algo hmac-md5 hmac-sha1 hmac-sha2-256 hmac-sha2-256-etm@openssh.com hmac-sha2-512 hmac-sha2-512-etm@openssh.com
end

Restore strong-crypto:

config system global
    set strong-crypto enable
    set ssh-mac-algo hmac-sha2-256 hmac-sha2-256-etm@openssh.com hmac-sha2-512 hmac-sha2-512-etm@openssh.com
end

Fortinet AU support: 1 800 043 218

Recovery password: 131703-234421-082764-435666-575696-441881-056496-671858

Keep this collapsed in shared screens.

What is VXLAN?

VXLAN (Virtual eXtensible LAN) is a way to carry a Layer 2 Ethernet network over a Layer 3 IP network. In simple terms, it lets you take an Ethernet frame from one switch or server, wrap it, send it across a routed network, and unwrap it at the other end. RFC 7348 describes it as a Layer 2 overlay scheme on a Layer 3 network.

How it works

VXLAN encapsulates the original Ethernet frame inside:

• Outer Ethernet
• Outer IP
• Outer UDP
• VXLAN header

So VXLAN is basically a Layer 2 overlay over a Layer 3 underlay. The standard VXLAN header is 8 bytes, and the VNI is a 24-bit field inside that header. VXLAN runs over UDP with IANA-assigned destination port 4789.

Why VXLAN is used

• Traditional VLANs are limited to 4094 IDs
• Large Layer 2 domains do not scale well
• It allows Layer 2 extension across routed networks
• It provides much larger segmentation using VNI (24 bits, around 16 million segments)
• Commonly used in datacentres and EVPN fabrics

VLAN vs VXLAN

• VLAN = local Layer 2 segmentation
• VXLAN = Layer 2 overlay carried across Layer 3
• VLAN ID = 12 bits
• VNI = 24 bits

A VLAN and a VNI can be mapped, for example VLAN 20 -> VNI 10020, but they are not the same field.

VXLAN packet structure

A VXLAN packet wraps an original Ethernet frame inside:

Outer Ethernet + Outer IP + Outer UDP + VXLAN header + Inner Ethernet frame

VXLAN header (8 bytes)

• Flags: 8 bits
• Reserved: 24 bits
• VNI: 24 bits
• Reserved: 8 bits

The VNI lives inside the VXLAN header, not inside the 802.1Q VLAN tag.

 0                  7 8                 15 16                23 24                31
+--------------------+--------------------+--------------------+--------------------+
|       Flags        |                           Reserved                           |
+--------------------+--------------------+--------------------+--------------------+
|                        VNI (24 bits)                         |      Reserved      |
+--------------------+--------------------+--------------------+--------------------+

Encapsulation overhead

VXLAN over IPv4 = 50 bytes total:

• Outer Ethernet = 14 bytes
• Outer IPv4 = 20 bytes
• Outer UDP = 8 bytes
• VXLAN header = 8 bytes

VXLAN over IPv6 = 70 bytes total:

• Outer Ethernet = 14 bytes
• Outer IPv6 = 40 bytes
• Outer UDP = 8 bytes
• VXLAN header = 8 bytes

Why UDP and not TCP?

• Lightweight
• Avoids TCP overhead and session handling
• Allows ECMP and load-balancing using UDP source-port entropy

Standard VXLAN uses UDP destination port 4789.

Mental model

• 802.1Q VLAN tag = sticker on the Ethernet frame
• VXLAN header = wrapper around the whole frame
• VNI = label on that wrapper

One-line summary

VXLAN is a Layer 2 overlay over a Layer 3 IP network that encapsulates Ethernet frames using UDP and identifies segments using a 24-bit VNI.

What it is

A password-based EAP method. The RADIUS server presents a certificate so the client can validate it. Once a TLS tunnel is built, the client sends its username and password inside the tunnel using MSCHAPv2. The RADIUS server validates the password against AD.

Server proves itself with a cert. Client proves itself with a password. One-way cert auth on the outside, password auth inside the tunnel.

The three players

EAP runs end-to-end between supplicant and RADIUS. The NAS is a postman. It wraps EAP in EAPOL on the wireless side, repackages it inside RADIUS attributes on the wired side, and forwards it. It cannot read the EAP payload.

What each side needs

End-to-end auth flow

1. User connects to SSID, supplicant prompted for credentials
2. AP wraps EAP-Identity in Access-Request (code 1) to RADIUS on UDP 1812
3. RADIUS replies with Access-Challenge (code 11) to start the PEAP TLS handshake
4. Multiple Request and Challenge round trips negotiate the TLS tunnel. Server presents its certificate, supplicant validates it, both derive session keys
5. Supplicant sends MSCHAPv2 challenge/response inside the encrypted TLS tunnel
6. RADIUS validates the MSCHAPv2 response against AD via the Netlogon Secure Channel to a domain controller
7. RADIUS queries AD via LDAP for group memberships
8. RADIUS walks Network Policies top to bottom, first match wins
9. Matched policy injects tunnel attributes into reply
10. RADIUS sends Access-Accept (code 2) with Tunnel-Type, Tunnel-Medium-Type, Tunnel-Assignment-Id, MS-MPPE keys
11. AP reads Tunnel-Assignment-Id, matches its VLAN Assignment Rule, bridges client into assigned VLAN
12. AP uses MS-MPPE keys to derive PMK for WPA2 4-way handshake
13. Client gets DHCP from the assigned VLAN's scope
14. AP sends Accounting-Request Start (code 4) on UDP 1813
15. RADIUS replies Accounting-Response (code 5)
16. Periodic Interim-Updates throughout session, Stop on disconnect

Critical truth about the request packet

The Access-Request does not contain a VLAN or tunnel ID. It carries only:

• User-Name
• EAP-Message (outer plaintext, inner encrypted once TLS is up)
• NAS-IP-Address, NAS-Identifier
• Called-Station-Id (AP BSSID + SSID)
• Calling-Station-Id (client MAC)
• NAS-Port-Type (Wireless-802.11)

The tunnel attributes only appear on the return leg (Access-Accept), generated by RADIUS based on policy match.

RADIUS message codes - Authentication (UDP 1812)

RADIUS message codes - Accounting (UDP 1813)

Acct-Status-Type values

Key RADIUS attributes for VLAN assignment

Aruba Instant typically uses attribute 82. Cisco WLC typically uses attribute 81. Both achieve the same outcome.

RADIUS packet structure

+--------+--------+----------------+--------------------+----------+
| Code   | ID     | Length         | Authenticator      | Attribs  |
| 1 byte | 1 byte | 2 bytes        | 16 bytes           | variable |
+--------+--------+----------------+--------------------+----------+

ID pairs requests with replies. The Authenticator field plus the Message-Authenticator attribute (80, HMAC-MD5 over the packet using the shared secret) provide integrity. Message-Authenticator is mandatory when EAP is in use.

Onboarding workflow for new user category

One-time setup (network team):

1. Create RADIUS Network Policy with condition: User Groups contains X
2. Set RADIUS attributes: Tunnel-Type=VLAN, Tunnel-Medium-Type=802, Tunnel-Assignment-Id=N
3. Create AP VLAN Assignment Rule: if Tunnel-Assignment-Id equals N then bridge VLAN N
4. Ensure VLAN N exists on switch, has DHCP scope, has SVI / gateway

Per user (systems team):

1. Create user in AD
2. Add to the appropriate AD group

AD touches identity only. RADIUS holds the policy. AP holds the rule.

Strengths

• No client-side PKI required, just username and password
• Easy onboarding, especially for BYOD where pushing a client cert is hard
• Works with existing AD passwords, no parallel credential lifecycle
• Cheap and fast to deploy at scale

Weaknesses

• Rogue RADIUS attack surface. If the supplicant is misconfigured (no CA trust enforcement, no server name validation), an attacker can stand up a fake RADIUS server, capture the MSCHAPv2 challenge/response, and crack it offline to recover the password
• Passwords are crackable offline. MSCHAPv2 is a known-weak hash protocol. Capture once, brute force forever
• User-typed credentials. Subject to phishing, shoulder surfing, weak passwords, reuse across services
• Password expiry causes mass WiFi failures. Every password rotation cycle generates support tickets
• Server cert validation is the only thing keeping it secure. If clients are not enforcing it (and many supplicants default to permissive), the whole method falls apart

Common NPS gotcha

On the policy Overview tab, the "Access permission" radio must be set to "Grant access". You can have perfect Conditions, Constraints, and Settings and the policy will still deny if this is wrong. Catches everyone exactly once.

Debug cheat sheet

Capture and log sources

• Wireshark on RADIUS NIC, filter radius. Cleanest view of all RADIUS traffic
• NPS Event Viewer. Custom Views > Server Roles > Network Policy and Access Services. Event 6272 = granted, 6273 = denied. Shows matched policy name and reason
• NPS log files in %SystemRoot%\System32\LogFiles\ in IAS format
• SPAN or mirror on AP uplink if RADIUS access not available

Combine packet capture (what flew on the wire) with RADIUS event log (which policy was picked and why) for full diagnosis.

Reference values

What it is

A certificate-based EAP method. Both sides present certificates. The RADIUS server validates the client's certificate. The client validates the RADIUS server's certificate. The TLS handshake itself is the authentication. No password is ever exchanged.

Server proves itself with a cert. Client proves itself with a cert. Mutual cert authentication.

The three players

Same as PEAP-MSCHAPv2. Supplicant, authenticator, authentication server. EAP runs end-to-end between supplicant and RADIUS. The NAS is a relay.

What each side needs

End-to-end auth flow

1. Client connects to SSID
2. AP wraps EAP-Identity in Access-Request (code 1) to RADIUS on UDP 1812
3. RADIUS replies with Access-Challenge (code 11) containing EAP-TLS Start
4. Client sends TLS ClientHello inside Access-Request
5. RADIUS replies with ServerHello, server Certificate, CertificateRequest, ServerHelloDone
6. Client validates server cert against its trusted CA store
7. Client sends its own Certificate, ClientKeyExchange, CertificateVerify (signs the handshake with its private key, proving possession), ChangeCipherSpec, Finished
8. RADIUS validates the client certificate: chains to a trusted CA, not expired, not revoked (CRL or OCSP check), CertificateVerify signature valid (proves client holds the private key)
9. RADIUS maps the certificate to an identity (cert-to-account mapping, usually via UPN or DNS name in the Subject Alternative Name)
10. RADIUS queries AD via LDAP for the mapped account's group memberships
11. RADIUS walks Network Policies top to bottom, first match wins
12. Matched policy injects tunnel attributes into reply
13. RADIUS sends Access-Accept (code 2) with Tunnel-Type, Tunnel-Medium-Type, Tunnel-Assignment-Id, MS-MPPE keys
14. AP reads Tunnel-Assignment-Id, matches its VLAN Assignment Rule, bridges client into assigned VLAN
15. AP uses MS-MPPE keys to derive PMK for WPA2 4-way handshake
16. Client gets DHCP from the assigned VLAN's scope
17. AP sends Accounting-Request Start (code 4) on UDP 1813
18. Periodic Interim-Updates, Stop on disconnect

What is different from PEAP-MSCHAPv2

The auth half changes, the policy half does not. Tunnel attributes, VLAN assignment, MS-MPPE key delivery, accounting, all identical.

Device certs vs user certs

This is the design decision that bites schools and corporate fleets. The cert determines what the network sees.

Implications for VLAN assignment:

• Device cert: VLAN follows the laptop. A staff laptop stays on the staff VLAN regardless of who logs in. If a teacher hands the laptop to a student, the student lands on the staff VLAN. This is by design, not a bug
• User cert: VLAN follows the user. When a different user logs in, the supplicant re-authenticates with the new user's cert and lands on a different VLAN

User certs solve the lending scenario but introduce operational cost: every user needs a cert on every device they use, provisioning is per-user-per-device, and the chicken-and-egg of "first login on a new device needs cert enrolment but the network needs the cert" usually requires a wired bridge or a staged onboarding flow.

The common compromise for managed-fleet environments: device certs for the standard fleet, plus a documented policy that loaner devices come from the appropriate VLAN's pool, not by lending across user categories.

Cert-to-account mapping

When EAP-TLS succeeds, the RADIUS server has a validated certificate but no AD account yet. Mapping the cert to an account is what lets policy evaluation proceed.

Cert-to-account mapping is where many EAP-TLS deployments quietly fail. Symptoms: cert validates fine, but RADIUS rejects because it cannot find a matching account. Look at NPS Event 6273 reason codes.

Strengths

• No password on the wire, ever. Removes phishing, password spray, MSCHAPv2 cracking, password expiry headaches
• Mutual authentication enforced by the protocol. Cannot be silently weakened by client misconfiguration the way PEAP can
• Private key never leaves the device. Often hardware-backed (TPM), so even stealing the device does not yield extractable credentials
• Cleanest path for zero-trust and posture-based access

Weaknesses

• PKI is real infrastructure. Issuing CA, enrolment automation, renewal lifecycle, revocation. None of it is free
• Onboarding friction. Every device needs a cert before it can connect. BYOD and unmanaged devices are painful
• Cert expiry causes silent dropouts. If renewal automation breaks, devices fall off the network with no obvious cause
• Non-domain devices need a separate path. Printers, IoT, AV gear usually fall back to MAB or a PSK SSID
• Troubleshooting is harder. Debugging PKI chains, revocation reachability, SAN mismatches, clock skew, cert-to-account mapping

Common deployment patterns

Pure-EAP-TLS-everywhere designs are rare outside high-security environments. Most real deployments mix EAP-TLS for managed devices with PEAP or MAB for everything else.

SCEPman and RADIUSaaS specifics

When using cloud-based RADIUS like RADIUSaaS with SCEPman issuing certs via Intune:

• SCEPman acts as the issuing CA, certs are deployed to devices via Intune SCEP profiles
• RADIUSaaS validates the cert chain and provides the RADIUS endpoint reachable over RadSec (RADIUS over TLS, TCP 2083) from the APs
• Cert-to-account mapping happens against Entra ID rather than on-prem AD
• VLAN assignment via Tunnel-Private-Group-Id or Tunnel-Assignment-Id works the same way

Product capabilities change, verify current SCEPman and RADIUSaaS docs before committing a design.

Debug cheat sheet (EAP-TLS specific)

Related concepts

• MAB (MAC Authentication Bypass) - fallback for devices that cannot do 802.1X. Same RADIUS server, different policy that matches on Calling-Station-Id (MAC) instead of EAP
• CoA (Change of Authorization) - RFC 5176, UDP 3799. RADIUS pushes session changes (force re-auth, terminate, bounce port) without the client reconnecting. Useful for posture, quarantine, role changes
• RadSec - RADIUS over TLS, TCP 2083. Replaces UDP plus shared-secret with a TLS tunnel. Used by RADIUSaaS and any cloud RADIUS service where AP-to-server traverses the public internet
• EAP-TTLS - similar to PEAP, server cert + inner method, but the inner method is more flexible. Rare in Microsoft shops, more common in mixed environments

Reference values

Scope

Deep reference for packet capture analysis, MTU/MSS sizing, and troubleshooting transport-layer issues. Covers IPv4, IPv6, TCP, UDP at the byte level. The diagrams below follow standard RFC bit-position notation: each character pair represents one bit, bits 0-31 left to right, MSB first.

IPv4 header (RFC 791, 20 bytes minimum, up to 60 with options)

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version|  IHL  |   DSCP    |ECN|         Total Length          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Identification         |Flags|     Fragment Offset     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      TTL      |   Protocol    |        Header Checksum        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Source Address                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Destination Address                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|             Options (if IHL > 5)              |    Padding    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

IPv4 fragmentation flags - what to look for in pcap

• DF set, MF clear - normal unfragmented packet. If too big for an MTU on path, gets dropped with ICMP Type 3 Code 4 (Fragmentation Needed). This is how PMTU Discovery works.
• DF clear, MF set - this is a fragment, more to come
• DF clear, MF clear, Fragment Offset > 0 - this is the LAST fragment
• DF set, MF set - illegal combination, indicates a broken sender or malicious packet

IPv6 header (RFC 8200, fixed 40 bytes)

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| Traffic Class |              Flow Label               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Payload Length         |  Next Header  |   Hop Limit   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                   Source Address (128 bits)                   |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                                                               +
|                                                               |
+                                                               +
|                Destination Address (128 bits)                 |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Key differences from IPv4

• Fixed 40-byte header - no IHL field, no options inside the header. Options go in extension headers (Hop-by-Hop, Routing, Fragment, AH, ESP, etc.) chained via Next Header.
• No header checksum - relies on L2 (Ethernet FCS) and L4 (TCP/UDP checksum). Routers don't have to recompute anything per hop.
• No router-level fragmentation - only the source can fragment, via the Fragment extension header. Path MTU Discovery is effectively mandatory.
• Hop Limit replaces TTL - same function, more honest name.
• Next Header replaces Protocol - same numbering scheme, but can point to an extension header rather than directly to L4.
• Flow Label (20 bits) - for ECMP hashing and QoS. Rarely populated by endpoints today.

Common protocol numbers (IPv4 Protocol / IPv6 Next Header)

TCP header (RFC 9293, 20 bytes minimum, up to 60 with options)

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |       Destination Port        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Sequence Number                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Acknowledgment Number                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Data Of| Rsvd  |C|E|U|A|P|R|S|F|          Window Size          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Checksum            |        Urgent Pointer         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Options (if Data Offset > 5)          |    Padding    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

TCP control bits (flags)

Common flag combinations seen in captures

TCP connection states (RFC 9293)

Normal 3-way handshake

Client                                Server
------                                ------
CLOSED                                LISTEN
   |                                    |
   |---- SYN  (seq=x)             ---->|
SYN-SENT                                |
   |                                    |
   |<--- SYN+ACK (seq=y, ack=x+1) -----|
   |                              SYN-RECEIVED
   |                                    |
   |---- ACK  (seq=x+1, ack=y+1) ----->|
ESTABLISHED                          ESTABLISHED

Normal 4-way close (active vs passive)

Active closer                         Passive closer
-------------                         --------------
ESTABLISHED                           ESTABLISHED
   |                                    |
   |---- FIN, ACK                ---->|
FIN-WAIT-1                              |
   |                                    |
   |<--- ACK                     -----|
FIN-WAIT-2                           CLOSE-WAIT
   |                                    |
   |                              (app calls close)
   |                                    |
   |<--- FIN, ACK                -----|
   |                                LAST-ACK
   |                                    |
   |---- ACK                     ---->|
TIME-WAIT                            CLOSED
   |
   |  (wait 2*MSL)
   |
CLOSED

TIME-WAIT and 2*MSL - what every senior engineer should know

• The side that sends FIN first is the active closer and ends up in TIME-WAIT.
• RFC says wait 2 * MSL (Maximum Segment Lifetime). RFC suggests MSL = 2 minutes, so 2*MSL = 4 minutes default.
• Linux: hardcoded at 60 seconds, not configurable (kernel constant TCP_TIMEWAIT_LEN).
• Windows: registry TcpTimedWaitDelay, default 240s, range 30-300s.
• Why it exists: absorb delayed segments from this connection so they don't pollute a new connection on the same 4-tuple, and ensure the peer's FIN gets retransmitted and ACKed if the final ACK was lost.
• Symptom of exhaustion: "Address already in use" when restarting a server on a well-known port. Or ephemeral port exhaustion on a busy client (HTTP load test, monitoring poller).
• Linux tunables: net.ipv4.tcp_tw_reuse=1 (safe to enable, reuses TIME-WAIT for outgoing). tcp_tw_recycle was removed in 4.12 (NAT-hostile, do not look for it).
• Quick check on Linux: ss -tan state time-wait | wc -l

UDP header (RFC 768, fixed 8 bytes)

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |       Destination Port        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            Length             |           Checksum            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

• Length covers header + data (so minimum 8).
• Checksum is optional in IPv4 (set to 0 to skip), mandatory in IPv6.
• No sequence numbers, no flow control, no retransmission. Anything above that lives in the application protocol (QUIC, DNS, RTP, etc.).

MTU / MSS math reference

Standard Ethernet MTU is 1500 bytes (payload after Ethernet header). MSS is negotiated in the SYN/SYN-ACK Options field and is what TCP advertises as its max segment size.

FortiGate tip: set tcp-mss-sender and set tcp-mss-receiver on the firewall policy or interface clamp the MSS in transit. Common for VPN paths where PMTU is unreliable.

Path MTU Discovery (PMTUD) - how it breaks

• Sender sets DF flag. Router along path with smaller MTU drops the packet and returns ICMP Type 3 Code 4 (Fragmentation Needed, DF Set) with the next-hop MTU.
• Sender caches the new MTU per destination and reduces segment size.
• Common breakage: firewall along path drops ICMP unreachable. Sender keeps retransmitting full-size DF-set packets, application hangs after the SYN works fine but data stalls. Classic "small requests work, large ones don't" symptom.
• Workaround: MSS clamping at the tunnel ingress (see FortiGate tip above), or set DF=0 to allow router fragmentation (IPv4 only, slow).

Wireshark display filters - deep troubleshooting picks

BPF capture filters (tcpdump / wireshark capture mode)

Wireshark TCP analysis flags - what they actually mean

Useful ports for network engineers

Reference RFCs

Scope

QoS only matters when there is congestion. When the pipe is wider than demand, QoS does nothing useful. When demand exceeds capacity, QoS decides what gets through first and what waits or drops. The full job is: classify, mark, then enforce (queue, police, shape, drop). This collapsible focuses on the practical reference values and FortiGate-specific behaviour, with deep dives into Microsoft Teams QoS and SD-WAN shaping.

The ToS byte: DSCP + ECN

Byte 1 of the IPv4 header, also Byte 1 of the IPv6 Traffic Class field. Same format, same semantics.

 0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|   DSCP    |ECN|
+-+-+-+-+-+-+-+-+

Top 6 bits = DSCP. Bottom 2 bits = ECN. The legacy "IP Precedence" field was the top 3 bits of DSCP, which is why CS values (Class Selector) line up: CS5 (5 in IPPrec) = 101000 in DSCP = decimal 40.

DSCP codepoint reference

How to read AFxy

• x (first digit) = the queue / class. 1=low priority data, 2=low-latency data, 3=streaming, 4=conferencing.
• y (second digit) = drop precedence within that class. 1=lowest probability of drop, 3=highest. WRED uses this to decide what to drop first under congestion.
• So AF43 means "conferencing class, drop me first if congested". AF41 means "conferencing class, drop me last".

ECN bits

An ECN-aware router sees congestion, finds ECT(0)/ECT(1), and flips both bits to 11 (CE). Receiver echoes via TCP ECE flag in the next ACK. Sender reduces window (sends CWR flag in next segment). No packet drop, no retransmit. ECN works only end-to-end if every hop and both endpoints support it. Many middleboxes still zero it out.

802.1Q PCP / CoS (Layer 2 priority)

Inside the 802.1Q tag, the TCI (Tag Control Information) field is 16 bits:

 0                   1
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PCP |D|     VLAN ID (VID)     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

PCP (Priority Code Point) = 3 bits, also called CoS or 802.1p. DEI = 1 bit, drop eligibility indicator. VID = 12 bits, VLAN ID 0-4095.

Standard DSCP ↔ CoS mapping

Most vendors copy the top 3 bits of DSCP into the 3-bit CoS field. Useful when a router strips the 802.1Q tag on egress to a routed interface and the L2 priority gets lost - the DSCP survives.

Trust boundaries

• The trust boundary is the point in your network where you start honouring whatever DSCP/CoS the upstream device set. Inside: trust. Outside: classify and re-mark.
• Access ports (user devices): generally don't trust. Re-mark from CoS 0 / DSCP 0 unless the device is known good (Teams client with GPO, SIP phone with config).
• SIP phone uplink with PC dangling off it: trust CoS from the phone (CDP/LLDP-MED tells you what VLAN and how to trust), don't trust the PC behind it.
• Inter-switch trunks within your admin domain: trust.
• WAN edge (to ISP / MPLS): re-mark to whatever the carrier expects. Some scrub on ingress (set to 0), some preserve, some map DSCP to MPLS EXP. Confirm in writing with the carrier.
• Internet egress: assume markings will be wiped by the first ISP hop. End-to-end DSCP only works inside one admin domain or across a contracted MPLS service.

Microsoft Teams QoS - the canonical configuration

These are source port ranges on the Teams client. You must enable QoS in the Teams Admin Centre (Meetings > Meeting settings) AND configure the client to insert DSCP markings via GPO/Intune. By default Teams uses any ephemeral port 1024-65535 and classification by port range fails.

Where to enable Teams QoS marking

• Teams Admin Centre tenant setting: Meetings > Meeting settings > Network > Quality of Service. Turn on "Insert Quality of Service (QoS) markers" and lock to the 50000-50059 port ranges.
• Windows clients (domain joined): GPO Policy-based QoS targeting ms-teams.exe (new Teams) and teams.exe (classic), DSCP 46, source ports 50000-50019, protocol TCP+UDP. Repeat for video and sharing.
• Windows clients (Intune managed): NetworkQoSPolicy CSP via Intune. Same OMA-URI settings, applied per-device.
• Windows PowerShell (one-off): New-NetQosPolicy -Name "TeamsAudio" -AppPathNameMatchCondition "ms-teams.exe" -IPProtocolMatchCondition Both -IPSrcPortStartMatchCondition 50000 -IPSrcPortEndMatchCondition 50019 -DSCPAction 46
• Teams Rooms on Android: tenant-level only. Note: Android Teams Rooms uses DSCP 34 (AF41) for both video AND screen sharing, not 18.
• Teams Rooms on Windows: same as Windows clients.
• macOS / Linux Teams clients: client-side DSCP marking is limited or absent. Mark at the network layer (FortiGate shaping policy matching the port ranges).

Teams QoS gotchas

Queueing strategies

Policing vs shaping

• Policing: hard ceiling. Excess is dropped (or re-marked) immediately. No buffering. Low memory, sharp edges, can be brutal on TCP. Use on ingress to protect downstream.
• Shaping: smooth ceiling. Excess is buffered and released at the configured rate. Adds latency but avoids loss. Use on egress to fit the next-hop pipe (ISP CIR, MPLS contract rate).
• FortiGate does shaping, not policing. Conceptually it's a token-bucket egress shaper.
• Rule of thumb: shape down, police up. Shape your egress to the carrier rate. Police user-side ingress to protect your shaper from overflowing.

WRED / RED - drop strategies

• Tail drop (default): queue full, drop new arrivals. Causes global TCP synchronisation - everyone halves window at once, queue drains, everyone ramps up together, queue refills, repeat. Sawtooth utilisation.
• RED (Random Early Detection): probabilistic drop as queue depth grows. Smooths TCP behaviour by hitting individual flows at different times.
• WRED (Weighted RED): per-class drop curves. AF11 starts dropping at 30% queue depth, AF13 at 60% - drop precedence in action. EF rarely uses WRED because it's normally strict-priority and policed instead.

FortiGate QoS - architecture and quirks

• FortiGate shapes on egress only. To limit download speed for users, apply a shaper on the LAN-side interface (the egress for return traffic) - or use the reverse field on the shaping policy.
• Three shaper types: shared (one bucket per policy or aggregate), per-IP (one bucket per source IP within a policy), interface-based / shaping profile (hierarchical, attached to physical interface).
• Shaping policies are separate from firewall policies, evaluated after the firewall policy matches. Order matters.
• outbandwidth must be set on the interface for any shaping profile to work. Without a ceiling, the shaper has nothing to apportion. This is the single most common QoS misconfig on FortiGate.
• SD-WAN service rules can match on DSCP and steer based on it (dscp-forward enable). Useful when CE-marked traffic should prefer MPLS over internet.
• Three priority levels: high, medium, low. Within a priority, guaranteed-bandwidth is honoured first, then maximum-bandwidth caps.

FortiGate shaper + shaping policy (Teams audio example)

config firewall shaper traffic-shaper
    edit "Teams-Audio-EF"
        set maximum-bandwidth 100000
        set per-policy enable
        set priority high
        set diffserv-forward enable
        set diffservcode-forward 101110
        set diffserv-reverse enable
        set diffservcode-rev 101110
    next
    edit "Teams-Video-AF41"
        set maximum-bandwidth 4000000
        set per-policy enable
        set priority medium
        set diffserv-forward enable
        set diffservcode-forward 100010
    next
    edit "Bulk-CS1"
        set guaranteed-bandwidth 1000
        set maximum-bandwidth 1048576
        set priority low
        set diffserv-forward enable
        set diffservcode-forward 001000
    next
end

config firewall shaping-policy
    edit 1
        set name "Teams audio (port 50000-50019)"
        set srcaddr "internal-subnets"
        set dstaddr "all"
        set service "ALL_UDP"
        set srcintf "internal"
        set dstintf "virtual-wan-link"
        set traffic-shaper "Teams-Audio-EF"
        set traffic-shaper-reverse "Teams-Audio-EF"
    next
end

For port-range matching you'll typically build a custom firewall service object with the UDP source port range, then reference it in the shaping-policy service field.

FortiGate hierarchical shaping profile (egress)

config firewall shaping-profile
    edit "wan-uplink"
        set type queuing
        set default-class-id 31
        config shaping-entries
            edit 1
                set class-id 5
                set priority high
                set guaranteed-bandwidth-percentage 30
                set maximum-bandwidth-percentage 100
            next
            edit 2
                set class-id 10
                set priority medium
                set guaranteed-bandwidth-percentage 40
                set maximum-bandwidth-percentage 100
            next
            edit 3
                set class-id 31
                set priority low
                set guaranteed-bandwidth-percentage 10
                set maximum-bandwidth-percentage 100
            next
        end
    next
end

config system interface
    edit "wan1"
        set egress-shaping-profile "wan-uplink"
        set outbandwidth 100000
    next
end

Match traffic into class IDs via shaping policies (set class-id 5) or via firewall policies. Class IDs 2-31 are admin-defined; class 0 is internal, class 1 is reserved.

FortiGate SD-WAN service rule with DSCP forwarding

config system sdwan
    config service
        edit 1
            set name "Teams-EF-prefer-MPLS"
            set mode priority
            set priority-members 1 2
            set dst "Microsoft-365-services"
            set tos 0xb8
            set tos-mask 0xfc
        next
        edit 2
            set name "Set-EF-on-egress"
            set priority-members 1
            set dscp-forward enable
            set dscp-forward-tag 101110
            set dst "voip-server"
        next
    end
end

The tos / tos-mask pair matches packets already marked. 0xb8 = 10111000 = EF (46) shifted left 2 bits because ToS field is 8 bits wide. Mask 0xfc ignores the bottom 2 ECN bits. dscp-forward-tag sets a new DSCP on egress through the SD-WAN.

FortiGate QoS troubleshooting

diagnose firewall shaper traffic-shaper list
diagnose firewall shaper per-ip-shaper list
diagnose firewall shaper interface-shaper list
diagnose firewall iprope list 100015
diagnose netlink interface list <intf>
get system performance status
diag sniffer packet any 'host x.x.x.x' 4 0 a    # 'a' includes timestamp; check TOS byte

• diag firewall shaper traffic-shaper list shows per-shaper counters: current bandwidth, packets dropped, bytes dropped. Drops here mean your shaper is doing its job.
• diag firewall iprope list 100015 shows shaping-policy rules with the shaper attached (forward and reverse). This is where you confirm a policy is actually getting the shaper you expect.
• get system performance status shows CPU. Shaping is CPU-bound on lower-end models, so check before blaming the shaper for slowness.

Wireshark filters for QoS work

Common gotchas across the board

Reference RFCs