Troubleshooting WLANs with Centralized Controllers Steve Acker Wireless Network Consulting Engineer CCIE #14097 CISSP #86844 BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Troubleshooting Wireless LANs Troubleshooting 101 Basic Concepts Getting it Right AP Troubleshooting Mobility Client Troubleshooting 802.11n Troubleshooting BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Troubleshooting 101 BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Problem Definition Having a clear understanding of the problem yourself will make life easier for the TAC engineer and you Examples: Good: client with an X card running Y driver version has issues authenticating to a WLAN on WLC version Z with WPA2-AES using MS-CHAPv2 due to no EAP-Identity-Request from AP Bad: client can’t connect to network BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Tools What Is Needed: L1: spectrum expert L2/L3: wireless sniffer trace: Omnipeek, Airpcap (multichannel), Wireshark, etc Packet replay Configuration analysis: WLC config analyzer Grep/editor tools Custom made BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Basic Concepts BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Steps to Building an 802.11 Connection 802.11 Listen for Beacons State 1: Unauthenticated, Unassociated Probe Request Probe Response Authentication Request Authentication Response State 2: Authenticated, Unassociated AP plete, m o C h t Au 802.11 datory n a M t o n Association Request Association Response 802.1 mplete o C c o Ass WLC State 3: Authenticated, Associated (Optional: EAPOL Authentication) (Optional: Encrypt Data) 10 Move User Data BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 802.1X Authentication Supplicant Authenticator Server EAPOL-START EAP-ID-Request EAP-ID-Response RADIUS (EAP-ID_Response) Rest of the EAP Conversation EAP-Success The Supplicant Derives the Session Key from User Password or Certificate and Authentication Exchange BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Radius-Access-Accept (Key) Session Key CAPWAPP Changes (5.2 and Above) From a client perspective, nothing is changed Traffic flow is encrypted (DTLS), and uses new ports: 5246 for control 5247 for data This means make sure to update ACL during upgrade! BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Radio Resource Management Refresher Dynamic Channel Assignment (DCA) Selects channels for the radios to use Responds to interference Dynamic Power Control (DPC) Reduces radio power, to ensure that each radio hears exactly three others at or above the tx-power-thresh value Can adjust for missing radio, increasing power Coverage Hole Detection (CHD) Detects coverage holes, by identifying clients from which we are receiving a poor signal, and accordingly increases radio power, to compensate BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 10 Successful 802.1X Client Authentication debug aaa events (WLC_CLI) >debug mac addr 00:13:ce:57:2b:84 (WLC_CLI) >debug aaa events enable [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 49) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57 [TIME]: DEBU CTRLR processIncomingMessages:3480 ****Enter processIncomingMessages: response code=11 [TIME]: DEBU CTRLR processRadiusResponse:3053 ****Enter processRadiusResponse: response code=11 [TIME]: * processRadiusResponse:3325 Access-Challenge received from RADIUS server 20.20.20.12 for mobile 00:13:ce:57:2b:84 receiveId = [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 59) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57 [TIME]: DEBU CTRLR processIncomingMessages:3480 ****Enter processIncomingMessages: response code=2 [TIME]: DEBU CTRLR processRadiusResponse:3053 ****Enter processRadiusResponse: response code=2 [TIME]: * processRadiusResponse:3325 Access-Accept received from RADIUS server 20.20.20.12 for mobile 00:13:ce:57:2b:84 receiveId = * DEBU STA 00:13:ce:57:2b:84 BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 77 Failed 802.1X Client Authentication debug aaa events—AAA Server Unreachable (Cisco Controller) >debug mac addr 00:13:ce:57:2b:84 (Cisco Controller) >debug aaa events enable [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57 [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57 [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57 [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57 [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57 [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57 [TIME]: * radiusProcessQueue:2735 Max retransmission of Access-Request (id 66) to 20.20.20.12 reached for mobile 00:13:ce:57:2b:84 [TIME]: * sendAAAError:323 Returning AAA Error 'Timeout' (-5) for mobile 00:13:ce:57:2b:84 * DEBU STA 00:13:ce:57:2b:84 AAA connectivity failure will generate an SNMP trap In the WLC GUI, Go to: Management | SNMP BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Trap Logs 78 Verify Complete 802.11/ 802.1X Connectivity debug pem state (WLC_CLI) >debug mac addr 00:13:ce:57:2b:84 (WLC_CLI) >debug pem state enable [TIME]: pem_api.c:1780 - State Update 00:13:ce:57:2b:84 from RUN (20) to START (0) [TIME]: pem_api.c:1836 - State Update 00:13:ce:57:2b:84 from START (0) to AUTHCHECK (2) [TIME]: pem_api.c:1859 - State Update 00:13:ce:57:2b:84 from AUTHCHECK (2) to 8021X_REQD (3) [TIME]: pem_api.c:3977 - State Update 00:13:ce:57:2b:84 from 8021X_REQD (3) to L2AUTHCOMPLETE (4) [TIME]: pem_api.c:4152 - State Update 00:13:ce:57:2b:84 from L2AUTHCOMPLETE (4) to RUN (20) BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 79 Troubleshooting 802.1X Make sure the RADIUS server is properly configured Make Sure the Correct Shared Secret Is Input Select the Correct RADIUS Port (Common Ports Are 1812 and 1645) Status Must Be Enabled Timeout May Be too Short Network User Auth Has to Be Enabled for This AAA Server to Be Used Globally, Otherwise, Select on WLAN In the WLC GUI, Go to: Security | AAA BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public RADIUS Authentication and Then Select Edit or New 80 Troubleshooting 802.1X Make sure the proper security policy is enabled for both encryption and authentication Step (1): Select the Desired Layer Security Configuration Step (2): Check Radius list per WLAN or Use Global list In the WLC GUI, Go to: WLANs | WLANs BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public WLANs and Then Select Edit for the WLAN of Interest 81 Troubleshooting 802.1X Enabled Logging in your ACS server to identify where issues might lie with backend authentication Make Sure at Least Logging for Failed Attempts Is Enabled on ACS So Server-side Debugging Can Be Performed In ACS, Select System Configuration | Logging and Enable Each Desired Option BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 82 DHCP Succeeds 2.1 1D a ta 802.11 Management RADIUS IP LWAPP WLC IP LWAPP ACS DH CP Di sc D DH HCIP ove r CP P O Re ffer DHCP DH que st CP Ac k EOIP radio driver supp EAP Chan DHCP Discover 802.11 Data DHCP Offer 802.11 Management ChDHCP Request an 80 DHCP Ack WLC Client probes for the SSID Client authenticates/associates in 802.11 to an AP EAP takes place 3.1 EAP dialog between client and authenticator 3.2 authenticator (radius) dialog to end-user DB DHCP address negotiation Client reaches RUN state BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 83 Client IP Provisioning via DHCP debug dhcp message (WLC_CLI) >debug mac addr 00:13:ce:57:2b:84 (WLC_CLI) >debug dhcp message enable [TIME]: dhcp option: received DHCP DISCOVER msg DHCP Discover message details [TIME]: Forwarding DHCP packet (332 octets) from 00:13:ce:57:2b:84 20.20.20.1 packet received on direct-connect port requires forwarding to external DHCP server Next-hop is [TIME]: dhcp option: received DHCP OFFER msg [TIME]: dhcp option: server id = 20.20.20.1 [TIME]: dhcp option: netmask = 255.255.255.0 [TIME]: dhcp option: gateway = 20.20.20.1 DHCP Offer message details [TIME]: dhcp option: received DHCP REQUEST msg [TIME]: dhcp option: requested ip = 20.20.20.113 [TIME]: dhcp option: server id = 1.1.1.1 DHCP Request message details [TIME]: Forwarding DHCP packet (340 octets) from 00:13:ce:57:2b:84 packet received on direct-connect port requires forwarding to external DHCP server Next-hop is 20.20.20.1 [TIME]: dhcp option: received DHCP ACK msg DHCP Ack message details BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 84 Troubleshooting DHCP If Clients Aren’t Getting Addresses Properly via DHCP, Ensure: Clients are not configured for static addressing DHCP scopes are properly configured (either external or internal DHCP) External servers: need to support DHCP proxy—if they don’t, turn on DHCP bridging: (WLC_CLI) >config dhcp proxy disable Internal DHCP server: after properly configuring the WLC’s scopes, each interface needs to have the WLC’s management IP as its DHCP server IP address, as below: Note: The WLC’s Internal DHCP Server Will Provide Addresses to APs, As Well, Provided the WLC Is Running 4.0 or Later and the AP DHCP Requests Can Find the Controller’s Management Interface In the WLC GUI, Go to: Controller | Interfaces and Select Edit for the Interface of Choice For Internal DHCP, Input the WLC’s Management IP Address Here BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 85 PING Succeeds!! EAP 802.11 Data 802.11 Management Ch an 80 RADIUS IP WLC 11 Da ta 802.11 Management ACS LWAPP IP LWAPP EOIP radio driver supp Chan IP DHCP WLC Client probes for the SSID Client authenticates/associates in 802.11 to an AP EAP takes place 3.1 EAP dialog between client and authenticator 3.2 authenticator (radius) dialog to end-user DB DHCP address negotiation Client reaches RUN state BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 86 Troubleshooting Roaming Does the roaming problem happen when using open auth/no crypto? (factor out: EAP, CCKM, PMK caching, etc.) Does the roaming problem with intracontroller roams, or only intercontroller? (factor out: VLAN config problems, CAM table, L3 mobility problems, mobility group config problems) Does the roaming problem occur only in specific locations? (factor out: RF coverage issues) Does the problem happen when using one supplicant, but not another? (factor out: specific supplicant issues) BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 87 Do Some Clients Have Roaming Problems, and Others Not? Try tuning the client roaming behavior Intel: roaming aggressiveness knob 7921: lock to 802.11a if you have the coverage CB21ag: turn down Scan Valid, BSS Aging in Device Manager (see “Optimize CB21AG/PI21AG Roaming Behavior”, Document ID 69403, cisco.com) Try upgrading the client code 7921: must have at least 1.0.5 Intel: drivers in latest (March ’09) 12.4 bundle BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 88 What if Some Clients Just Don’t Roam Right, No Matter What? Prove that another wireless adapter (CB21AG?) in the identical application, works fine Escalate to your laptop/device vendor Open a case with Cisco, if TAC assistance is needed in setting up the back end debugging At times, we can help with vendor communications (Broadcom, Intel) however, not rely solely on us BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 89 Q&A BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 90 BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public 91 .. .Troubleshooting Wireless LANs Troubleshooting 101 Basic Concepts Getting it Right AP Troubleshooting Mobility Client Troubleshooting 802.11n Troubleshooting BRKAGG-3011_c2 © 2009 Cisco. .. BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Troubleshooting 101 BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Problem Definition Having a clear understanding... Custom made BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Basic Concepts BRKAGG-3011_c2 © 2009 Cisco Systems, Inc All rights reserved Cisco Public Steps to Building an