Troubleshooting DHCP Pool Exhaustion on Cisco IOS

 It quite common to see a Cisco Switch or a Router being used as a DHCP server, generally in a small office or a remote branch office, where there is no need of a dedicated DHCP server.

If you are managing one such network, you may have run into a issue, where clients fail to get an IP from the Cisco Switch/Router. If you are lucky, you might get the Syslogs showing a lot of IP conflicts being logged. That will be your hint.  Most often that not, you may not see anything in the logs, if the conflicts are old or if logging is incorrectly setup.

I came across one such case during my tenure at Cisco as a TAC engineer with Cisco LAN Switching, The customer, unlike many others,  was a curious being  and wanted to dig deeper and find out why its happening.

NOTE: The term "server" will be used to refer  the Switch or the Router, which has the active DHCP Server service.

So first thing I did was to run a dhcp debug on the switch. And we observed something interesting.

cmd : debug dhcp server


A.  DHCP Request was received at Switch from the Client

B,  DHCP Assignment Failed stating that the Pool is exhausted.



Nov 27 03:36:30.154: DHCPD: DHCPDISCOVER received from client 011c.66aa.c414.71 on interface Vlan666.
Nov 27 03:36:30.154: DHCPD: using received relay info.Nov 27 03:36:30.154: DHCPD: Sending notification of DISCOVER:

Nov 27 03:36:30.154:   DHCPD: htype 1 chaddr 1c66.aac4.1471Nov 27 03:36:30.154:   DHCPD: interface = Vlan666Nov 27 03:36:30.154:   DHCPD: class id 64686370636420342e302e3135Nov 27 03:36:30.154:   

DHCPD: out_vlan_id 0Nov 27 03:36:30.154: DHCPD: subnet [192.168.66.1,192.168.66.254] in address pool VLAN666 is empty.
Nov 27 03:36:30.154: DHCPD: Sending notification of ASSIGNMENT FAILURE:

Nov 27 03:36:30.154:   DHCPD: htype 1 chaddr 1c66.aac4.1471Nov 27 03:36:30.154:   DHCPD: remote id 020a0000c0a842020d000000Nov 27 03:36:30.154:   DHCPD: interface = Vlan666Nov 27 03:36:30.154:   DHCPD: class id 64686370636420342e302e3135Nov 27 03:36:30.154:   DHCPD: out_vlan_id 0Nov 27 03:36:30.154: DHCPD: Sending notification of ASSIGNMENT_FAILURE:Nov 27 03:36:30.154:  DHCPD: due to: POOL EXHAUSTED



Next I check the DHCP Pool Statistics to see why such a big pool is exhausted.


Switch#sh ip dhcp pool

Pool VLAN666 :

 Utilization mark (high/low)    : 100 / 0

 Subnet size (first/next)       : 0 / 0

 Total addresses                : 254

 Leased addresses               : 16

 Excluded addresses             : 238

 Pending event                  : none

 1 subnet is currently in the pool :

 Current index   IP address range           Leased/Excluded/Total

 0.0.0.0    192.168.66.1 - 192.168.66.254    16    / 238   / 254


The IPs from the pool would be excluded when a duplicate IP is found or a conflict is detected. So next I check the IP conflicts, and see that the a lot of address are under conflict. 


IS50SWH002#sh ip dhcp conflict

IP address        Detection method   Detection time 

192.168.66.135    Gratuitous ARP     Oct 02 2013 06:45 PM

192.168.50.102    Gratuitous ARP     Oct 03 2013 06:34 AM

192.168.66.137    Gratuitous ARP     Oct 05 2013 06:07 PM

192.168.66.219    Gratuitous ARP     Oct 06 2013 03:29 PM

192.168.66.147    Gratuitous ARP     Oct 06 2013 04:12 PM

192.168.66.151    Gratuitous ARP     Oct 08 2013 07:05 PM

192.168.10.127    Gratuitous ARP     Oct 09 2013 05:48 PM

192.168.66.146    Gratuitous ARP     Oct 09 2013 11:48 PM

192.168.66.160    Gratuitous ARP     Oct 10 2013 01:19 AM

192.168.66.161    Gratuitous ARP     Oct 10 2013 06:36 PM

192.168.66.167    Gratuitous ARP     Oct 10 2013 06:40 PM

192.168.66.159    Gratuitous ARP     Oct 11 2013 09:16 AM



I cleared the DHCP conflicts and the excluded IPs from the pool were available in the Pool again.


IS50SWH002#sh ip dhcp pool VLAN666


Pool VLAN666 :

 Utilization mark (high/low)    : 100 / 0

 Subnet size (first/next)       : 0 / 0

 Total addresses                : 254

 Leased addresses               : 17

 Excluded addresses             : 128

 Pending event                  : none

 1 subnet is currently in the pool :

 Current index        IP address range        Leased/Excluded/Total

 192.168.66.2   192.168.66.1 - 192.168.66.254    17 / 0 / 254





So what is happening here ?

              The DHCP Server sends and ICMO Echo to the IP address it intends to allocate to a client before replying to the DHCP request. If it receives ICMP Echo Reply message to its ping, then the IP address is obviously in use by another client. And hence the DHCP server tries the next available address from the pool and so on, till it finds a free IP which can assigned to the requesting client.

If the DHCP conflict logging feature is enabled (which is default), then the server will log the conflict with a syslog message and puts the address on the list of conflicts. The addresses on that list (displayed with show ip dhcp conflict) are not used in the future. To reuse a conflicting address, the network admin has to manually clear it from conflict the list with the "clear ip dhcp conflict" cmd.



Resolutions

1, Turn off conflict logging

  DHCP Database agents are basically any storage location like  FTP, TFTP server or a storage media on the server itself. The DHCP agents are used for storing the DHCP lease info like addresses leased, lease expiry, client MAC etc.

 If you don't use DHCP agents, then it's best to turn off conflict logging with the "no ip dhcp conflict logging" configuration command.

  The DHCP conflict logging makes sense if the router uses external DHCP agents to store the DHCP binding database, otherwise any addresses that is allocated prior to a router reload would be reported as conflicts after the bindings are lost !

NOTE: Even with conflict logging disabled, there's no DHCP functionality loss and no chance of duplicate address allocation, as the server would still check whether an IP address is being used by another client before allocating it ! :)

The more recent Cisco IOS have auto conflict clearing command " ip dhcp conflict resolution", which will check for conflicts at regular intervals of time and clear them automatically. 

  2.  You can configure a lease time of  8-10 hours for DHCP client ( as that's is generally the working office hours !)  Which will ensure that the clients renew the DHCP IPs every 8-10 hours, so that there is a lesser probability of clients retaining a stale entry and creating a conflict.


  3. Ignore the BOOTP requests

Configure the following command to prevent BOOTP requests from getting an IP from the DHCP server. By Default, the BOOTP leased IPs do not have a lease time expiry. Hence any client which uses BOOTP instead of DHCP, will fetch an IP and will retain it for infinite time.

               "ip dhcp bootp ignore"





Hope it helps !

Until next time..... Cheers !

Troubleshooting POE on old Cisco IP Phones and Cisco Aironet Access points

Mondays are the days when we have to get a lot of things done !

So yesterday I was onsite and my task at hand was to provision couple of IP phones for a new bunch of people who had just moved in to our premises.

I setup a new subnet and vlans for them and plugged in the Cisco 7960G IP phones to the POE switch.

The Phones didn't bootup as they should ideally have.  I kept staring at the phone ... and so were the clients staring at me !  This was strange. I had IP phones rebooting, but this one was just dead. For a minute, I thought I had fried up the phone. 

Without any trace of panic, I got into troubleshooting mode.  Checked the cables. Then the POE status on the ports. POE budget on the global level. 
Logs to see any negotiation issues.

Swapped the ports with known working POE ports, where Aruba APs were earlier powered up.

All done but the Phones just wouldn't power up. It was time to seek help form the almighty and the all-knowing Google Baba.

Quick search of Datasheets of both the IP phone and the POE switch, gave me a clue of what was happening.

So here is the story.

A set of old Cisco Phones and old access points, run on on what is now called as Pre-Standard POE.

A standard which was defined way before the IEEE 802.3af and works only on CDP.

Hence any new third party switch that complies with IEEE 802.3af wouldnt be able to power these old phones.

Following is a snippet of good set of explanation that I found on the Cisco POE FAQ.

Q. What is the difference between pre-standard power and PoE?

A. They are the same. When Cisco Systems® first introduced powered Ethernet ports, the technology was called inline power ( now reffered as pre-standard POE ).  In order to allow for universal compatibility, Cisco now uses "Power over Ethernet" or "PoE" for all deployments, standard or pre-standard.

Q. What is the difference between the Cisco original pre-standard PoE products and the IEEE 802.3af standard?

A. The differences include:
the amount of power that is available to the connected device 
the method used for device discovery 
the way that power is removed from the wire when a powered device is removed

Q. What IP phone models support the Cisco pre-standard method of powered device detection?

A. These Cisco IP phones can accept Cisco pre-standard PoE from a card integrated with a Cisco Catalyst switch or a Catalyst in-line power patch panel:
7985G
7960G
7940G
7910G
7910G + SW
7912G
7905G
7902G
7962G
7975G

So next time if there is a old Cisco IP Phone or an Aironet AP that is being brought into the network, then do check if it supports an IEEE standard or not.

Reference Doc:

http://www.cisco.com/c/en/us/support/docs/voice-unified-communications/unified-ip-phone-7900-series/97869-poe-requirement-faq.html#qa5

Hope it helps.  

Until next time ... Cheers !