moving ip addresses to another device without downtime: scenarios

For the impatient: arpsend !

We all know the problem: you replace a firewall with another model, correctly configure everything and lose connectivity.

After a few hours, things automagically start working again.

You may also know the reason: the upstream router has an entry in its arp cache, incorrectly linking the ip address you’re trying to reach with the mac address of the old network card.

possible reasons this may happen:

• replacing a firewall with a new/other model.
• moving an ip address to another network card on the same firewall (note: you will not have the issue with secondary ip’s as the ethernet address won’t change in this case).
• Replacing a broken network card in a firewall
• moving an ip address to another device, while keeping the existing device up and running

Now you may also know the solution. Easy: get rid of the old arp entry. Then, a new arp request will be sent by the upstream router and the new mac address will get sent in reply.

So, logically, you would clear or delete the arp entry in the upstream router. Problem: when this is the provider’s router, you cannot access it.

There are a few ways to go about this…

First, the basic, logical way:

• call the provider and get them to delete the offending arp entry. Could be tricky as you will have to cross the first line helpdesk and they may not have the correct procedure…

Next, you could take action yourself, although there is a difference between action, correct action and efficient action.

• reboot the router.
• remove the ethernet cable (pointing to your firewall) from the router’s network card

Both these actions would cause the arp cache (or at least part of it) te be reset. Both actions will also cause some amount of downtime, not to mention the risc of losing access to the device while administering it remotely.

Now for the more interesting actions…The typical way to handle this issue is “the gratuitous arp”.

Clustered systems (depending on the used clustering mechanism) use this as a feature instead of as a defect.

When an active/passive system needs to fail over, the ip addresses need to be moved to the other node. This is more or less the same scenario as before: the addresses will be moved, but then the arp cache of the upstream router needs to be updated IMMEDIATELY or clients will lose connectivity (undermining the ide of a cluster of course).

This is typically done by sending a gratuitous arp.

The idea is to send an arp packet (best to send both a reply and a request due to the way the router os could handle these packets).

A typical arp process would be:

1) device a wants to send data to device b. it determines device b is on the same subnet so no routers need to be crossed. It therefore sends out a request for the mac address of device b. This packet is broadcast (keep in mind arp is a layer-II protocol so this is not an ip but a layer-2 broadcast) and contains: my ip = XYZ, my mac address is 123, I’m looking for IP ABC.

2) device b picks up on the broadcast and says: I need to reply. It sends a reply with: my ip = ABC, my mac address = 456, destination mac address is 123, destination IP = XYZ. As a bonus, device b will put the mac address and ip of device a in its arp cache for future use.

3) device a receives the reply and puts the information ‘ip address ABC is at mac address 456’ in its local arp cache.

for subsequent requests, it will consult its local cache.

but the experts can explain this much clearer: click.

So the question is: when do we time out the entry in the local arp cache? This is of course both operating system-dependent and tunable.

Now, how to update a neighbour’s arp cache to reflect a changed ip <-> mac binding on a local device ?

We could send a gratuitous arp. This can be sent to a certain mac address (or can be broadcast to update all neighbours) and you would put both the source and destination mac address to the same data: the mac address of your local device.

Neighbours will see this packet and update their arp caches.

This is what you would typically do.

This requires of course accessibility to both local devices (the old one and the new one) AND the possibility to send a gratuitous arp at all.

Usually this is done with the arping command, linux or bsd systems should have it onboard.

But what if one of these devices is no longer accessible OR does not have the arping command?

Here it becomes interesting.

Some possible solutions…

1. You might think: do I not have a tool which allows me to craft custom packets? Like netcat ?
   Unfortunately netcat only does tcp or udp…
2. You might decide to send a ping from the new device, sourcing the ping from your moved IP. ON cisco firewalls for example, you can not do this by default, but you CAN send a TCP ping.
   Unfortunately, devices like an ASA do not allow your to configure secondary ip’s (strangely enough) and use proxy arp for adding ip’s for incoming services. So no ball..
3. I could install a system which does support an arping-like command, patch it into the same subnet, spoof the network card’s mac address with the mac address of the new interface/device and send the gratuitous arp from there.
   This would however cause a duplicate mac address which is always something to avoid…

So what we need is a netcat-like tool but it needs to operate on both layer-II and layer-III. If we had this, we could just send the gratuitous arp, hand-craft all the required fields (source mac and IP) and send this custom-made gratuitous arp out to the upstream router, all without causing downtime on any level.

Enter arpsend !

It will get the ip moved smoothly without any downtime and with the ability update only select devices, or broadcast it out to the entire local subnet. There’s even a switch to allow you to send both replies and requests, maximizing your chances of success !