ARP (Address Resolution Protocol) provides dynamic mapping from an IP address to the corresponding hardware or MAC address. Using ARP, VxWorks implements a proxy ARP scheme over the shared-memory network that can make distinct networks appear as one logical network (that is, the networks share the same address space). This proxy ARP scheme is an alternative to the use of explicit subnets for accessing the shared-memory network. See 3.5 Shared-Memory Network on the Backplane. ¹

Previously, the shared-memory network (backplane) had to be partitioned as a separate subnet, and routes to that subnet had to be added to each host that required access to the shared-memory network. Each shared-memory network took up an individual subnet number; therefore, if a large number of shared-memory networks were present on a network, precious subnet numbers were rapidly consumed. However, with proxy ARP, the shared-memory network appears to use the same subnet/network as the Ethernet. Therefore, subnet numbers are not assigned.

If the shared-memory network is attached to a large network with many networks and subnets, network configuration becomes difficult. Proxy ARP simplifies network configuration because there is only one network to deal with and additional configuration on the host is unnecessary.

ARP is used to resolve a host's IP address into a hardware address. This is done by broadcasting an ARP request on the physical medium (typically Ethernet). The destination host sees the request and recognizes the destination IP address as its own. It then sends a reply with its hardware address.

In the example in Figure 4-7, host h1 wants to communicate with host h4. It needs h4's hardware address, so it broadcasts an ARP request. Host h4 sees the ARP request and replies with its hardware address. h1 records h4's IP-to-hardware mapping and proceeds to communicate with it.

Figure 4-7:  ARP Example

For a host to communicate with another host on a different subnet or network (as indicated by the IP addresses and the subnet mask), it must use a gateway. In Figure 4-8, vx3 acts as a gateway between Network A and Network B. Each host must have a routing entry for the gateway in its routing table. The routing table for vx1 to communicate with Network B includes entries like the following:

node           destination            gateway 
vx1            150.12.2.0             150.12.1.1 (network) 
 

The routing table for h1 to communicate with Network A includes entries like the following:

node           destination            gateway 
h1             150.12.1.0             150.12.2.1 (network) 
 

A sender cannot send an ARP request for a host on another subnet or network. Instead, if it doesn't know the hardware address for the gateway listed in its routing table, it sends an ARP request for the gateway's hardware address.

Figure 4-8:  Subnets and ARP

With proxy ARP, nodes on different physical networks are assigned addresses with the same subnet number. Because they appear to reside on the same network, they can communicate directly and use ARP to resolve hardware addresses. The gateway node provides this network transparency by watching for and answering ARP requests that cross network boundaries. The node providing this transparency is the proxy server.

The example configuration shown in Figure 4-8 changes slightly when proxy ARP is used. As shown in Figure 4-9, the nodes vx1 and h1 now look as if they are on the same subnet. Nodes h1 and vx1 are fooled by vx3 into thinking they can send directly to each other, when they are actually sending to vx3. The gateway node, vx3, ensures that the packets get to the correct destination.

Figure 4-9:  Proxy ARP Example

The proxy server provides network transparency by listening to and answering ARP messages, and by manipulating its routing tables. Suppose the proxy server had two interfaces: shared-memory network and Ethernet. Nodes residing on different interfaces can have the same network address if network-specific routes with an explicit mask of 0xFFFFFFFF were used on one interface (shared-memory network) and network routing was done on the other (Ethernet).

The proxy clients in the shared-memory network are added with a subnet mask of 0xFFFFFFFF so that arp route entries to the proxy clients can be cloned from the route added by the proxyLib. In the proxy server, the backplane IP address should also have an explicit mask of 0xFFFFFFFF. For example:

Inet on the backcplane (b): 150.12.1.199:FFFFFFFF

In the example in Figure 4-10, vx1 and h1 have the same network address, 150.12.1.0. The proxy server, vx3, has a routing table like the following example:

Destination                                                  Gateway 
150.12.1.6 (network route with netmask 0xffffffff).......) 150.12.1.8 
150.12.1.7 (network route with netmask 0xffffffff........) 150.12.1.8 
150.12.1.0 (network)                                        150.12.1.60

Figure 4-10:  Proxy Server Example

The network on which the proxy server performs network-specific routing (or for which it is acting as a proxy) with the mask 0xFFFFFFFF is referred to as the proxy network. The proxy server has a network-specific route with mask 0xFFFFFFFF to each node on the proxy network. The network interface on which the proxy server performs network routing is called the main network. In the example in Figure 4-10, the shared-memory network is the proxy network and the Ethernet is the main network. The routing table of vx3 has network-specific routes with mask 0xFFFFFFFF for both vx1 and vx2. To send to nodes h1 and h2, it uses the network route (150.12.1.0). There can be multiple proxy networks per main network. However, there can only be one main network per network/subnet number.

Although network-specific routes with netmask 0xFFFFFFFF can be used on all interfaces for complete generality, a VxWorks shared-memory network usually is configured so that one side of the proxy server contains the majority of nodes (the Ethernet side). Therefore, in this case it is reasonable to use this network as the main network.

The following subsections describe how the proxy ARP protocol responds to ARP requests for proxy clients, non-proxy clients, and how it responds to replies from the main network.

ARP Requests for Proxy Clients

If the proxy server receives an ARP request from the main network for a node on a proxy network (proxy client), the proxy server generates an ARP reply with its own hardware address as the source hardware address. If the node that generated the request is also on the proxy network, the destination proxy client answers for itself. In the example in Figure 4-10, if vx1 broadcasts an ARP request for 150.12.1.7, vx2 replies to the request, not the proxy server vx3. However, if h1 broadcasts an ARP request for 150.12.1.7, the proxy server (vx3) replies with its own hardware address.

ARP Requests from Proxy Clients for Non-proxy Clients

If an ARP request comes from a proxy network and the destination address is not a proxy client, the proxy server tries to resolve the request. If the hardware address of the destination is known, the server generates and sends an ARP reply to the source proxy client. If the hardware address is unknown, the server forwards the ARP request to the proxy network's corresponding main network, replacing the source hardware address in the ARP message with its own outgoing interface hardware address. For example, in Figure 4-10, vx1 sends an ARP request for 150.12.1.62. If vx3 knows the hardware address, it sends an ARP reply to vx1. Otherwise it forwards the request to the Ethernet.

ARP Replies from the Main Network

If the proxy server gets an ARP reply, the server checks to see if the destination is a proxy client. If it is, and the server previously forwarded this request, then the server forwards the ARP reply back to the proxy client (replacing the source hardware address in the ARP reply message with its own). In the previous example, if h2 replies to the request for the Ethernet address of 150.12.1.62, the proxy server (vx3) records the address for itself and then forwards the reply to vx1 (with vx3's own hardware address substituted for h2's).

All nodes on a logical network are expected to receive an IP broadcast for that network (for example, 150.12.1.255). Thus, broadcasts must be passed through the proxy server so that nodes on both the proxy network and the main network receive them. Because most broadcast traffic is extraneous, it is desirable to minimize the number of forwarded shared-memory network broadcasts, thus keeping shared-memory network traffic to a minimum.

To minimize and control shared-memory network broadcast traffic, the proxy server can be configured to forward broadcasts only to a specified set of destination UDP ports. Ports are enabled using the routine proxyPortFwdOn( ), and are disabled with proxyPortFwdOff( ). Only the BOOTP/DHCP server port (67) is enabled by default.

If a broadcast datagram originates from a proxy network (and the port is enabled), the server forwards the broadcast to the main network, and to all other proxy networks that have the same main network. For example, in Figure 4-11, if a datagram comes from sm1, it gets forwarded to ln0 and sm0.

Figure 4-11:  Broadcast Datagram Forwarding

If the datagram originates from a main network (and the port is enabled), the server forwards the broadcasts to all the main network's proxy networks. For example, in Figure 4-11, a datagram from ln0 is forwarded to both sm0 and sm1. To prevent forwarding loops, broadcasts forwarded onto proxy networks are given a time-to-live value of 1.

Although forwarding broadcasts between interfaces is potentially dangerous (due to broadcast storms and forwarding loops), the restrictions put on the configuration make these situations unlikely. Even so, forwarding broadcasts between proxy and main interfaces is not recommended. Therefore, forward broadcasts only on necessary ports.

Using multi-homed proxy clients requires that you make changes to the routing and broadcast configuration of your VxWorks system. These changes are described in the following subsections.

Routing Configuration Considerations for Multi-Homed Proxy Clients

If a proxy client also has an interface to the main network, some additional configuration is required for optimal communications. The proxy client's routing tables must have network-specific routes with netmask 0xFFFFFFFF for nodes on the proxy network, and a network-specific route for the main network. Otherwise traffic travels an extra unnecessary hop through the proxy server.

In the example shown in Figure 4-12, vx1 is the proxy server and vx2 is a proxy client with an interface on the main network. vx2 must be configured to have network-specific routes with mask 0xFFFFFFFF to each of the other proxy clients (vx4 and vx5), and a network-specific route to the main network. Otherwise any traffic from vx2 to vx4 (or vx5) unnecessarily travels over the main network through the proxy server (vx1).

Figure 4-12:  Routing Example

The following is an example of vx2's routing table. The routing table is manipulated using routeAdd( ) and routeDelete( ). For more information, see the reference entry for routeLib.

Destination                                                 Gateway 
150.12.0.4 (network with netmask   0xffffffff)            150.12.0.6 
150.12.0.5 (network with netmask   0xffffffff)            150.12.0.6 
150.12.0.0 (network)                                       150.12.0.7 

Broadcasts Configuration Considerations for Multi-Homed Proxy Clients

A proxy client that also has an interface connected to the main network must disable broadcast packets from the proxy interface. Otherwise, it receives duplicate copies of broadcast datagrams (one from Ethernet and one from the shared-memory network). Broadcasts can be disabled on an interface using ifFlagChange( ). (See the reference entry.)

Proxy ARP works only for a single tier of shared-memory networks. That is, only interfaces directly attached to the proxy server are supported. Example configurations that work are shown in Figure 4-13 and Figure 4-15.

Figure 4-13:  Single-Tier Example Using Proxy ARP with Two Branches

However, the configuration shown in Figure 4-14 does not work because ARP requests are not forwarded over proxy networks, and there can be only one proxy server per shared-memory network. This single-tier restriction means that problems such as network circles, broadcast storms, and continually forwarded ARP requests are avoided.

To work, the configuration in Figure 4-14 requires a combination of proxy ARP and IP (or standard subnet) routing. The modified configuration is shown in Figure 4-16, where Proxy Network 1 has become an IP routing network with a different network address. For vx6 to send to h2 in the modified configuration, it requires the following entry in its routing table:

Figure 4-14:  Multi-Tier Configuration that CANNOT Be Used with Proxy ARP

Destination             Gateway 
150.12.0.0 (network)      161.27.0.1 

For h2 to send to vx6, it requires the following entry in its routing table:

Destination             Gateway 
161.27.0.0 (network)      150.12.0.6

If the main network on which the proxy server is connected is subnetted, then all the interfaces (both proxy and main) must reside on the same subnet as the main network. That is, the main network interface and the proxy network interface on the proxy server and all the proxy clients must have the same subnet mask.

Figure 4-15:  Another Single-Tier Example Using Proxy ARP

To enable proxy ARP for the shared-memory network, reconfigure VxWorks and rebuild it with the proxy server. The relevant configuration macro is INCLUDE_PROXY_SERVER. If the target is processor zero (the shared-memory network master), the proxy server is enabled using the boot parameter inet on ethernet (e) as the main network, and the boot parameter inet on backplane (b) as the proxy network. From the example in Figure 4-15, vx1's corresponding boot parameters are as follows:

inet on ethernet (e)    : 150.12.7.3:ffffff00 
inet on backplane (b)   : 150.12.7.4

Proxy ARP Server Configuration

The proxy server for the shared-memory network must be the master board. As previously mentioned, the server must be configured for proxy servers. The relevant configuration macro is INCLUDE_PROXY_SERVER. If sequential addressing is not used, then the master backplane inet address must be specified as well as the slaves' backplane and gateway inet addresses. This configuration gives you greater control over the addresses that are assigned to the target boards.

Sequential and Default Addressing

If such control is not required, it is possible to have the proxy server assign the inet addresses to the proxy clients. When VxWorks is configured for sequential addressing, the proxy server assigns incremental inet addresses to the slave boards based on the proxy server's backplane inet address. The relevant configuration macro is INCLUDE_SM_SEQ_ADDR. For example, if the proxy server has a backplane inet address of 150.12.0.4, the inet address assigned to the first slave is 150.12.0.5, to the second slave 150.12.0.6, and so on. (See Figure 4-16.)

Using sequential addressing frees you from having to specify a backplane or a gateway inet address for each proxy client. All the addresses are assigned by the proxy server at boot time.

It is also possible to have the proxy server's backplane address configured by default. This allows for greater flexibility in the assignment of backplane inet addresses. You are only required to assign the inet address for the proxy server's interface to the main network. The backplane address is assigned automatically by adding 1 (one) to the network interface address.

To assign the proxy server's backplane address by default, you must use a configuration with default addressing as well as sequential addressing. The relevant configuration macro is INCLUDE_PROXY_DEFAULT_ADDR. This frees you from having to specify the backplane inet address of the proxy server and the proxy clients, and the gateway address of the proxy clients.

For example, with VxWorks so configured: if the proxy server is given the inet network address of 150.12.0.3, its backplane address is 150.12.0.4. The first proxy client is assigned the inet address 150.12.0.5, the second 150.12.0.6, and so on.

Figure 4-16:  Multi-Tier Example Using Proxy ARP and IP Routing

Note that with proxy ARP it is no longer necessary to specify the gateway. Each target on the shared-memory network (except the proxy server) can register itself as a proxy client by specifying the proxy ARP flag, 0x100, in the boot flags instead of specifying the gateway. For additional information on booting with proxy ARP, see 13.5 Booting from the Shared-Memory Network.

VxWorks Images for Proxy ARP with Shared Memory and IP Routing

Even if you are using the same board for the master and the slaves, the master and slaves need separate BSP directories because they have different configurations. For more information on configuring VxWorks, see the Tornado User's Guide: Projects.

Proxy ARP and Shared Memory Definition in the VxWorks Configuration:

1. PING client (configuration constant: INCLUDE_PING)

1. Shared memory network initialization (INCLUDE_SM_NET)

1. Proxy ARP server (INCLUDE_PROXY_SERVER)

1. Auto address setup (INCLUDE_SM_SEQ_ADDR) - required only for default addressing

1. Default address for bp (INCLUDE_PROXY_DEFAULT_ADDR) - required only for default addressing

Master Definition in config.h:

#define PROXY_ARP_MASTER 
#define SM_OFF_BOARD=FALSE

Slave definition in config.h:

#define PROXY_ARP_SLAVE 
#define SM_OFF_BOARD=TRUE

Setting Up Boot Parameters and Booting

See 3.5 Shared-Memory Network on the Backplane for information on booting shared memory networks. After booting vx1 (the master), use smNetShow( ) to find the shared memory anchor, which will be used with the slave boot device (for vx2, vx3, and vx4). You will need to run sysLocalToBusAddr( ) on the master and sysBusToLocalAddr( ) on each type of target to get the correct bus address for the anchor. For general information on boot parameters, see the Tornado User's Guide: Getting Started.

Creating Network Connections

From vx1 (the master):

Use routeAdd( ) to tell the master (the proxy server) about the IP routing network by running the following:

-> routeAdd ("161.27.0.0", "150.12.0.6") 
value = 0 = 0x0

From vx3:

Since vx3 boots from the shared memory network, it needs to have its connection to the IP routing network brought up explicitly. The following example shows how to do this for vx3 in Figure 4-16:

-> userNetIfAttach ("ln", "161.27.0.1") 
Attaching network interface ln0...done. 
value = 0 = 0x0 
-> userNetIfConfig ("ln", "161.27.0.1", "t0-1", 0xffffff00) 
value = 0 = 0x

	NOTE: Substitute the appropriate network boot device for "ln". The correct boot device is the first one given by ifShow( ).

Diagnosing Shared Memory Booting Problems

See Troubleshooting for information on debugging the shared memory network.

Diagnosing Routing Problems

The following routines can be useful in locating the source of routing problems:

ping( )

Starting from vx1, ping other processors in turn to see if you get the expected result. The function returns OK if it reaches the other machine, or ERROR if the connection fails.

smNetShow( )

This routine displays cumulative activity statistics for all attached processors.

arpShow( )

This routine displays the current Internet-to-Ethernet address mappings in the system ARP table.

arptabShow( )

This routine displays the known Internet-to-Ethernet address mappings in the ARP table

routeShow( )

This routine displays the current routing information contained in the routing table.

ifShow( )

This routine displays the attached network interfaces for debugging and diagnostic purposes.

proxyNetShow( )

This routine displays the proxy networks and their associated clients.

proxyPortShow( )

This routine displays the ports currently enabled.

1:  Proxy ARP is described in Request For Comments (RFC) 925 "Multi LAN Address Resolution," and an implementation is discussed in RFC 1027 "Using ARP to Implement Transparent Subnet Gateways." The ARP protocol is described in RFC 826 "Ethernet Address Resolution Protocol: Or converting network protocol addresses to 48-bit Ethernet address for transmission on Ethernet hardware." The implementation of Proxy ARP for VxWorks is based on RFC 925. However, it is a limited subset of that proposal.