Friday, June 19, 2009

Driver Configuration File to Force the Sun GigaSwift Ethernet Adapter's Parameters

For ease of administration it is advisable to put the configuration file in the same directory as the driver itself.There is a 64-bit driver located in the /platform/sun4u/kernel/drv/sparcv9 directory and a 32-bit driver in the /platform/sun4u/kernel/drv directory.

Example 1

As a first example let's look at the devices on a Sun Fire 480R. If we look at the path_to_inst file on this type of system we may see something similar to the following.

# grep '"ce"' /etc/path_to_inst
"/pci@9,700000/network@2" 0 "ce"
"/pci@9,600000/network@1" 1 "ce"

Each line in the file contains the following three fields as described in the man page for path_to_inst(4).

"physical-name" instance-number "driver-binding-name"

So, we see two instances of the device, 0 and 1, which will be seen as ce0 and ce1 respectively in the output from the ifconfig -a command if the interfaces are plumbed.

To force ce0 to run at 100 Mbit/s full duplex and ce1 to run at 1 Gbit/s full duplex we would need to place the following two lines in the ce.conf file.

name="ce" parent="/pci@9,700000" unit-address="2" adv_autoneg_cap=0 adv_1000fdx_cap=0
adv_1000hdx_cap=0 adv_100fdx_cap=1 adv_100hdx_cap=0 adv_10fdx_cap=0 adv_10hdx_cap=0;
name="ce" parent="/pci@9,600000" unit-address="1" adv_autoneg_cap=0 adv_1000fdx_cap=1
adv_1000hdx_cap=0 adv_100fdx_cap=0 adv_100hdx_cap=0 adv_10fdx_cap=0 adv_10hdx_cap=0;

The important points to note here are:

  1. The name, parent and unit-address parameters should be enclosed in double quotes. Should you wish to force all ce interfaces to the same speed and duplex you do not need to include these parameters.
    • The "name" parameter is the driver binding name for the device and will always be "ce". You may have seen previous revisions of this document which state that it should be "pci108e,abba". While this works you should use the driver binding name (ce) due to its simplicity and the fact that it does not tie the configuration to a specific type of PCI card as defined in the /etc/driver_aliases file.
    • The "parent" parameter is that part of the device path (or physical-name) prior to the "/network@" part. In a Sun Cluster environment you may see the device path prefixed with "node@<n>" where <n> is replaced by a digit. This prefix should not be included in the parent parameter.
    • The "unit-address" parameter is the number immediately after the "network@" part in the device path, it is not the device instance.
  2. We have defined more than the minimum set of parameters to effect this change since we are being deliberately verbose for the sake of completeness and to reduce the scope for error.
  3. Each entry in the file should be on a separate line and terminated with a semi-colon. While the example above may at first sight appear to be spread over several lines there are in fact only two lines, one for each instance of the device.
  4. These parameters will take effect following a system reboot, only the ndd command may be used to effect dynamic changes.

Example 2

As a second example let's look at the interfaces on a Sun Fire 4800 domain where the /etc/path_to_inst file contains the following entries.

"/ssm@0,0/pci@18,600000/pci@1/network@0" 0 "ce"
"/ssm@0,0/pci@18,600000/pci@1/network@1" 1 "ce"
"/ssm@0,0/pci@19,700000/pci@3/network@0" 2 "ce"
"/ssm@0,0/pci@1c,600000/pci@1/network@0" 3 "ce"
"/ssm@0,0/pci@1c,600000/pci@1/network@1" 4 "ce"
"/ssm@0,0/pci@1d,700000/network@3" 5 "ce"

In order to force ce5 to 1000 Mbit/s full duplex and ce0 and ce3 to 100Mbit/s full duplex the ce.conf file should contain the following three lines.

name="ce" parent="/ssm@0,0/pci@1d,700000" unit-address="3" adv_autoneg_cap=0 adv_1000fdx_cap=1
adv_1000hdx_cap=0 adv_100fdx_cap=0 adv_100hdx_cap=0 adv_10fdx_cap=0 adv_10hdx_cap=0;
name="ce" parent="/ssm@0,0/pci@18,600000/pci@1" unit-address="0" adv_autoneg_cap=0
adv_1000fdx_cap=0 adv_1000hdx_cap=0 adv_100fdx_cap=1 adv_100hdx_cap=0 adv_10fdx_cap=0 adv_10hdx_cap=0;
name="ce" parent="/ssm@0,0/pci@1c,600000/pci@1" unit-address="0" adv_autoneg_cap=0
adv_1000fdx_cap=0 adv_1000hdx_cap=0 adv_100fdx_cap=1 adv_100hdx_cap=0 adv_10fdx_cap=0 adv_10hdx_cap=0;

In order to clarify point 1 above, if we wanted to force all six interfaces to operate at 1Gbit/s full duplex we would only need to place the following single line in the file.

adv_autoneg_cap=0 adv_1000fdx_cap=1 adv_1000hdx_cap=0 adv_100fdx_cap=0 adv_100hdx_cap=0 adv_10fdx_cap=0

To check the interface settings:

netstat -k ce0

To get the NIC's capabilities:

kstat -p ce:0::"/^cap_/"

To get advertised capabilities:

ndd -set /dev/ce instance
ndd -get /dev/ce adv_autoneg_cap
ndd -get /dev/ce \? | grep "adv_"

Parameter Name Value Meaning
adv_autoneg_cap 0 The interface has been forced directly into a particular operating mode without regard to auto-negotiation

1 The interface is currently auto-negotiating
adv_1000fdx_cap 0 The interface is not capable of operating at 1000 Mbit/s full duplex

1 The interface is capable of operating at 1000 Mbit/s full duplex
adv_1000hdx_cap 0 The interface is not capable of operating at 1000 Mbit/s half duplex

1 The interface is capable of operating at 1000 Mbit/s half duplex
adv_100fdx_cap 0 The interface is not capable of operating at 100 Mbit/s full duplex

1 The interface is capable of operating at 100 Mbit/s full duplex
adv_100hdx_cap 0 The interface is not capable of operating at 100 Mbit/s half duplex

1 The interface is capable of operating at 100 Mbit/s half duplex
adv_10fdx_cap 0 The interface is not capable of operating at 10 Mbit/s full duplex

1 The interface is capable of operating at 10 Mbit/s full duplex
adv_10hdx_cap 0 The interface is not capable of operating at 10 Mbit/s half duplex

1 The interface is capable of operating at 10 Mbit/s half duplex


To get the actual link mode:

kstat -p ce:0::"/^link_/"

Parameter Name Value Meaning
link_up 0 The link is down

1 The link is up
link_speed 1000 The link is operating at 1000 Mbit/s

100 The link is operating at 100 Mbit/s

10 The link is operating at 10 Mbit/s
link_duplex 0 The link is down

1 The link is operating in half duplex mode

2 The link is operating in full duplex mode


To get the link partner capabilities:

kstat -p ce:0::"/^lp_cap_/"

Parameter Name Vlaue Meaning< /b>
lp_cap_autoneg 0 The link partner operational parameters have been forced or the link partner simply does not support auto-negotiation

1 Auto-negotiation information has been received from the link partner
lp_cap_1000fdx 0 The link partner appears to be incapable of operating at 1000 Mbit/s full duplex

1 The link partner is capable of operating at 1000 Mbit/s full duplex
lp_cap_1000hdx 0 The link partner appears to be incapable of operating at 1000 Mbit/s half duplex

1 The link partner is capable of operating at 1000 Mbit/s half duplex
lp_cap_100fdx 0 The link partner appears to be incapable of operating at 100 Mbit/s full duplex

1 The link partner is capable of operating at 100 Mbit/s full duplex
lp_cap_100hdx 0 The link partner appears to be incapable of operating at 100 Mbit/s half duplex

1 The link partner is capable of operating at 100 Mbit/s half duplex
lp_cap_10fdx 0 The link partner appears to be incapable of operating at 10 Mbit/s full duplex

1 The link partner is capable of operating at 10 Mbit/s full duplex
lp_cap_10hdx 0 The link partner appears to be incapable of operating at 10 Mbit/s half duplex

1 The link partner is capable of operating at 10 Mbit/s half duplex


Methods to determine link status of ethernet interfaces

- check /var/adm/messages (including when IPMP is used)

---------------------
checking link status with ndd

# ndd -set /dev/qfe instance 0 (instance set to "0" checks qfe0 status; "1" for qfe1,etc...)
# ndd /dev/qfe link_status
1
( 1 -up, 0 = down )

--------------------

Checking link status for Gigaswift Ethernet(ce) is different from other Ethernet interfaces.   netstat
-k is used:

netstat -k ce0

link_up (1=up,0=down)
-------------

checking with kstat (netstat -k)

# kstat qfe:0::link_up
module: qfe instance: 0
name: qfe0 class: net
link_up 1

# kstat eri |grep link_up
link_up 1

# netstat -k qfe0 |grep link_up
...... link_up 1

-------------

examining LED states

newer network interface adapters have light indicators.

on "qfe" quad fast ethernet and "ge" Sun Gigabit Ethernet 2.0 adapters

     LED           Status                 Description
--------------------------------------------------------------
LINK Off Port not connected or Link partner down
Green Steady Link detected & active
on "ce" Gigaswift:
--------------------------------------------------------------
LINK QUALITY Purple Gigabit link quality good
PHY 1000X Green Link 1Gbit/s up
PHY 100X Green Link 100Mbit/s up
PHY 10X Green Link 10Mbit/s up
-------------

checking with SNMP ( requires snmp Software)
./snmpwalk netlab11 public interfaces.ifTable

...
interfaces.ifTable.ifEntry.ifDescr.1 = lo0
interfaces.ifTable.ifEntry.ifDescr.2 = ce3
interfaces.ifTable.ifEntry.ifDescr.3 = hme0
interfaces.ifTable.ifEntry.ifDescr.4 = qfe0
...
interfaces.ifTable.ifEntry.ifOperStatus.1 = up(1)
interfaces.ifTable.ifEntry.ifOperStatus.2 = up(1)
interfaces.ifTable.ifEntry.ifOperStatus.3 = up(1)
interfaces.ifTable.ifEntry.ifOperStatus.4 = down(2)



Ethernet auto-negotiation

Use the following commands to check that the link partner (switch / link / router) has auto-negotiation capability enabled.

for ge   # ndd /dev/ge lp_1000autoneg_cap
for hme # ndd /dev/hme lp_autoneg_cap
for qfe # ndd /dev/qfe lp_autoneg_cap
for eri # ndd /dev/eri lp_autoneg_cap
for ce # kstat ce:::lp_cap_autoneg
for dmfe # ndd /dev/dmfe0 lp_autoneg_cap

If the lp_autoneg_cap is 0 (ie, auto-negotiate is disabled on the link partner), action should be taken to either enable autoneg_cap on the link partner, or disable autoneg_cap on the host.

Note: the kstat command is only available in Solaris 8 and later. For previous releases the information is included in the "netstat -k" command (see the ce relevant section there).