SSL Certificates and NetApp ActiveIQ Unified Manager
I had the opportunity to work a little bit with the new Nvidia SN2100 that NetApp have certified for Metrocluster use as an alternative to the Cisco and Broadcom switches.
The SN2100 comes with Cumulus linux installed and configuring them is more like a normal linux machine than a switch OS like the NXOS so that took some getting used you. Tab completion only works within a command, so one needs to know “net show [tab]” to have a starting point.
I have compiled here some of the commands that I used to check that the switch was correctly configured and ready to use.
There are a few things that I wanted to check before using the switches in production.
Version, hostname, management ip settings, port configs (vlan, mode, mtu, etc), port status (link, speed, etc), rcf file version.
Showing the version can mean a few things in cumulus but normally when we on a Cisco do a “show version” we just want the NXOS version so we can compare it to a support matrix, and that version number can be found either by “cat /etc/lsb-release” or with the hostnamectl command. The hostnamectl is not the obvious place to look, but if we want to check the hostname anyway then it can be a useful place.
ronnie@SWITCH-A1:mgmt:~$ cat /etc/lsb-release DISTRIB_ID="Cumulus Linux" DISTRIB_RELEASE=4.4.3 DISTRIB_DESCRIPTION="Cumulus Linux 4.4.3"
4.4.3 is the number we need to look up in the support matrix.
ronnie@SWITCH-A1:mgmt:~$ hostnamectl Static hostname: SWITCH-A1 Icon name: computer-desktop Chassis: desktop Machine ID: 4949e6ec559d507493bbc1eed319e570 Boot ID: d595aa55efc825e5594eed52f61e1cfad Operating System: Cumulus Linux CPE OS Name: cpe:/o:cumulusnetworks:cumulus_linux:4.4.3 Kernel: Linux 4.19.0-cl-1-amd64 Architecture: x86-64
Here the version is listed as part of the CPE OS Name.
Normally it is pretty easy to find the hostname on a linux machine. It is right in the prompt. But sometimes the prompt is broken, or the system has not been rebooted since it was changed, or you want to put it into a script as a failsafe.
Seeing basic hardware status of psu, fans and temperature can be seen with “net show system sensors”
ronnie@SWITCH-A1:mgmt:~$ net show system sensors Fan1 (Fan 1 ): OK Fan2 (Fan 2 ): OK Fan3 (Fan 3 ): OK Fan4 (Fan 4 ): OK PSU1 : OK PSU2 : OK Temp1 (Asic Temp Sensor ): OK Temp2 (Port Ambient Sensor ): OK Temp3 (Main Board Ambient Sensor ): OK Temp4 (core0 Sensor ): OK Temp5 (core1 Sensor ): OK Temp6 (core2 Sensor ): OK Temp7 (core3 Sensor ): OK
And the status of the LEDS on the front of the switch “net show system leds”
ronnie@SWITCH-A1:mgmt:~$ net show system leds System: green Psu1: green Psu2: green Fan: green
The dedicated management interface on the SN2100 is eth0 and we can show a lot of information with “net show” like on modern linux.
ronnie@SWITCH-A1:mgmt:~$ net show interface eth0 Name MAC Speed MTU Mode -- ---- ----------------- ----- ---- ---- UP eth0 11:71:fd:c1:d5:11 100M 1500 Mgmt IP Details ------------------------- ---------------- IP: 192.168.41.143/24 IP Neighbor(ARP) Entries: 1 cl-netstat counters ------------------- RX_OK RX_ERR RX_DRP RX_OVR TX_OK TX_ERR TX_DRP TX_OVR ------- ------ ------ ------ ------ ------ ------ ------ 9597447 0 0 0 669746 0 0 0 Routing ------- Interface eth0 is up, line protocol is up Link ups: 1 last: 2019/05/01 20:10:07.45 Link downs: 0 last: (never) PTM status: disabled vrf: mgmt index 2 metric 0 mtu 1500 speed 100 flags: <UP,BROADCAST,RUNNING,MULTICAST> Type: Ethernet HWaddr: 10:70:fd:c0:d4:10 inet 192.168.41.143/24 inet6 fe80::1270:fdff:fec0:d410/64 Interface Type Other protodown: off
It will also list a few things we normally don’t care about at first (like netstat counters) but gives a nice easy overview. If we just want the ip an net show interface will also list it, but if multiple address and interfaces has been configured it can be difficult to be sure what interface is for what.
The gateway can be found in /etc/network/interfaces under the interface.
ronnie@SWITCH-A1:mgmt:~$ cat /etc/network/interfaces | grep -A 3 eth0 auto eth0 iface eth0 address 192.168.41.143/24 gateway 192.168.41.1 vrf mgmt.
NetApp has elected to not configure routing so some of the normal ways to also show gateway like net show route does not work in the default NetApp RCF configuration.
Now we have the basic information needed to management switch. So moving on to the port config. Most of the config we want can be seen with net show interface.
ronnie@SWITCH-A1:mgmt:~$ net show interface State Name Spd MTU Mode LLDP Summary ----- ------------- ---- ----- ---------- --------------------- ------------------------- UP lo N/A 65536 Loopback IP: 127.0.0.1/8 lo IP: ::1/128 UP eth0 100M 1500 Mgmt Master: mgmt(UP) eth0 IP: 192.168.41.143/24 UP swp1 100G 9216 Access/L2 clusterA-01 (e3a) Master: bridge-CL-101(UP) UP swp2 100G 9216 Access/L2 clusterA-02 (e3a) Master: bridge-CL-101(UP) UP swp7 100G 9216 Trunk/L2 clusterA-01 (e1a) UP swp7.10 100G 9216 Access/L2 Master: bridge-DR-10(UP) UP swp8 100G 9216 Trunk/L2 clusterA-02 (e1a) UP swp8.10 100G 9216 Access/L2 Master: bridge-DR-10(UP) UP swp13 100G 9216 BondMember SWITCH-A2 (swp13) Master: bond-DR(UP) DN swp14 N/A 9216 BondMember Master: bond-DR(UP) UP swp15 100G 9216 BondMember SWITCH-B1 (swp15) Master: bond-CL(UP) UP swp16 100G 9216 BondMember SWITCH-B1 (swp16) Master: bond-CL(UP) UP bond-CL 200G 9216 802.3ad Bond Members: swp15(UP) bond-CL Bond Members: swp16(UP) UP bond-CL.101 200G 9216 Access/L2 Master: bridge-CL-101(UP) UP bond-DR 100G 9216 802.3ad Bond Members: swp13(UP) bond-DR Bond Members: swp14(DN) UP bond-DR.10 100G 9216 Access/L2 Master: bridge-DR-10(UP) UP bridge-CL-101 N/A 9216 Bridge/L2 UP bridge-DR-10 N/A 9216 Bridge/L2 UP mgmt N/A 65536 VRF IP: 127.0.0.1/8 mgmt IP: ::1/128
When we have an port listed as .xx like swp8.10 that means port 8 is an trunk port with vlan 10 tagged. We can also see the MTU setting and if the port is up or down. If the port is part of a bond or bridge remember to also check that. The port might be down but the bond can be up, and then at least something is working.
We have ports swp1 and 2. Configured as access ports and ports swp7 and 8 as trunks with vlan 10 tagged.
This is as we would expect if the switch is configured for MetroCluster IP. Ports 1 and 2 are for cluster traffic that uses access ports and ports 7 and 8 are for the MetroCluster iSCSI connections. Remember to check the MetroCluster configuration is using the same vlan settings as the switch. NetApp change the iSCSI vlans in the RCF generator some time ago, but the default created vlan in ONTAP when doing metrocluster configuration has not changed yet.
Port 13, 14, 15 and 16 are ISL / uplink connections to the other switches. 13 and 14 are for local uplink to switch at same site. 15 and 16 are for the other site. They are configured with LACP(=802.3ad) for the bond.
All this config can also be found in the file /etc/network/interfaces some might find it easier to read.
Checking the NetApp RCF file version can be done in kind of around a bout way, not sure how else to check it. But the RCF will configure a banner to be displayed when logging with the SSH. This is configured in sshd_config. This banner will display the RCF version.
ronnie@SWITCH-A1:mgmt:~$ cat /etc/ssh/sshd_config | grep banner # no default banner path Banner /etc/banner.txt Banner /etc/banner.txt ronnie@SWITCH-A1:mgmt:~$ cat /etc/banner.txt ****************************************************************************** * NetApp Reference Configuration File (RCF) * Switch : SN2100 (direct storage, L2 Networks, direct ISL) * Filename : SN2100_v1.0_Switch-A1.txt * Version : v1.0 * Date : Generator: v1.5a_2022-12-09_001, file creation: 2023-03-09, 15:42:18 * ******************************************************************************
If this is not as expected, then the RCF has not been applied correctly and I would recommend doing it again.
When troubleshooting a specific link or we can get a lot of information with the first command we used. Net show interface xx and if we add detail to the command we get more then we need.
ronnie@SWITCH-A1:mgmt:~$ net show interface eth0 detail Name MAC Speed MTU Mode -- ---- ----------------- ----- ---- ---- UP eth0 11:71:fd:c1:d5:11 100M 1500 Mgmt IP Details ------------------------- ---------------- IP: 192.168.41.143/24 IP Neighbor(ARP) Entries: 1 cl-netstat counters ------------------- RX_OK RX_ERR RX_DRP RX_OVR TX_OK TX_ERR TX_DRP TX_OVR ------- ------ ------ ------ ------ ------ ------ ------ 9610386 0 0 0 672533 0 0 0 Routing ------- Interface eth0 is up, line protocol is up Link ups: 1 last: 2019/05/01 20:10:07.45 Link downs: 0 last: (never) PTM status: disabled vrf: mgmt index 2 metric 0 mtu 1500 speed 100 flags: <UP,BROADCAST,RUNNING,MULTICAST> Type: Ethernet HWaddr: 11:71:fd:c1:d5:11 inet 192.168.41.143/24 Interface Type Other protodown: off Ethtool ------- Settings for eth0: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 0 Transceiver: internal Auto-negotiation: on MDI-X: off (auto) Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes Ethtool Statistics ------------------ NIC statistics: rx_packets: 9610387 tx_packets: 672533 rx_bytes: 997768246 tx_bytes: 198297704 rx_broadcast: 7941578 tx_broadcast: 0 rx_multicast: 1648795 tx_multicast: 660487 multicast: 1648795 collisions: 0 rx_crc_errors: 0 rx_no_buffer_count: 0 rx_missed_errors: 0 tx_aborted_errors: 0 tx_carrier_errors: 0 tx_window_errors: 0 tx_abort_late_coll: 0 tx_deferred_ok: 0 tx_single_coll_ok: 0 tx_multi_coll_ok: 0 tx_timeout_count: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 rx_align_errors: 0 tx_tcp_seg_good: 9 tx_tcp_seg_failed: 0 rx_flow_control_xon: 0 rx_flow_control_xoff: 0 tx_flow_control_xon: 0 tx_flow_control_xoff: 0 rx_long_byte_count: 997768246 tx_dma_out_of_sync: 0 tx_smbus: 0 rx_smbus: 0 dropped_smbus: 0 os2bmc_rx_by_bmc: 0 os2bmc_tx_by_bmc: 0 os2bmc_tx_by_host: 0 os2bmc_rx_by_host: 0 tx_hwtstamp_timeouts: 0 tx_hwtstamp_skipped: 0 rx_hwtstamp_cleared: 0 rx_errors: 0 tx_errors: 0 tx_dropped: 0 rx_length_errors: 0 rx_over_errors: 0 rx_frame_errors: 0 rx_fifo_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_queue_0_packets: 3609 tx_queue_0_bytes: 402131 tx_queue_0_restart: 0 tx_queue_1_packets: 2360 tx_queue_1_bytes: 237092 tx_queue_1_restart: 0 tx_queue_2_packets: 4582 tx_queue_2_bytes: 703593 tx_queue_2_restart: 0 tx_queue_3_packets: 661982 tx_queue_3_bytes: 194226174 tx_queue_3_restart: 0 rx_queue_0_packets: 7929212 rx_queue_0_bytes: 718752118 rx_queue_0_drops: 0 rx_queue_0_csum_err: 0 rx_queue_0_alloc_failed: 0 rx_queue_1_packets: 1594679 rx_queue_1_bytes: 214072200 rx_queue_1_drops: 0 rx_queue_1_csum_err: 0 rx_queue_1_alloc_failed: 0 rx_queue_2_packets: 2627 rx_queue_2_bytes: 229561 rx_queue_2_drops: 0 rx_queue_2_csum_err: 0 rx_queue_2_alloc_failed: 0 rx_queue_3_packets: 83869 rx_queue_3_bytes: 26272819 rx_queue_3_drops: 0 rx_queue_3_csum_err: 0 rx_queue_3_alloc_failed: 0
In addition to the LLDP info that is shown in “net show interface” we can view more information with this command “sudo lldpcli show neighbors ports [port #]”
ronnie@SWITCH-A1:mgmt:~$ sudo lldpcli show neighbors ports swp1 ------------------------------------------------------------------------------- LLDP neighbors: ------------------------------------------------------------------------------- Interface: swp1, via: LLDP, RID: 5, Time: 0 day, 12:01:50 Chassis: ChassisID: mac d1:30:ea:a1:c0:5a SysName: clusterA-01 SysDescr: AFF-A400, NetApp Release 9.11.1P4: Thu Oct 27 07:37:48 EDT 2022 Capability: Station, on Port: PortID: ifname e3a TTL: 121 ------------------------------------------------------------------------------- ronnie@SWITCH-A1:mgmt:~$ sudo lldpcli show neighbors ports swp15 ------------------------------------------------------------------------------- LLDP neighbors: ------------------------------------------------------------------------------- Interface: swp15, via: LLDP, RID: 8, Time: 0 day, 12:00:41 Chassis: ChassisID: mac 11:71:fd:c1:d5:a9 SysName: SWITCH-B1 SysDescr: Cumulus Linux version 4.4.3 running on Mellanox Technologies Ltd. MSN2100 MgmtIP: 192.168.41.144 MgmtIface: 2 Capability: Bridge, on Capability: Router, on Port: PortID: ifname swp15 PortDescr: Intra-Cluster ISL Port TTL: 120 -------------------------------------------------------------------------------
Depending on the equipment in the other end, we get different information. I really like that we can see the management ip of the switch here.
https://mysupport.netapp.com/site/tools/tool-eula/rcffilegenerator
https://docs.netapp.com/us-en/ontap-metrocluster/install-ip/task_switch_config_nvidia.html
English (UK) 
Dansk