I recently came across a situation where I needed to check if a frame is received and forwarded correctly on a Nexus 9K. There are multiple ways we can achieve this but I found ELAM (Embedded Logic Analyzer Module) capture is very powerful and granular. ELAM can look inside Cisco ASIC and understand how a packet is forwarded. It can capture a packet in real-time without disruptions to performance or control-plane resources.
There are four main steps involved:
- Identify the Ingress forwarding engine
- Configure ELAM trigger
- Start the ELAM capture
- View the capture report
Example
Let's say you are pinging from 10.1.1.1 to 10.100.100.100 but the ping fails and you wanted to see if the packet is arriving on Eth1/10 interface on the Nexus and leaving on Eth1/11. If everything configured correctly the packet arrives on Eth1/10 will have:
src.mac =bb.bb
src.ip=10.1.1.1
dst.mac=cc.cc
dst.ip=10.100.100.100
ELAM capture procedure
Step 1 - Identify port's ASIC, Slice and Srcld
93180YC-EX# show hardware internal tah int eth1/10
#########################################
IfIndex: 436234560
DstIndex: 6079
IfType: 26
Asic: 0
Asic: 0
AsicPort: 20
SrcId: 40
Slice: 0
PortOnSlice: 20
Step 2 - Attach to the module
93180YC-EX# attach module 1
Step 3 - Enter ELAM configuration mode and assign the ASIC
93180YC-EX# debug platform internal tah elam asic 0
Step 4 - Configure the trigger
module-1(TAH-elam)# trigger init asic 0 slice 0 use-src-id 40
param values: start asic 0, start slice 0, lu-a2d 1, in-select 6, out-select 0, src_id 40
Step 5 - Set the ELAM triggers using SRC & DEST IP and start the ELAM
module-1(TAH-elam-insel6)# reset
module-1(TAH-elam-insel6)# set outer ipv4 dst_ip 10.100.100.100 src_ip 10.1.1.1
module-1(TAH-elam-insel6)# start
Step 6 - Verify if your ELAM triggered and review the results
module-1(TAH-elam-insel6)# report
SUGARBOWL ELAM REPORT SUMMARY
slot - 1, asic - 0, slice - 0
============================
Incoming Interface: Eth1/10
Src Idx : 0x23, Src BD : 4158
Outgoing Interface Info: dmod 1, dpid 24
Dst Idx : 0x601, Dst BD : 116
Packet Type: IPv4
Dst MAC address: CC:CC
Src MAC address: BB:BB
.1q Tag0 VLAN: 10, cos = 0x7
Dst IPv4 address: 10.100.100.100
Src IPv4 address: 10.1.1.1
Ver = 4, DSCP = 0, Don't Fragment = 1
Proto = 1, TTL = 255, More Fragments = 0
Hdr len = 20, Pkt len = 8028, Checksum = 0x495d
L4 Protocol : 1
ICMP type : 8
ICMP code : 0
Drop Info:
----------
LUA:
LUB:
LUC:
LUD:
Final Drops:
Outgoing Interface Info shows the egress interface. Here in our case, the correct egress interface is Eth1/11. This can be checked with,
93180YC-EX# show system internal ethpm info interface ethernet 1/11 | grep index
Ethernet1/11 - if_index: 0x1A006A00 IF_STATIC_INFO:
port_name=Ethernet1/11,if_index:0x1a006a00,ltl=6075,slot=0,
nxos_port=212,dmod=1,dpid=24,unit=0,queue=65535,
xbar_unitbmp=0x0,ns_pid=255,slice_num= 0,
port_on_slice=24,src_id=48
Looking at the ELAM capture we can conclude that the ICMP packet ingress on Eth1/10 and egress on Eth1/11.
Thanks for the read
Suresh