CCIE Service Provider #1 {Yocto&Qnx.Embedded}
I suppose you already know but Cisco Systems, inc. is a Software Development company. They believe in their software to provide services. such as SD-WAN with Viptela/Meraki, SSE/SASE with Cisco Umbrella, Intent Network with DNA Center, observability with ThousandEyes, 2FA with DUO, and many many more.
One of the most know software that Cisco provides is their Network Operation System (NOS): Internetwork Operation System AKA IOS. In summary, IOS evolution comes from 80’s Classic IOS, 2000s IOS-XE, 2003–4 IOS-XR 32bits Kernel Unix, present-day virtual IOS-XR 64bits kernel Linux.
Network Operation System
- Classic IOS used to be a monolith kernel with one big memory space. If anything got wrong or crashed the whole system reloads as it was not modular.
- Classic IOS XR (cXR), is a modular 32bits OS running on top of QNX/Neutrino Microkernel. That doesn't provide support to virtualization and ISSU.
- eXR and XR7 both run on top of Yocto-based 64 bits Open Embedde-based Linux Kernel. The difference is that eXR brings containers and XR brings Virtual Machines. IOS XR 64 bits is built using Wind River 7 distribution. Feature includes: Hierarchical Configuration, Two stage configuration model, Package based installation (RPM), Multiprocess, MicroKernel, separate planes for MGMT, Control and Data, High availability NFS & SSO, ISSU support and Modular chassis.
_ The kernel manages communication among softwares and hardware devices.
_ OS has as a function: Process scheduling, Interrupt handling, Memory management, Interprocess Communications.
Different than 32bits that run control and Admin plane on 64bits not run on the same environment.
Also, 64 bits XR has one instance of each Line Card
All process runs outside of the microkernel, which allows the ability to restart any process when required. like SysMgr, SysDB, or BGP are restartable.
Sysmgr — It's responsible for starting, monitoring, stopping, and restarting almost every process on the box;
SysDB — This is a distributed memory. When something is configured, the new config will be stored in memory in SysDB where the process owning that sysDB directory (Interfaces, BGP, SNMP etc) will be notified that something has been updated. It looks like afilesystem with directories and files.
To allow distribution system architecture, IOS-XR has internal IP network for communications among RSP, and Line cards:
_ Admin Plane Guest OS 10.0.2.z/24
_ Control Plane 192.0.x.y/8 (x Slot RP LC, Y (1 admin, 4,6 XR on RSP) ( 1 admin, 3,4 on LC)
RP/0/RP0/CPU0:basquiat-9kV#show platform vm
Sun Oct 1 17:21:33.065 UTC
Node name Node type Partner name SW status IP address
--------------- --------------- --------------- --------------- ---------------
0/RP0/CPU0 RP (ACTIVE) NONE FINAL Band 192.0.0.4
0/0/CPU0 LC (ACTIVE) NONE FINAL Band 192.0.0.6
sysadmin-vm:0_RP0# show vm
Sun Oct 1 17:29:03.427 UTC+00:00
Location: 0/RP0
Id Status IP Address HB Sent/Recv
-------------------------------------------------------------
sysadmin running 192.0.0.1 NA/NA
default-sdr running 192.0.0.4 307085/307085
default-sdr running 192.0.0.6 15334/15334Platforms that support Cisco IOS XR include ASR9k, NCS540, 560, 5700, and 8000 Series Routers. I will focus on ASR9k.
System Architecture
Related to ASR9K, there are a couple of chassis for both first and second generation that could be fixed or modular and would support up to 20 Line cards plus 2 RSP/RP. One good resource to know what ASR9K architecture is the Cisco live BRKARC-2000.
Other components include Router processor or Router switch processor, line cards and Switch Fabric.
RP/RSP — High performance control plane processor. Where Control plane runs: Routing protocolos (OSPF, RSVP, BGP…), Management (SSH, XML, SNMP). RP standard for router processor and S means Switch Fabric because that chassi there are no dedicated Fabric cards.
RSP/RP/NPU/ Generations
- 1st Gen: Trident line cards are 32 bit Freescale PPC LC CPUs.
- 2nd Gen: Typhoon line cards are 32 bit Freescale PPC CPUs (P4040 4 cores @ 1.5Ghz)
- 3rd Gen: Tomhawk line cards use 6-core Intel x86 64 bit CPUs (Ivy Bridge EN 6 cores @ 2Ghz)
- 4th Gen: Lightspeed line cards use a 8-core x86 64 bit CPU
- 5th Gen: LightSpeed+ Line cards use 8-core x86 64bit@ 2.2Ghz
- RSP880 uses an x86 64 bit Intel CPU (8 cores @ 1.9Ghz Ivy Bridge EP “Tomahawk” (3rd) generation fabric and RSP.
- The ASR9001 RSP uses a 32 bit quad-core Freescale PPC CPU (P4040 @ 1.5Ghz) Use Typhoon
- A9K-RSP5-SE/TR uses an x86 64 bit Intel CPU (8 cores @ 1.9Ghz Skylake EP) “Lightspeed” (4th) generation fabric and RSP.
Distributed system — router in today' Service provider environments
Packet forwarding
When packet reach ingress NPU, things like forwarding lookup, ACL, QoS, PBR, uRRF etc is done. Egress Line card actions include L2 re-write.
2-stage forward:
Transit traffic
interface PHY on LC → NPU mapped to incoming interface on LC → → FIA on LC (queued via VOQ) → Crossbar switch on RSP → (ingress via virtual egress queue) FIA on LC →NPU mapped to outgoing interface — -> Outgoing Interface PHY
Put traffic
incoming Interface PHY on LC → NPU → LC CPU → NPU → LC FIA → RSP Crossbar → Punt FPGA on RSP → RSP CPU
Local interface ping
RSP CPU → RSP FIA → RSP Crossbar → LC FIA → LC CPU → NP0 →LC FIA→Crossbar →RSP FIA→ RSP CPU
Maximum Transmission Unit (MTU)
MTU for the higher layer must fit MTU for the lower layer. Different than Classical IOS, IOS XR MTU command includes L2 header. Where mtu +14 is l2 header for non-dot1q interface and MTU + 18 for dot1q subinterface.
RP/0/RP0/CPU0:basquiat-9kV#show interface GigabitEthernet0/0/0/1 | in MTU
Sun Oct 1 18:55:43.667 UTC
MTU 1514 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
RP/0/RP0/CPU0:basquiat-9kV#show interface GigabitEthernet0/0/0/0.10 | in MTU
Sun Oct 1 18:55:58.105 UTC
MTU 1518 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)
RP/0/RP0/CPU0:basquiat-9kV#
RP/0/RP0/CPU0:basquiat-9kV#show im database interface GigabitEthernet0/0/0/0.10
Sun Oct 1 19:10:07.702 UTC
View: OWN - Owner, L3P - Local 3rd Party, G3P - Global 3rd Party, LDP - Local Data Plane
GDP - Global Data Plane, RED - Redundancy, UL - UL
Node 0/0/CPU0 (0x10c0)
Interface GigabitEthernet0/0/0/0.10, ifh 0x01000028 (up, 1518)
Interface flags: 0x0000000001000597 (ROOT_IS_HW|IFINDEX
|SUP_NAMED_SUB|BROADCAST|CONFIG|VIS|DATA|CONTROL)
Encapsulation: dot1q
Interface type: IFT_VLAN_SUBIF
Control parent: GigabitEthernet0/0/0/0
Data parent: GigabitEthernet0/0/0/0
Views: GDP|LDP|L3P|OWN
Protocol Caps (state, mtu)
-------- -----------------
None vlan_jump (up, 1518)
None dot1q (up, 1518)
RP/0/RP0/CPU0:basquiat-9kV#show im database interface GigabitEthernet0/0/0/1
Sun Oct 1 19:11:04.417 UTC
View: OWN - Owner, L3P - Local 3rd Party, G3P - Global 3rd Party, LDP - Local Data Plane
GDP - Global Data Plane, RED - Redundancy, UL - UL
Node 0/0/CPU0 (0x10c0)
Interface GigabitEthernet0/0/0/1, ifh 0x01000020 (up, 1514)
Interface flags: 0x000000000110059f (ROOT_IS_HW|IFCONNECTOR
|IFINDEX|SUP_NAMED_SUB|BROADCAST|CONFIG|HW|VIS
|DATA|CONTROL)
Encapsulation: ether
Interface type: IFT_GETHERNET
Control parent: None
Data parent: None
Views: GDP|LDP|L3P|OWN
Protocol Caps (state, mtu)
-------- -----------------
None ether (up, 1514)
arp arp (up, 1500)
ipv4 ipv4 (up, 1500)
ether_sock ether_sock (up, 1500)For Classic IOS mtu command configures L2 payload size. To changes MTU for Layer, have to use IPv4/IPv6 mtu.
interface GigabitEthernet0/0
mtu 9000
ip address 192.0.2.2 255.255.255.0
ip mtu 1500
NOC#show interface g0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is iGbE, address is 5254.000f.b11e (bia 5254.000f.b11e)
Internet address is 192.0.2.2/24
MTU 9000 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
NOC#show ip interface g0/0
GigabitEthernet0/0 is up, line protocol is up
Internet address is 192.0.2.2/24
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1500 bytes
Helper address is not setXR device that has a routed interface to cIOS, must have MTU 1514 or higher to be able to allow IP communications:
# Classic IOS
NOC#ping 192.0.2.1 size 1500 df
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.0.2.1, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms
NOC#ping 192.0.2.1 size 1501 df
Type escape sequence to abort.
Sending 5, 1501-byte ICMP Echos to 192.0.2.1, timeout is 2 seconds:
Packet sent with the DF bit set
.....
# XR7
RP/0/RP0/CPU0:basquiat-9kV#show interfaces MgmtEth0/RP0/CPU0/0
Sun Oct 1 19:06:20.790 UTC
MgmtEth0/RP0/CPU0/0 is up, line protocol is up
Interface state transitions: 1
Hardware is Management Ethernet, address is 5254.000e.f433 (bia 5254.000e.f433)
Internet address is 192.0.2.1/24
MTU 1514 bytes, BW 0 KbitManagement Plane Protection MPP
LPTS is a kind of built-in firewall for traffic that is punted ( already there are different policers for different type of traffic. Actually, in most cases is not necessary to change LPTS value we're using in for all traffic punted. Example.: For management traffic like SSH, SNMP or route protocols. LTPS will allow more traffic for established BGP peers than non-established sessions.
# MPP configuration
RP/0/RP0/CPU0:basquiat-asr9k#show running-config control-plane
Fri Sep 29 23:43:39.576 UTC
control-plane
management-plane
inband
interface all
allow SSH peer
address ipv4 192.0.2.0/24
!
!
!
out-of-band
interface MgmtEth0/RP0/CPU0/0
!
!
!
!
# NOC team access
# No Access from Data networks
NOC#ssh -l admin 172.29.0.1
% Connection refused by remote host
# enable to access from mgmt networks
NOC#
NOC#ssh -l admin 192.0.2.1
Password:
% Password: timeout expired!Two-Stage commit model
On XR, there is no startup-config file. Instead, We've the running-config, the first stage is created when changes are entered (target configuration). And then, once committed, Target configurations are merged to running-config.
RP/0/RP0/CPU0:basquiat-asr9k(config)#commit ?
best-effort Commit the configuration changes via best-effort operation
comment Assign a comment to this commit
confirmed Rollback this commit unless there is a confirming commit
force Override the memory checks
label Assign a label to this commit
replace Replace the contents of running configuration
save-running Save running configuration to a file
show-error Displays commit failures immediately
<cr> Commit the configuration changes via pseudo-atomic operation
RP/0/RP0/CPU0:basquiat-asr9k(config)#show configuration ?
changes Show configuration changes to be made during a replace operation
commit Show commit information
failed Contents of failed configuration
formal Formal (large) configuration format. No submode indentation.
history Display history of configuration events (up to 1500 events)
merge Contents of merged configuration
rollback Show rollback information
running Contents of running configuration
| Output Modifiers
<cr> Contents of uncommitted configuration
RP/0/RP0/CPU0:basquiat-asr9k(config)#interface g0/0/0/0
RP/0/RP0/CPU0:basquiat-asr9k(config-if)#description SHUTDOWN
RP/0/RP0/CPU0:basquiat-asr9k(config-if)#show configuration
Fri Sep 29 23:58:10.744 UTC
Building configuration...
!! IOS XR Configuration 7.7.1
interface GigabitEthernet0/0/0/0
description SHUTDOWN
!
end
RP/0/RP0/CPU0:basquiat-asr9k(config-if)#commit comment COMMENT-INTERFACE-G0/0/$
Fri Sep 29 23:58:38.596 UTC
RP/0/RP0/CPU0:basquiat-asr9k(config-if)#show configuration commit list
Fri Sep 29 23:58:53.387 UTC
SNo. Label/ID User Line Client Time Stamp
~~~~ ~~~~~~~~ ~~~~ ~~~~ ~~~~~~ ~~~~~~~~~~
1 1000000009 cisco con0_RP0_CPU0 CLI Fri Sep 29 23:58:38 2023
2 1000000008 cisco con0_RP0_CPU0 CLI Fri Sep 29 23:41:42 2023
3 1000000007 cisco con0_RP0_CPU0 CLI Fri Sep 29 23:39:22 2023
4 1000000006 cisco con0_RP0_CPU0 CLI Fri Sep 29 23:29:22 2023
5 ssh cisco con0_RP0_CPU0 CLI Fri Sep 29 23:25:12 2023
6 1000000004 cisco con0_RP0_CPU0 CLI Fri Sep 29 23:23:07 2023
7 1000000003 cisco con0_RP0_CPU0 CLI Fri Sep 29 23:22:30 2023
8 1000000002 CVAC UNKNOWN CLI Fri Sep 29 22:51:40 2023
9 1000000001 root N/A CALLHOME Fri Sep 29 22:49:58 2023
RP/0/RP0/CPU0:basquiat-asr9k(config)#show configuration commit list 1 detail
Fri Sep 29 23:59:53.160 UTC
1) CommitId: 1000000009 Label: NONE
UserId: cisco Line: con0_RP0_CPU0
Client: CLI Time: Fri Sep 29 23:58:38 2023
Comment: COMMENT-INTERFACE-G0/0/0/0
RP/0/RP0/CPU0:basquiat-asr9k(config)#That's it. As I'm going ahead into Service Provider topics, I would like to take notes and share them here.
