Complex state parsing example using Juniper inventory Part 1

Sometimes you need to parse out information that is not in the NQE data model for various reasons. It maybe device specific information that is not generalizable across vendors, or not generally useful in dataplane analysis but maybe important for operations. Importing this data is beyond the scope of this post but you can read more about it here: Getting Custom Commands

First, I am going to demonstrate how you build up your parsing chops by leveraging a testing approach to NQE, then will demonstrate how this can be applied to your collection data.
 

• Setting up your testing. 

We need to represent the data that our custom command will import into the snapshot. We can represent this by setting a variable to a multiline string using the delimiter ”””

output =
  """
Item             Version  Part number  Serial number     FRU model number
Midplane         REV 03   710-013698   TR0001            CHAS-BP-MX960-S
FPM Board        REV 03   710-014974   JZ2323            CRAFT-MX960-S
PEM 0            Rev 07   740-029344   QZY2122ABCD       PWR-MX960-4100-DC-S
PEM 1            Rev 07   740-029344   QZY2123ABCE       PWR-MX960-4100-DC-S
PEM 2            Rev 07   740-029344   QZY2124ABCF       PWR-MX960-4100-DC-S
PEM 3            Rev 07   740-029344   QZY2125ABCG       PWR-MX960-4100-DC-S
Routing Engine 0 REV 20   750-054758   ZAUY1234          RE-S-X6-64G-S
Routing Engine 1 REV 20   750-054758   ZAUY1235          RE-S-X6-64G-S
CB 0             REV 12   750-062572   XYZE1234          SCBE2-MX-S
CB 1             REV 12   750-062572   XYZE1235          SCBE2-MX-S
CB 2             REV 12   750-062572   XYZE1236          SCBE2-MX-S
FPC 0            REV 57   750-053323   ZYGB2222          MPC7E-10G
FPC 1            REV 46   750-056519   ZYGB2223          MPC7E-MRATE
FPC 3            REV 01   750-136058   ZYGB2223          MPC7E-10G
FPC 7            REV 26   750-028467   BVCC1234          MPC-3D-16XGE-SFPP
FPC 8            REV 46   750-056519   CABC2912          MPC7E-MRATE
Fan Tray 0       REV 08   740-031521   ACCA1234          FFANTRAY-MX960-HC-S
Fan Tray 1       REV 08   740-031521   ACCA1235          FFANTRAY-MX960-HC-S
""";

Next we define some helper functions:

fixRev(s) = replaceMatches(s, "REV 20", "    REV 20");

fixBlocks(blocks) =
  foreach line in blocks
  select if matches(line.text, "Routing Engine*")
         then replaceMatches(fixRev(line.text), "\\b\\s\\b", "-")
         else replaceMatches(line.text, "\\b\\s\\b", "-");

So you may ask yourself why do we need these helper functions. Well the reason is parsing data can be very complicated and NQE tries to make this as simple as possible but it can’t account for every possible scenario. For instance in our output you may notice that we have a table of 5 columns labeled Item, Version, Part number, Serial number and FRU model number

The output for these columns vary from simple strings e.g. Midplane to a list of strings PEM 0 or Routing Engine 0 . In order for NQE to group these appropriately we need to manipulate the strings or marshal them into a more structured representation.

Let’s break down what fixBlocks is doing. It takes in a variable called blocks, which is of type List<ConfigLine> and iterates over the collection with a comprehension. When we encounter a line matching the glob “Routing Engine*”, we use the replaceMatches function to first give us a little room between the 0 and 1 and the next word block so we can use our word boundary regex then we convert the spaces between Routing Engine <number> to Routing-Engine-<number>. You may notice that we don’t have to do this for Fan Tray 1 because there is sufficient space between the <number> and the next column.

The rest of the query should be self explanatory if you are familiar with NQE pattern matching and data extraction. 
 

pattern =
  `{!"Item"} {item:string} {version:string} {partNo:string} {serialNum:string} {FRUModel:string}`;

foreach x in [0]
let blocks = parseConfigBlocks(OS.UNKNOWN, output)
let newBlocks = fixBlocks(blocks)
foreach block in newBlocks
let matches = patternMatch(block, pattern)
where matches?.item != null : String
select {
  item: matches?.item,
  version: matches?.version,
  partNo: matches?.partNo,
  serialNo: matches?.serialNum,
  FRUMpdel: matches?.FRUModel
}

Here is the full query 
 

/**
 * @intent Provide structure report on inventory
 * @description This query is a test to parse out the outputs of JunOS "show chassis hardware models"
 */
output =
  """
Item             Version  Part number  Serial number     FRU model number
Midplane         REV 03   710-013698   TR0001            CHAS-BP-MX960-S
FPM Board        REV 03   710-014974   JZ2323            CRAFT-MX960-S
PEM 0            Rev 07   740-029344   QZY2122ABCD       PWR-MX960-4100-DC-S
PEM 1            Rev 07   740-029344   QZY2123ABCE       PWR-MX960-4100-DC-S
PEM 2            Rev 07   740-029344   QZY2124ABCF       PWR-MX960-4100-DC-S
PEM 3            Rev 07   740-029344   QZY2125ABCG       PWR-MX960-4100-DC-S
Routing Engine 0 REV 20   750-054758   ZAUY1234          RE-S-X6-64G-S
Routing Engine 1 REV 20   750-054758   ZAUY1235          RE-S-X6-64G-S
CB 0             REV 12   750-062572   XYZE1234          SCBE2-MX-S
CB 1             REV 12   750-062572   XYZE1235          SCBE2-MX-S
CB 2             REV 12   750-062572   XYZE1236          SCBE2-MX-S
FPC 0            REV 57   750-053323   ZYGB2222          MPC7E-10G
FPC 1            REV 46   750-056519   ZYGB2223          MPC7E-MRATE
FPC 3            REV 01   750-136058   ZYGB2223          MPC7E-10G
FPC 7            REV 26   750-028467   BVCC1234          MPC-3D-16XGE-SFPP
FPC 8            REV 46   750-056519   CABC2912          MPC7E-MRATE
Fan Tray 0       REV 08   740-031521   ACCA1234          FFANTRAY-MX960-HC-S
Fan Tray 1       REV 08   740-031521   ACCA1235          FFANTRAY-MX960-HC-S
""";

fixRev(s) = replaceMatches(s, "REV 20", "    REV 20");

fixBlocks(blocks) =
  foreach line in blocks
  select if matches(line.text, "Routing Engine*")
         then replaceMatches(fixRev(line.text), "\\b\\s\\b", "-")
         else replaceMatches(line.text, "\\b\\s\\b", "-");

pattern =
  `{!"Item"} {item:string} {version:string} {partNo:string} {serialNum:string} {FRUModel:string}`;

foreach x in [0]
let blocks = parseConfigBlocks(OS.UNKNOWN, output)
let newBlocks = fixBlocks(blocks)
foreach block in newBlocks
let matches = patternMatch(block, pattern)
where matches?.item != null : String
select {
  item: matches?.item,
  version: matches?.version,
  partNo: matches?.partNo,
  serialNo: matches?.serialNum,
  FRUModel: matches?.FRUModel
}

Now that we have tested our parsing we can apply this to the snapshot with the following modifications.

• Running against your snapshots

/**
 * @intent Provide structure report on JunOS "show chassis hardware models"
 * @description Report for JunOS "show chassis hardware models"
 */

command_eval = "show chassis hardware models";

fixRev(s) = replaceMatches(s, "REV 20", "    REV 20");

fixBlocks(blocks) =
  foreach line in blocks
  select if matches(line.text, "Routing Engine*")
         then replaceMatches(fixRev(line.text), "\\b\\s\\b", "-")
         else replaceMatches(line.text, "\\b\\s\\b", "-");

pattern =
  `{!"Item"} {item:string} {version:string} {partNo:string} {serialNum:string} {FRUModel:string}`;

foreach device in network.devices
where device.platform.os == OS.JUNOS
foreach output in device.outputs.commands
where output.commandText == command_eval
let blocks = parseConfigBlocks(OS.UNKNOWN, output.response)
let newBlocks = fixBlocks(blocks)
foreach block in newBlocks
let matches = patternMatch(block, pattern)
where matches?.item != null : String
select {
  item: matches?.item,
  version: matches?.version,
  partNo: matches?.partNo,
  serialNo: matches?.serialNum,
  FRUModel: matches?.FRUModel
}

In Part 2 we will look at an even more complex example. Thanks for hanging in..