Skip to main content

Complex state parsing example using Juniper inventory Part 2

  • October 20, 2023
  • 0 replies
  • 195 views
  • Translate

Forum|alt.badge.img+1

In Part 1 we introduced several techniques for dealing with semi-structured data to parse a Juniper Inventory. In Part 2 we will leverage a few more techniques to parse a more complex output. 

1. Let’s setup our test by grabbing the output from the “show chassis hardware” command. remember to use the multi-string block delimiter “””

We have 5 columns Item, Version, Part number, Serial number and Description, but we are only interested in three: Item, Serial number and Description

output =
  """
Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                JN2221837ABC      MX960
Midplane         REV 03   710-013698   TR0123            MX960 Backplane
FPM Board        REV 03   710-014974   JZ1111            Front Panel Display
PDM              Rev 03   740-013110   QCS1012345U       Power Distribution Module
PEM 0            Rev 07   740-029344   QCS1012346U       DC 4.1kW Power Entry Module
PEM 1            Rev 07   740-029344   QCS1012347U       DC 4.1kW Power Entry Module
Routing Engine 0 REV 20   750-054758   CAPS1234          RE-S-2X00x6
Routing Engine 1 REV 20   750-054758   CAPS1235          RE-S-2X00x6
CB 0             REV 12   750-062572   CAMZ1235          Enhanced MX SCB 2
CB 1             REV 12   750-062572   CAMZ1245          Enhanced MX SCB 2
CB 2             REV 12   750-062572   CAMZ1255          Enhanced MX SCB 2
FPC 0            REV 57   750-053323   CAZZZ255          MPC7E 3D 40XGE
  CPU            REV 22   750-057177   CZZM0123          SMPC PMB
  PIC 0                   BUILTIN      BUILTIN           20x10GE SFPP
    Xcvr 1       REV 01   740-031980   011263B00023      SFP+-10G-SR
    Xcvr 4       REV 01   740-031980   ZAC0ZAA           SFP+-10G-SR
  PIC 1                   BUILTIN      BUILTIN           20x10GE SFPP
    Xcvr 0       REV 01   740-031980   1RT511231076      SFP+-10G-SR
FPC 1            REV 46   750-056519   CAZE1231          MPC7E 3D MRATE-12xQSFPP-XGE-XLGE-CGE
  CPU            REV 21   750-057177   CAZE1232          SMPC PMB
  PIC 0                   BUILTIN      BUILTIN           MRATE-6xQSFPP-XGE-XLGE-CGE
    Xcvr 0       REV 01   740-032986   QE443531          QSFP+-40G-SR4
    Xcvr 1       REV 01   740-046565   QE443532          QSFP+-40G-SR4
  PIC 1                   BUILTIN      BUILTIN           MRATE-6xQSFPP-XGE-XLGE-CGE
    Xcvr 2       REV 01   740-058732   1GCQA42007G       QSFP-100GBASE-LR4
FPC 3            REV 01   750-136058   CASS5520          MPC7E 3D 40XGE
  CPU            REV 22   750-057177   CASR7694          SMPC PMB
  PIC 0                   BUILTIN      BUILTIN           20x10GE SFPP
    Xcvr 1       REV 01   740-031980   1YT517100173      SFP+-10G-SR
    Xcvr 3       REV 01   740-031980   093363A00235      SFP+-10G-SR
    Xcvr 4       REV 01   740-031980   083363A22223      SFP+-10G-SR
  PIC 1                   BUILTIN      BUILTIN           20x10GE SFPP
    Xcvr 0       REV 01   740-031980   023213B00001      SFP+-10G-SR
    Xcvr 1       REV 01   740-031980   023213B00002      SFP+-10G-SR
    Xcvr 3       REV 01   740-031980   023213B00003      SFP+-10G-SR
FPC 7            REV 26   750-028467   ABBG3242          MPC 3D 16x 10GE
  CPU            REV 10   711-029089   ABBG3243          AMPC PMB
  PIC 0                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
  PIC 1                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0       REV 01   740-031980   ABBG3245           SFP+-10G-SR
  PIC 2                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0       REV 01   740-031980   012343A00000      SFP+-10G-SR
  PIC 3                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 2       REV 01   740-031980   012343A00001      SFP+-10G-SR
    Xcvr 3       REV 01   740-031980   012343A00002      SFP+-10G-SR
Fan Tray 0       REV 08   740-031521   ASZDS7154          Enhanced Fan Tray
Fan Tray 1       REV 08   740-031521   AADSA1292          Enhanced Fan Tray
""";


Next we will write our helper functions and patterns
 

pattern5=
  `{!"Item"} {!"Hardware inventory"} {item:string} {string} {string} {serialNum:string} {description:string}`;

pattern3 = `{!"Item"} {item:string} {serialNum:string} {description:string}`;

pattern4 = `{item:string} {string} {serialNum:string} {description:string}`;

fixRev(s) = replaceMatches(s, "REV 20", "    REV 20");

findTokenCount(s) = length(patternMatch(s, `{string*}`));

stripSpaces(s) = replaceMatches(s, "\\s\\s\\s\\s\\b", "");

flattenBlocks(blocks) =
  max(foreach x in [0]
      let matches = patternMatches(blocks, pattern3)
      let a = (foreach match in matches
               select stripSpaces(match.line.text))
      let b = (foreach match in matches
               foreach child in match.line.children
               select stripSpaces(child.text))
      let c = (foreach match in matches
               foreach child1 in match.line.children
               foreach child2 in child1.children
               select child2.text)
      let newBlocks = a + b + c
      select newBlocks);

fixBlocks(blocks) =
  foreach line in blocks
  select if matches(line, "Routing Engine*")
         then replaceMatches(fixRev(line), "\\b\\s\\b", "-")
         else replaceMatches(line, "\\b\\s\\b", "-");

normalizeBlocks(blocks) =
  foreach line in blocks
  let var_count = findTokenCount(line)
  select if var_count == 3
         then patternMatch(line, pattern3)
         else if var_count == 4
              then patternMatch(line, pattern4)
              else if var_count == 5
                   then patternMatch(line, pattern5)
                   else null : {item: String, serialNum: String, description: String};


We have seen fixRev before in Part 1 so let’s discuss findTokenCount and stripSpaces.

findTokenCount is going to be a handy function for us because it is going to tell us how many string tokens are in a line. We can use this to select which pattern to use to parse information out using the function normalizeBlocks

stripSpaces - Well this does what it says it is used to clean up some of those hanging indents to align all those columns together. 

We used something similar to fixBlocks in Part 1, but here we have modified it to work on strings, since we are converting List<ConfigLine> to List<String>.  

So now lots cover the heart of the query, normalizeBlocks. As much as we would like all the strings to fall nicely into their columns we have a problem. Some lines will have 3 strings, some 4 and some 5. So we use the result of the function findTokenCount to tell us which pattern to use.

At this point we have everything we need so the main body of the query is pretty simple.
 

foreach x in [0]
let blocks = parseConfigBlocks(OS.UNKNOWN, output)
let newBlocks = flattenBlocks(blocks)
let fixBlocks = fixBlocks(newBlocks)
let normalizeBlocks = normalizeBlocks(fixBlocks)
foreach block in normalizeBlocks
select {
  item: block.item, serialNo: block.serialNum, description: block.description
}


Success!!


Full query here:
 

/**
 * @intent Parse out item, serialNo, and description from chassing inbentory
 * @description Parse out item, serialNo, and description from chassing inbentory
 */

output =
  """
Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                JN1096837AFA      MX960
Midplane         REV 03   710-013698   TR0185            MX960 Backplane
FPM Board        REV 03   710-014974   JZ6891            Front Panel Display
PDM              Rev 03   740-013110   QCS1103501U       Power Distribution Module
PEM 0            Rev 07   740-029344   QCS1619V0YT       DC 4.1kW Power Entry Module
PEM 1            Rev 07   740-029344   QCS1619V115       DC 4.1kW Power Entry Module
PEM 2            Rev 07   740-029344   QCS1619V1CH       DC 4.1kW Power Entry Module
PEM 3            Rev 07   740-029344   QCS1619V07Z       DC 4.1kW Power Entry Module
Routing Engine 0 REV 20   750-054758   CAPS2212          RE-S-2X00x6
Routing Engine 1 REV 20   750-054758   CAPV3694          RE-S-2X00x6
CB 0             REV 12   750-062572   CAMW5050          Enhanced MX SCB 2
CB 1             REV 12   750-062572   CAMX8697          Enhanced MX SCB 2
CB 2             REV 12   750-062572   CAMY1945          Enhanced MX SCB 2
FPC 0            REV 57   750-053323   CARN6138          MPC7E 3D 40XGE
  CPU            REV 22   750-057177   CARM0155          SMPC PMB
  PIC 0                   BUILTIN      BUILTIN           20x10GE SFPP
    Xcvr 1       REV 01   740-031980   083363A00023      SFP+-10G-SR
    Xcvr 4       REV 01   740-031980   AJC0BLU           SFP+-10G-SR
  PIC 1                   BUILTIN      BUILTIN           20x10GE SFPP
    Xcvr 0       REV 01   740-031980   1YT517101076      SFP+-10G-SR
FPC 1            REV 46   750-056519   CALE9001          MPC7E 3D MRATE-12xQSFPP-XGE-XLGE-CGE
  CPU            REV 21   750-057177   CAKX3479          SMPC PMB
  PIC 0                   BUILTIN      BUILTIN           MRATE-6xQSFPP-XGE-XLGE-CGE
    Xcvr 0       REV 01   740-032986   QE443531          QSFP+-40G-SR4
    Xcvr 1       REV 01   740-046565   QH2102DM          QSFP+-40G-SR4
    Xcvr 2       REV 01   740-046565   QH2102DF          QSFP+-40G-SR4
  PIC 1                   BUILTIN      BUILTIN           MRATE-6xQSFPP-XGE-XLGE-CGE
    Xcvr 2       REV 01   740-058732   1GCQA42007G       QSFP-100GBASE-LR4
FPC 3            REV 01   750-136058   CASS5520          MPC7E 3D 40XGE
  CPU            REV 22   750-057177   CASR7694          SMPC PMB
  PIC 0                   BUILTIN      BUILTIN           20x10GE SFPP
    Xcvr 1       REV 01   740-031980   1YT517100173      SFP+-10G-SR
    Xcvr 3       REV 01   740-031980   093363A00235      SFP+-10G-SR
    Xcvr 4       REV 01   740-031980   083363A00043      SFP+-10G-SR
    Xcvr 5       REV 01   740-031980   1YT517100717      SFP+-10G-SR
    Xcvr 7       REV 01   740-031980   093363A00636      SFP+-10G-SR
  PIC 1                   BUILTIN      BUILTIN           20x10GE SFPP
    Xcvr 0       REV 01   740-031980   073363A00041      SFP+-10G-SR
    Xcvr 1       REV 01   740-031980   093363A00557      SFP+-10G-SR
    Xcvr 3       REV 01   740-031980   073363A00057      SFP+-10G-SR
    Xcvr 5       REV 01   740-031980   073363A00607      SFP+-10G-SR
    Xcvr 7       REV 01   740-031980   133363A01488      SFP+-10G-SR
    Xcvr 15      REV 01   740-031980   133363A01901      SFP+-10G-SR
    Xcvr 17      REV 01   740-031980   1YT517100720      SFP+-10G-SR
    Xcvr 19      REV 01   740-031980   133363A01671      SFP+-10G-SR
FPC 7            REV 26   750-028467   ABBF8962          MPC 3D 16x 10GE
  CPU            REV 10   711-029089   ABBF8821          AMPC PMB
  PIC 0                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
  PIC 1                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0       REV 01   740-031980   AHS0842           SFP+-10G-SR
  PIC 2                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0       REV 01   740-031980   083363A00041      SFP+-10G-SR
  PIC 3                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 2       REV 01   740-031980   083363A00036      SFP+-10G-SR
    Xcvr 3       REV 01   740-031980   223363A01252      SFP+-10G-SR
FPC 8            REV 46   750-056519   CALD2998          MPC7E 3D MRATE-12xQSFPP-XGE-XLGE-CGE
  CPU            REV 21   750-057177   CAKX3028          SMPC PMB
  PIC 0                   BUILTIN      BUILTIN           MRATE-6xQSFPP-XGE-XLGE-CGE
    Xcvr 0       REV 01   740-046565   QG4303PE          QSFP+-40G-SR4
    Xcvr 1       REV 01   740-046565   QH2102CX          QSFP+-40G-SR4
    Xcvr 2       REV 01   740-046565   QH2102DJ          QSFP+-40G-SR4
  PIC 1                   BUILTIN      BUILTIN           MRATE-6xQSFPP-XGE-XLGE-CGE
    Xcvr 2       REV 01   740-058732   1GTQA44202E       QSFP-100GBASE-LR4
Fan Tray 0       REV 08   740-031521   ACAD7154          Enhanced Fan Tray
Fan Tray 1       REV 08   740-031521   ACAD4792          Enhanced Fan Tray
""";

pattern5 =
  `{!"Item"} {!"Hardware inventory"} {item:string} {string} {string} {serialNum:string} {description:string}`;

pattern3 = `{!"Item"} {item:string} {serialNum:string} {description:string}`;

pattern4 = `{item:string} {string} {serialNum:string} {description:string}`;

fixRev(s) = replaceMatches(s, "REV 20", "    REV 20");

findTokenCount(s) = length(patternMatch(s, `{string*}`));

stripSpaces(s) = replaceMatches(s, "\\s\\s\\s\\s\\b", "");

flattenBlocks(blocks) =
  max(foreach x in [0]
      let matches = patternMatches(blocks, pattern3)
      let a = (foreach match in matches
               select stripSpaces(match.line.text))
      let b = (foreach match in matches
               foreach child in match.line.children
               select stripSpaces(child.text))
      let c = (foreach match in matches
               foreach child1 in match.line.children
               foreach child2 in child1.children
               select child2.text)
      let newBlocks = a + b + c
      select newBlocks);

fixBlocks(blocks) =
  foreach line in blocks
  select if matches(line, "Routing Engine*")
         then replaceMatches(fixRev(line), "\\b\\s\\b", "-")
         else replaceMatches(line, "\\b\\s\\b", "-");

normalizeBlocks(blocks) =
  foreach line in blocks
  let var_count = findTokenCount(line)
  select if var_count == 3
         then patternMatch(line, pattern3)
         else if var_count == 4
              then patternMatch(line, pattern4)
              else if var_count == 5
                   then patternMatch(line, pattern5)
                   else null : {item: String, serialNum: String, description: String};

foreach x in [0]
let blocks = parseConfigBlocks(OS.UNKNOWN, output)
let newBlocks = flattenBlocks(blocks)
let fixBlocks = fixBlocks(newBlocks)
let normalizeBlocks = normalizeBlocks(fixBlocks)
foreach block in normalizeBlocks
select {
  item: block.item, serialNo: block.serialNum, description: block.description
}

And finally now we can integrate with our snapshot after adding the custom command.

 

/**
 * @intent Parse out item, serialNo, and description from chassis inventory
 * @description Parse out item, serialNo, and description from chassis inventory
 */

command_eval = "show chassis hardware";

pattern5 =
  `{!"Item"} {!"Hardware inventory"} {item:string} {string} {string} {serialNum:string} {description:string}`;

pattern3 = `{!"Item"} {item:string} {serialNum:string} {description:string}`;

pattern4 = `{item:string} {string} {serialNum:string} {description:string}`;

fixRev(s) = replaceMatches(s, "REV 20", "    REV 20");

findTokenCount(s) = length(patternMatch(s, `{string*}`));

stripSpaces(s) = replaceMatches(s, "\\s\\s\\s\\s\\b", "");

flattenBlocks(blocks) =
  max(foreach x in [0]
      let matches = patternMatches(blocks, pattern3)
      let a = (foreach match in matches
               select stripSpaces(match.line.text))
      let b = (foreach match in matches
               foreach child in match.line.children
               select stripSpaces(child.text))
      let c = (foreach match in matches
               foreach child1 in match.line.children
               foreach child2 in child1.children
               select child2.text)
      let newBlocks = a + b + c
      select newBlocks);

fixBlocks(blocks) =
  foreach line in blocks
  select if matches(line, "Routing Engine*")
         then replaceMatches(fixRev(line), "\\b\\s\\b", "-")
         else replaceMatches(line, "\\b\\s\\b", "-");

normalizeBlocks(blocks) =
  foreach line in blocks
  let var_count = findTokenCount(line)
  select if var_count == 3
         then patternMatch(line, pattern3)
         else if var_count == 4
              then patternMatch(line, pattern4)
              else if var_count == 5
                   then patternMatch(line, pattern5)
                   else null : {item: String, serialNum: String, description: String};

foreach device in network.devices
where device.platform.os == OS.JUNOS
foreach output in device.outputs.commands
where output.commandText == command_eval
let blocks = parseConfigBlocks(OS.UNKNOWN, output.response)
let newBlocks = flattenBlocks(blocks)
let fixBlocks = fixBlocks(newBlocks)
let normalizeBlocks = normalizeBlocks(fixBlocks)
foreach block in normalizeBlocks
select {
  item: block.item, serialNo: block.serialNum, description: block.description
}

Thanks for reading, Let us know if you have any improvements to this or if this helped you solve a problem.

0 replies

Be the first to reply!

Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings