In Part 1 we introduced several techniques for dealing with semi-structured data to parse a Juniper Inventory. In Part 2 we will leverage a few more techniques to parse a more complex output.
1. Let’s setup our test by grabbing the output from the “show chassis hardware” command. remember to use the multi-string block delimiter “””
We have 5 columns Item, Version, Part number, Serial number and Description, but we are only interested in three: Item, Serial number and Description.
output =
"""
Hardware inventory:
Item Version Part number Serial number Description
Chassis JN2221837ABC MX960
Midplane REV 03 710-013698 TR0123 MX960 Backplane
FPM Board REV 03 710-014974 JZ1111 Front Panel Display
PDM Rev 03 740-013110 QCS1012345U Power Distribution Module
PEM 0 Rev 07 740-029344 QCS1012346U DC 4.1kW Power Entry Module
PEM 1 Rev 07 740-029344 QCS1012347U DC 4.1kW Power Entry Module
Routing Engine 0 REV 20 750-054758 CAPS1234 RE-S-2X00x6
Routing Engine 1 REV 20 750-054758 CAPS1235 RE-S-2X00x6
CB 0 REV 12 750-062572 CAMZ1235 Enhanced MX SCB 2
CB 1 REV 12 750-062572 CAMZ1245 Enhanced MX SCB 2
CB 2 REV 12 750-062572 CAMZ1255 Enhanced MX SCB 2
FPC 0 REV 57 750-053323 CAZZZ255 MPC7E 3D 40XGE
CPU REV 22 750-057177 CZZM0123 SMPC PMB
PIC 0 BUILTIN BUILTIN 20x10GE SFPP
Xcvr 1 REV 01 740-031980 011263B00023 SFP+-10G-SR
Xcvr 4 REV 01 740-031980 ZAC0ZAA SFP+-10G-SR
PIC 1 BUILTIN BUILTIN 20x10GE SFPP
Xcvr 0 REV 01 740-031980 1RT511231076 SFP+-10G-SR
FPC 1 REV 46 750-056519 CAZE1231 MPC7E 3D MRATE-12xQSFPP-XGE-XLGE-CGE
CPU REV 21 750-057177 CAZE1232 SMPC PMB
PIC 0 BUILTIN BUILTIN MRATE-6xQSFPP-XGE-XLGE-CGE
Xcvr 0 REV 01 740-032986 QE443531 QSFP+-40G-SR4
Xcvr 1 REV 01 740-046565 QE443532 QSFP+-40G-SR4
PIC 1 BUILTIN BUILTIN MRATE-6xQSFPP-XGE-XLGE-CGE
Xcvr 2 REV 01 740-058732 1GCQA42007G QSFP-100GBASE-LR4
FPC 3 REV 01 750-136058 CASS5520 MPC7E 3D 40XGE
CPU REV 22 750-057177 CASR7694 SMPC PMB
PIC 0 BUILTIN BUILTIN 20x10GE SFPP
Xcvr 1 REV 01 740-031980 1YT517100173 SFP+-10G-SR
Xcvr 3 REV 01 740-031980 093363A00235 SFP+-10G-SR
Xcvr 4 REV 01 740-031980 083363A22223 SFP+-10G-SR
PIC 1 BUILTIN BUILTIN 20x10GE SFPP
Xcvr 0 REV 01 740-031980 023213B00001 SFP+-10G-SR
Xcvr 1 REV 01 740-031980 023213B00002 SFP+-10G-SR
Xcvr 3 REV 01 740-031980 023213B00003 SFP+-10G-SR
FPC 7 REV 26 750-028467 ABBG3242 MPC 3D 16x 10GE
CPU REV 10 711-029089 ABBG3243 AMPC PMB
PIC 0 BUILTIN BUILTIN 4x 10GE(LAN) SFP+
PIC 1 BUILTIN BUILTIN 4x 10GE(LAN) SFP+
Xcvr 0 REV 01 740-031980 ABBG3245 SFP+-10G-SR
PIC 2 BUILTIN BUILTIN 4x 10GE(LAN) SFP+
Xcvr 0 REV 01 740-031980 012343A00000 SFP+-10G-SR
PIC 3 BUILTIN BUILTIN 4x 10GE(LAN) SFP+
Xcvr 2 REV 01 740-031980 012343A00001 SFP+-10G-SR
Xcvr 3 REV 01 740-031980 012343A00002 SFP+-10G-SR
Fan Tray 0 REV 08 740-031521 ASZDS7154 Enhanced Fan Tray
Fan Tray 1 REV 08 740-031521 AADSA1292 Enhanced Fan Tray
""";
Next we will write our helper functions and patterns
pattern5=
`{!"Item"} {!"Hardware inventory"} {item:string} {string} {string} {serialNum:string} {description:string}`;
pattern3 = `{!"Item"} {item:string} {serialNum:string} {description:string}`;
pattern4 = `{item:string} {string} {serialNum:string} {description:string}`;
fixRev(s) = replaceMatches(s, "REV 20", " REV 20");
findTokenCount(s) = length(patternMatch(s, `{string*}`));
stripSpaces(s) = replaceMatches(s, "\\s\\s\\s\\s\\b", "");
flattenBlocks(blocks) =
max(foreach x in x0]
let matches = patternMatches(blocks, pattern3)
let a = (foreach match in matches
select stripSpaces(match.line.text))
let b = (foreach match in matches
foreach child in match.line.children
select stripSpaces(child.text))
let c = (foreach match in matches
foreach child1 in match.line.children
foreach child2 in child1.children
select child2.text)
let newBlocks = a + b + c
select newBlocks);
fixBlocks(blocks) =
foreach line in blocks
select if matches(line, "Routing Engine*")
then replaceMatches(fixRev(line), "\\b\\s\\b", "-")
else replaceMatches(line, "\\b\\s\\b", "-");
normalizeBlocks(blocks) =
foreach line in blocks
let var_count = findTokenCount(line)
select if var_count == 3
then patternMatch(line, pattern3)
else if var_count == 4
then patternMatch(line, pattern4)
else if var_count == 5
then patternMatch(line, pattern5)
else null : {item: String, serialNum: String, description: String};
We have seen fixRev before in Part 1 so let’s discuss findTokenCount and stripSpaces.findTokenCount is going to be a handy function for us because it is going to tell us how many string tokens are in a line. We can use this to select which pattern to use to parse information out using the function normalizeBlocks.
stripSpaces - Well this does what it says it is used to clean up some of those hanging indents to align all those columns together.
We used something similar to fixBlocks in Part 1, but here we have modified it to work on strings, since we are converting List<ConfigLine> to List<String>.
So now lots cover the heart of the query, normalizeBlocks. As much as we would like all the strings to fall nicely into their columns we have a problem. Some lines will have 3 strings, some 4 and some 5. So we use the result of the function findTokenCount to tell us which pattern to use.
At this point we have everything we need so the main body of the query is pretty simple.
foreach x in u0]
let blocks = parseConfigBlocks(OS.UNKNOWN, output)
let newBlocks = flattenBlocks(blocks)
let fixBlocks = fixBlocks(newBlocks)
let normalizeBlocks = normalizeBlocks(fixBlocks)
foreach block in normalizeBlocks
select {
item: block.item, serialNo: block.serialNum, description: block.description
}
Success!!
Full query here:
/**
* @intent Parse out item, serialNo, and description from chassing inbentory
* @description Parse out item, serialNo, and description from chassing inbentory
*/
output =
"""
Hardware inventory:
Item Version Part number Serial number Description
Chassis JN1096837AFA MX960
Midplane REV 03 710-013698 TR0185 MX960 Backplane
FPM Board REV 03 710-014974 JZ6891 Front Panel Display
PDM Rev 03 740-013110 QCS1103501U Power Distribution Module
PEM 0 Rev 07 740-029344 QCS1619V0YT DC 4.1kW Power Entry Module
PEM 1 Rev 07 740-029344 QCS1619V115 DC 4.1kW Power Entry Module
PEM 2 Rev 07 740-029344 QCS1619V1CH DC 4.1kW Power Entry Module
PEM 3 Rev 07 740-029344 QCS1619V07Z DC 4.1kW Power Entry Module
Routing Engine 0 REV 20 750-054758 CAPS2212 RE-S-2X00x6
Routing Engine 1 REV 20 750-054758 CAPV3694 RE-S-2X00x6
CB 0 REV 12 750-062572 CAMW5050 Enhanced MX SCB 2
CB 1 REV 12 750-062572 CAMX8697 Enhanced MX SCB 2
CB 2 REV 12 750-062572 CAMY1945 Enhanced MX SCB 2
FPC 0 REV 57 750-053323 CARN6138 MPC7E 3D 40XGE
CPU REV 22 750-057177 CARM0155 SMPC PMB
PIC 0 BUILTIN BUILTIN 20x10GE SFPP
Xcvr 1 REV 01 740-031980 083363A00023 SFP+-10G-SR
Xcvr 4 REV 01 740-031980 AJC0BLU SFP+-10G-SR
PIC 1 BUILTIN BUILTIN 20x10GE SFPP
Xcvr 0 REV 01 740-031980 1YT517101076 SFP+-10G-SR
FPC 1 REV 46 750-056519 CALE9001 MPC7E 3D MRATE-12xQSFPP-XGE-XLGE-CGE
CPU REV 21 750-057177 CAKX3479 SMPC PMB
PIC 0 BUILTIN BUILTIN MRATE-6xQSFPP-XGE-XLGE-CGE
Xcvr 0 REV 01 740-032986 QE443531 QSFP+-40G-SR4
Xcvr 1 REV 01 740-046565 QH2102DM QSFP+-40G-SR4
Xcvr 2 REV 01 740-046565 QH2102DF QSFP+-40G-SR4
PIC 1 BUILTIN BUILTIN MRATE-6xQSFPP-XGE-XLGE-CGE
Xcvr 2 REV 01 740-058732 1GCQA42007G QSFP-100GBASE-LR4
FPC 3 REV 01 750-136058 CASS5520 MPC7E 3D 40XGE
CPU REV 22 750-057177 CASR7694 SMPC PMB
PIC 0 BUILTIN BUILTIN 20x10GE SFPP
Xcvr 1 REV 01 740-031980 1YT517100173 SFP+-10G-SR
Xcvr 3 REV 01 740-031980 093363A00235 SFP+-10G-SR
Xcvr 4 REV 01 740-031980 083363A00043 SFP+-10G-SR
Xcvr 5 REV 01 740-031980 1YT517100717 SFP+-10G-SR
Xcvr 7 REV 01 740-031980 093363A00636 SFP+-10G-SR
PIC 1 BUILTIN BUILTIN 20x10GE SFPP
Xcvr 0 REV 01 740-031980 073363A00041 SFP+-10G-SR
Xcvr 1 REV 01 740-031980 093363A00557 SFP+-10G-SR
Xcvr 3 REV 01 740-031980 073363A00057 SFP+-10G-SR
Xcvr 5 REV 01 740-031980 073363A00607 SFP+-10G-SR
Xcvr 7 REV 01 740-031980 133363A01488 SFP+-10G-SR
Xcvr 15 REV 01 740-031980 133363A01901 SFP+-10G-SR
Xcvr 17 REV 01 740-031980 1YT517100720 SFP+-10G-SR
Xcvr 19 REV 01 740-031980 133363A01671 SFP+-10G-SR
FPC 7 REV 26 750-028467 ABBF8962 MPC 3D 16x 10GE
CPU REV 10 711-029089 ABBF8821 AMPC PMB
PIC 0 BUILTIN BUILTIN 4x 10GE(LAN) SFP+
PIC 1 BUILTIN BUILTIN 4x 10GE(LAN) SFP+
Xcvr 0 REV 01 740-031980 AHS0842 SFP+-10G-SR
PIC 2 BUILTIN BUILTIN 4x 10GE(LAN) SFP+
Xcvr 0 REV 01 740-031980 083363A00041 SFP+-10G-SR
PIC 3 BUILTIN BUILTIN 4x 10GE(LAN) SFP+
Xcvr 2 REV 01 740-031980 083363A00036 SFP+-10G-SR
Xcvr 3 REV 01 740-031980 223363A01252 SFP+-10G-SR
FPC 8 REV 46 750-056519 CALD2998 MPC7E 3D MRATE-12xQSFPP-XGE-XLGE-CGE
CPU REV 21 750-057177 CAKX3028 SMPC PMB
PIC 0 BUILTIN BUILTIN MRATE-6xQSFPP-XGE-XLGE-CGE
Xcvr 0 REV 01 740-046565 QG4303PE QSFP+-40G-SR4
Xcvr 1 REV 01 740-046565 QH2102CX QSFP+-40G-SR4
Xcvr 2 REV 01 740-046565 QH2102DJ QSFP+-40G-SR4
PIC 1 BUILTIN BUILTIN MRATE-6xQSFPP-XGE-XLGE-CGE
Xcvr 2 REV 01 740-058732 1GTQA44202E QSFP-100GBASE-LR4
Fan Tray 0 REV 08 740-031521 ACAD7154 Enhanced Fan Tray
Fan Tray 1 REV 08 740-031521 ACAD4792 Enhanced Fan Tray
""";
pattern5 =
`{!"Item"} {!"Hardware inventory"} {item:string} {string} {string} {serialNum:string} {description:string}`;
pattern3 = `{!"Item"} {item:string} {serialNum:string} {description:string}`;
pattern4 = `{item:string} {string} {serialNum:string} {description:string}`;
fixRev(s) = replaceMatches(s, "REV 20", " REV 20");
findTokenCount(s) = length(patternMatch(s, `{string*}`));
stripSpaces(s) = replaceMatches(s, "\\s\\s\\s\\s\\b", "");
flattenBlocks(blocks) =
max(foreach x in b0]
let matches = patternMatches(blocks, pattern3)
let a = (foreach match in matches
select stripSpaces(match.line.text))
let b = (foreach match in matches
foreach child in match.line.children
select stripSpaces(child.text))
let c = (foreach match in matches
foreach child1 in match.line.children
foreach child2 in child1.children
select child2.text)
let newBlocks = a + b + c
select newBlocks);
fixBlocks(blocks) =
foreach line in blocks
select if matches(line, "Routing Engine*")
then replaceMatches(fixRev(line), "\\b\\s\\b", "-")
else replaceMatches(line, "\\b\\s\\b", "-");
normalizeBlocks(blocks) =
foreach line in blocks
let var_count = findTokenCount(line)
select if var_count == 3
then patternMatch(line, pattern3)
else if var_count == 4
then patternMatch(line, pattern4)
else if var_count == 5
then patternMatch(line, pattern5)
else null : {item: String, serialNum: String, description: String};
foreach x in b0]
let blocks = parseConfigBlocks(OS.UNKNOWN, output)
let newBlocks = flattenBlocks(blocks)
let fixBlocks = fixBlocks(newBlocks)
let normalizeBlocks = normalizeBlocks(fixBlocks)
foreach block in normalizeBlocks
select {
item: block.item, serialNo: block.serialNum, description: block.description
}
And finally now we can integrate with our snapshot after adding the custom command.
/**
* @intent Parse out item, serialNo, and description from chassis inventory
* @description Parse out item, serialNo, and description from chassis inventory
*/
command_eval = "show chassis hardware";
pattern5 =
`{!"Item"} {!"Hardware inventory"} {item:string} {string} {string} {serialNum:string} {description:string}`;
pattern3 = `{!"Item"} {item:string} {serialNum:string} {description:string}`;
pattern4 = `{item:string} {string} {serialNum:string} {description:string}`;
fixRev(s) = replaceMatches(s, "REV 20", " REV 20");
findTokenCount(s) = length(patternMatch(s, `{string*}`));
stripSpaces(s) = replaceMatches(s, "\\s\\s\\s\\s\\b", "");
flattenBlocks(blocks) =
max(foreach x in <0]
let matches = patternMatches(blocks, pattern3)
let a = (foreach match in matches
select stripSpaces(match.line.text))
let b = (foreach match in matches
foreach child in match.line.children
select stripSpaces(child.text))
let c = (foreach match in matches
foreach child1 in match.line.children
foreach child2 in child1.children
select child2.text)
let newBlocks = a + b + c
select newBlocks);
fixBlocks(blocks) =
foreach line in blocks
select if matches(line, "Routing Engine*")
then replaceMatches(fixRev(line), "\\b\\s\\b", "-")
else replaceMatches(line, "\\b\\s\\b", "-");
normalizeBlocks(blocks) =
foreach line in blocks
let var_count = findTokenCount(line)
select if var_count == 3
then patternMatch(line, pattern3)
else if var_count == 4
then patternMatch(line, pattern4)
else if var_count == 5
then patternMatch(line, pattern5)
else null : {item: String, serialNum: String, description: String};
foreach device in network.devices
where device.platform.os == OS.JUNOS
foreach output in device.outputs.commands
where output.commandText == command_eval
let blocks = parseConfigBlocks(OS.UNKNOWN, output.response)
let newBlocks = flattenBlocks(blocks)
let fixBlocks = fixBlocks(newBlocks)
let normalizeBlocks = normalizeBlocks(fixBlocks)
foreach block in normalizeBlocks
select {
item: block.item, serialNo: block.serialNum, description: block.description
}
Thanks for reading, Let us know if you have any improvements to this or if this helped you solve a problem.
