Do blockMatches require consistent line patterns?

Hi,

please check my posts how to start with Parsers:

it explains how to take the texts blob and make it parsable. 

Regarding parser structure, there is your exact example described in part 2 

I hope it helps.

In your case, you actually have two pattern.

The first one is when subline 12 not present: 

stats_pattern1 =
```
line one {main:string}
    sub-line3 {value1:string} is target and sub-line3 will always be there
    sub-line8 {value2:string} is target and sub-line8 will always be there
         sub-sub-line9 {value3:string} is target and sub-sub-line9 will always be there    
```;

The second one of subline 12 is present:

stats_pattern2 =
```
line one {main:string}
    sub-line3 {value1:string} is target and sub-line3 will always be there
    sub-line8 {value2:string} is target and sub-line8 will always be there
         sub-sub-line9 {value3:string} is target and sub-sub-line9 will always be there    
    sub-line12 {value4:string} is target and may or may not be present
         sub-sub-line13 {value5:string} is target and sub-sub-line13 will be there if sub-line12 is present
```;

The reason you need two pattern is that in your select statement you are asking for the output of value 4 and value 5 that do not exist in stats_pattern1:

select {
  main: r.data.main,
  value1: r.data.value1,
  value2: r.data.value2,
  value3: r.data.value3,
  value4: r.data.value4,
  value5: r.data.value5
}

additionally you have to replace:

let  my_data = blockMatches(parsed_resonse, pattern)

with 

foreach r in blockMatches(parsed_respone, stats_pattern)

to allow NQE to iterate over each line of block code.

Here is your NQE with both patterns that matches subline 12 pattern:

stats_pattern1 =
```
line one {main:string}
    sub-line3 {value1:string} is target and sub-line3 will always be there
    sub-line8 {value2:string} is target and sub-line8 will always be there
         sub-sub-line9 {value3:string} is target and sub-sub-line9 will always be there    
```;
stats_pattern2 =
```
line one {main:string}
    sub-line3 {value1:string} is target and sub-line3 will always be there
    sub-line8 {value2:string} is target and sub-line8 will always be there
         sub-sub-line9 {value3:string} is target and sub-sub-line9 will always be there    
    sub-line12 {value4:string} is target and may or may not be present
         sub-sub-line13 {value5:string} is target and sub-sub-line13 will be there if sub-line12 is present
```;

foreach device in network.devices
foreach command in device.outputs.commands
where command.commandText == "show my lines"
let parsed_respone = parseConfigBlocks(OS.UNKNOWN, command.response)
foreach r in blockMatches(parsed_respone, stats_pattern2)

select {
  main: r.data.main,
  value1: r.data.value1,
  value2: r.data.value2,
  value3: r.data.value3,
  value4: r.data.value4,
  value5: r.data.value5
}

Thank you. I’ll study this today and follow-up with “Best answer”

I apologize for the delayed response; I haven’t have time to get this to work with both patterns.  Pattern2 runs fine when there is a match but if not if value 4 and value 5 are missing, i.e. matches pattern1.  I hope to have some time this week to research the solution for myself. Thank you.

Hi @dakotaglory ,

try this one. It addresses the missing value 4 and 5

stats_pattern =
  ```
line one {main:string}
  sub-line3 {value1:string} is target and sub-line3 will always be there
  sub-line8 {value2:string} is target and sub-line8 will always be there
    sub-sub-line9 {value3:string} is target and sub-sub-line9 will always be there
```;

stats_pattern2 =
  ```
line one {main:string}
  sub-line3 {value1:string} is target and sub-line3 will always be there
  sub-line8 {value2:string} is target and sub-line8 will always be there
    sub-sub-line9 {value3:string} is target and sub-sub-line9 will always be there
  sub-line12 {value4:string} is target and may or may not be present
    sub-sub-line13 {value5:string} is target and sub-sub-line13 will be there if sub-line12 is present
```;

foreach device in network.devices
foreach command in device.outputs.commands
where command.commandText == "show my lines"
let parsed_respone = parseConfigBlocks(OS.UNKNOWN, command.response)
foreach r in blockMatches(parsed_respone, stats_pattern)
let rPlus = max(foreach match in blockMatches(parsed_respone, stats_pattern2)
            where match.data.main == r.data.main
            select match)
select {
  main: r.data.main,
  value1: r.data.value1,
  value2: r.data.value2,
  value3: r.data.value3,
  value4: rPlus?.data?.value4,
  value5: rPlus?.data?.value5
}

credits for this enhancement is to @Tyson Henrie 

Thank you Tyson

can you post the entirety of the NQE with the enhancement in 1 code block?

Hi @CarlB 

Can you use the code block above?

having to modify to match my environment and hitting the usual challenges.  Is that final block 100% with all replacements/modifications?  

@CarlB That example is fairly generic.  If you are having trouble adapting that to your use case.  Can you share a sanitized example of the data file you are trying to parse and the query you are currently using? That might be good as a new post.

I think I have it working.

NHRPnoFWpattern =

  ```

{IPandPrefix:string} via {VPNpeer: ipv4Address}

  {tunnelnum: string} created

  NBMA address: {TunPublic: ipv4Address}

```;

NHRPwithNATpattern =

  ```

{IPandPrefix:string} via {VPNpeer: ipv4Address}

  {tunnelnum: string} created

  NBMA address: {TunPublic: ipv4Address}

   {string} NBMA address: {NoNATtunPublic: string}

```;

cleanParen(s) = cleanLParen(cleanRParen(s));

cleanLParen(s) = replaceMatches(s, "(", "");

cleanRParen(s) = replaceMatches(s, ")", "");

foreach device in network.devices

foreach command in device.outputs.commands

  where command.commandType == CommandType.NHRP_STATE

foreach x in [1]

//let ParsedResponse = parseConfigBlocks(OS.UNKNOWN, command.response)

let ParsedResponse = parseConfigBlocks(OS.UNKNOWN, cleanParen(command.response))

foreach r in blockMatches(ParsedResponse, NHRPnoFWpattern)

let rPlus = max(foreach match in blockMatches(ParsedResponse, NHRPwithNATpattern)

            where match.data.IPandPrefix == r.data.IPandPrefix

            select match)

select {

  Prefix: r.data.IPandPrefix,

  IP: r.data.VPNpeer,

  ARIN: r.data.TunPublic,

  LAN: rPlus?.data?.NoNATtunPublic

}

​@CarlB going to use the code embedding so its easier for others to read. Thanks for posting!!

 

NHRPnoFWpattern =

  ```

{IPandPrefix:string} via {VPNpeer: ipv4Address}

  {tunnelnum: string} created

  NBMA address: {TunPublic: ipv4Address}

```;



NHRPwithNATpattern =

  ```

{IPandPrefix:string} via {VPNpeer: ipv4Address}

  {tunnelnum: string} created

  NBMA address: {TunPublic: ipv4Address}

   {string} NBMA address: {NoNATtunPublic: string}

```;



cleanParen(s) = cleanLParen(cleanRParen(s));



cleanLParen(s) = replaceMatches(s, "(", "");



cleanRParen(s) = replaceMatches(s, ")", "");




foreach device in network.devices

foreach command in device.outputs.commands

  where command.commandType == CommandType.NHRP_STATE



foreach x in [1]

//let ParsedResponse = parseConfigBlocks(OS.UNKNOWN, command.response)

let ParsedResponse = parseConfigBlocks(OS.UNKNOWN, cleanParen(command.response))

foreach r in blockMatches(ParsedResponse, NHRPnoFWpattern)

let rPlus = max(foreach match in blockMatches(ParsedResponse, NHRPwithNATpattern)

            where match.data.IPandPrefix == r.data.IPandPrefix

            select match)



select {

  Prefix: r.data.IPandPrefix,

  IP: r.data.VPNpeer,

  ARIN: r.data.TunPublic,

  LAN: rPlus?.data?.NoNATtunPublic

}