Best way to start writing a Parser - Part 1

Are you new to NQE?
Do you need to write a pattern match parser but don't know where to start?
Do you want to test your pattern match without running it against your snapshot, because it takes too long for each iteration?

Well, I have been there and here's what I've learned.

If you start writing a parser, you'll most likely run into multiple iterations that won't give you the desired results. If you're as impatient as me, you'd like to get your test results quickly back so you can adjust the parser.

To do this, you can:

1. Take a sample of the config or device output and store it in a variable.
2. Write the parser to run against this sample.
3. Check if the result is what you expect.
4. Modify the parser and run step 3 until you get your expected results.

Well, that sounds easy enough! But how do you actually do each of these steps?

Here's an example.

For example, lets assume you have to lookup information from the "show interfaces" command on Cisco devices, something like this:

GigabitEthernet0/0 is up, line protocol is up (connected)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 25w3d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/64/0 (size/max total/threshold/drops) 
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 4/4 (allocated/max allocated)
     Available Bandwidth 0 kilobits/sec
  5 minute input rate 4000 bits/sec, 6 packets/sec
  5 minute output rate 3000 bits/sec, 4 packets/sec
     35659777 packets input, 3682764911 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 33245532 multicast, 0 pause input
     0 input packets with dribble condition detected
     19564830 packets output, 2596316333 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

You can define the “show interfaces” output as a variable in following format:

<variable name> = """ <content> """;

In our example it would look like this:

test_string = 
"""
GigabitEthernet0/0 is up, line protocol is up (connected)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 25w3d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/64/0 (size/max total/threshold/drops) 
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 4/4 (allocated/max allocated)
     Available Bandwidth 0 kilobits/sec
  5 minute input rate 4000 bits/sec, 6 packets/sec
  5 minute output rate 3000 bits/sec, 4 packets/sec
     35659777 packets input, 3682764911 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 33245532 multicast, 0 pause input
     0 input packets with dribble condition detected
     19564830 packets output, 2596316333 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/1 is up, line protocol is up (connected)
  MTU 1501 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 25w3d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/64/0 (size/max total/threshold/drops) 
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 4/4 (allocated/max allocated)
     Available Bandwidth 0 kilobits/sec
  5 minute input rate 4000 bits/sec, 6 packets/sec
  5 minute output rate 3000 bits/sec, 4 packets/sec
     35659777 packets input, 3682764911 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 33245532 multicast, 0 pause input
     0 input packets with dribble condition detected
     19564830 packets output, 2596316333 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
""";

The content of test_string can be way longer, but the variable definition stays the same.

The task is to lookup all interfaces and to list all MTU settings.

If you look at the "show interface" data, the Information you need is in following two lines:

GigabitEthernet0/0 is up, line protocol is up (connected)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 

Let’s write our first pattern match parser:

To make it easy, we would like to list all interface names.

In the first line we have the interface and the interface status with link and line protocol.

If we want to list all interfaces, we have to tell the parser that the value for interface can change.

The parser format looks line this:

<parser name> = ``` <parser content> ```;

our parser will look like this:

interface_parser = 
```
{string} is up, line protocol is up (connected)
```;

Now with this parser, it will match all interfaces, but you will not get any results, because {string} changes with every interface, but it is not stored in a variable.

In order to store it in a variable, the format looks like this:

{<variable name>:string}

Let's create a variable "int" to get the interface name.

The new parser looks like this:

interface_parser = 
```
{int:string} is up, line protocol is up (connected)
```;

Great, we have out first parser!

Let's test it out.

Now we come to the next important three lines of code:

foreach x in r1]
let lines = parseConfigBlocks(OS.UNKNOWN, test_string)
foreach line in blockMatches(lines, interface_parser2)

What do they mean?

foreach x in r1]

NQE is a Data structure query language. therefore it requires to itterate over a set of data.

In our case we do not iterate over a data model, but a single variable.

Since NQE still needs to iterate over something, it just needs that expression.

next line of code:

let lines = parseConfigBlocks(OS.UNKNOWN, test_string)

when we defined the variable test_string, we just pasted a blob of text into it.

with this code, we create a variable lines, that will parste the blob "test_string" and format it as lines.

next line of code:

foreach line in blockMatches(lines, interface_parser)

here we do the pattern match parsing. We compare the content of lines with the pattern defined in interface_parser

Each matched content is being stored into the variable "line" as a list of data.

Great. now lets generate some output:

select {
 Interface: line.data.int   
}

Here is the NQE without the test_string content:

test_string =
"""
<snip content>
""";

interface_parser = 
```
{int:string} is up, line protocol is up (connected)
```;

foreach x in >1]
let lines = parseConfigBlocks(OS.UNKNOWN, test_string)
foreach line in blockMatches(lines, interface_parser2)

select {
 Interface: line.data.int   
}

Great, you got your first parser test environment running.

But, wait!

We see only interfaces that are up? Why?

well, because the parser is looking for:

{int:string} is up, line protocol is up (connected)

If we want to list all interfaces including those that are down or in any other state, we can change the parser to:

interface_parser =

```

{int:string} is {string*}

```;

or

interface_parser =

```

{int:string} is

```;

Both will list all interfaces.

Here the final NQE:

test_string =
"""
GigabitEthernet0/0 is up, line protocol is up (connected)
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 25w3d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/64/0 (size/max total/threshold/drops) 
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 4/4 (allocated/max allocated)
     Available Bandwidth 0 kilobits/sec
  5 minute input rate 4000 bits/sec, 6 packets/sec
  5 minute output rate 3000 bits/sec, 4 packets/sec
     35659777 packets input, 3682764911 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 33245532 multicast, 0 pause input
     0 input packets with dribble condition detected
     19564830 packets output, 2596316333 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
GigabitEthernet0/1 is up, line protocol is up (connected)
  MTU 1501 bytes, BW 1000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 25w3d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/64/0 (size/max total/threshold/drops) 
     Conversations  0/2/256 (active/max active/max total)
     Reserved Conversations 4/4 (allocated/max allocated)
     Available Bandwidth 0 kilobits/sec
  5 minute input rate 4000 bits/sec, 6 packets/sec
  5 minute output rate 3000 bits/sec, 4 packets/sec
     35659777 packets input, 3682764911 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 33245532 multicast, 0 pause input
     0 input packets with dribble condition detected
     19564830 packets output, 2596316333 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
""";
interface_parser = 
```
{int:string} is 
```;
foreach x in r1]
let lines = parseConfigBlocks(OS.UNKNOWN, test_string)
foreach line in blockMatches(lines, interface_parser)
select {
 Interface: line.data.int   
}

In part 2, I show how to parse the output for MTU length and CRC errors.