A Basic Guide to Extracting Data From Custom Commands

  • 24 April 2024
  • 7 replies
  • 184 views

Userlevel 3
Badge

The Forward Networks data model has an extensive collection of normalized vendor agnostic data for each of your devices. There are times though, when you may need a piece of information about a device that you can’t seem to find within the data model, or you need to pull information from a configuration that is not part of the running config or standard collection. This is where custom commands come in. Forward Networks allows you to add additional show commands to each snapshot to tailor the platform to your organization’s specific needs. Knowing how to pull information out of these files gives you flexibility to extract what you need to answer urgent questions, taking your data management to the next level.

 

The basic formula that will get you started

 

foreach device in network.devices
foreach command in device.outputs.commands
where command.commandText == "<YOUR COMMAND HERE>"
let parsed_response = parseConfigBlocks(device.platform.os, command.response)
let my_data = blockMatches(parsed_response, pattern)

Almost all NQE’s are going to start with foreach device in network.devices

Then we’re going to look at the outputs of all the commands Forward runs on your device.

For a command that is standard within the data model we have commandType, but for a custom command we need the commandText. All data collected from devices are stored as a text file with type string. We are going to be using the blockMatches function which uses NQE specific data type List<ConfigLine> which adds additional properties to each line of the raw text.

In order to achieve this, we use the parseConfigBlocks function that takes the device OS and the output of the command and converts it to this format.  And then we can use the pattern for the data we want to extract with the blockMatches function to collect the information needed.

 

The Sample Dataset

 

The example I’m using will be on getting a list of APs out of an Aruba WLC. The command we want to run on the controller is show ap database and the output is as follows.

 

Command: show ap database

Name Group AP Type IP Address Status Flags Switch IP Standby IP
---- ----- ------- ---------- ------ ----- --------- ----------
AP203R default 203R 192.168.1.9 Down 2 10.8.36.30 0.0.0.0
AP203RP default 203RP 192.168.1.10 Up 12h:21m:22s 2f 10.8.36.30 0.0.0.0
AP203RP default 203RP 192.168.1.8 Down N2 10.8.36.30 0.0.0.0
AP203RP-new default 203RP 192.168.1.12 Up 12h:21m:25s 2f 10.8.36.30 0.0.0.0
AP205 default 205 192.168.1.97 Down N 10.4.183.78 0.0.0.0
AP205 default 205 192.168.1.128 Down 10.4.183.78 0.0.0.0
AP207 default 207 192.168.1.4 Down 2 10.8.36.30 0.0.0.0
AP207 default 207 192.168.1.115 Down N2 10.4.183.78 0.0.0.0
AP207 default 207 192.168.1.123 Down N2 10.4.183.78 0.0.0.0
AP225 default 225 192.168.1.117 Down 2 10.4.183.78 0.0.0.0
AP303H default 303H 192.168.1.120 Down N2 10.4.183.78 0.0.0.0
AP303H default 303H 192.168.1.129 Down N2 10.4.183.78 0.0.0.0
AP303H default 303H 192.168.1.2 Up 12h:19m:13s 2 10.8.36.30 0.0.0.0
AP303H-Ash default 303H 1.1.1.2 Down Rc2 10.4.183.78 0.0.0.0
AP305 default 305 192.168.1.131 Down 2 10.4.183.78 0.0.0.0
AP315 default 315 192.168.1.100 Down 2 10.4.183.78 0.0.0.0
AP325 default 325 192.168.1.95 Down N2 10.4.183.78 0.0.0.0
AP325 default 325 192.168.1.10 Down 2 10.8.36.30 0.0.0.0
AP335 default 335 192.168.1.4 Down 2 10.8.36.30 0.0.0.0
mk-ap335 default 335 192.168.1.120 Down 2 10.4.183.78 0.0.0.0
X4 default 335 192.168.1.121 Down 2 10.4.183.78 0.0.0.0

 

The Data Pattern

 

Once we know what the output looks like, we can determine the pattern we’re going to search for. I like to start creating my block pattern variable by copying one section of the output and replacing each item with a variable. Each AP is its own line, so this one is easy. 

ap_config =```
AP203R       default  203R     192.168.1.9    Down            2      10.8.36.30   0.0.0.0
```;

After the replacement, the pattern that will catch our data will look like this:

ap_config =```
{name:string} {group:string} {type:string} {ip:ipv4Address} {status: ("Up" string | "Down")} {flag:string} {ipSwitch:ipv4Address} {ipStandby:ipv4Address}
```;

Notice that each item has a variable name and type so we can access the information later, but anything you don’t care to collect can be skipped by just adding the data type - {string} is a good catch all.

 

The Final Query

 

ap_config =```
{name:string} {group:string} {type:string} {ip:ipv4Address} {status: ("Up" string | "Down")} {flag:string} {ipSwitch:ipv4Address} {ipStandby:ipv4Address}
```;

status(stat) =
if isPresent(stat) then "Up" else "Down"
;

foreach device in network.devices
foreach command in device.outputs.commands
where command.commandText == "show ap database"
let parsed_response = parseConfigBlocks(OS.ARUBA_WIFI, command.response)
let my_data = blockMatches(parsed_response, ap_config)
foreach ap in my_data
select {
name: ap.data.name,
group: ap.data.group,
type: ap.data.type,
ip: ap.data.ip,
status: status(ap.data.status.left),
uptime: ap.data.status.left,
flags: ap.data.flag,
switch_ip: ap.data.ipSwitch,
standby_ip: ap.data.ipStandby,
controller: device.name
}

With NQE, every foreach statement takes us to a more granular level.

We want to pull the data for every AP, so the last line foreach ap in my_data and our select statement will then access the pattern variables.

 

Caveats

These are specific to this query to account for variable length output but will likely not be necessary in other queries.

Pattern matching is based on matching each item after a space. Since the status has 2 variants either with one string Down or 2 strings Up 12h:21m:22s we can use an “or” notated by the “|” to account for both possibilities.

status(stat) = 
if isPresent(stat) then "Up" else "Down"
;

This function is then called on by status: status(ap.data.status.left) to determine the output by detecting whether uptime is present. 

 

The Results

 

 

As you can see, we now have a complete list of all APs on the network and the information as it relates to them.

 

TLDR

 

This basic formula will provide you with a jumping off point when searching for data in any custom command output.

The example above searches at the data level, to search at the device level, you can create a list for every data point in the select statement.

Searching at the Device Level

pattern =```
<PATTERN YOU WANT TO MATCH - Possible variables: {name:string} {ip:ipv4Address} {other:string}>
```;

foreach device in network.devices
foreach command in device.outputs.commands
where command.commandText == "<YOUR COMMAND HERE>"
let parsed_response = parseConfigBlocks(device.platform.os, command.response)
let my_data = blockMatches(parsed_response, pattern)
select {
device: device.name,
dataName: foreach item in my_data select item.data.name,
dataIp: foreach item in my_data select item.data.ip,
dataOther: foreach item in my_data select item.data.other
}

I hope this enables you to enhance your network management by expanding the data you are able to collect from Forward Networks.

I’d love to hear if you found this useful and what custom commands you intend to run this against!


7 replies

Userlevel 1
Badge +1

Really nice post, I wonder if we can build on top of this and grab the entire matched content. 

I.e: 
String to match:
foo bar string argz bruh
foo foo foo foo foo foo foo
bar bar bar bar bar bar 

pattern defined:

{ foo foo| bar bar | foo bar | bar foo } {s: string} {random : (string*)}

would be nice to be able to get item.data.all be the entire “foo bar string argz bruh”,“foo foo foo foo foo foo foo”,”bar bar bar bar bar bar”( without having to write 3 NQE methods or define other pattern to match each separately) or the entire of the 2 pattern instead of just item.data.name or item.data.random. 

Userlevel 3
Badge

@huutho5011 Check out my series on pattern matching → 

Some is possible but as of now doing option handling can be a bit verbose.

Userlevel 1
Badge +1

@GaryB Actually, you gave me a great idea, never thought about it this way.

So for context, I am trying to do this: 

logging {"host inside"|"host"|"server"|"vrf MGMT host"|empty} {ip:ipv4Address} {misc:(string*)}


Since the environment is not 100% templatized, and this is feed into automation for decom and roll back, so we need to get the proper full line.

Looking at your code, I remember FWD only ever allow 1 pair of L/R logic in a block/pattern matches, and you all do save it as a match.data output ( as left/right).

All I need to do is reverse your shift left logic to know which of the 4 was matched( although it is a bit of a pain)

Sharing if someone else ever run into this edge case.

Userlevel 1
Badge +1

Mainly because this is a thing in IOS 

logging ip
logging ip
logging ip
logging host ip transport tra port port
logging host ip transport tra port port

Userlevel 3
Badge

@huutho5011 in this case, if you want to pull every line that starts with logging you can use

foreach device in network.devices
foreach match in patternMatches(device.files.config, `logging`)
select{
device:device.name,
match:match.line.text
}

and if you only want the logging lines that contain an ip 

pattern = `logging {(!ipv4Address string)*} {ip:ipv4Address}`;

foreach device in network.devices
foreach match in patternMatches(device.files.config, pattern)
select{
device:device.name,
match:match.line.text
}

 

Userlevel 1
Badge +1

 @AricaFN Thanks Arica, 

Mine is a bit like this, for specific use case that I have, in getting only the lines with ip and also getting the entire line for decom/roll back 

pattern=```
logging {"host inside"|"host"|"server"|"vrf MGMT host"|empty} {ip:ipv4Address} {garbo:(string*)}
```;
foreach device in network.devices
foreach command in device.outputs.commands
where command.commandType==CommandType.CONFIG
let cfg=parseConfigBlocks(OS.OTHER,command.response)
let match=(foreach match in blockMatches(cfg,pattern)
    let log_text= if isPresent(match.data.right) then ""
        else if isPresent(match.data.left?.right) then "vrf MGMT host"
              else if isPresent(match.data.left?.left?.right) then "server"
                  else if isPresent(match.data.left?.left?.left?.right) then "host"
                      else if isPresent(match.data.left?.left?.left?.left) then "host inside
                            else "not handled"
                 select "logging "+log_text +" "+toString(match.data.ip)+" "+join(" ",match.data.garbo))
//                  select match.data.ip)
where length(match)>0
select{
  deviceName:device.name,
  os: device.platform.os,
  cmd.resp: command.response,
  match:match
}
 

Userlevel 3
Badge

@huutho5011 I see, thanks for the insight! Your query is great for accessing each possibility and extracting it if you need to know each type later on. In order to achieve the result you’re looking for in this specific query, this should do the trick. It will search each line in the config to find all lines that begin with logging and have an ip address and the result will return the entire line for every match. This way will also show you where exactly it was found in the config. 

pattern = `logging {(!ipv4Address string)*} {ip:ipv4Address}`;

foreach device in network.devices
let lines = (foreach match in patternMatches(device.files.config, pattern) select match.line.text)
where length(lines) > 0
select{
device:device.name,
lines
}

 

 

Reply