Skip to main content

A Basic Guide to Extracting Data From Custom Commands

  • April 24, 2024
  • 14 replies
  • 967 views
  • Translate

AricaFN
Employee
Forum|alt.badge.img+2

The Forward Networks data model has an extensive collection of normalized vendor agnostic data for each of your devices. There are times though, when you may need a piece of information about a device that you can’t seem to find within the data model, or you need to pull information from a configuration that is not part of the running config or standard collection. This is where custom commands come in. Forward Networks allows you to add additional show commands to each snapshot to tailor the platform to your organization’s specific needs. Knowing how to pull information out of these files gives you flexibility to extract what you need to answer urgent questions, taking your data management to the next level.

 

The basic formula that will get you started

 

foreach device in network.devices
foreach command in device.outputs.commands
where command.commandText == "<YOUR COMMAND HERE>"
let parsed_response = parseConfigBlocks(device.platform.os, command.response)
let my_data = blockMatches(parsed_response, pattern)

Almost all NQE’s are going to start with foreach device in network.devices

Then we’re going to look at the outputs of all the commands Forward runs on your device.

For a command that is standard within the data model we have commandType, but for a custom command we need the commandText. All data collected from devices are stored as a text file with type string. We are going to be using the blockMatches function which uses NQE specific data type List<ConfigLine> which adds additional properties to each line of the raw text.

In order to achieve this, we use the parseConfigBlocks function that takes the device OS and the output of the command and converts it to this format.  And then we can use the pattern for the data we want to extract with the blockMatches function to collect the information needed.

 

The Sample Dataset

 

The example I’m using will be on getting a list of APs out of an Aruba WLC. The command we want to run on the controller is show ap database and the output is as follows.

 

Command: show ap database

Name         Group    AP Type  IP Address     Status          Flags  Switch IP    Standby IP
----         -----    -------  ----------     ------          -----  ---------    ----------
AP203R       default  203R     192.168.1.9    Down            2      10.8.36.30   0.0.0.0
AP203RP      default  203RP    192.168.1.10   Up 12h:21m:22s  2f     10.8.36.30   0.0.0.0
AP203RP      default  203RP    192.168.1.8    Down            N2     10.8.36.30   0.0.0.0
AP203RP-new  default  203RP    192.168.1.12   Up 12h:21m:25s  2f     10.8.36.30   0.0.0.0
AP205        default  205      192.168.1.97   Down            N      10.4.183.78  0.0.0.0
AP205        default  205      192.168.1.128  Down                   10.4.183.78  0.0.0.0
AP207        default  207      192.168.1.4    Down            2      10.8.36.30   0.0.0.0
AP207        default  207      192.168.1.115  Down            N2     10.4.183.78  0.0.0.0
AP207        default  207      192.168.1.123  Down            N2     10.4.183.78  0.0.0.0
AP225        default  225      192.168.1.117  Down            2      10.4.183.78  0.0.0.0
AP303H       default  303H     192.168.1.120  Down            N2     10.4.183.78  0.0.0.0
AP303H       default  303H     192.168.1.129  Down            N2     10.4.183.78  0.0.0.0
AP303H       default  303H     192.168.1.2    Up 12h:19m:13s  2      10.8.36.30   0.0.0.0
AP303H-Ash   default  303H     1.1.1.2        Down            Rc2    10.4.183.78  0.0.0.0
AP305        default  305      192.168.1.131  Down            2      10.4.183.78  0.0.0.0
AP315        default  315      192.168.1.100  Down            2      10.4.183.78  0.0.0.0
AP325        default  325      192.168.1.95   Down            N2     10.4.183.78  0.0.0.0
AP325        default  325      192.168.1.10   Down            2      10.8.36.30   0.0.0.0
AP335        default  335      192.168.1.4    Down            2      10.8.36.30   0.0.0.0
mk-ap335     default  335      192.168.1.120  Down            2      10.4.183.78  0.0.0.0
X4           default  335      192.168.1.121  Down            2      10.4.183.78  0.0.0.0

 

The Data Pattern

 

Once we know what the output looks like, we can determine the pattern we’re going to search for. I like to start creating my block pattern variable by copying one section of the output and replacing each item with a variable. Each AP is its own line, so this one is easy. 

ap_config =```
AP203R       default  203R     192.168.1.9    Down            2      10.8.36.30   0.0.0.0
```; 

After the replacement, the pattern that will catch our data will look like this:

ap_config =```
{name:string} {group:string} {type:string} {ip:ipv4Address} {status: ("Up" string | "Down")} {flag:string} {ipSwitch:ipv4Address} {ipStandby:ipv4Address}
```;

Notice that each item has a variable name and type so we can access the information later, but anything you don’t care to collect can be skipped by just adding the data type - {string} is a good catch all.

 

The Final Query

 

ap_config =```
{name:string} {group:string} {type:string} {ip:ipv4Address} {status: ("Up" string | "Down")} {flag:string} {ipSwitch:ipv4Address} {ipStandby:ipv4Address}
```;

status(stat) = 
if isPresent(stat) then "Up" else "Down"
;

foreach device in network.devices
foreach command in device.outputs.commands
where command.commandText == "show ap database"
let parsed_response = parseConfigBlocks(OS.ARUBA_WIFI, command.response)
let my_data = blockMatches(parsed_response, ap_config)
foreach ap in my_data
select { 
  name: ap.data.name,
  group: ap.data.group,
  type: ap.data.type,
  ip: ap.data.ip,
  status: status(ap.data.status.left),
  uptime: ap.data.status.left,
  flags: ap.data.flag,
  switch_ip: ap.data.ipSwitch,
  standby_ip: ap.data.ipStandby,
  controller: device.name
}

With NQE, every foreach statement takes us to a more granular level.

We want to pull the data for every AP, so the last line foreach ap in my_data and our select statement will then access the pattern variables.

 

Caveats

These are specific to this query to account for variable length output but will likely not be necessary in other queries.

Pattern matching is based on matching each item after a space. Since the status has 2 variants either with one string Down or 2 strings Up 12h:21m:22s we can use an “or” notated by the “|” to account for both possibilities.

status(stat) = 
if isPresent(stat) then "Up" else "Down"
;

This function is then called on by status: status(ap.data.status.left) to determine the output by detecting whether uptime is present. 

 

The Results

 

 

As you can see, we now have a complete list of all APs on the network and the information as it relates to them.

 

TLDR

 

This basic formula will provide you with a jumping off point when searching for data in any custom command output.

The example above searches at the data level, to search at the device level, you can create a list for every data point in the select statement.

Searching at the Device Level

pattern =```
<PATTERN YOU WANT TO MATCH - Possible variables: {name:string} {ip:ipv4Address} {other:string}>
```;

foreach device in network.devices
foreach command in device.outputs.commands
where command.commandText == "<YOUR COMMAND HERE>"
let parsed_response = parseConfigBlocks(device.platform.os, command.response)
let my_data = blockMatches(parsed_response, pattern)
select { 
  device: device.name,
  dataName: foreach item in my_data select item.data.name,
  dataIp: foreach item in my_data select item.data.ip,
  dataOther: foreach item in my_data select item.data.other
}

I hope this enables you to enhance your network management by expanding the data you are able to collect from Forward Networks.

I’d love to hear if you found this useful and what custom commands you intend to run this against!

14 replies

Forum|alt.badge.img+1
  • Ramping Up
  • 6 replies
  • April 26, 2024

Really nice post, I wonder if we can build on top of this and grab the entire matched content. 

I.e: 
String to match:
foo bar string argz bruh
foo foo foo foo foo foo foo
bar bar bar bar bar bar 

pattern defined:

{ foo foo| bar bar | foo bar | bar foo } {s: string} {random : (string*)}

would be nice to be able to get item.data.all be the entire “foo bar string argz bruh”,“foo foo foo foo foo foo foo”,”bar bar bar bar bar bar”( without having to write 3 NQE methods or define other pattern to match each separately) or the entire of the 2 pattern instead of just item.data.name or item.data.random. 

Translate

Forum|alt.badge.img+1
  • Employee
  • 55 replies
  • April 26, 2024

@huutho5011 Check out my series on pattern matching → 

Some is possible but as of now doing option handling can be a bit verbose.

Translate

Forum|alt.badge.img+1
  • Ramping Up
  • 6 replies
  • April 26, 2024

@GaryB Actually, you gave me a great idea, never thought about it this way.

So for context, I am trying to do this: 

logging {"host inside"|"host"|"server"|"vrf MGMT host"|empty} {ip:ipv4Address} {misc:(string*)}


Since the environment is not 100% templatized, and this is feed into automation for decom and roll back, so we need to get the proper full line.

Looking at your code, I remember FWD only ever allow 1 pair of L/R logic in a block/pattern matches, and you all do save it as a match.data output ( as left/right).

All I need to do is reverse your shift left logic to know which of the 4 was matched( although it is a bit of a pain)

Sharing if someone else ever run into this edge case.

Translate

Forum|alt.badge.img+1
  • Ramping Up
  • 6 replies
  • April 26, 2024

Mainly because this is a thing in IOS 

logging ip
logging ip
logging ip
logging host ip transport tra port port
logging host ip transport tra port port

Translate

AricaFN
Employee
Forum|alt.badge.img+2
  • Author
  • Employee
  • 13 replies
  • April 29, 2024

@huutho5011 in this case, if you want to pull every line that starts with logging you can use

foreach device in network.devices
foreach match in patternMatches(device.files.config, `logging`)
select{
  device:device.name,
  match:match.line.text
}

and if you only want the logging lines that contain an ip 

pattern = `logging {(!ipv4Address string)*} {ip:ipv4Address}`;

foreach device in network.devices
foreach match in patternMatches(device.files.config, pattern)
select{
  device:device.name,
  match:match.line.text
}

 

Translate

Forum|alt.badge.img+1
  • Ramping Up
  • 6 replies
  • April 30, 2024

 @AricaFN Thanks Arica, 

Mine is a bit like this, for specific use case that I have, in getting only the lines with ip and also getting the entire line for decom/roll back 

pattern=```
logging {"host inside"|"host"|"server"|"vrf MGMT host"|empty} {ip:ipv4Address} {garbo:(string*)}
```;
foreach device in network.devices
foreach command in device.outputs.commands
where command.commandType==CommandType.CONFIG
let cfg=parseConfigBlocks(OS.OTHER,command.response)
let match=(foreach match in blockMatches(cfg,pattern)
    let log_text= if isPresent(match.data.right) then ""
        else if isPresent(match.data.left?.right) then "vrf MGMT host"
              else if isPresent(match.data.left?.left?.right) then "server"
                  else if isPresent(match.data.left?.left?.left?.right) then "host"
                      else if isPresent(match.data.left?.left?.left?.left) then "host inside
                            else "not handled"
                 select "logging "+log_text +" "+toString(match.data.ip)+" "+join(" ",match.data.garbo))
//                  select match.data.ip)
where length(match)>0
select{
  deviceName:device.name,
  os: device.platform.os,
  cmd.resp: command.response,
  match:match
}
 

Translate

AricaFN
Employee
Forum|alt.badge.img+2
  • Author
  • Employee
  • 13 replies
  • May 2, 2024

@huutho5011 I see, thanks for the insight! Your query is great for accessing each possibility and extracting it if you need to know each type later on. In order to achieve the result you’re looking for in this specific query, this should do the trick. It will search each line in the config to find all lines that begin with logging and have an ip address and the result will return the entire line for every match. This way will also show you where exactly it was found in the config. 

pattern = `logging {(!ipv4Address string)*} {ip:ipv4Address}`;

foreach device in network.devices
let lines = (foreach match in patternMatches(device.files.config, pattern) select match.line.text)
where length(lines) > 0
select{
  device:device.name,
  lines
}

 

 

Translate

cariddir
Spotter
Forum|alt.badge.img+4
  • Spotter
  • 24 replies
  • May 28, 2024

Arica and @huutho5011 
This is exactly what I was hoping for this community to be!

I’ve been trying to solve for this kind of ‘config sprawl’  for a few months.  huutho5011, we are in the same boat, of ‘not being templatized’ and trying to find the outliers has been difficult with the simple checks, b/c there is so many variation, examples below. Some using VRF’s, some using the wrong UDP/TCP port etc… 

The combo of the examples you guys put together have gotten me closer. 
 

I would like to see which devices have ‘NO LOGGING configured’, but I’m not sure that’s a simple add-on to this. But for now, at least I can export and do some excel manipulations to find the misconfigured devices.


Thanks for the work!
Rich

Translate

Tyson Henrie
Employee
Forum|alt.badge.img+2

Rich,

If you use “foreach” or “where” to see which devices have logging configured, then you don’t get to see the devices that “don’t” match.  That is where “let” statements are useful.  The “let” does not filter out data.  It just files in a variable.

You could do something like

pattern01 = ```
logging server {serverIp:ipv4Address}
```;

pattern02 = ```
logging host {serverIp:ipv4Address}
```;

pattern03 = ```
logging {serverIp:ipv4Address }
```;

foreach device in network.devices
where device.platform.vendor == Vendor.CISCO
let logServ = blockMatches(device.files.config, pattern01)
let logHost = blockMatches(device.files.config, pattern02)
let logIp = blockMatches(device.files.config, pattern03)
let logging = logServ + logHost + logIp
let violation = length(logging) == 0 && device.snapshotInfo.result == DeviceSnapshotResult.completed
select {
  violation: violation,
  device: device.name,
  servers: (foreach record in logging
            select record.data.serverIp),
  logging: (foreach record in logging
            select record.blocks),
  Collection_Status: device.snapshotInfo.result
}

It is not beautiful.  I didn’t work too hard on how the output looks.  But it is a way to view the data.  The important part is that now you can choose “true” for violation and see the devices that are NOT configured with a logging server.

-Tyson

Translate

Tyson Henrie
Employee
Forum|alt.badge.img+2

Another one is to use the query Arica put above, but remove the “where” clause

pattern = `logging {(!ipv4Address string)*} {ip:ipv4Address}`;

foreach device in network.devices
let lines = (foreach match in patternMatches(device.files.config, pattern) select match.line.text)
==>> where length(lines) > 0 <<== remove this
select{
  device:device.name,
  lines
}

Then it is useful to add a “violation” to quickly sort to the devices that are incorrect.

-Tyson

Translate

Tyson Henrie
Employee
Forum|alt.badge.img+2

There are other things you can do if you have an “approved” list of logging servers.  You can make a list of “found servers”.  Then you something like

let missingServers = approved - foundServers
let extraServer = foundServers - approved

 

Then you can add those to a violation like:

violationMissing = length(missingServers) > 0,
violationExtra = length(extraServers) > 0

Just to illustrate that you can have more than one violation column.  If you make it a Verification check, then it is really only useful to have a single violation.  but for investigation it is useful to create multiple violation columns.

To make them a single violation you could do something like:

let violation = length(logging) == 0 && device.snapshotInfo.result == DeviceSnapshotResult.completed || length(missingServers) > 0 || length(extraServers) > 0

Sorry for a bunch of messages.  I had a few different thoughts come up.

-Tyson

Translate

cariddir
Spotter
Forum|alt.badge.img+4
  • Spotter
  • 24 replies
  • May 29, 2024
Tyson Henrie wrote:

Another one is to use the query Arica put above, but remove the “where” clause

pattern = `logging {(!ipv4Address string)*} {ip:ipv4Address}`;

foreach device in network.devices
let lines = (foreach match in patternMatches(device.files.config, pattern) select match.line.text)
==>> where length(lines) > 0 <<== remove this
select{
  device:device.name,
  lines
}

Then it is useful to add a “violation” to quickly sort to the devices that are incorrect.

-Tyson

I’d like to Fist Bump this. one…. THANK YOU!. Perfect

Translate

cariddir
Spotter
Forum|alt.badge.img+4
  • Spotter
  • 24 replies
  • May 29, 2024

Thanks Tyson/Arica et al - 

 

This is what I used with all different permutations of ‘logging’ in our environment. And it worked perfectly - 

pattern = `logging {"host "|"host inside"|"host"|"server"|"vrf MGMT host"|empty} {ip:ipv4Address} {misc:(string*)}`;

foreach device in network.devices
let lines = (foreach match in patternMatches(device.files.config, pattern) select match.line.text)
//where length(lines) > 0
select{
  device:device.name,
  Platform: device.platform.os,
  lines
}

 

Translate

cariddir
Spotter
Forum|alt.badge.img+4
  • Spotter
  • 24 replies
  • May 29, 2024
Tyson Henrie wrote:

There are other things you can do if you have an “approved” list of logging servers.  You can make a list of “found servers”.  Then you something like

let missingServers = approved - foundServers
let extraServer = foundServers - approved

 

Then you can add those to a violation like:

violationMissing = length(missingServers) > 0,
violationExtra = length(extraServers) > 0

Just to illustrate that you can have more than one violation column.  If you make it a Verification check, then it is really only useful to have a single violation.  but for investigation it is useful to create multiple violation columns.

To make them a single violation you could do something like:

let violation = length(logging) == 0 && device.snapshotInfo.result == DeviceSnapshotResult.completed || length(missingServers) > 0 || length(extraServers) > 0

Sorry for a bunch of messages.  I had a few different thoughts come up.

-Tyson

I’ll try this as well, if I can nail down a list in 4 global regions x 6 different options of servers ;-)

Translate

Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings