I have several questions around the NQE api endpoint for running a queries. This endpoint has parameters for limit and offset for the number of records to skip and how many records to request.

(example from the documentation)

{
  "query": "foreach d in network.devices select { Name: d.name }",
  "queryId": "FQ_ac651cb2901b067fe7dbfb511613ab44776d8029",
  "commitId": "84f84b0c0a0a1805ddff0ca5451c2c55c58605e5",
  "queryOptions": {
    "offset": 20,
    "limit": 100,
    "sortBy": {
      "columnName": "Name",
      "order": "ASC"
    },
    "columnFilters": b
      {
        "columnName": "Name",
        "value": "MyDeviceName"
      }
    ]
  },
  "parameters": {
    "mtuThreshold": 123,
    "ntpServers": "
      "10.22.2.3",
      "192.33.4.1"
    ]
  }
}

I have a few questions, and would appreciate any guidance the community can provide

1. how do i know the total number of records that is available ? do i just ask for the next number of records and if none are returned then i have the full set ?
2. if i run the api for a 2nd or 3rd time, with the offset amended, will it re-run the query from scratch or are the results held somewhere in a cache.
3. is there any way to increase the amount of time allowed before a response can occur. e.g. via the API or even in the UI. I’ve noticed in the UI, for queries that do logic against ACLs, etc then it can time out. Note I’m using python (requests library)
4. Would it better if we are using the API to filter outside of the NQE query for efficiency reasons. I know there are options in the API endpoint to perform some filtering.

My use case is about finding firewall ACL/rules that match IP Addresses, using a parameterized query

// represents all networks
allNetworks = ipSubnet("0.0.0.0/0");

// which vendors are relevant
includeVendors = uVendor.CHECKPOINT];

// convert list of strings to ip addresses
queryIPAddresses(addresses:List<String>) =
foreach address in addresses
select ipAddress(address);

// returns true for each address that matches the subnet
subnetMatchIP(subnet, addresses) =
  foreach ip in addresses
  where ip in subnet
  select true;

getICMPCodes(icmptype, icmpcode) =
  min(foreach x in s0]
      let code = if icmpcode.start == 0 && icmptype.end == 255
                 then ""
                 else toString(icmpcode.start)
      let type = toString(icmptype.start)
      let response = if code == ""
                     then "ICMP-" + type
                     else "ICMP-" + type + "-" + code
      select response);

protocolLookup(protocol) =
  min(foreach x in  0]
      let protocolString = if protocol.start == 6
                           then "TCP"
                           else if protocol.start == 17
                                then "UDP"
                                else if protocol.start == 1
                                     then "ICMP"
                                     else "OTHER"
      select protocolString);

@query
getCSVRules(fromHosts: List<String>, allowAllNetworks: Bool)=
foreach x in C0]
let ipaddresses = queryIPAddresses(fromHosts)
foreach device in network.devices
where device.platform.vendor in includeVendors
foreach aclEntry in device.aclEntries
foreach src in aclEntry.headerMatches.ipv4Src
// default match src addresses, ignore all networks, unless overridden
where src != allNetworks || allowAllNetworks
foreach dst in aclEntry.headerMatches.ipv4Dst
// default match dst addresses, ignore all networks, unless overridden
where dst != allNetworks || allowAllNetworks
// make sure a match against src or dst or a rule
let matches = subnetMatchIP(dst, ipaddresses) + subnetMatchIP(src, ipaddresses)
// more than one match means the rule is valid
where length(matches) >= 1
foreach protocol in aclEntry.headerMatches.ipProtocol
foreach port in aclEntry.headerMatches.tpDst
let protocolString = protocolLookup(protocol)
let portString = if protocolString == "ICMP"
                 then getICMPCodes(min(aclEntry.headerMatches.icmpType), min(aclEntry.headerMatches.icmpCode))
                 else if port.start == port.end
                      then protocolString + "-" + toString(port.start)
                      else protocolString + "-" + toString(port.start) + "-" +
                           toString(port.end)
select distinct {
  deviceName: device.name,  
  name: aclEntry.name,
  sources: src,
  destinations: dst,
  ports: portString,
  action: aclEntry.action
};

so i know the NQE above is a little on the complex side, as i wanted to make the output nice for a basic demo. I’ll happily remove this if i’m doing this inside python. The main challenge I would think is the number of iterations generated as I turn each access rule into it’s atomic components. This equates to a number of lines directly related to the multiplication of sources * destinations * protocol * ports, which is a huge number of permutations when consider a large estate.

Hence asking for some guidance here, as the functionality above is could be really useful.

For example, requestor wants to replace a server, that is in security controls. They may not know where though. However we can identify all the existing places where such a server (or sets of servers) are configured using this query. (Please note I think you can remove the filtering for CHECK POINT, etc and cover all security type controls for network devices).

Another optimisation i thought about if using the API endpoint to grab the data would to feed the script the device or devices to check acl’s against, allowing us to reduce the amount of data being queried in one NQE query.