Forward NQE includes several under-the-hood optimizations that can improve query performance. This post focuses on the _lookup optimization, which can reduce execution time when filtering on a single key field using an exact equality comparison. Leveraging _lookup is considered a best practice whenever possible.

In a recent query I wrote joining device CVE information with the EPSS database in a network containing 10,000 devices, re-writing the NQE to take advantage of _lookup reduced the execution time by over 50%. While this kind of optimization isn’t as dramatic as parallelization—where a multi-minute query might drop to seconds—it can still offer a significant and worthwhile improvement, particularly in queries with repeated key-based lookups over large lists.

What _lookup Does

Normally, when you write a loop in NQE that scans a list for a match, the query engine will iterate over every item in that list and check the condition. This can be slow if the list or number of devices in the network is large.

When a query uses a foreach on a list of records followed by a where clause that filters that list using an exact equality comparison on a single field, NQE may rewrite the loop into a _lookup operation. This changes the behavior from scanning the entire list for each match to performing a direct, indexed lookup.

 The index used for this optimization comes from two possible sources:

• Data model lists that already have a predefined index on that field (key fields are not exposed in the UI).
• User-defined lists that are saved to a variable and iterated over more than once by that same field, in which case NQE will automatically build an index.

Building the index requires one initial pass over the list (O(n)), but all subsequent accesses are O(1). This makes repeated key-based lookups significantly faster, especially when the list is large or the same field is queried multiple times.

How to Tell If _lookup Is Happening

You can see whether _lookup is being used by looking at the Query Debug Panel by holding either alt(windows) or option(mac) and left clicking on Execute from the NQE IDE. If you see _lookup(...) in the Query Debug Panel, the optimization is being leveraged for that section of the query.

The Query Debug Panel also displays useful details such as execution duration, allowing you to see the exact NQE run time and compare how different changes to the query impact performance.

If you don’t see _lookup, then the engine is performing a full scan.

Simple Example: No _lookup Optimization

Using a function like matches() or any non-exact match breaks the optimization:

Syntax

foreach device in network.devices

foreach interface in device.interfaces

where matches(interface.name, "ge-0/0/0")

select { device: device.name, interface: interface.name }

Query Debug Panel

_lookup does not exist in the below syntax, indicating that the optimization is not being leveraged.

@juniper(INDENTATION);



parallel_foreach device in network.devices

⟪foreach interface in ⟪device.interfaces⟫

 where @builtin/matches(interface.name, "ge-0/0/0")

 select ⟪{ device: ⟪⟪device⟫.name⟫, interface: ⟪⟪interface⟫.name⟫ }⟫⟫

Simple Example: _lookup Optimization Enabled

Switching to a simple equality check on a unique key enables _lookup:

Query

foreach device in network.devices

foreach interface in device.interfaces

where interface.name == "ge-0/0/0"

select { device: device.name, interface: interface.name }

Query Debug Panel

@juniper(INDENTATION);



parallel_foreach device in network.devices

⟪foreach interface in _lookup(device.interfaces, "name", "ge-0/0/0")

 select ⟪{ device: ⟪⟪device⟫.name⟫, interface: ⟪⟪interface⟫.name⟫ }⟫⟫

Here, _lookup means the engine jumps directly to the matching interface instead of scanning all interfaces.

Advanced Example: _lookup Optimization in Larger Queries

The _lookup optimization isn’t just for simple interface-name matches — it also applies in larger, more complex queries when you’re joining one dataset to another via a unique key using ==. In fact, it can make a huge difference in scenarios where you’re performing the same lookup for many records.

Here’s an example using EPSS data which is a list 

Optimized Version (with _lookup)

EPSS = network.externalSources.first_org.ePSS_LATEST;



foreach device in network.devices

foreach cveFinding in device.cveFindings

let epssRecord = max(foreach epssRecord in EPSS

                     where epssRecord.cve == cveFinding.cveId

                     select epssRecord)

select {

  deviceName: device.name,

  cveFindingCveId: cveFinding.cveId,

  isVulnerable: cveFinding.isVulnerable,

  basis: cveFinding.basis,

  "EPSS Score": epssRecord?.epss,

  "EPSS Percentile": epssRecord?.percentile,

  "EPSS Date": epssRecord?.date

}

Query Debug Panel

parallel_foreach device in network.devices

⟪foreach cveFinding in ⟪device.cveFindings⟫

 let epssRecord = @builtin/max(

   foreach epssRecord

     in _lookup(./EPSS, "cve", cveFinding.cveId)

   select epssRecord)

 select { ... }

Not Optimized

If you wrap the same logic in a function:

EPSS = network.externalSources.first_org.ePSS_LATEST;

getEPSSRecord(cveId) =
  max(foreach epssRecord in EPSS
      where epssRecord.cve == cveId
      select epssRecord);

foreach device in network.devices
foreach cveFinding in device.cveFindings
let epssRecord = getEPSSRecord(cveFinding.cveId)
select {
  deviceName: device.name,
  cveFindingCveId: cveFinding.cveId,
  isVulnerable: cveFinding.isVulnerable,
  basis: cveFinding.basis,
  "EPSS Score": epssRecord?.epss,
  "EPSS Percentile": epssRecord?.percentile,
  "EPSS Date": epssRecord?.date
}

There is no _lookup optimization — even though the logic is the same as the inline version. The optimizer is conservative and when the lookup logic is wrapped in a separate function, the optimizer can’t verify there’s no other logic or side effects, so it won’t rewrite it to a _lookup.

Query Debug Panel

parallel_foreach device in network.devices

⟪foreach cveFinding in ⟪device.cveFindings⟫

 let epssRecord = ./getEPSSRecord(cveFinding.cveId)

 select { ... }

Practical Implications

• In the optimized version, each CVE lookup is an O(1) key access into the EPSS list.
• In the non-optimized version, each lookup scans the entire EPSS list, making it O(n) per finding.
• On large datasets (e.g., hundreds of thousands of EPSS records × thousands of device findings), this difference can mean minutes of runtime saved.

Best Practice Takeaways

 Inline the loop when possible if you want _lookup optimization.

1. Inline the loop when possible if you want _lookup optimization.
2. Avoid wrapping direct key lookups in helper functions unless you’re willing to trade speed for reuse/readability.
3. Always check your debug output for _lookup when performance matters.
4. Use exact equality on unique keys (==) to trigger the optimization — no regex or pattern match.

​