Accessing NQE query output via the API for a large lists of results

The default number of results returned for an NQE query executed from the API is 1000 (returned as a list assigned to the key “items”)

You can increase the limit to 10,000 items. For retrieving results of more than 10,000 items, you will need to use the limit and offset parameters as part of your script, and concatenate the results of multiple API calls within the script.

Below is a simple script in Python that concatenates successive chunks of 10,000 items into a single output:

import requests

def nqe_query_with_offset():
    my_result = []
    limit= 10000
    offset = 0
    network_id = "101"
    url = "https://fwd.app/api/nqe?networkId=" + network_id
    total_extracted = False
    while total_extracted == False:
        payload ={"queryId": "FQ_ac651cb2901b067fe7dbfb511613ab44776d8029", "queryOptions": { "limit": limit, "offset": offset}}
        print(payload)
        headers = { 'Content-Type': 'application/json', 'Authorization': 'Basic #KEY#'}
        response = requests.request("POST", url, headers=headers,json=payload, verify=True).json()
        returned_data_count = len(response['items'])
        if returned_data_count == 0:
            total_extracted = True
        else:
            total_extracted = False
        offset += limit
        my_result.extend(response['items'])
    my_dictionary = {"items": my_result}
    return my_dictionary

if __name__ == "__main__":
    nqe_results = nqe_query_with_offset()
    print(nqe_results)