I need to export millions of rows of data. How to proceed?

There are several possibilities.

Data Flow

This option enables you to gather all data over the last 48 hours.
If you wish to gather historical data beyonf the last 48 hours, you will need to go through a custom export or a series of queries via the REST API or Data Query.

More information on Data Flow can be found here.

REST API

The REST API enables you to query up to 10 000 rows per API call.
However, you are not limited in the total number of rows. To gather more data, you will have to change the page number parameter.

Example:

  • &max-results=10000&page-num=1. You will obtain the results from rows 1 to 10 000
  • &max-results=10000&page-num=2. You will obtain the results from rows 10 000 to 20 000 etc.

We recommend setting up a script to make this dynamic. You can do this by querying the total number of rows; replace getData by getRowCount in your query. This number, divided by 10 000, will give you the total number of pages.

More information here.

Data Query

Data Query enables you to export a larger number of rows at once than the REST API. Depending on the options you have activated, the upper limit can vary from 200 000 to 1 000 000 rows. You can get around this limit by splitting your analysis period so that each chunk is under the export limit, and combining the data in an external tool. This can however be time-consuming.

More information here.

Custom export

Our technical teams can develop a custom export tailored to suit your needs. You will need to define the KPIs so we can look into feasability and pricing.

For more information, contact the Support Centre.

Have more questions? Submit a request