Working within the GA4 Data API's token system
The Analytics Data API provides programmatic access to GA4 report data. You can create reports, automate complex reporting tasks, integrate GA4 with other business applications, and more.
When loading data from the Google Analytics 4 Data API, you are subject to a quota system. This quota system is designed to limit how much data you can extract per hour and per day, and to prevent dashboards from using the GA4 Data API as a data warehouse.
Google wants you to store the data outside of GA4, either in BigQuery through the [GA4] BigQuery Export, or in your own databases and files.
This article shows you how to use Analytics Canvas to build your GA4 report tables so that they can be used in visualization tools like Looker Studio, Tableau, or Power BI, or stored in your own BigQuery project, databases, or files.
Getting started with the Analytics Canvas GA4 API connector
The first step is to make your initial connection and authorization to Google Analytics. This article from our Knowledge Base shows you how to make a GA4 API query.
The data is landed into Smart Tables, which exist within your Analytics Canvas subscription. These tables can be analyzed within Analytics Canvas, used in data workflows, or exported to BigQuery, Sheets, or Looker Studio for reporting and analysis.
Loading data from the GA4 Data API
Things to consider when making your queries:
Analytics Canvas is subject to the same API quota limits as every other 3rd party client that connects to GA4, including Looker Studio and Google Sheets. Some other things to consider when using the API to power your reports instead of BigQuery:
- You start each day with 25,000 tokens that can be used for your Property
- You start each hour with 1,750 tokens that can be used by any one application (like Analytics Canvas) and 5,000 tokens that can be used by all applications that connect to your GA4 Property, including Looker Studio.
- Every request to the API that is made within Analytics Canvas will consume tokens
- Working within the token system, it may take several hours to load your historic data from the API. For now, this is a manual process and something you will have to manage.
- Once loaded, you will never have to run the exercise again if you schedule automatic updates
- If your scheduled refreshes consume more than 1,750 tokens per hour, you will need to stagger them over several hours.
- Scopes matter! While you can make certain queries, not all tables are summable, and while you can get 9 dims and 10 metrics per query, arbitrarily adding dims and metrics may skew your results.
- Your report will need a minimum of 2 tables, but likely 5-7 summary tables to feed all of the various charts and tables in your dashboard.
Don't know what query to make? Start with the Explorer tool in Google Analytics or look to your existing Looker Studio dashboard and make the EXACT SAME query in Canvas as you make there.
🛟 Tip from Support: reports will generally need 3-5 summary tables. Make as many tables as are required to feed your report. Most mistakes happen when a table has dimensions that cross scope, or when trying to summarize on the User metric.
Making your first query
With these considerations in mind, we suggest making your first query over a small date range and set the reload period to 1 day so that we don't exhaust the hourly quota on your first query!
This small query will give you a sense for how many tokens the query will consume and how long it will take to load the historic data.
Monitoring your GA4 API quota use
Having made your first GA4 API query by following the steps above, you will now have a GA4 data block on your Canvas. Click on the center of the GA4 API block to bring up the control for it. Then click on the "GA4 Quota Use" tab.
The GA4 Quota Use section will show the number of tokens used by this first query and how many tokens are available.
⚠️ Project Remaining Tokens is based on all queries that were run within the hour by Analytics Canvas.
Due to the complexity of this API, in most cases you cannot get the full history for your report within the quota limits. Now is the time to develop a query plan that includes the tables you need for your reports.
Developing a Query Plan within the Token Limits
A query plan is simply an outline of how you are going to load your historic data working within the quota limits.
For example, if it took 60 tokens to load 1 month of data for a given query, you can estimate that it will take 60 x 12 = 720 tokens to load 1 year of history, and 1080 tokens to load 1.5 years. Allowing for seasonal adjustments in data volumes, this is the maximum that you should attempt to load over the first hour.
In most cases, the dashboard itself will likely have a dozen or more elements on each page, each at either the User, Session, or Event scope. As a result, for each report, you will need 5 or more summary tables from which to build your dashboard, so you must repeat this exercise for each summary table used in your report.
Loading Historic Data
Having made your test queries and developed the queries needed for your report, it is time to load the historic data. Returning to each query, click on the centre to bring up the control, then click on "Edit Query".
Next, click Date Range and adjust the date range back by 1 increment according to your query plan. In this case, we are going back by 3 months to October 1st, 2022.
Click "Next Step: Submit" to load the additional data. Analytics Canvas will automatically keep the data that has already been loaded and simply append the older data to the same table.
⚠️ Working within the token system, it may take several hours to load your historic data from the API
Continue this exercise until all of your tables have been updated. This may require several sessions over several days if your site has a lot of traffic.
Incremental Load - keeping the data updated using the automation scheduler
With the historic data now loaded, it is time to prepare the canvas for daily refreshes to keep the data set up to date. We recommend a 5 day incremental load to allow time for GA4 data reprocessing to occur.
For example, GA4 will accept late arriving hits for up to 72 hours after the event was triggered. In order to account for that hit, your query must include the last 3 days.
🛑 You must reload at least the last 3 days or risk falling out of date with the Data API.
To setup your GA4 API query for an incremental load, go back to Edit the query, go to Date Range, and this time change the reload period to 5 days.
Click "Next Step: Submit" to run the query. Note that this will reload the last 5 days.
Incremental Load Token Consumption
Repeat this process for each query in your report. Upon completion, inspect each query to see how many tokens it consumed when performing the incremental load. If the total of ALL queries being updated across the property, not just in this Canvas, is less than 1,250, you can run the daily incremental load in one package.
If the sum of the tokens required for the incremental load is more than 1,250, you will have to move some queries to a new Canvas file or files, and schedule them to run at least 1 hour apart.
⚠️ Token consumption will vary depending on seasonal traffic volumes, etc. Google does not publish how it calculates the tokens, nor do they tell you how many tokens a query will cost before it is made.
You can schedule a maximum of 20 runs over 20 hours that each consume 1,250 tokens for your daily refreshes. That would bring you to the maximum of 25,000 tokens per day for your property.
Beyond that, you can:
- Re-assess your queries. Are they optimized, or can you reduce the number of tables and aggregate from fewer summary tables?
- Assess your granularity - does your data contain more fields than are used in your reports? Are there any high cardinality dimensions that can be removed from one or more datasets? Can you switch your reporting from hourly to daily, or daily to weekly?
- Give up on the API and move to the BigQuery export (recommended).
Having made one or more canvases that have the queries you'd like to run, use the Automation scheduler to configure the daily updates.
Scheduling GA4 Data API updates
The Automation Scheduler is used to keep data sets up to date. Add a Canvas containing one or more queries, or one or more completed workflows to the automation scheduler, and set the schedule to run daily.
If you have more than 10 queries in your report, we recommend separating them out into multiple packages (not canvas files but individual automation packages).
Learn more about automating data tasks in this article from our knowledge-base.
Support & Troubleshooting
If your setup includes one or more Canvas files that simply load data, while other Canvas files write that data to an output, be sure to un-check the box that says "Refresh table for every Canvas run". If this box is deselected, the table will only update once per day.
If you run into any issues at all, please contact support@analyticscanvas.com.