Oncrawl – Help centre

Important

This article shows how the connector works with the Data Query 2 API

About Oncrawl

OnCrawl is a powerful SEO crawler and log analyzer for enterprise SEO audits and daily monitoring. With over 400 customers in Europe, including the half of the TOP 20 French e-commerce websites, and in North America, OnCrawl helps e-commerce websites, SEO agencies and publishers improve their SEO performance, rankings and revenues.

OnCrawl also combines your logs and crawl data to provide comprehensive analytics about the on-page SEO factors that have a positive or negative impact on your indexation and crawl budget. OnCrawl gives you detailed data and analysis you can act on.

The team aims to democratize its tools, thanks to technological innovations, to let small but also bigger companies get access to the SEO data they need at affordable prices.

To learn more: http://www.oncrawl.com/

Discover the connector

Oncrawl and AT Internet have developed a bidirectional bridge that allows you to:

Send AT Internet data to Oncrawl platform in order to compare your SEO, SMO with your analytics metrics such as visitors, page views, …
Gather in a AT Internet dashboard main Oncrawl KPI using Oncrawl API

Prerequisites

Prerequisites on AT Internet side:

Activate URL+Querystring dimension (technical validation is required)
Have access to API account

Export data to Oncrawl

1. Configuration of cross analysis (step 1)

Setting up AT Connect Oncrawl is really easy to implement. To do so, you will need to connect to your Oncrawl and AT Internet interfaces.

Connect to your Oncrawl interface
Select a project and click on LAUNCH CRAWL

Select Avanced report, click on AT Internet and check the box Enable AT Internet cross analysis

Once you are here, you will need to access to your AT Internet interface in order to create the two API requests (see section below)

2. API creation

On your AT Internet interface, go on API accounts

Create a new flow, define an account name, choose your site, define a start date and end date. (If you don’t want to define an end date, let the field blank). Click on OK

Two flows are required for the AT Connect Oncrawl: getData and getRowCount. You will create those two flows by giving a label name getData and getRowCount and by copy / paste the two URL available on Oncrawl interface:

To create the two flows on AT Internet interface:

Create a label name and copy the URL. Then click on Save

3. Configuration of cross analysis (step 2)

As soon as your two flows are created on AT Internet interface, you will have a GUID code:

Copy this GUID code and paste it on Oncrawl interface. Do not forget to write down you ID Site:

Then click on SAVE button.

You are now able to launch the crawl. As soon as the crawl is finalized, you will see your crawled SEO data crossed with your AT Internet data.

Import data to Dashboard

Oncrawl provides an API that you will be able to use to create a dashboard in Analytics Suite 2 and import some main KPI.

1. Authentication

Authentication is OAuth. To connect your Oncrawl account to your AT Internet dashboard:

Go to your profile and select EXTERNAL LOGINS then Add a connection

Choose Oncrawl provider and click on Create a connection

Click on OK on the popup confirmation

Fill your Oncrawl login and password

Click on ALLOW

Give a name to your connection and click on Modify

Your new connection will appear on the list

2. Create a dashboard

Oncrawl provides a list of KPI that you will be able to use on an AT Internet dashboard. The KPI can be imported to AT Internet dashboard thanks to Oncrawl API.

All flows will allow you to get the last crawl KPI on your project.

Example of flow:

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=<aggs>&fmt=row_objects

/<project_id>/ is the ID of your projet. You will be able to find the ID of your project on Oncrawl interface URL (underlined in blue below):

The list of all Oncrawl KPI available is described on the next section.

Note: Due to Oncrawl API protection to limit DDOS attack, you are allowed to only get 10 KPI in the same time in your dashboard sheet. It means that you will have to create a dashboard sheet with only 10 boxes. If you want to integrate in your dashboard 30 Oncrawl KPI you will have to create 3 sheets with 10 boxes per sheet.

3. Dashboard template

A dashboard template with Oncrawl KPI is available on the file attached.

Dashboard SEO AT Internet - Oncrawl.pdf (300 KB)

API List

The list of Oncrawl API are listed below.

Oncrawl crossed with AT

SEO active pages

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((oql:(field:!(ati_entrances_seo,gte,'1'))))&fmt=row_objects

SEO active orphan pages

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((oql:(and:!((field:!(depth,has_no_value,'')),(field:!(ati_entrances_seo,gte,'1'))))))&fmt=row_objects

SEO visits from orphan pages

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((oql:(and:!((field:!(depth,has_no_value,'')),(field:!(ati_entrances_seo,gte,'1')))),value:'ati_entrances_seo:sum'))&fmt=row_objects

SEO active pages by depth

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:depth)),oql:(and:!((field:!(ati_entrances_seo,gte,'1')),(field:!(depth,has_value,''))))))&fmt=row_objects

Average inrank by SEO visits range

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:ati_entrances_seo,ranges:!((from:1,name:one,to:2),(from:2,name:'2_9',to:10),(from:10,name:'10_99',to:100),(from:100,name:'100_999',to:1000),(from:1000,name:'1000_9999',to:10000),(from:10000,name:'10000_99999',to:100000),(from:100000,name:more_100000)))),oql:(and:!((field:!(ati_entrances_seo,gte,'1')),(field:!(depth,has_value,'')))),value:'inrank:avg'))&fmt=row_objects

Load time evaluation by SEO visits range

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:ati_entrances_seo,ranges:!((from:1,name:one,to:2),(from:2,name:'2_9',to:10),(from:10,name:'10_99',to:100),(from:100,name:'100_999',to:1000),(from:1000,name:'1000_9999',to:10000),(from:10000,name:'10000_99999',to:100000),(from:100000,name:more_100000))),(name:load_time_range)),oql:(and:!((field:!(ati_entrances_seo,gte,'1')),(field:!(depth,has_value,''))))))&fmt=row_objects

Average SEO bounce rate by load time evaluation

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:load_time_range)),oql:(and:!((field:!(ati_entrances_seo,gte,'1')),(field:!(depth,has_value,'')))),value:'ati_bounce_rate_seo:avg'))&fmt=row_objects

Structure

Pages crawled: Pages discovered and fetched by OnCrawl during the crawl. Oncrawl bot considers robots.txt and no follow rules.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((oql:(field:!(fetched,equals,true))))&fmt=row_objects

Indexable pages: Pages fetched by OnCrawl bot without noindex in meta robots. The content has been analyzed unlike pages whose meta robots is noindex.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((oql:(and:!((field:!(fetched,equals,true)),(field:!(meta_robots_index,equals,true)),(field:!(status_code_range,equals,ok))))))&fmt=row_objects

Indexable compliant pages: Indexable HTML pages that returned a status code 200 to Oncrawl robot. Non compliant pages can be redirections, errors (status code 3xx or 404) or pages that are not HTML (xml, pdf ...)

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((oql:(and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true))))))&fmt=row_objects

Depth distribution: Shows how pages are distributed by depth. Page depth refers to the number of clicks you need to reach a specific page from the homepage using the shortest path. Depth matters insofar search engines are less likely to find deepest pages and so to crawl them

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:depth)),oql:(and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true))))))&fmt=row_objects

Links breakdown

Internal links follow: Links from pages of the site, pointing to other pages of the site. Follow attribute informs bots to crawl the link’s target and thus distribute any page rank to the destination page.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((oql:(and:!((field:!(depth,has_value,'')),(field:!(internal_follow_outlinks,gt,'0')))),value:'internal_follow_outlinks:sum'))&fmt=row_objects

Internal links nofollow: Links from pages of the site, pointing to other pages of the site. Nofollow attribute informs bots to not crawl the link’s target and thus do not distribute any page rank to the destination page

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((oql:(and:!((field:!(depth,has_value,'')),(field:!(internal_nofollow_outlinks,gt,'0')))),value:'internal_nofollow_outlinks:sum'))&fmt=row_objects

External links follow: Links from pages of the site pointing to pages of another site. Follow attribute informs bots to crawl the link’s target and thus distribute any page rank to the destination page.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((oql:(and:!((field:!(depth,has_value,'')),(field:!(external_follow_outlinks,gt,'0')))),value:'external_follow_outlinks:sum'))&fmt=row_objects

External links nofollow: Links from pages of the site pointing to pages of another site. Nofollow attribute informs bots to not crawl the link’s target and thus do not distribute any page rank to the destination page.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((oql:(and:!((field:!(depth,has_value,'')),(field:!(external_nofollow_outlinks,gt,'0')))),value:'external_nofollow_outlinks:sum'))&fmt=row_objects

Inrank by depth: The inrank is a kind of internal pagerank. It is a score (on a 0 to 10 scale) developed by Oncrawl that helps to understand how popular pages are regarding the internal link architecture. Shows the impact of the depth on the internal pagerank

Main tags

Title evaluation: Provides a clear view of your titles state of play. Titles tags are the main tags on which you should focus. They have a real impact on your search engine results and your page ranking, so it is necessary to fill them correctly.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:title_evaluation)),oql:(and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true))))))&fmt=row_objects

Description evaluation: Provides a clear view of your main meta descriptions state of play. Even if Google has said that keywords in your meta description are not affecting your rankings, it can compel visitors to click on your site through the SERP if the keywords are the ones they were looking for.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:description_evaluation)),oql:(and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true))))))&fmt=row_objects

H1 evaluation: Provides a clear view of your titles state of play. H1 tags are as important as titles. They have a real impact on your search engine results and your page ranking, so it is necessary to fill them correctly.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:h1_evaluation)),oql:(and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true))))))&fmt=row_objects

Title length evaluation: Allows to verify if you have respected the Google guidelines in terms of title length.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:title_length,ranges:!((from:40,name:good,to:70),(name:too_short,to:40),(from:70,name:too_long)))),oql:(and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true))))))&fmt=row_objects

Description length evaluation: Allows to verify if you have respected the Google guidelines in terms of description length.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:description_length,ranges:!((from:110,name:good,to:170),(name:too_short,to:110),(from:170,name:too_long)))),oql:(and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true))))))&fmt=row_objects

Open Graph distribution : The Open Graph allows you to add metadatas to the pages of a website to improve the presentation of a web page when it is shared on a social network (Facebook, LinkedIn or Twitter…) (even if this last one has developped its own one, the Twitter Cards). Officially, Open Graph does not have a direct impact on your on-page SEO and it just seems to be here to optimize your social links.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:ogp_type)),oql:(and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true))))))&fmt=row_objects

Twitter Cards distribution: Twitter Cards also help to improve your Twitter shares. This protocol allows tweets containing links to display in a richer format, with more content visible and integrated as part of the tweet. As the Open Graph, Twitter Cards are a great way to grow engagement and click-through rate.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:twc_type)),oql:(and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true))))))&fmt=row_objects

Content

Duplicate content: Show the number of pages with content duplication issue: pages whose text content is considered duplicated, or very similar to another page. Duplicate content has a negative impact on a website ranking and the crawl frequency

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:cluster_canonical_status)),oql:(and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true))))))&fmt=row_objects

Structured datas distribution: Page’s structured datas help search engines to understand what your content is about and contribute to the display of rich snippets in SERPs While there is no direct correlation between those data and a ranking improvement, structured datas can really help you boost your visibility in SERPs.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:semantic_types)),oql:(and:!((and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true)))),(field:!(semantic_item_count,gt,'0'))))))&fmt=row_objects

Performance

HTTP status code distribution: For each HTTP request, the web server sends a response as a status codes. The main ones are:
- 200: success
- 301 or 302: redirections
- 404: Error page does not exist
- 500: server error

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:status_code)),oql:(field:!(depth,has_value,''))))&fmt=row_objects

Loadtime distribution: Time to fetch the entire HTML of the page (excluding external resources). Page speed is regarded as a ranking factor.

https://app.oncrawl.com/api/v2/data/project/<project_id>/last_crawl/pages/aggs?aggs=!((fields:!((name:load_time_range)),oql:(and:!((field:!(meta_robots_index,equals,true)),(field:!(status_code,equals,'200')),(field:!(canonical_evaluation,not_equals,not_matching)),(field:!(parsed_html,equals,true))))))&fmt=row_objects

Privacy

Activating this partnership activates the import of Oncrawl data into AT Internet tools and the export of AT Internet data into Oncrawl. Only the data necessary for the proper functioning of the bridge are imported and exported.