The Google Search Console API offers a gold mine of free search data for those who are able to get at it. The first step in being able to get at that data is to connect to the API with Python.
Why is the Search Console API so great?
If you’re an SEO’er or even a webmaster, then you’ve probably used Google Search Console at some time to see what search terms people are using to find your website on Google. SC also offers vast amount of historic data for each page of your website that details things like CTR, impressions, search rankings and even what devices your visitors are using.
By accessing Search Console through your browser, you are limited in the amount of actual data you are able to extract due to a general limitation. For example, you are only able to see about 1000 keywords for your site over a time period.
The Search Console API has a much bigger limit and with some clever programming, you can basically remove the limit all together.
What are the steps needed to connect to Search Console with Python?
- Verifying a domain in Google Search Console
- Setting up the python scripts and environment
- Activating your Google Search Console API
- Authenticating your Python app to access your search data
Before we begin, I am assuming you have a basic understanding of how to use Python. If not there are tons of online courses like the one over at Codecademy where you can learn the basics and even take it further if you feel daring.
Verifying a domain in Google Search Console
Before you can do anything, you’ll need to make sure that the Google account you use is a verified owner of the domain property in Search Console. You can learn more about this here
There are two main types of verification offered currently by Google. The easiest way is to verify by updating your DNS records with a TXT file that Google gives you. However, I find that this way only verifies a property without the full URL E.G digitalhop.co.uk as opposed to https://digitalhop.co.uk
You will need to be verified for a full URL such as https://yourdomain.com for your scripts to work. So if you don’t have that yet or you only have a property without the https part verified, then go ahead and verify a full domain property using the URL prefix option.
There are two ways to verify your website in Search Console. You will need to have URL prefix option for the Python scripts to work properly. Sometimes it is easier to do the domain verification first and then the url prefix verification
Setting up the python scripts and environment
- You’ll need some Python scripts to do the heavy lifting. So head over to my Github Search Console repository.
- Clone the repository and store it on your local machine.
- Open up your command prompt and navigate to the directory where you stored your cloned repository
- We’re going to install all the modules you need to run these Python scripts
- This might also be a good time to mention that you might want to make a virtual environment for these Python modules so you can keep them separate from your main Python modules. Corey Schafer has a video that explains how to set up a virtual environment
- Enter this command to install the modules from the requirements.txt file
- pip install -r requirements.txt
Activating your Google Search Console API
Google let’s you set up free API’s for its tools like Search Console in the Google API Dashboard. We’re going to setup a Search Console API and also give it the credentials it needs to connect your python scripts to Search Console API.
Step1 – Follow this link to create your Google API project.
Here you create your API Project in Google.
Step2 – Click the go to credentials button
Step 3 – don’t worry about too much here, just click the OAuth consent screen link in the left hand navigation
We will not be worrying about this now. Click the OAuth consent screen in the left hand navigation
Step 4 – We are going to select External here
Select External
Step 6 – Enter a name for the App and click save
Step 7 – On left hand navigation click Credentials and then – click create credentials at the top and then click Oath client ID
Select OAuth client ID from the drop down menu
Step 8 – Select Other for the application type and give it a unique short name – (if ‘other’ is not an option it could be because you already have an Oath credential in your project using the ‘other’ property, you can always make a new project)
The client id and secret should appear, but we are going to use the json file
Step 9 – Click the down arrow to download the json file with your client id and your client secret. Put this file into the config folder found in the repository you downloaded from Github
Authenticating your Python app to access your search data
Now that we have set up our Google Search Console API we need to authenticate it.
- Run the check_verified_sites.py file in the folder you cloned from the repository
- The terminal will print out a URL for you to visit to verify that you are accepting your new Python app to connect to your Google personal account. Visit the URL and login to your personal Google account and accept the conditions
- Google then gives you an authentication code. Copy this code and paste it into the terminal and hit enter
If Everything has worked, then the terminal will print out a list of the sites that are verified for you to access the search console data.
What next?
Now that you’re set up with the Google API and basic Search Console Python script, it’s a good time to see how many keywords you have stored in Search Console. It’s possible that you have over 16 months worth of Keyword data waiting for you.
Every time a visitor has used Google to find your website on Google, this has been recorded in your Search Console webmaster account. Using the Top keywords extractor script we can get up to 25 000 keywords if a few seconds.