Scraping Google SERPs can be incredibly useful for SEO competitor analysis. In this blog post, I will show you how to leverage Python’s rich ecosystem of libraries to scrape Google Search Results efficiently, providing a powerful way to gather valuable data for SEO and research purposes.
Scraping Google Search Results using Python is a powerful way to gather valuable data for SEO and research. With tools like Playwright, you can automate the process and extract URLs and Google Titles.
How to Scrape Google Search Results: A Step-by-Step Guide
My code focuses on extracting the title displayed on Google and the corresponding URL for a set of keywords placed in the Python code. The results are saved in a CSV file namedgoogle_search_results.csv
, where each row contains the keyword, Google Title, and URL.
Step 1: Install Python
Make sure you have Python 3.8 or higher installed. You can download it from python.org.
Step 2: Install Required Python Libraries
Install the following Python libraries:
- Playwright: For browser automation.
- Pandas: For saving the extracted data to a CSV file.
Install the required libraries by running:
pip install playwright
playwright install
Step 3: Download this Python Code & Execute
If you’d like to get started quickly with the code, you can download the complete Python script here. Simply click the link, save the file (google_search_scraper.py), and run it in your Python environment. Make sure to install the required dependencies outlined in Step 2.
To run the script, save the script to a file (e.g., google_search_scraper.py) and execute the below command on the command prompt.
python google_search_scraper.py

Step 4: Open Downloaded Output File

To perform a competitor analysis, you can take the output sheet and run a crawl of the URLs using Screaming Frog. By crawling the URLs, you can easily extract valuable metadata such as page titles, meta descriptions, and H1 tags. This allows you to compare and analyse your competitors’ optimisation strategies against your URL ranking for the same keyword set, providing actionable insights to refine and optimise your SEO approach.
Let me know if you have any questions or suggestions for improving this guide! Remember to always respect Google’s terms of service, use scraping responsibly for small-scale tasks or research, and consider APIs like SerpAPI for large-scale scraping of SERPs.