How to Scrape Google Search Results for Free [2024 Edition]
Ever needed Google SERP data for hundreds of keywords at once? Traditional SEO tools can’t do it, and it's frustrating.
Collecting data from Google search results for lots of keywords can be a headache. Doing it manually might take eons. And traditional SERP analysis tools are pretty much useless for this.
In this article, we’ll learn how to extract all Google search results from hundreds of keywords for free.
But is it useful for SEOs only? Let’s explore some other vital use cases of scraping Google SERPs.
Google dominates the search engine market with almost 92% market share. Every month more than 84 billion people use the world’s #1 search engine.
This makes the most visited website in the world a goldmine of data for various use cases, including:
But does Google allow web scraping? Let’s find out.
Google does prohibit scraping but they never tried to enforce it seriously. Like a stackoverflow user said; no lawsuits till now:
It’s not just Bing, most SEO tools scrape Google. As Semrush mentions in their knowledgebase:
Why is Google silent about it? The simple and sweet answer is – Google doesn’t care. As r/LopsidedNinja says:
Google itself scrapes that data from the web. But that’s possible because the websites allow crawling in their robots.txt file. So if it’s harmless, it’s allowed.
Also Google search results are publicly accessible information. After LinkedIn vs HiQ Labs judgment, it is established that scraping publicly available data is completely legal.
Hence scraping Google search results is legal. But you can get in trouble for copyright infringement if you reuse copyrighted content without consent.
But Google has tons of APIs, why scrape it if we can use the official API?
Yes, Google does have an API for collecting and displaying search results. You can use Custom Search JSON API for this. But like any other official API, it has limitations.
Let’s elaborate on each limitation.
Firstly,the official Google API is extremely expensive.
You can only perform 100 requests per day for free. With a paid plan, the API allows only 10k requests per day and you have to pay $5 per 1000 requests.
Secondly, the data is not useful. You won’t get real-time search results. The API data differs from actual SERP data. This makes it useless for commercial use, especially SEO.
Lastly, even if it fits your requirements, the usefulness remains uncertain. Google keeps updating the limits and pricing.
Other than Google official API, we can also use 3rd party SERP APIs. There are a lot of them available. But again, they’re expensive and require coding.
So why not just code a scraper without any API?
If you’re a good programmer, you can scrape Google search using python or any other programming language. But it’s hectic, complex, and non sustainable.
Look at Google’s web page structure. Even if you manage to parse this html using beautifulsoup or lxml, Google keeps updating the page structure.
If you use selenium, you’ll have to deal with captchas. But that’s not the worst part.
Google will simply block your IP address. Using proxies can be extremely expensive too. As Mr. John shares his experience in this stackoverflow post:
So for peeps like me who don’t have a fortune to spend and don’t like coding, both these solutions are useless. The only solution we have is using a no-code tool.
There are a lot of no–code web scraping tools. But for this article, I’ll be using the one that I believe is best in the market – Google Search Scraper by Lobstr.io.
Lobstr offers a transparent and affordable pricing range that suits every pocket.
Let’s learn how to scrape Google search results using Lobstr.io
We’ll scrape Google search results within a few minutes in 6 simple steps.
Let’s go! 🏃
To scrape Google SERP data, all you need is a SERP URL. How do we get that? Let me hit 2 birds with 1 arrow. I’ll track how our latest article is performing in different regions in the US.
I recently published how to scrape TikTok ads without coding. If you were looking for a perfect TikTok ads scraper, do check it out.
We’ll track its SERP position in New York, California, and Texas. Let’s enter our search query in Google and copy the URL.
Copy the URL and go to Lobstr.io, and create new squid. Here’s our URL: https://www.google.com/search?q=scraping+tiktok+ads
To launch Google Search scraper, click the create new squid button on the dashboard. Now search “Google search” and select Google Search Scraper.
Tada! ✨Your scraper is ready.
Next, we’ll add tasks. In this case, task means the SERP URL. Paste the URL you copied here. You can also add URLs in bulk using the upload file option.
You can add as many tasks as you want. Once done, click the Save button to proceed to settings.
Let’s adjust our scraper’s behavior using settings. We have 3 options to adjust our Squid’s behavior.
Firstly, we have basic settings. You can specify how many pages you want to scrape using the Max Pages option.
Leave it blank to scrape all available search result pages. You can toggle Mobile Results to scrape mobile search results.
Next is – When to end run. Select end run once no credit left to stop scraping when you’ve consumed your daily credits. This is best for getting fresh data every time you initiate a run.
If you’ve got hundreds of URLs to scrape, select end run once all tasks consumed. This will pause the run once you’ve used all the credits and resume it the next day.
I’ve explained how this works in my how to scrape TikTok ads article. Do check it out.
Since we only have 1 URL and we need fresh data on every launch, I’ll choose the first option.
With that, we move to advanced settings. First option is Results per page, which allows you to set number of results displayed per page. It’s not so useful in our case.
The second option is where it gets interesting. This is where we choose the country, language, and specific location of our SERP data.
I need to know what position my article is ranking in New York. Let’s adjust our boi.
Next is concurrency i.e. number of bots deployed per run. You can deploy multiple bots for faster data extraction.
Check the pricing page to learn about the number of bots your plan includes. For example: with premium plan, you can either create 10 Squids or deploy 10 bots on a single Squid.
You can remove duplicate results using the Unique Results filter. If you prefer Excel for viewing data, toggle No Line Breaks to remove line breaks from text fields.
Once done with settings, click save and you’ll see the notifications menu.
You can opt to receive email notifications upon success or failure of a run.
You don’t have to open your dashboard to know how your last run ended. Select On success to receive an email notification when the run ends successfully. Choose On error for vice versa.
With that, we’re ready to launch our scraper.
This is our final step. Here we configure our launch sequence 🚀 and boom!!! Google Search Scraper starts collecting data. There are 2 ways to launch the Squid:
If you want to launch the scraper instantly, select Manually and click Save & Extract button. Next thing you see is a live console for monitoring progress and real-time results.
But what if I want to track the SERP position daily or weekly? I obviously want to do that. That’s where the schedule feature will help us.
You can schedule your Squid to run automatically on time and frequency of your choice. To do that, select Repeatedly and set a schedule.
I’m scheduling my Squid to run every Monday at 12PM. This way I can track weekly changes in rankings.
Since I’ve to track 3 states, I created 2 more Squids to track Florida and California. They’ll all run on scheduled time every Monday.
But I can’t wait till Monday. After launching all 3 squids manually. Let’s download the .csv files and see the results.
Starting with California, our scraper collected 352 pages, and we rank #8 in SERP woohoo!!
In Texas, Google gave us 268 search results, and our boi ranks at #8, again ✨
And finally New York, the region we targeted specifically; 228 total results, and we rank… #6. That’s some great news.
So the tool is awesome, but I prefer Google Sheets. Downloading a csv file, then importing it to Sheets is hectic. Is it possible to sync a Google Sheet to our Squid?
Lobstr’s all Squids can be synced to Google Sheets, Amazon S3, SFTP, and Webhook. To do this, just click the delivery button in the console and configure your preferred service.
Now you don’t have to open the dashboard and download results manually. They’ll be updated to your preferred service automatically after every run.
But is it actually better than SEO tools? Let’s find out!
For comparison, I’m using comparing this boi with 2 SEO giants i.e. Ahrefs and Semrush. Both tools offer the rank tracking feature.
But why are they the best? Well instead of searching for “best seo tool”, I prefer reading consumers’ responses. Here is 1 TwitterX polls for example:
Let’s draw a side-by-side comparison first:
Features | Lobstr.io | Semrush | Ahrefs |
---|---|---|---|
SERP Updates | Real-time | Daily | Weekly |
Local Tracking | Yes | Yes | No |
Accuracy Complains | None | Some | None |
Pricing | Affordable | Moderate | Expensive |
Now let’s dive deeper. First, I’m going to use Ahrefs’ rank tracker.
Ahrefs rank tracker doesn’t support local rank tracking. You can’t see your ranking in a particular city or state.
This can often mislead you while doing competitor research or local optimization. Let’s understand it with an example.
Here I added my targeted keyword in the rank tracker. It says we’re ranking on #10 in the US. But this seems incorrect 🤔. We just confirmed we’re ranking #6 in New York.
Why this inaccuracy? That’s because Google SERP rankings may vary city to city. Like we rank #8 in 2 states and #6 in New York.
For local SEO, knowing your SERP position and your top competitors in that targeted region is crucial. You can read this Ahrefs guide to learn more about local SEO.
Lobstr allows you to choose the city, state, and language you’re targeting. Giving you accurate rankings for local SEO optimization.
Second issue with Ahref is outdated results.
As the screenshot says, these rankings were last updated on 9th of January. Which means they’re almost 1 week old. Some users find it frustrating.
Lobstr provides real-time SERP data from Google. Every time you initiate a run, the results you get are exactly what your audience would see when they search that keyword.
Another issue with Ahrefs is – it’s expensive.
The basic plan costs $99 per month. Plus you’ll have to pay extra $100 per month for daily rank updates. Adding additional keywords costs you extra $50 per month.
Lobstr costs you only €50 per month. You can track unlimited keywords. Plus get real-time SERP updates. No extra charges.
Let’s see how Semrush’s rank tracker performs.
Semrush allows you to target a specific city or even a specific zip code. Its data is also updated daily.
But users do complain about data inaccuracy.
This was further validated by other users in comments:
One thing I really love about Semrush – is their support. They’ll respond to you whether you’re on LinkedIn, X, Reddit, or any other social media. Check the reddit comment out there.
But the main issue is – It’s the most expensive tool.
Pro plan costs you $130 per month. You can only create 5 projects and each project supports only 1 location. This means you can track 5 locations.
It also limits total keywords. You can only monitor a maximum of 500 keywords in Pro plan.
It’s way more expensive if you compare it to Lobstr.io which allows unlimited keywords in all plans. Plus with Lobstr’s pro plan, you can track up to 10 locations.
But Lobstr also has a limitation. Let’s discuss that too.
While Lobstr.io is the best SERP in the market, it does have a limitation. Google shows a max 500 results per search, so the scraper can only extract up to 500 results per URL.
You can split the search into multiple URLs using related keywords.
SERP stands for Search Engine Results Page. It's the page you see after you enter a query into a search engine like Google. In our context, it’s the Google search results page.
Yes, this scraper extracts both paid and organic results. This means you can scrape Google ads too.
No, Google doesn’t display volume and KD in SERP. In fact keyword difficulty is an estimate by SEO tools. So, no you won’t get these details. This tool is to scrape data from SERP.
Yes, you get People Also Ask and Related Searches data too. Which makes it a perfect choice for keyword research.
No, if you want to scrape news articles specifically, this tool is not the ideal choice. You can try the Google News Scraper.
No, this scraper specifically collects SERP data only. To scrape Google maps listings, you can use the Google Maps scraper. For scraping reviews, use Google Maps review scraper.
Unfortunately No. If you want to scrape Google SERP data at scale, there’s no reliable AI solution available yet. But you can use GPT to help you code a scraper.
No, with Lobstr, you don’t need proxies. We’ve got you covered. But yes, if you’re creating your own scraper, you may need proxies.
That’s a wrap on how to scrape Google search results without coding. You can use Lobstr.io Google Search Scraper not only for data collection but also a perfect, affordable, accurate SEO tool.
Go on, create a free account and give it a spin.
Self-proclaimed Head of Content @ lobstr.io. I write all those awesome how-tos, listicles, and (they deserve) troll our competitors.