How to Scrape Google Search Results: Unlimited Keywords, No Code

Shehriar Awan●
30 Sept 2026

(updated)

●
12 min read

15-Second Summary

  1. This guide shows you how to scrape Google search results from unlimited keywords, no code, ranks and People Also Ask included
  2. Google has no public API for google.com results. The Custom Search JSON API is closed to new customers, gone January 1, 2027
  3. Building your own scraper is a nightmare, between JavaScript walls, CAPTCHAs and IP blocks
  4. lobstr.io does it without code, at about 250-350 results a minute on one Slot in my runs, from $2 down to $0.50 per 1,000 results
  5. You'll get the 5-step setup, search operators, city-level results and AI integrations

Ever tried plugging Google Search into your local LLM, your LLM API, your agent or your SEO app?

And Google isn't helping. It closed its own search API to new customers, then deleted the docs of its partner-only one.

The third-party scrapers and SERP APIs? Mostly super expensive and terribly slow.

Hacker News comment on Google's missing search API

Don't worry, I got you 😉 I'll show you how to collect all search results from unlimited queries, fast, at scale and affordably.

But wait, doesn't Google have an official API for this?

Does Google have an official search API?

I'll give it to you straight: No. Google gives the public no official API for google.com results.

The closest thing was the Custom Search JSON API. It's closed to new customers, and existing ones have until January 1, 2027 to move off it.
Custom Search JSON API closed to new customers

Even when it was open, it was a toy:

  1. 100 free queries a day, then $5 per 1,000, up to 10K a day
  2. 10 results per call, and never more than 100 per query
  3. Results from "a subset of the total Google Web Search corpus", with no real-time results

Google's suggested replacement, Vertex AI Search, covers up to 50 domains. That's site search, not google.com.

Then came the Web Search Service API. Late this summer, Google documented a partner-only, full-web search API: partner agreement required, the end user's IP on every request, 20 results per request max, no public pricing.
By September 20, the docs were gone. Hard 404, no redirect, no statement.
Web Search Service API docs returning 404

And that's exactly why you need a scraper.

But is it even legal to scrape Google search results?

Is it legal to scrape Google search results?

Disclaimer

This section is for general informational purposes only. It's based on publicly available sources and practical interpretation, not legal advice.

Laws vary by country and change over time. If compliance, contracts, or platform risk matter to you, talk to a qualified legal professional.

Does Google allow scraping?

No. Google's Terms of Service ban "using automated means to access content from any of our services in violation of the machine-readable instructions on our web pages".
Google Terms of Service automated access clause

But does that make it illegal?

Is it actually legal?

Absolutely yes! It's generally legal. Google search results are public data anyone can see.

Google did sue SerpApi in December 2025 under the DMCA's anti-circumvention rules, for bypassing its SearchGuard system. SerpApi won round one.

On July 20, 2026, the court dismissed the claims for good on results with no copyrighted content.

Google got to amend only the copyrighted parts, like Knowledge Panel snippets, and its narrower complaint faces a second motion to dismiss, still pending.

The legal series goes deeper, from Google's terms and the SerpApi case to whether data scraping is illegal at all.
Google v. SerpApi dismissal order, July 2026

But legal doesn't mean reckless. You still need to:

  1. Respect rate limits
  2. Comply with GDPR when results involve EU citizens
  3. Never republish scraped content as your own
  4. Use the data responsibly

But how do I actually scrape Google search results at scale?

2 ways to scrape Google search results

You've got two options: build your own scraper, or use a ready-made one.

Build your own scraper

This one's for the nerds. Push any real volume and Google hits you with CAPTCHAs and IP blocks, so now you're paying for proxies.

It gets worse. Since January 2025, Google requires JavaScript to search, which broke simple request-based scrapers. In September 2025 it retired the num=100 parameter, and the layout keeps changing under your parser.
TechCrunch: Google requires JavaScript to search

And you maintain all of it. Forever.

P.S. If you want me to write a proper tutorial on how to scrape Google search results with Python, ping me on LinkedIn.

Use a ready-made scraper

This is what smart people do. APIs are for nerds, and this article is no-code. The best Google search scrapers get a full comparison on the blog, so here's the best one.

Best Google search scraper: lobstr.io

lobstr.io is a no-code cloud scraping platform with 50+ ready-made scrapers. One of them is the Google Search Scraper.
lobstr.io Google Search Scraper store page
Scrape Google search results from unlimited keywords

Features

  1. Full results pages with organic results, ads, People Also Ask and related searches
  2. The rank of every result in RESULT POSITION
  3. No keyword cap: you pay per unique result, and each Squid holds up to 10,000 keywords
  4. Search operators like site:, intitle:, filetype: and OR
  5. Any country, city or language, with 250 countries and 114 languages
  6. Up to 300-400 results per keyword, as many as Google shows
  7. About 250-350 results a minute on one Slot, in my runs
  8. No Google account needed
  9. Scheduling, exports and auto-delivery to Google Sheets, S3, SFTP, email or webhook
  10. MCP server, n8n, Make and API integrations

Data

🏆 RESULT POSITION 📝 TITLE 🔗 URL
🌐 DISPLAYED URL 📄 DESCRIPTION ✂️ SNIPPET SEGMENTS
❓ QUESTION 💬 ANSWER 🙋 IS QUESTION ANSWER
🔁 IS RELATED QUERY 🌱 IS ORGANIC 💰 IS PAID
🔢 TOTAL RESULTS 📅 DATE ✨ EMPHASIZED KEYWORDS
📑 PAGE

Pricing

1 credit = 1 unique result. Google repeats results across its pages, but duplicates are never charged: 179 rows collected, 104 billed in my test.

  1. 100 free results a month, with exports capped at 30 rows and no scheduling
  2. Starts at $2 per 1,000 results
  3. Drops to $0.50 per 1,000 results at scale
  4. Community Story Rewards... share a genuine lobstr.io story on Reddit or LinkedIn and Free accounts get 2,000 credits plus 30 days without the cap (how it works)
lobstr.io pricing plans

So how do you set it up?

How to scrape Google search results using lobstr.io [Step by Step Guide]

Scraping Google search results with lobstr.io takes less than 2 minutes. Here's the process:

  1. Create a Squid
  2. Add tasks
  3. Adjust behavior
  4. Launch
  5. Enjoy

Step 1: Create a Squid

A Squid is just a scraper instance. To create one:

  1. Log in to your lobstr.io dashboard (no account yet? start here)
  2. Click New Squid
  3. Type google search in the search bar
  4. Select Google Search Scraper
Creating a Google Search Scraper Squid
That's it. Your Squid is ready to configure (more on Squids).

Step 2: Add tasks

A task is your input. Here, one task is one keyword.

Type it in the Keyword field and click Add +.

Keywords added as tasks
Got hundreds? Put them in a .csv, .tsv or .txt file, one keyword per row under a keyword column, and click Upload file. The header has to match, case included, or nothing imports.
One Squid takes up to 10,000 keywords, and duplicates already in the list are skipped (how to upload tasks).
CSV column label on the Upload file button

And yes, search operators work. Type them in like you would on Google. Here's what I tested:

  1. "web scraping" filetype:pdf... only PDF files
  2. intitle:"google search scraper" -github... every title has the phrase, zero GitHub
  3. crab cakes recipe -reddit... recipe sites, no Reddit
  4. site:reddit.com serp api... Reddit threads only
  5. "serp api" after:2026-01-01... every organic result dated 2026
  6. best crab restaurants OR crab shack boston... results for both terms
filetype:pdf results, PDFs only
Want more operators? Here's my guide to Google's operators. They're the same trick behind finding LinkedIn profiles with Google, LinkedIn X-ray search and searching Twitter without an account.

Step 3: Adjust behavior

The Settings tab controls how your Squid behaves. Let me cover the ones you'll actually use.

Max pages... how many result pages to crawl per keyword. Leave it empty to get every page Google shows.

Google shows up to 300-400 results per keyword, but it decides how many. Often it's fewer, and it sometimes omits results. My test keyword got 10 pages.

When to end run... what happens when your credits run out. End run once no credit left (the default) marks the run Done, and relaunching starts fresh. End run once all tasks consumed pauses it and resumes where it stopped once credits refresh (details).
Max pages and When to end run settings
Pro tip... pair End run once all tasks consumed with daily credit allocation, which splits your monthly credits over the days left in the cycle. A huge keyword list then runs a slice a day, pausing and resuming on its own.

Under Advanced settings:

  1. Max Unique Results... caps unique results across the whole run (empty means unlimited)
  2. Country... 250 options, sets the IP origin and the Google domain
  3. Location... a state, city or district, like "Santa Clara, California"
  4. Language... 114 options (default English)
  5. Slots... 1 per Squid for this scraper
  6. Unique Results... removes duplicate rows (on by default)
  7. No Line Breaks... keeps text fields Excel-friendly (on by default)
Country, Location and Language settings

Last, the Notification tab emails you on success, on error, or never.

Step 4: Launch

In the Launch tab, you've got two options: Manually or Repeatedly.

Manually means right now. Click Save & Extract and watch the live console fill up.

Live console of a Google Search run

But what if you want to track rankings over time?

Pick Repeatedly and set how often the Squid runs. That's a rank tracker or ad monitor on autopilot, and it needs a paid plan (how scheduling works).
Repeatedly launch schedule

Step 5: Enjoy

Once the run finishes, download your results as CSV, Excel, JSON or JSONL. Every row has its RESULT POSITION, with flags for organic results, ads, People Also Ask and related searches.
Results with RESULT POSITION and People Also Ask rows

But downloading by hand every time? Nah.

Set up delivery once and every run ships itself. Google Sheets is my personal favorite. Amazon S3, email, SFTP and webhooks work too.
Google Sheets delivery settings
👉 How to export data to Google Sheets?

But a spreadsheet is just the start.

What can you do with scraped Google search results?

You've got the results. Now what?

Plug them straight into your AI. The lobstr.io MCP server (
https://mcp.lobstr.io/mcp
) gives Claude, ChatGPT, Cursor, Grok, VS Code and Codex 22 tools. Reading is free, and running a scraper estimates the cost and waits for your OK first.
Connecting is a quick browser approval on lobstr.io. Setup guides: Claude, ChatGPT, Cursor, Grok and the MCP docs.
The lobstr.io MCP server
Or automate it with n8n. The official lobstr.io node creates Squids, adds tasks, launches runs and pulls results, and its trigger fires when a run finishes.
The lobstr.io n8n node's operations
Make has an official lobstr.io app too: it launches runs, watches for finished ones and pushes results to any Make app (guide). If you code, the API, CLI and Python SDK are there too:
lobstr go google-search-scraper "crab" -o results.csv
f
All of it lives on the integrations page. Here's what I'd do with the data.

1. Feed LLMs and agents live Google results. The problem from the top, solved: Claude, ChatGPT, Cursor or Grok through the MCP, or a local LLM through n8n, Make or the API.

2. Check SEO ranks by city. Set Country and Location, run the Squid whenever you want, and you get the whole results page with every RESULT POSITION, city by city, no keyword cap. It's the organic side of local SEO, next to Google Maps rankings and the Google Maps Leads Scraper.
Page-1 ranks in three cities
Sure, Ahrefs and Semrush track ranks too, and they add keyword difficulty and volume. But they're super expensive: Ahrefs Rank Tracker starts at $129 a month for 750 keywords updated weekly, Semrush Position Tracking at $139 a month for 500 updated daily. And you don't control the tracking, they do (rank tracking APIs, if you need one).
3. Mine content ideas. Every People Also Ask question comes with its answer, next to the related searches: your next FAQ and your next article. For news instead of web results, here's how to scrape Google News with its own scraper.
4. Monitor competitors and ads. The is_paid flag splits ads from organic results. Schedule your money keywords and see who ranks and who advertises, run after run.
If you want me to build a complete AI agent for Google search results, ping me on LinkedIn.

Still got questions?

FAQs

Is it legal to scrape Google search results?

Generally yes, for public results anyone can see. SerpApi won round one of Google's lawsuit on July 20, 2026, and a second motion to dismiss is pending. More in the legal series.

How many results can I scrape per keyword?

Up to 300-400, if Google shows them. Google decides, often shows fewer and sometimes omits some. My test keyword returned 10 pages.

Can I scrape Google results for a specific city?

Yes, down to a state, city or district, with the Country, Location and Language settings.

Can I use it with ChatGPT, Claude or my own LLM?

Yes, through the lobstr.io MCP server, or n8n, Make and the API for a local LLM.

Do I pay for duplicate results?

No, 1 credit per unique result. In my test, 179 rows collected, 104 billed.

How does it compare to Ahrefs or Semrush rank tracking?

lobstr.io charges per unique result, with no keyword cap, while Ahrefs starts at $129 a month (750 keywords, weekly), Semrush at $139 (500). They add difficulty and volume, but you don't control the tracking.

Is there a free plan?

Yes, 100 free credits a month, with a 30-row export cap and no scheduling. Community Story Rewards adds 2,000 credits and lifts the cap for 30 days.

Final thoughts

That's a wrap on how to scrape Google search results... unlimited keywords, no code, no waiting on Google for a public API. Set it up once and your agent, your rank checks and your content research run on fresh results.

Want me to cover a related topic, like the Python version of this? Ping me on LinkedIn.

Happy scraping,
Shehriar

Related Articles

Related Squids

Try lobstr.io for free today!

No captcha free data

Start now