How to Analyze Competitor Keywords: Tools, Limits and the Gap List

To analyze competitor keywords you take the search terms that already send traffic to the sites beating you, line them up against the terms you rank for, and work the difference. Nothing about it is clever. The reason most people end up with a useless spreadsheet is that they skip the two steps that decide the outcome: choosing the right competitors, and filtering the export down to the handful of queries they can realistically win this quarter.

This guide runs the process with named tools and the limits those tools actually enforce as of August 2026. Every number below comes from the vendor’s own page.

What analyzing competitor keywords actually gets you

Three things, and it is only three.

• Demand that is already proven. If a rival ranks page one for a term and has kept that page live for two years, someone is searching it. You do not have to guess.
• A format brief. The top ten tells you what shape of page Google currently rewards: a comparison table, a short definition, a long guide, a calculator.
• A gap list. Terms where two or three competitors rank and you do not appear at all. That list is the actual deliverable.

What it does not give you is a ranking. Copying a rival’s list without beating their page gets you position 34.

Pick search competitors, not business competitors

The company that takes your customers and the domain that takes your clicks are often different companies. A regional accounting firm loses SERP space to national comparison sites and to the tax authority, not to the firm down the road. Three types are worth separating:

• Direct business competitors. Same product, same buyer. Useful for commercial and transactional terms.
• Content competitors. Publishers, review sites, tool blogs. They own the informational half of the funnel and are usually the ones you can beat, because they write about your topic once and move on.
• SERP competitors. Whoever happens to rank for one specific query. This set changes query by query, and it is the set that tells you which format wins.

Building the list needs no tool. Take five or six core terms, run each in an incognito window, and write down every domain in the top ten. Domains appearing for four terms out of six are your search competitors. Keep the working list at three to five: one direct rival, one publisher, one smaller site that is clearly gaining. Ten competitors produces an export you will never open.

While you are there, harvest two free signals. Every question in the People Also Ask box is a query Google has confirmed people ask around your topic, and expanding one loads more. The related searches block at the foot of the page does the same job phrased as queries.

Start with your own Search Console data

Before you buy anything, mine the data you already own. Google Search Console shows the queries where your site earned impressions, with clicks, click-through rate and average position. Queries too rare to report are filtered out to protect user privacy, so the list is close to complete rather than complete. The band that matters is positions 11 to 50. Those are queries where Google already treats you as a candidate and you are losing to someone specific. For each one you can open the SERP, see exactly who is ahead, and read their page. That is competitor keyword analysis with a sample size of one, and it is the most actionable version of it.

If you outgrow the interface, the Search Console API accepts a rowLimit between 1 and 25,000 per request, defaults to 1,000, and pages through results with a zero-based startRow. Google publishes a Search Analytics quota of 1,200 queries per minute per site and per user, so a nightly pull of your full query set sits well inside the limits.

The tools, their free limits and what they cost

Every vendor has a free tier. None of them is generous, and knowing where each one stops saves you an afternoon.

Semrush gives ten competitor reports a day for free, enough to profile three competitors in a week. Ubersuggest gives three. Ahrefs gives deep data on your own site but keeps competitor domains behind the subscription. When the free tiers stop, these are the current list prices:

Semrush states that its prices are in US dollars and Screaming Frog bills in pounds, so your local page may show a converted figure. One seat on one mid-tier tool for three months finishes this job for a normal site.

Pull competitor keywords with no keyword tool at all

If the budget is zero the data is still reachable. It just takes longer.

1. Open the competitor’s sitemap. Try /sitemap.xml and check /robots.txt for the sitemap directive. You get their full published URL set, and the slugs are their target keywords written out in plain text.
2. Crawl it with the free Screaming Frog SEO Spider. Up to 500 URLs per crawl, which covers most small and mid-sized sites. Export title tags, meta descriptions and H1s: a keyword list with the intent already labeled.
3. Sort by internal links. The pages a competitor links to most from navigation and body copy are the pages they care about. Free priority signal.

The manual route gives terms and intent but not volumes. Pair it with the Ahrefs Keyword Generator or Keyword Planner to attach rough numbers.

Run the keyword gap properly

A keyword gap compares your ranking set against two or three competitors and returns the terms where they rank and you do not. Every major platform ships this as a named feature. Two mistakes make the output useless:

• Leaving branded terms in. A competitor’s own brand name will be their single biggest traffic driver, and you will never rank for it. Filter out anything containing their brand before you look at the list.
• Comparing against the wrong weight class. Run a five-page service site against a domain with 40,000 indexed pages and the gap is the entire English language.

Run the gap twice: against your two closest peers, then against the biggest site in the niche. The first run gives this month’s work, the second the two-year map.

Filter the gap list down to what you can win

A raw gap export runs to thousands of rows. Three filters cut it to something a human can act on:

• Winnable. Look at who holds the top ten. If eight of ten slots belong to national publishers, government sites or Wikipedia, skip it regardless of volume.
• Relevant. The term has to map to something you sell or something your buyer asks before they buy. Traffic that never converts still costs you a page.
• Realistic. Match difficulty to your current authority. Every tool’s difficulty score is proprietary and they disagree with each other, so read them as bands.

Read the SERP before you write a word

Keyword in hand, open the results and answer four questions:

• What intent is Google serving? Guides mean informational. Product pages mean transactional. If the top ten is all product pages, a blog post will not rank there however good it is.
• What format repeats? Listicle, comparison table, step-by-step, calculator. Match the format, then beat it on depth or clarity.
• What subtopics do all of them cover? Table stakes. Missing one is why a good page stalls at position 12.
• What does none of them cover? That is your edge. Real limits, real prices, a screenshot of the actual tool, a number nobody else bothered to check.

Check the keywords they pay for

Organic research tells you what a competitor earned. Paid research tells you what they are willing to buy, a much sharper signal of commercial value. If a rival has run the same ad copy against the same term for six months, that term converts. They measured it and kept paying. Most SEO platforms expose competitor ad history under an advertising or PPC research tab, and Google’s Ads Transparency Center shows the ads a given advertiser is currently running.

Treat any term a competitor buys and also ranks for organically as a priority target. Hedging on both channels means it matters to their revenue.

Group the keywords and map them to pages

Five hundred loose keywords are not a plan. Group terms that share a search intent and a SERP into one cluster, and give each cluster one page. “Best running shoes”, “top running shoes” and “running shoe reviews” return near-identical results, so they are one page, not three. Splitting them across three URLs makes all three weaker.

The quick test for whether two keywords belong together: run both and count how many of the top ten URLs overlap. Five or more shared URLs means one page. Two or fewer means two pages. There is a full walkthrough in our guide to keyword clustering. Then map clusters to page types: informational to articles, comparison to versus pages, transactional to service or product pages.

AI Overviews changed what the click is worth

Any competitor keyword list built in 2026 has to account for AI answers. Where Google shows an AI Overview, the first organic result sits further down the page and part of the answer is on screen before the reader scrolls. Nobody publishes a reliable cross-industry figure for what that costs, and it varies by query, so do not plan around a percentage. Treat it as a tie-breaker: between two otherwise equal candidates, take the one with a clean ten blue links SERP.

On what to do about it, Google is unusually direct. Its documentation on AI features states there are no additional requirements to appear in AI Overviews or AI Mode and no special optimizations necessary, and that you do not need to create new machine readable files, AI text files, or markup to appear in these features. The stated requirement is that the page is indexed and eligible to be shown in Google Search with a snippet.

So the practical move is not a new tactic. Check whether an AI Overview fires for each shortlisted keyword and record it as a column in the sheet, next to difficulty.

Score it, ship it, and re-run it on a schedule

Score each surviving keyword on three axes, with business value carrying the most weight:

1. Business value. Does this term belong to someone who buys? Weight it heaviest.
2. Difficulty. Can you outbuild the current top ten with the resources you have?
3. Volume. Weight it lightest. A hundred searches from buyers beats ten thousand from students.

Start with the low-difficulty, high-intent rows. Early rankings give the domain signal and the next one gets easier. Re-run the gap quarterly: rankings move, competitors publish, and a list built in January is stale by June. Keep one sheet with a status column reading planned, in progress, published, ranking. That column is what stops a good analysis from becoming an unopened file.

FAQ

Can I analyze competitor keywords for free?

Yes, with limits. Google Search Console covers your own queries, the Ahrefs Keyword Generator returns 100+ ideas with volumes and no account, a free Semrush account allows 10 reports per day, and a free Ubersuggest account allows 3 searches per day. The free Screaming Frog SEO Spider crawls 500 URLs per crawl, enough to export a small competitor’s full title and H1 set.

How many competitors should I analyse?

Three to five. One direct business rival, one content publisher, and one smaller site that is gaining. Keyword gap tools typically compare your domain against two to four competitors at once, and past five domains the overlap list becomes unreadable.

How often should I re-run the analysis?

Quarterly for the full gap analysis. Monthly for a lighter check on your own Search Console positions 11 to 50, where the fastest wins live. Google Trends lets you compare up to 8 groups of terms at once if you want to watch seasonality between full runs.

Which metric matters most in a competitor keyword export?

Intent, then business value, then difficulty. Volume comes last. Keyword Planner volumes are rounded and averaged over 12 months, so treat them as bands rather than precise figures, and never let a big volume number override a mismatch in intent.

Do I need to do anything special to rank in AI Overviews?

According to Google’s own documentation on AI features, no. It states there are no additional requirements to appear in AI Overviews or AI Mode and no special optimizations necessary, and that no new machine readable files or markup are needed. The page has to be indexed and eligible to appear with a snippet.