SEO News

How to Use Python to Identify Keyword Cannibalization

Jennifer Law · · 8 min read

How to Use Python to Identify Keyword Cannibalization

Keyword cannibalization is a critical SEO issue where multiple pages on the same website target the same primary keyword or related semantically similar clusters. Instead of search engines distributing link equity and ranking power across relevant pages, they become confused, often ranking only one page highly, and severely diminishing the potential traffic for all related content.

While manual auditing is possible, the sheer volume of modern websites makes automated identification necessary. Python, combined with web scraping and data analysis libraries, offers a powerful solution.

This guide outlines the technical workflow for using Python to detect potential instances of cannibalization.

⚙️ Prerequisites and Data Acquisition

Before writing a single line of code, you must gather the right data. Python cannot magically know what your website ranks for; it needs to know what content exists and what keywords are associated with that content.

1. Data Sources Required

  • URL List: A comprehensive list of all pages you suspect are targeting related keywords.
  • Content: The body text of each page (for semantic analysis).
  • SEO Metadata: Page Title Tags and Meta Descriptions (These often contain the explicit target keywords).
  • Google Search Console (GSC) Data (Ideal): Direct access to GSC performance data showing which keywords your pages are currently appearing for.

2. Essential Python Libraries

You will need the following libraries installed:

bash
pip install requests pandas nltk beautifulsoup4

  • requests: To download the HTML content of the URLs.
  • beautifulsoup4 (bs4): To parse the HTML and extract clean text, titles, and meta tags.
  • pandas: For structuring, cleaning, and manipulating the dataset.
  • nltk: (Natural Language Toolkit) Useful for text cleaning, tokenization, and potential stemming/lemmatization.

🐍 Step-by-Step Python Implementation

We will structure the process into three main phases: Data Extraction, Keyword Aggregation, and Analysis.

Phase 1: Extracting On-Page Data (The Scraper)

The goal is to create a structured DataFrame where each row is a URL and the columns contain its key SEO elements.

“`python
import requests
from bs4 import BeautifulSoup
import pandas as pd
from urllib.parse import urljoin

Assume you have a list of URLs to check

url_list = [“https://example.com/article-a”, “https://example.com/guide-b”, “https://example.com/best-practices”]
data = []

def scrape_seo_data(url):
“””Fetches and extracts title, meta description, and body text.”””
try:
response = requests.get(url, timeout=10)
response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)
soup = BeautifulSoup(response.content, ‘html.parser’)

    # 1. Extract Title Tag
    title = soup.find('title').get_text(strip=True) if soup.find('title') else ""

    # 2. Extract Meta Description
    meta_desc = soup.find("meta", attrs={"name": "description"})
    description = meta_desc.get("content", "") if meta_desc else ""

    # 3. Extract Main Content (This is tricky and highly site-dependent)
    # A common heuristic is to look for article bodies or remove navigation/footers.
    main_content = soup.find('article')
    if main_content:
        body_text = main_content.get_text(separator=' ', strip=True)
    else:
        # Fallback: grab all visible text if no <article> tag exists
        body_text = soup.get_text(separator=' ', strip=True)

    return {
        'URL': url,
        'Title': title,
        'Description': description,
        'Content': body_text
    }
except requests.exceptions.RequestException as e:
    print(f"Error scraping {url}: {e}")
    return None

for url in url_list:
data.append(scrape_seo_data(url))

Clean the list to remove any failures (None values)

data = [item for item in data if item is not None]

Create the main DataFrame

seo_df = pd.DataFrame(data)
print(“Data extraction complete.”)
“`

Phase 2: Keyword Aggregation and Scoring

Now, we need to process the raw text fields (Title, Description, Content) to find the dominant topics/keywords for each URL.

“`python
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
import re

Download necessary NLTK data if running for the first time

nltk.download(‘punkt’)

nltk.download(‘stopwords’)

STOP_WORDS = set(stopwords.words(‘english’))

def extract_keywords(text, source_type=’combined’):
“””Tokenizes text, removes stopwords, and counts word frequency.”””
if not text:
return {}

# Simple cleaning: lowercase, remove special characters
text = re.sub(r'[^a-zA-Z\s]', '', text.lower())
tokens = word_tokenize(text)

# Filter out common words and short tokens
keywords = [word for word in tokens if word not in STOP_WORDS and len(word) > 2]

if source_type == 'combined':
    # Combine Title, Desc, and Content into one corpus for basic analysis
    # In a real-world scenario, you might analyze them separately.
    pass

# Simple frequency counter for demonstration
from collections import Counter
return pd.Series(keywords).value_counts().to_dict()

Apply keyword extraction to the DataFrame

seo_df[‘Keywords_Title’] = seo_df[‘Title’].apply(lambda x: extract_keywords(x, ‘title’))
seo_df[‘Keywords_Desc’] = seo_df[‘Description’].apply(lambda x: extract_keywords(x, ‘description’))

Note: Content can be massive; limiting its keyword analysis might be necessary.

seo_df[‘Keywords_Content’] = seo_df[‘Content’].apply(lambda x: extract_keywords(x, ‘content’))

Function to find the MOST dominant word across all sources for a page

def get_dominant_keyword(row):
# This combines all word counts into a single tally for the URL
combined_counts = {}
for col in [‘Title’, ‘Description’, ‘Content’]:
keywords = row[f’Keywords_{col}’]
for keyword, count in keywords.items():
combined_counts[keyword] = combined_counts.get(keyword, 0) + count

# Find the top keyword
if combined_counts:
    return max(combined_counts, key=combined_counts.get)
return ""

seo_df[‘Dominant_Keyword’] = seo_df.apply(get_dominant_keyword, axis=1)
“`

Phase 3: Detecting Cannibalization

The core logic resides here: grouping pages that share a dominant or highly recurring keyword.

“`python

Initialize a dictionary to hold potential clusters

cannibalization_clusters = {}

1. Primary Detection (Single Keyword Match)

Group by the single most dominant keyword

keyword_groupings = seo_df.groupby(‘Dominant_Keyword’)

for keyword, group in keyword_groupings:
# A cluster exists if more than one URL shares this keyword
if len(group) > 1:
cluster_urls = “, “.join(group[‘URL’].tolist())
if keyword not in cannibalization_clusters:
cannibalization_clusters[keyword] = {
“count”: len(group),
“urls”: cluster_urls,
“details”: group[[‘Title’, ‘Description’]].to_dict(‘records’)
}

2. Secondary Detection (Semantic/Thematic Clustering – Advanced)

This requires analyzing the overlap of the keyword lists, not just the single dominant word.

We can identify URLs that share 3 or more high-frequency keywords (e.g., ‘SEO’, ‘guide’, ‘best’).

overlap_keywords = set()
for i in range(len(seo_df)):
for j in range(i + 1, len(seo_df)):
urls_a = seo_df.iloc[i:i+1]
urls_b = seo_df.iloc[j:j+1]

    # Merge the keyword dictionaries and find common keys
    k_a = urls_a['Keywords_Content'].iloc[0]
    k_b = urls_b['Keywords_Content'].iloc[0]

    common_keys = list(set(k_a.keys()) & set(k_b.keys()))

    # If a significant number of common keywords are found (e.g., > 3)
    if len(common_keys) >= 3:
        overlap_keywords.add(f"{urls_a['URL'].iloc[0]} | {urls_b['URL'].iloc[0]}")

=========================================================

print(“\n\n========== 🔍 CANNIBALIZATION ANALYSIS COMPLETE ==========”)

if cannibalization_clusters:
print(“\n✅ PRIMARY CANNIBALIZATION CANDIDATES FOUND (Based on Single Dominant Keyword):”)
for keyword, data in cannibalization_clusters.items():
print(f”\n— KEYWORD: ‘{keyword}’ —“)
print(f”Potential Conflict: {data[‘count’]} pages target this term.”)
print(“Affected URLs:”)
for url, title, desc in data[‘details’]:
print(f” – {url} (Title: {title[:50]}… | Desc: {desc[:50]}…)”)
print(“Action Needed: Consolidate or differentiate the focus for these pages.”)
else:
print(“\n✨ No strong primary keyword overlaps detected among the tested URLs.”)

if overlap_keywords:
print(“\n\n🔍 SECONDARY CANNIBALIZATION CANDIDATES FOUND (Based on Content Overlap):”)
print(“The following pairs share significant thematic keyword overlap (3+ keywords):”)
for overlap in overlap_keywords:
print(f” – {overlap}”)
“`

📊 Interpretation and Next Steps

How to Read the Output

  1. Primary Candidates: The dictionary cannibalization_clusters provides the easiest wins. If ‘best espresso machine’ is the dominant keyword for three different URLs, those three URLs are competing with each other for the same traffic.
  2. Secondary Candidates: This list is more nuanced. It signals that two pages talk about the same thing using similar vocabulary, even if their single dominant keyword is different (e.g., one focuses on “espresso” and the other on “brewing,” but they use “guide,” “best,” and “machine” together).

Mitigation Strategy

Identifying the problem is only half the battle. The next steps require strategic SEO effort:

  1. Consolidation (The Merge): If multiple pages are covering the exact same core topic, merge them into one definitive pillar page. Redirect the old, merging pages to the new, comprehensive resource.
  2. Deeper Differentiation (The Split): If the pages cover related but distinct sub-topics, clarify their focus. For example, if the main keyword is “Gardening Tips,” one page should be dedicated only to “Indoor Gardening,” another only to “Vegetable Beds,” and so on.
  3. Update Metadata: Ensure that the Title Tag and Meta Description for every single URL clearly signals its unique focus to both the user and the search engine.

By automating the tedious process of data gathering and keyword comparison using Python, you transform a manual, error-prone auditing task into a scalable, data-driven diagnostic process.

Post Views: 4