SEO News

How to Use Python to Automate Broken Anchor Text Analysis

Jennifer Law · · 6 min read

How to Use Python to Automate Broken Anchor Text Analysis

In the world of Search Engine Optimization (SEO), link equity is king. Outbound links not only build authority but also serve as critical indicators of content relevance and quality. A major source of SEO pain is the prevalence of “broken anchor text”—text that appears to be a clickable link but points to a non-existent (404) or incorrect destination.

Manually auditing hundreds or thousands of pages for these broken links is time-consuming, tedious, and prone to human error. This is where Python shines. By leveraging Python’s powerful libraries for web scraping and network requests, you can build a robust, automated system to crawl your site, identify all anchor links, and systematically test their validity.

This detailed guide will walk you through the conceptual framework and practical steps using core Python libraries to automate broken anchor text analysis.

🏗️ Prerequisites: Setting Up Your Environment

Before diving into the code, ensure you have a Python environment set up with the necessary libraries.

Required Libraries:
1. requests: For making HTTP requests to check the status of links.
2. beautifulsoup4 (or bs4): For parsing HTML and efficiently extracting anchor text and URLs.
3. urllib.parse: For robust handling of URLs (joining base URLs with relative links, etc.).
4. pandas: (Optional, but highly recommended) For organizing and exporting the final results into a clean spreadsheet.

You can install them using pip:

bash
pip install requests beautifulsoup4 pandas

🕸️ Step 1: Crawling the Target Pages

Your script needs a starting point. Whether you are analyzing a small, defined set of URLs or building a full-scale crawler, the core mechanism is fetching the page content.

Key Concept: To identify all anchor links, you must first crawl the page content and parse the HTML structure.

“`python
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

def get_page_content(url):
“””Fetches the content of a given URL, handling potential connection errors.”””
try:
# Use headers to mimic a real browser, which can prevent blocking
headers = {
‘User-Agent’: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36’
}
response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status() # Raises an HTTPError for bad responses (4xx or 5xx)
return response.text
except requests.exceptions.RequestException as e:
print(f”Error accessing {url}: {e}”)
return None
“`

🔗 Step 2: Extracting All Anchor Links (The Core Logic)

Once you have the HTML content, you use BeautifulSoup to find all <a> tags and extract their href attributes. This process is critical because the href attribute contains the link destination.

“`python
def extract_all_links(html_content, base_url):
“””
Parses HTML content and extracts all valid anchor links.
Handles relative URLs by joining them to the base URL.
“””
if not html_content:
return []

soup = BeautifulSoup(html_content, 'html.parser')
links = set() # Use a set to automatically handle duplicate links

for a_tag in soup.find_all('a'):
    href = a_tag.get('href')
    if href:
        # Use urljoin to resolve relative URLs (e.g., "/about-us" becomes "http://example.com/about-us")
        absolute_url = urljoin(base_url, href)
        links.add(absolute_url)

return list(links)

“`

🔍 Step 3: Testing Link Validity and Status Codes

The most crucial step is testing these extracted URLs. We use the requests library to perform a HEAD or GET request and check the HTTP status code.

Status Code Meanings:
* 200 OK: The link is working (Success).
* 301/302: The link is working but redirects (Often acceptable, but worth noting).
* 404 Not Found: BROKEN LINK (Failure).
* 410 Gone: BROKEN LINK (Failure).
* 5xx: Server error (Failure).

“`python
def check_link_status(url):
“””Checks the HTTP status code of a given URL.”””
try:
# Using HEAD request is efficient as it only fetches headers, not the whole page body.
# We still use GET as a fallback if HEAD is blocked by the server.
response = requests.head(url, timeout=10, allow_redirects=True)
status_code = response.status_code

    if 400 <= status_code < 500:
        if status_code == 404 or status_code == 410:
            return "Broken (404/410)"
        else:
            return f"Client Error ({status_code})"
    elif 500 <= status_code < 600:
        return f"Server Error ({status_code})"
    else:
        # 200-299 range
        return "OK"

except requests.exceptions.RequestException as e:
    return f"Error ({e.__class__.__name__})"

“`

📊 Step 4: Orchestrating the Analysis (The Main Script)

We combine the previous functions into a main execution flow. We will analyze a single target page for demonstration.

“`python
import pandas as pd
import time # To prevent rate limiting

def analyze_page_links(target_url):
“””
Analyzes all links found on the target_url for broken anchor text.
“””
print(f”— Starting analysis for: {target_url} —“)

# 1. Get the raw HTML content
html_content = get_page_content(target_url)
if not html_content:
    return pd.DataFrame()

# 2. Extract all links
links = extract_all_links(html_content, target_url)

results = []

# 3. Test each link
for i, link in enumerate(links):
    print(f"[{i+1}/{len(links)}] Testing {link}...")
    status = check_link_status(link)

    results.append({
        'Source Page': target_url,
        'Broken Link URL': link,
        'Status Code': status,
        'Link Type': 'Anchor Text'
    })

    # Be polite: delay requests to avoid being blocked
    time.sleep(0.5)

# Convert list of dictionaries to a DataFrame
df = pd.DataFrame(results)

# Filter and categorize the output
df['Broken'] = df['Status Code'].apply(lambda x: 'Yes' if 'Broken' in str(x) else 'No')

return df

— EXECUTION —

Define the target URL to audit

TARGET_URL = “http://quotes.toscrape.com/”

Run the analysis

results_df = analyze_page_links(TARGET_URL)

Optional: Save the results to a CSV file

if not results_df.empty:
output_filename = “broken_anchor_text_report.csv”
results_df.to_csv(output_filename, index=False)
print(f”\n✅ Analysis complete. Report saved to {output_filename}”)
else:
print(“\n❌ Analysis failed or no content was retrieved.”)
“`

💡 Best Practices and Enhancements

  1. Rate Limiting (Crucial): Never run link checking against a large site without adding delays (time.sleep(seconds)). Excessive requests can trigger IP bans or be perceived as a DDoS attack.
  2. Error Handling: The provided code includes basic try...except blocks. For professional use, you should add logging to track failures (e.g., tracking why a request failed due to timeouts vs. connection errors).
  3. Full Site Crawl: To analyze an entire website, you would need to implement a Queue System. Start with a list of seed URLs. In each iteration, process the URLs in the queue, extract all internal links found, and add any unvisited, internal links to the queue.
  4. Excluding Assets: Remember to filter out non-page assets like image links (.jpg), social media links (t.co), and mailto links, as these are not typically the focus of traditional SEO anchor text analysis.
  5. Handling Authentication: If your site requires a login, you must adapt your get_page_content function to handle sessions (using requests.Session()) to maintain cookies and authentication headers.
Post Views: 12