Automating Broken Anchor Text Analysis with Python

As a web developer or SEO specialist, you’re likely familiar with the importance of internal linking and anchor text optimization for website navigation and user experience. However, broken anchor texts can lead to frustration and errors on your site, affecting both users and search engines alike.

In this article, we’ll explore how to use Python to automate broken anchor text analysis, making it easier to identify and fix these issues on your website.

Understanding Broken Anchor Texts

Broken anchor texts occur when a link’s target URL is no longer valid or has changed. This can happen due to various reasons, such as:

• A page or resource being deleted or moved
• A typo in the URL or anchor text
• A server-side issue or network error

Identifying and fixing broken anchor texts can improve user experience, reduce bounce rates, and enhance overall website performance.

Python Library for Web Scraping: BeautifulSoup

For web scraping and parsing HTML documents, we’ll use beautifulsoup4. This powerful library allows us to navigate through a webpage’s structure and extract relevant data.

bash
pip install beautifulsoup4 requests

Here’s an example of how to scrape the links from a given page:

“`python
import requests
from bs4 import BeautifulSoup

def get_page_source(url):
response = requests.get(url)
soup = BeautifulSoup(response.text, ‘html.parser’)
return soup

url = “https://example.com”
soup = get_page_source(url)

links = [a[‘href’] for a in soup.find_all(‘a’)]
print(links)
“`

Scraping Anchor Texts and URLs

To extract anchor texts and their corresponding URLs, we can modify the previous code snippet:

“`python
import requests
from bs4 import BeautifulSoup

def get_anchor_texts(url):
response = requests.get(url)
soup = BeautifulSoup(response.text, ‘html.parser’)

anchor_texts = []
for a in soup.find_all('a'):
    if a.has_attr('href'):
        href = a['href']
        text = a.text
        anchor_texts.append((text, href))

return anchor_texts

url = “https://example.com”
anchor_texts = get_anchor_texts(url)
print(anchor_texts)
“`

Broken Anchor Text Detection and Reporting

Now that we have the anchor texts and their URLs, let’s create a function to detect broken links and report them:

“`python
import requests
from bs4 import BeautifulSoup

def detect_broken_links(anchor_texts):
broken_links = []

for text, href in anchor_texts:
    try:
        response = requests.get(href)
        if not response.ok:
            print(f"Broken link: {href}")
            broken_links.append((text, href))
    except requests.RequestException as e:
        print(f"Error checking link: {href} - {e}")
        broken_links.append((text, href))

return broken_links

anchor_texts = get_anchor_texts(“https://example.com”)
broken_links = detect_broken_links(anchor_texts)
print(broken_links)
“`

Putting it all Together

To automate the process of detecting and reporting broken anchor texts, we can combine the previous functions into a single script:

“`python
import requests
from bs4 import BeautifulSoup

def get_page_source(url):
response = requests.get(url)
soup = BeautifulSoup(response.text, ‘html.parser’)
return soup

def get_anchor_texts(soup):
anchor_texts = []
for a in soup.find_all(‘a’):
if a.has_attr(‘href’):
href = a[‘href’]
text = a.text
anchor_texts.append((text, href))
return anchor_texts

def detect_broken_links(anchor_texts):
broken_links = []

for text, href in anchor_texts:
    try:
        response = requests.get(href)
        if not response.ok:
            print(f"Broken link: {href}")
            broken_links.append((text, href))
    except requests.RequestException as e:
        print(f"Error checking link: {href} - {e}")
        broken_links.append((text, href))

return broken_links

url = “https://example.com”
soup = get_page_source(url)
anchor_texts = get_anchor_texts(soup)
broken_links = detect_broken_links(anchor_texts)

print(“Broken links:”)
for text, href in broken_links:
print(f”{text}: {href}”)
“`

This script will scrape the anchor texts and URLs from a given webpage, detect any broken links, and report them.

By automating this process with Python, you can regularly scan your website for broken anchor texts, improving user experience and reducing errors on your site.