Post History

How to scrape all thread links from a XenForo forum board?

I'm trying to extract all thread links from a specific board on a XenForo forum (like this one: [Creative Writing](https://forum.questionablequesting.com/forums/creative-writing.19/)), but a simple script I tried stops working after the second page.

Why does the script fail on the second page, and how can I fix this script to reliably scrape all pages?

What I've tried is a JavaScript snippet that can run directly in the browser's console. It automatically clicks through all the pages of a forum board and collects every unique thread link it finds.

Here is the complete script. Copy, navigate to the first page of the forum board to scrape, open the browser's developer console (usually by pressing F12), paste the script, and press Enter.

```javascript
(async function() {
    const allLinks = new Set();
    let currentPage = 1;
    let hasNextPage = true;

    while (hasNextPage) {
        console.log(`Scraping page ${currentPage}...`);

        // Get all thread links on current page
        const links = document.querySelectorAll('a[href*="/threads/"]');
        links.forEach(link => {
            if (link.href && !link.href.includes('#') && link.href.includes('/threads/')) {
                allLinks.add(link.href);
            }
        });

        // Check for next page button - XenForo usually uses rel="next"
        const nextButton = document.querySelector('a.pageNav-jump--next, a[rel="next"]');

        if (nextButton) {
            nextButton.click();
            // Wait for the next page to load
            await new Promise(resolve => setTimeout(resolve, 5000));
            currentPage++;
        } else {
            hasNextPage = false;
        }
    }

    // Display and copy results
    console.log(`Finished! Found ${allLinks.size} unique thread links.`);
    const linksArray = Array.from(allLinks);
    const linksText = linksArray.join('\n');

    // Copy to clipboard
    try {
        await navigator.clipboard.writeText(linksText);
        console.log('✅ All links have been copied to your clipboard!');
    } catch (err) {
        console.error('❌ Failed to copy links to clipboard:', err);
        // Fallback: log the links to the console
        console.log('Links found:', linksArray);
    }

    return linksArray;
})();
```