Welcome to Software Development on Codidact!
Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.
Tackling net::ERR_NAME_NOT_RESOLVED and timeout error on browser object creation when using Puppeteer
Used to work with PhantomJS. Want to upgrade to Puppeteer. Started with some code:
"use strict";
const puppeteer = require("puppeteer");
const capture = async () => {
let browser;
try {
console.time('myTimer');
console.info("Opening browser ...");
const browser = await puppeteer.launch({
headless:true,'ignoreHTTPSErrors':true,devtools:false
});
const page = await browser.newPage();
page.on('console', (msg) => console.log('DEBUG:', msg.text()) );
console.info("browser instance created.");
await page.evaluate(() => console.log(`URL="${location.href}".`));
await page.goto("https://www.chunkbase.com/apps/seed-map");
// await page.goto("https://checkip.amazonaws.com/");
await page.evaluate(() => console.log(`URL="${location.href}".`));
console.info("getting webpage dimensions.");
const getDimensions = await page.evaluate(() => {
return {
pxH: document.documentElement.clientHeight,
pxW: document.documentElement.clientWidth,
scale: window.devicePixelRatio
};
});
console.info("Dimensions: ", getDimensions);
console.info("capturing.");
await page.screenshot({ path:"D:/screenshot.png", type:"png", fullPage:false });
console.timeEnd('myTimer');
} catch (err) {
console.warn("Could not create a browser instance: ", err);
return;
} finally {
await browser.close();
}
};
capture();
Output:
Opening browser ...
browser instance created.
DEBUG: URL="about:blank".
DEBUG: [.WebGL-000022F60095A300]GL Driver Message (OpenGL, Performance, GL_CLOSE_PATH_NV, High): GPU stall due to ReadPixels
DEBUG: fun-hooks: referenced 'adpod' but it was never created
DEBUG: %cVideoManagerComponent::noStickyPlaylistOrSekindo color: #999; font-weight: bold; JSHandle@object
DEBUG: %cBaseDynamicAdsInjector::_logDensityInfo color: #999; font-weight: bold; JSHandle@object
DEBUG: fun-hooks: referenced 'checkAdUnitSetup' but it was never created
DEBUG: fun-hooks: referenced 'checkAdUnitSetup' but it was never created
DEBUG: fun-hooks: referenced 'checkAdUnitSetup' but it was never created
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
DEBUG: Failed to load resource: net::ERR_NAME_NOT_RESOLVED
Could not create a browser instance: TimeoutError: Navigation timeout of 30000 ms exceeded
at ~\node_modules\puppeteer\lib\cjs\puppeteer\common\LifecycleWatcher.js:108:111 ~\scripts\chunkbase.js:51
await browser.close();
^
TypeError: Cannot read properties of undefined (reading 'close')
Notes:
- With the current
page.gotocode Puppeteer- Throws a lot of output.
- Times out after 30 seconds.
- The page dimensions are skipped.
- The screen capture never occurs.
- If I set
headlessto false, the page renders for the most part. Some parts dont load.
- If I swap around the URLs in the
page.gotoPuppeteer- works fast without a ton of
DEBUGs in output. - The page dimensions are calculated.
- The screen capture occurs.
- works fast without a ton of
Questions:
- What's the proper way to initialize
browserto correct the.close()error within the try-catch? - How can I troubleshoot this further? Especially where these
net::ERR_NAME_NOT_RESOLVEDerrors come from (is it DNS-related)?
Open to all sorts of suggestions, on-line reading material, etc.
Thanks.
1 answer
To fix TypeError: Cannot read properties of undefined (reading 'close'), remove the const in the following code, which ensures your let browser outside the block will be assigned in the common case when browser.launch() doesn't throw:
- const browser = await puppeteer.launch({
+ browser = await puppeteer.launch({
But if browser.launch() throws, then let browser will be undefined, so a throw is still possible in the finally block. Handle this in your finally block by using optional chaining:
- await browser.close();
+ await browser?.close();
Also, as you're seeing, the
console.warn("Could not create a browser instance: ", err);
log is misleading, since you're clearly able to create a browser instance. This catch block will run if any errors throw within the main automation block, which could be more than just failing to create a browser instance. That's only the first line of the block. I would make it:
- console.warn("Could not create a browser instance: ", err);
+ console.error(err);
As for your other question, the following listener forwards all logs from the website console into your Node console:
page.on('console', (msg) => console.log('DEBUG:', msg.text()) );
The output you're seeing is mostly harmless noise logged to a greater or lesser extent from site to site. I generally wouldn't add this listener since most large sites spam the console with random resource loading failures like this.
Generally speaking, I recommend the following goto:
await page.goto(url, {waitUntil: "domcontentloaded"});
which is least likely to get stuck and throw a loading timeout. If you're taking a screenshot, wait for all images to load manually before doing so ("networkidle0" can get stuck, as can the default "load" predicate).
Furthermore, the many sites take anti-bot measures, and headless mode makes bots much easier to detect. The default user agent basically says "I am a bot", so you can start by setting it to a normal mobile browser user agent
const ua =
"Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Mobile Safari/537.3";
await page.setUserAgent(ua);
If that doesn't work, you can try the Puppeteer stealth plugin or other fingerprint obfuscating measures. Or use headful mode.
Note that browser automation has few silver bullets, so a script that can automate one site may not work on another. You'll generally need to adapt your code a bit to determine when a particular page is fully loaded or otherwise in an actionable state, depending on what you want to accomplish on it.
0 comment threads
1 comment thread