how to customize http headers when fetch html page ?

[lamysafari]

For example, I want to set UA to 'GoogleBot' to get same result like search engine's crawling .
Thank you!

[B4nan]

You can set the user agent via crawler options, e.g. https://crawlee.dev/api/playwright-crawler/interface/PlaywrightLaunchContext#userAgent

[lamysafari]

thank you. I found that javascript rendering crawler has this option, but not found in sample http crawler like cheerioCrawler that I need.

[B4nan]

there is no browser with cheerio, but you can set it as a header. to do that, define a preNavigationHandler and adjust the second parameter - the gotOptions, which is the options objects for got, the HTTP client we use to run requests

https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#preNavigationHooks

[lamysafari]

What's the difference between gotOptions.headers in preVavigationHooks and request.headers in requestHandler? This is my code and how to verify if it really works?

const cheerioCrawl  = new CheerioCrawler({
    preNavigationHooks: [
        async(_, gotOptions)=> {
            gotOptions.headers = {
                'user-agent': 'GoogleBot'
            }
        }
    ],
    async requestHandler(inputs) {
        const { enqueueLinks, request} = inputs;
        console.log(request.headers); // output: {}
        await enqueueLinks();
    }
})

[B4nan]

When you get to request handler, the request has been already made.