User Agent Articles

Page 1 of 1 (3 articles)

4/13/2019 • EN

A guide to building a polite web crawler that respects robots.txt, manages crawl frequency, and avoids overloading servers, with an example in .NET.

Crawl Frequency http requests robots.txt User Agent web scraping

3/21/2017 • EN

How to set a custom User-Agent string for all requests in a UWP WebView control using a Win32 API.

Urlmondll User Agent Uwp Webview Win32 API

4/20/2013 • EN

A historical and technical look at the evolution of User-Agent strings, born from the browser wars and used for feature detection.

browser wars Firefox internet explorer User Agent Web Development

Select Language