If you own a website and would like to be visible in search results, for example, Google search results, you need to be visited by its crawlers first. Often they map content that they find to use later for search purposes (indexing), or help developers diagnose issues with their websites. In other words, they visit webpages, find links to further pages, and visit them, too. Nonetheless, crawlers (called spiders sometimes) are computer programs (bots) that crawl the web. If you’re seeking a way to detect and verify crawlers, you probably already know what they are. To learn more about search-friendly JavaScript delivery, you can watch this Google I/O ‘18 presentation. In order to do that, we have to be able to detect whether a request is made by a user or a bot. This technique is not considered cloaking and is allowed by Google.
One solution is to present crawlers with a pre-rendered version of the HTML file instead of the JavaScript code. Googlebot doesn’t interact with your website like a normal user would, and this may prevent it from discovering some of your content, particularly if it’s dependent on JavaScript. However, even Google has some limitations. It leads not only in market share but also in technology. In the search market, Google is the unquestionable leader. While users get the benefit of modern technology (if they’re willing to update their browsers), web crawlers struggle with it.
Every day, more and more sites switch from pure HTML to JavaScript-enriched websites.
0 kommentar(er)
