SEO News

Web Crawling

RSS
  1. Why Yahoo Wants to Move Into Contextual Search and How it Might Work For Them

    Unfortunately, it's not as simple as taking their old search algorithm, start crawling the web again, and expect they can compete against Google. Marissa Mayer, CEO of Yahoo, has been commenting a lot recently about how she wants to get Yahoo back...

  2. Matt Cutts: Facebook, Twitter Social Signals Not Part of Google Search Ranking Algorithms

    We have to crawl the web in order to find pages on those two web properties and we've had at least one experience where we were blocked from crawling for about a month and a half. Cutts also noted a big problem with crawling social profiles: they...

  3. Google Webmaster Tools Now Finds Smartphone Crawl Errors

    They’ve made some changes to their crawl errors page to include specific smartphone crawl errors that the Googlebot-Mobilebot discovers while crawling the web as a mobile useragent. Blocked URLs: A blocked URL is when the site's robots.txt...