RG: For many years now, I have written that the game to watch in modern search is to find out which entity has both the best crawler-based engine technology, combined with the best human social layer.
DuckDuckGo gets its results from over 50 sources, including DuckDuckBot (our own crawler), crowd-sourced sites (in our own index), Yahoo! Tech dreamers with aspirations of creating a search engine to rival Google: take heart in the story of...
Earlier this summer, Blekko upgraded their Scoutjet crawler to bring back more comprehensive, real-time data for SEOs. Last week, the slashtag search engine changed to a paid model for SEO data. Blekko CEO Rich Skrenta confirmed the change to a...
URLs as Bing, but given Google treats smartphones as desktop browsers (even with the new smartphone crawler) and your content rankings are not gaining any benefit from your shared m.links and.and.well again, why would you want two domain URLs?
FREE – Xenu: Run a crawler over your site and find all your broken links. Search Engine Watch: I'm not an employee of Search Engine Watch but I am a biased author. Search Engine Land: Another good daily news source.
The discussion around the increased emphasis of Google+ in the SERPs has also brought with it some controversy around other social networks in the SERPs (such as Twitter and Facebook), and questions as to what Google does and doesn’t have crawler...
But now that a generation has passed and crawler technology is not as primitive as it was. Developers of content management systems, long the arch enemy of the crawler, build them with being "crawler friendly" in mind.
Grieselhuber told Search Engine Watch their crawler is also sensitive to differences by market, customizing recommendations on a market-by-market basis. Global SEO software company GinzaMetrics recently announced conversion tracking improvements...
All the other cool metrics like inbound links, subscriber stats, and Googlebot crawler reports are still in Webmaster Tools. All three can be found under the Traffic Sources section of Google Analytics, in a section labeled Search Engine Optimization.
Check out your Robots.txt (Site configuration -> Crawler access). Google does a nice job with their tools, but allows you to get the viewpoint from a second search engine, and you will see different things.
Ultimately, speaking as a "search expert", I found following a mixture of users and media types on Twitter on a small selection of keywords was more effective than a crawler-based search engine. All of this was done via social networks rather than...
The image and video sitemaps are great ways to speak crawler, (XML) and inform Googlebot about the images on your site, their residing URL, ALT text and similarly, the pages where your videos can be found, any corresponding tags, duration...
An article could easily be dedicated to just the topic of SEO crawler tools. Xenu is a great little crawler that isn't built specifically for SEO. It's designed to be a powerful crawler that can be configured with regular expression to literally...
Xenu Link Sleuth -- Best described as a PC-based crawler, this handy tool spiders your website for links and ensures they are valid, executable (when pointed at files), and search engine friendly (if redirects).
However, the real power of Goodzer comes from it's crawler which is able to parse any online store regardless of it's design and if it changes it's website. Today, software engineers Dmitry Pakhomkin and Artem Zarutskiy, have launched a location...
While Blekko's crawler indexes the web itself, the social layer built into the service gives users the power to create their own search engine. However, this is not a spurious analogy - if you think of it like a flat packed search engine, you'll be...
If you previously blocked the Bing crawler but allowed the Yahoo crawler access to your site, you'll need to allow access to BingBot to continue to be seen in Yahoo search results. This transition period is designed to provide time for Bing to...
Especially when you're concerned with SEO: you want to know when, if, and how the crawler based search engines spider your pages, which search queries (a.k.a.keywords) are pulling visitors to your site, whether someone has linked to you out of the...
If there was an activity stream for the Google Crawler calculating PageRank, arguably it would say something very similar. A little over a month ago, we reported this story "Facebook In Early Stage Search Engine Tests?
It's undeniably exciting to see companies like BrightEdge and Conductor aiming to beat the black box of search engines by building their own crawler based solutions to monitor rankings. It's unusual and reminiscent of the traditional PR metric of...
