RG: For many years now, I have written that the game to watch in modern search is to find out which entity has both the best crawler-based engine technology, combined with the best human social layer.
By comparing the two numbers and filtering out any user-agent that declares itself a bot (spider or crawler) we were able to determine the total percentage of visits without JavaScript enabled. Looking at aimClear’s total organic traffic from every...
In December, Google introduced the smartphone crawler (User agent string: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot...
A Key Discovery from The Find’s Crawler Siva Kumar, CEO and Co-Founder of The Find told SEW that their crawler has discovered "50 million product pages have the Facebook Like button code, but only 1.8 million products have one or more Likes.
I asked Google for clarification on how they recommend the user agent detection for their crawler should be implemented. The technical challenges in supporting a wide array of form factors / screen sizes might make it easier from a technology...
The discussion around the increased emphasis of Google+ in the SERPs has also brought with it some controversy around other social networks in the SERPs (such as Twitter and Facebook), and questions as to what Google does and doesn’t have crawler...
But now that a generation has passed and crawler technology is not as primitive as it was. Developers of content management systems, long the arch enemy of the crawler, build them with being "crawler friendly" in mind.
Ultimately, speaking as a "search expert", I found following a mixture of users and media types on Twitter on a small selection of keywords was more effective than a crawler-based search engine. All of this was done via social networks rather than...
An article could easily be dedicated to just the topic of SEO crawler tools. Xenu is a great little crawler that isn't built specifically for SEO. It's designed to be a powerful crawler that can be configured with regular expression to literally...
Google Crawler May Have Helped Leak Colorado Drug Informants by Frank Watson Microsoft co-founder Paul Allen believes that most of the Silicon Valley tech giants have built their businesses around technology that he owns the patents on.
Efficiency: Enable compression and http conditional GET with If-Modified-Since to give the crawler a break. Certainly Bing is playing catch-up to Google's superior technology and earlier start. They have control of the press (can you think of a day...
However, the real power of Goodzer comes from it's crawler which is able to parse any online store regardless of it's design and if it changes it's website. Zarutskiy had been perfecting the brick and mortar search technology ultimately used to...
Before Yahoo had their own crawler, their backfill results were powered by Inktomi, AltaVista and finally Google (in that order). will continue to drive technology innovation in the search experience to bring more value to users and advertisers alike.
It's undeniably exciting to see companies like BrightEdge and Conductor aiming to beat the black box of search engines by building their own crawler based solutions to monitor rankings. Their technology combines two sources of data via web crawlers...
This is Google's fault because the crawler should always check that pages linked to from sitelinks aren't returning a 404 or other error code. Unfortunately, as with all tests of new technology, this feature has some pretty serious bugs.
Additionally, companies will recognize the importance of doing their homework on keyword research and creating "crawler-friendly" site architectures. US consumers will start catching up with the rest of the world regarding the boundless potential...
Search Wikia Launches Open Source, Distributed Crawler Microsoft acquired aQuantive, Yahoo acquired Right Media, and AOL went on an ad technology buying binge. AOL on Ad Technology Buying Binge Blended and universal search, quality scores, social...
This can prevent a search engine crawler from seeing the link. She particularly likes NetTracker, a technology acquired by Unica and rolled into the NetInsight family of products. In part 1, we looked at three key problem areas with sites that have...
Make sure that you have a crawler friendly web site The Man Responsible for NY Times Going Free, Technology Evangelist Patricia Hursh, Justin Sanger, and Matt Van Wagner made great presentations on the latest in local search and provided some...
Creating an intelligent crawler is one thing, but what about other, more basic tools that can be used to help improve SEO efficiencies? This week, SiLC, the “Super-Intelligent Link Crawler,"™ introduced in SMTrends in early 2007, became fully...
