The Abridged Version: Independent Report On Google’s Click Fraud Detection Practices

Last Friday, an independent report on how Google deals with click fraud was
published as part of the ongoing
Lane’s Gifts v.
class action lawsuit over click fraud. To my knowledge, it is the
most comprehensive, detailed public look into how Google deals with click fraud
that’s ever come out. It finds that Google’s efforts to combat the issue have
been reasonable, though there are some eyebrow raising bits on how the author
only finds the situation was in control by the end of 2005 and how it’s
impossible to fully know whether some clicks are invalid — and thus,
potentially — impossible to prevent some types of fraud through purely
automated means.

The report is long, a 47 page PDF file. Anyone interested in click fraud
issues should give it a thorough read. But given how everyone’s always busy, I
thought I’d highlight below a number of sections that stood out in my review of
the document.

The report is by Dr.
Alexander Tuzhilin
, Professor of Information Systems at New York University.
To prepare it, he says in the Executive Summary at the beginning (page 1):

I have been asked to evaluate Google?s invalid click detection efforts and to
conclude whether these efforts are reasonable or not. As a part of this
evaluation, I have visited Google?s campus three times, examined various
internal documents, interviewed several Google?s employees, have seen different
demos of their invalid click inspection system, and examined internal reports
and charts showing various aspects of performance of Google?s invalid click
detection system. Based on all these studied materials and the information
narrated to me by Google?s employees, I conclude that Google?s efforts to combat
click fraud are reasonable. In the rest of this report, I elaborate on this

Immediately, the first thing that comes to mind is that he makes no mention
of talking with individual advertisers, which could lead you to think that if
he’s only talking with Google, of course he’s likely to come away with the idea
that Google is doing everything just fine.

When you read the report, it’s clear this isn’t the case. Google does come
under criticism. It’s also important to realize Tuzhilin was not employed by Google to
create this report. He’s an independent expert appointed to my knowledge by the
court. Exactly how he was selected is unclear, and I do think it would be a
better report if advertiser data had been involved. But there’s still plenty of
good stuff here to digest.

Page 2 covers his background and materials reviewed from Google to prepare
the report.

Page 3 and some of page 4 covers those he talked with at Google. Interesting
details are that Google’s click quality team consists of about 36 people,
one-third engineers looking to design detection systems and the remaining
two-thirds dedicated to doing manual investigations of suspected fraud.

Pages 4 through 6 cover the history of the internet, search engines and
Google, most of which isn’t that necessary for most experienced search
marketers. Page 7 talks about three main ways of purchasing advertising:

  • CPM – cost per impression
  • CPC – cost per click
  • CPA – cost per action

Again, basic stuff. But it’s worth touching on because of some of the current
debate that Google and other search engines will be forced to go to CPA pricing
to fully eliminate fraud.

On page 8, Tuzhilin lends some support of this, or at least the problems that
others have raised with CPC:

Although currently popular, the CPC/PPC model has two fundamental problems:

  • Although correlated, good click-through rates (CTRs) are still not
    indicative of good conversion rates, since it is still not clear if a
    visitor would buy an advertised product once he or she clicked on the ad. In
    this respect, the CPA-based models provide better solutions for the
    advertisers (but not necessarily for the search engines), since they are
    more indicative that their ads are ?working.?
  • It does not offer any ?built-in? fundamental protection mechanisms
    against the click fraud since it is very hard to specify which clicks are
    valid vs. invalid in general, as will be explained in Section 8 (it can be
    done relatively easily in some special cases, but not in general). For this
    reason, major search engines launched extensive invalid click detection
    programs and still face problems combating click fraud.

In response to these two problems and for various other business reasons,
Google is currently testing a CPA payment model, according to some reports in
the media. Some analysts believe that the conversion-based CPA model is more
robust for the advertisers and also less prone to click fraud. Therefore, they
believe that the future of the online advertising payments lies with the CPA
model. Although this is only a belief that is not supported by strong evidence
yet, Google is getting ready for the next stage of the online advertising

What Will Replace Pay-Per-Click Advertising?
over at Publishing 2.0 from
Scott Karp is a good roundup and debate on some of the issues of CPA perhaps as
the solution to CPC issues.

I’ve posted lots of comments in Karp’s post, but my personal
view is this. Currently, Google is offering all three major payment systems:
CPC, CPM and CPA. It is offering all three not just because of fraud issues but
because advertisers have different goals with advertising, where different
payment models may be required.

Building brand? You want impressions perhaps more than clickthrough, and
suddenly CPM makes sense. Really savvy with conversion tracking? CPA might make
more sense for you, as a way for you to feel less likely to be exposed to fraud
and more likely to really be paying only for key traffic. Fairly rudimentary
with conversion tracking? Doing low-cost CPC ads might make a lot of sense, for
your situation. And beyond the three big ones, I’m sure we’ll see other options
emerge. The unifying goal around all of them, from Google’s perspective, will be
figuring out a way to help advertisers track that the ads are working according
to some type of metrics that the advertisers want.

Skipping down past background on how AdWords works and the AdSense program (AdSense
For Domains doesn’t get mentioned, though it’s a major program), page 13 starts
in on what Google can tell about clicking activities.

Google is apparently
making use of conversion data that advertisers provide to determine if
fraudulent clicks are happening. My understanding was that conversion data was
supposed to be ringfenced and not used by Google for anything, not even in the
aggregate. But perhaps the policy has changed or perhaps I misunderstood this.
I’ll check on that (and also note that confusingly, the report says on page 34
that "None of the filters uses the conversion information that Google collects"). Certainly Google made no such restrictions when it launched
Google Checkout. But even with conversion data, the report notes using this info
isn’t perfect.

Google collects various types of information about querying and clicking
activities, including certain types of ?post-clicking? data about conversion
actions on the advertiser?s website where the visitor is taken following the
click. All this data accumulated by Google is extracted from various sources and
contains comprehensive information about visitor?s activities on the Google

As stated before, the conversion data ? the ?post-clicking? data about
conversion actions on the advertiser?s website ? constitutes an important piece
of this collected data. In particular, if the advertiser formally agrees to
provide this information, Google collects data on whether or not the user
visited certain designated pages on the advertised website that the advertiser
marked as ?conversion? pages, such as the checkout page and certain form filling
pages. This conversion data is limited to what the advertiser decided to provide
to Google and is not as rich as the clickstream data collected by advertisers
themselves on their websites. Also, many advertisers decide to opt out from
providing this conversion data. In this case, Google does not have any
conversion information and therefore does not know what happened after a visitor
clicked on the ad. Nevertheless, this post-clicking conversion data is important
for Google even in its limited form because it conveys some intentions of the
visitors on the advertised website and provides good insights into whether or
not the visitor is seriously considering purchasing the advertised product or

This ?raw? clicking data described above is subsequently cleaned,
preprocessed and stored in various internal logs by Google for different types
of subsequent analysis conducted on this data.

One inherent weakness of Google?s (or any other search engine) data
collection effort that is important for detecting invalid clicks, is inability
to get full access to all the clicking activities of the visitors of the
advertised website. In other words, the conversion data that Google collects
provides only a partial picture of all the post-clicking activities of the
visitor on the advertised website. This data is important for detecting invalid
clicks since better invalid click detection methods can be developed using this
data. Unfortunately, Google (and other search engines) does not have full access
to this data, unless the advertised website decides to provide its clickstream
data to Google, which many websites are reluctant to do. However, this is not
Google?s fault ? this is an inherent limitation of the types of data available
to Google.

While it might not be perfect, the report also notes at the end of this
section that no one has the perfect collection of information:

However, this lack of full conversion data available to Google is compensated
by various types of querying and clicking data that Google can collect, whereas
advertisers and third-party vendors cannot. Therefore, there exists a tradeoff
between the types of data relevant for detecting invalid clicks that is
available to Google, advertisers and the thirdparty vendors. None of these three
groups have the most comprehensive set of data pertinent to detecting invalid
clicks, and each of them needs to settle for the invalid click detection methods
possible only with the data that they have.

On page 14, the report addresses the frustration advertisers feel over the
relatively non-granular nature of Google’s reporting versus Google’s need to
keep some things carefully protected:

The smallest unit of analysis is one day. For example, the number of invalid
clicks on an ad detected by Google (or any other related statistic) can only be
reported on a daily basis (although there are certain alternative methods of
obtaining aggregation granularity that is smaller than a day). In other words,
advertisers cannot know if a particular click on a particular ad was marked as
valid or invalid by Google, and Google refuses to provide this information to

This is a source of contention and dispute between Google and the
advertisers, and one can understand both parties in this dispute. On one hand,
the advertiser has the right to know why a particular click was marked as valid
by Google (when the advertiser thinks that it is invalid) because the advertiser
pays for this click. On the other hand, if Google discloses this information, it
opens itself to click fraud on a massive scale because, by doing so, it provides
certain hints about how its invalid click detection methods work. This means
that unethical users will immediately take advantage of this information to
conduct more sophisticated fraudulent activities undetectable by Google?s

This conflicting dilemma between advertisers? right to know and Google?s
inability to provide the appropriate information to advertisers because of the
security concerns is part of the Fundamental Problem of the PPC advertising
model to be discussed in the next section. More recently, Google tried to bridge
this gap between Google and the advertisers.

Page 15 spends time looking at various definitions of click fraud, bringing
us to page 16 which raises the bigger issue that it is impossible to know the
intent of ALL clicks, which is crucial to understand what chunk of them might be

Unfortunately, in several cases it is hard or even impossible to determine
the true intent of a click using any technological means. For example, a person
might have clicked on an ad, looked at it, went somewhere else but then decided
to have another look at the ad shortly thereafter to make sure that he/she got
all the necessary information from the ad. Is this second click invalid? To make
things even more complicated, the second click may not be strictly necessary
since the person remembers the content of the ad reasonably well (hence there is
no real need for the second click). However, the person may not really like or
care about the advertiser and decides to make this second click anyway (to make
sure that he/she did not miss anything in the ad and his/her information is
indeed correct) without any concerns that the advertiser may end up paying for
this second click (since the person really does not care about the advertiser
and his/her own interests of not missing anything in the ad overweigh the
concerns of hurting the advertiser). Therefore, in some cases the true intent of
a click can be identified only after examining deep psychological processes,
subtle nuances of human behavior and other considerations in the mind of the
clicking person.

Soon after this, on page 17, comes the first real bombshell to me. As said
above, you can’t detect the intent of all clicks. Given this, there’s no
reasonable way to be certain that technological fixes for click fraud detection
are working:

In summary, between the obviously clear cases of valid and invalid clicks,
lies the whole spectrum of highly complicated cases when the clicking intent is
far from clear and depends on a whole range of complicated factors, including
the parameter values of the click. Therefore, this intent (and thus the validity
of a click based on the above definitions) cannot be operationalized and
detected by technological means with any reasonable measure of certainty.

What? Didn’t the report find Google was acting reasonably? Yes, and I think
this is is because as the report goes on, it’s because Google’s not relying
solely on automated means to stop click fraud, which might allow some clicks to
get through, if that were only the case.

Page 18 picks of the issue even more strongly, and I’ve bolded this section
because it deserves special attention. Note that the italics were originally

The last statement has one important implication: given a particular click
in a log file, it is impossible to say with certainty if this click is valid
or not in
all the cases. This means that

  • It is impossible to measure the true rates of invalid clicking
    activities, and all the reports published in the business press are only
    at best.
  • The invalid click detection methods need to be developed without
    a proper operationalizable conceptual definition of invalid clicks.

The important word above is all the cases since in some cases it
can be stated with certainty if a particular click is valid or not. For example,
it is easy to detect a doubleclick using relatively simple technological means,
assuming that the doubleclick is invalid.

Again, it seems to be a case that automation can catch some, perhaps lots of click
fraud, but it can’t catch all of it because the intent problem. Also crucial in
the above is the stressing that rates we’ve been given from various sources are
simply guesses, since the intent of clicks aren’t know to some of these other

Indeed, in the case of the recent Outsell
report, you
don’t even have to worry about figuring out the intent of particular clicks.
Click fraud stats from that report come from half the panel entirely guessing
about what click fraud rates they might have — guessing, because that half does
not auditing of clicks at all.

Page 19 deals with ways of identifying invalid clicks, at least according to
operational approaches — IE, automated criteria. Do the clicks show some type

  1. Anomaly from past clicking patterns for a site or ad?
  2. Violate certain predefined rules?
  3. Fall into certain classes of behavior that make them deemed invalid?

Page 20 explains that Google primarily depends on the first two approaches —
looking for anomalies and using rules — but then gets into what it stresses as
the "Fundamental Problem" of fraudulent clicks:

We conclude that there is a fundamental problem associated with the
definition of invalid clicks for the Pay-per-Click model. This problem can be
summarized as follows:

  • There is no conceptual definition of invalid clicks that can be
    operationalized in the sense defined above.
  • An operational definition cannot be fully disclosed to the general
    public because of the concerns that unethical users will take advantage of
    it, which may lead to a massive click fraud. However, if it is not
    disclosed, advertisers cannot verify or even dispute why they have been
    charged for certain clicks.

This problem lies at the heart of the click fraud debate and constitutes the
main problem of the CPC model: it is inherently vulnerable to click fraud.

Page 21 poses solutions to the problem:

  • The ?trust us? approach of the search engines. The search engines can
    assure advertisers that they are doing everything possible to protect them
    against the click fraud. This is not easy because of the inherent conflict of
    interest between the two parties: the money from invalid clicks directly
    contribute to the bottom lines of the search engines. Nevertheless, it may be
    possible for the search engines to solve this trust problem by developing
    lasting relationships with the advertisers. However, the discussion of how
    this can be done lies outside of the scope of this report.
  • Third-party auditors. Independent third-party vendors, who have no
    financial conflicts of interest, can work with advertisers and audit their
    clickstream files to detect invalid clicks.

These two approaches would still constitute only a partial solution to the
Fundamental Problem because there is no conceptual definition of invalid clicks
that can be operationalized.

Page 21 continues on looking at how Google does click fraud detection,
covering a range of general preventative measure and more active things done
when clicks actually happen.

On page 23, a look at filtering systems begins, ending with this summary
that’s positive for Google, at the moment. It also stresses that filtering will
always come under new challenges:

The current set of Google filters is fairly stable and only requires periodic
?tuning? and ?maintenance? rather than a radical re-engineering, even when major
fraudulent attacks are launched against the Google Network. It also demonstrates
that various recent efforts of the Click Quality team to improve performance of
their filters produce only incremental improvements. Thus, the Click Quality
team currently reached a stability point since additional efforts to enhance
filters produce only marginal improvements.

Having said this, the Click Quality team also realizes that this is only a
local stability point in the sense that major future modifications in clicking
patterns of online users and new types of fraudulent attacks against Google can
lead to radically new types of invalid clicks that the current set of filters
can miss. Therefore, the Click Quality team is working on the next generation of
more powerful filters that will monitor a broader set of signals and more
complex monitoring conditions. These new filters will require a more powerful
computing infrastructure than is currently available, and the Click Quality team
also participates in developing this infrastructure. Their overall goal is to
make click spam hard and unrewarding for the unethical users thus making it
uneconomical for them and turning many of them away from Google and the Google

At page 28, the expert notes that Google’s filters are relatively simple in
nature, yet they work:

The structure of most of Google?s filters, with a few exceptions, is
surprisingly simple. I was initially puzzled and thought that Google did not do
a reasonable job in developing better and more sophisticated filters. I was
initially certain that these simple filters should miss many types of more
complicated attacks. However, the evidence reported in the previous two sections
indicates that these simple filters perform reasonably well.

Why? A variety of reasons, such unsophisticated attacks:

Although some of the coordinated attacks can be quite sophisticated, the
majority of the invalid clicks usually come from relatively simple sources and
less experienced perpetrators….Still, there are certain types of attacks that Google filters will miss; but
these attacks should be quite sophisticated and would require significant
ingenuity to launch. Therefore, there cannot be too many of these, unless
perpetrators become much more imaginative….

The Long Tail / Search Tail even
gets a mention, with the idea being that — if I understand correctly — most
activity focuses around the same type of things that the filters work well to
detect. IE, the filters do well at cutting off the head of click fraud — and if
tail activity gets through, it’s relatively little in comparison:

Despite its current reasonable performance, this situation may change
significantly in the future if new attacks will shift towards the Long Tail of
the Zipf distribution by becoming more sophisticated and diverse.

At the bottom of page 29, the report starts examining whether Google is
letting stuff slide to earn more money:

Since Google does not charge advertisers for invalid clicks, this means that
it loses money by filtering out these clicks. Thus, there is a financial
incentive for Google not to forgo some of these revenues and simply be ?easy?
Long Tail Left Part Frequency Rank 30 on filtering out invalid clicks.
Therefore, it is important to know if any business considerations entered into
the filter specification process or is it entirely determined by Google?s
engineers in an objective manner with a single purpose to protect the advertiser
base. This is one of the important issues that I investigated as a part of my
studies of how Google manages detection of invalid clicks….

The conclusion is that Google isn’t trying to favor itself:

I have spent a significant amount of time trying to understand who sets these
threshold parameters, how, and what are the procedures and processes for setting
them. In particular, I tried to understand if it is an entirely engineering
decision that tries to protect the advertisers from invalid clicks or any of the
business groups at Google are involved in this decision process with the purpose
of influencing it towards generating extra revenues for Google.

As a result of these investigations, I realized that it constitutes
exclusively an engineering decision with no inputs from the finance department
or the business units, except the following two cases:

  • The first one was a special case when one particular IP address was
    disabled because of inappropriate clicking activities, and a business unit
    requested the Click Quality team to conduct an additional investigation since
    it was an important customer associated with that IP address, and restore it
    if the investigation results were negative. When I was explained what had
    happened, I felt that Google?s actions were reasonable in this particular
  • The change in the doubleclick policy that was considered in Winter 2005
    and implemented in March 2005. It turned out that the change in the
    doubleclick policy (i.e., not to charge advertisers for the immediate second
    click in a doubleclick) had non-trivial financial implications for Google.
    Being a publicly traded company at that time, this change would have had a
    noticeable effect on Google?s total revenues with corresponding implications
    for the financial performance of the company. Therefore, this policy change
    had legitimate concerns for Google?s management, and these financial
    implications have been discussed in the company. Still, despite its noticeable
    negative effects on its financial performance, Google decided to abandon the
    old doubleclick policy and not to charge advertisers for the second click,
    which was an appropriate action to take.

In conclusion, with the exception of the doubleclick, I found Google?s
processes for specifying filters and setting parameters in these filters driven
exclusively by the consideration to protect the advertiser base, and, therefore,
being reasonable.

Doubleclick constitutes a special case. For me, the second click in the
doubleclick is invalid, as I argued in Section 8, and the advertisers should not
be charged for it. It is not clear to me why it took Google so long to revise
the policy of charging for doubleclicks. Nevertheless, this policy was revised
in March 2005 despite the fact that the company lost ?noticeable? revenues by
taking this action.

I find the conclusion that Google wasn’t trying to benefit itself doesn’t
mesh well with the expert’s own concern/confusion/uncertainty about why Google
took so long to change its policy on doubleclicks. Moreover, that entire policy
isn’t well explained. Way back up on page 20, there’s this very brief mention:

It turns out that Google had a history associated with the definition of a
doubleclick: at some point doubleclick was considered to be a valid click and
advertisers were charged for it, while subsequently Google reconsidered and
treated doubleclick as invalid.

And that’s it until the section later in the report, where Google’s
effectively accused of footdragging on changing its policy, where business
discussions about the change were made, but Google then seems to be given the
all clear because eventually it did the right thing.

The entire matter is something that feels like it should have been explored
more, but page 31 sheds light as to why this might have been difficult. Google’s
apparently had a complete staff change in relation to click fraud detection
since it began charging by the click:

In this subsection, I will describe the history of development of Google
filters. First of all, I would like to point out that most of the descriptions
in this subsection are not based on documents provided to me by Google but
rather on the verbal descriptions by the members of the Click Quality team based
on their recollections of the past events and on the ?folklore? evidence since
none of the team members I interviewed were even around or involved in the click
fraud effort when the AdWords program was introduced in February 2002.

The section continues with detection divided into these groupings — and I’ve
bolded a key part:

  • The Early Days (February 2002 ? Summer 2003). These were the early days of
    the PPC model and of the click fraud characterized by extensive learning about
    the problem and determining ways to deal with it.
  • The Formation Stage (Summer 2003 ? Fall 2005). This stage started with the
    introduction of the AdSense program in March 2003, formation of the Google
    Click Quality team in the Spring/Summer 2003, launch of new filters and the
    intent to take the invalid click detection efforts to the ?next level.? It
    ended with the development of the whole infrastructure for combating invalid
    clicks and the consolidation of Google?s invalid click detection efforts. This
    stage was characterized by significant progress in combating invalid clicking
    activities and developing mature systems and processes for accomplishing this
    task. Although the Click Quality team?s solutions were still not perfect,
    based on the information provided to me by Google, I reached the conclusion
    that the invalid clicking problem at Google was ?under control? by the end of
  • The Consolidation Stage (Fall 2005 ? present). By this time, Google had
    enough filters and perfected them to the level when they would detect most of
    the invalid clicking activities in the Left Part of the Zipf distribution (see
    Figure 1) and some of the attacks in the Long Tail. They would still miss more
    sophisticated attacks 32 in the Long Tail, and the Click Quality team
    continued working on the neverending process of improving their filters to
    detect and prevent new attacks. The Click Quality team has also been working
    on enhancing their infrastructure and improving their processes….

What? Click fraud wasn’t under control until the end of 2005, yet Google is
said to have acted reasonably by the report? How does this make sense? The best
explanation seems to be that as the report goes on, the author feels click fraud
was an evolving problem, and that Google was reasonably reacting to prevent it
even though it wasn’t "under control" until the end of last year. In contrast,
had Google been doing nothing, then it might have been deemed not to have been
taking reasonable steps to gain control.

Page 32 looks at the early days and notes that for a year and a half, no new
filters were added other than the three original ones that CPC-based AdWords
started with. Why? Maybe click fraud was less understood at that time since it
was so new (though Search Engine Watch was citing articles on the problem like
this one
from Wired as far back as 2001). That’s one suggestion, along with Google having
fewer resources, lacking the right infrastructure or click fraud being on a
smaller scale. But these are all guesses, since as the author notes (again, I’ve
bolded a key part):

Not a single person on the Click Quality team was either around or involved
in the click fraud detection back in 2002. The only person from this era who is
still at Google is on an extended leave and was not available for comments
during my visits to Google.

It is hard to judge reasonableness of Google?s invalid click detection
efforts between 2002 and summer 2003 because there is simply not enough
information available for this time period for me to form an informed judgment
about this matter.
One exception is the doubleclick policy that I have described
before. As I have already stated, the second click in the doubleclick is invalid
in my opinion, and Google should have identified it as such well before March
2005 (however, the detection and filtering out the third, fourth and other
subsequent clicks was there since the introduction of the PPC model, and
advertisers were not charged for these extra clicks).

Again, I get confused by the report declaring that Google operated reasonably
when it also states that it can’t judge if it indeed acted reasonably for part
of the claim period.

The middle period finds progress with far more confidence, as covered on page

The Formation Stage (Summer 2003 ? Fall 2005). This stage started with the
introduction of the AdSense program in March 2003 and the formation of the
Google Click Quality team in the Spring/Summer 2003 (the first person was hired
in April 2003 with the mandate to form the Click Quality team; several people
joined the team during the summer of 2003, and the initial ?core? team
consisting of Operations and Engineering groups was consolidated by Fall 2003).

During this time period, two new filters were introduced in Summer 2003 and
one more in January 2004. These three new filters remedied several problems that
existed since the launch of the first three filters and significantly advanced
Google?s invalid click detection efforts. Besides the development of new and
better filters, there was a separate effort launched to develop the whole
infrastructure for doing the offline analysis of invalid clicks and managing
customer inquiries about invalid clicks and billing charges.

Despite all these efforts, the new filters and the offline analysis methods
still failed to detect some of the more sophisticated attacks (presumably from
the Long Tail of the Figure 1) launched against the Google Network in 2004 and
the first half of 2005. In response to these activities and as a part of the
overall invalid click detection effort, Google engineers introduced some
additional filters around Winter and Spring 2005, including the filter
identifying the second immediate click in a doubleclick as invalid.

As a result of all of these efforts by the Click Quality team, a significant
progress has been made in combating invalid clicking activities and developing
mature systems and processes to accomplish this task. Although the Click Quality
team?s solutions were still not perfect, based on the information provided to me
by Google, I reached the conclusion that the invalid clicking problem at Google
was ?under control? by the end of 2005.

And overall filtering is given this conclusion at the top of page 35:

Google put much effort in developing infrastructure, methods and processes
for detecting invalid clicks since the Click Quality team was established in
2003. These efforts were not perfect since Google missed certain amounts of
invalid clicks over these years and it adhered to the doubleclicking policy for
too long in my opinion. However, click fraud is a very difficult problem to
solve, Google put a significant effort to solve it, and I find their efforts to
filter out invalid clicks as being reasonable, especially after the doubleclick
policy was reversed in March 2005.

Page 35 then begins looking at "offline" or non-automated ways to find click
fraud that’s gotten past filters. By page 37, it gets into systems applied to
review what happens on some AdSense sites:

Auto-Termination System is an automated offline system for detecting the
AdSense publishers who are engaged in inappropriate behavior violating the Terms
and Conditions of the AdSense program. It examines online behavior of various
publishers and either immediately terminates or warns the publishers who are
engaged in the activities that the system finds to be inappropriate.

Interestingly, the system is still relatively new, only about a year old, as
explained on page 38:

The first prototype of the auto-termination system was built in the early
2005 and the system was launched in the summer 2005. Recently, Google has
developed major enhancements to the current version of the auto-termination
system deploying an alternative set of technologies.

Page 38 also starts a look at the manual review that the click fraud team
does, with this positive summary coming on page 40:

I have personally observed several such inspections
and can attest to how successfully they have been conducted by Google?s
investigators. This success can be attributed to (a) the quality of the
inspection tools, (b) the extensive experience and high levels of
professionalism of the Click Quality inspectors, and (c) the existence of
certain investigation processes, guidelines and procedures assisting the
investigators in the inspection process.

However, using humans also poses a bottleneck, as covered on page 41:

My only concern with these manual inspections is about scalability of the
inspection process. Since the number of inquiries grows rapidly, so does the
number of inspections required to investigate these inquiries. As stated before,
Google tries to automate this process by letting software systems do a sizable
number of inspections. Still, the number of manual inspections keeps growing
significantly over time, based on the numbers that I have seen. This means that
Google has a challenging task of expanding and properly training its team of
inspectors to assure rapid high-quality inspections of inquiries in the future.

Page 41 also revisits the tug-of-war between advertisers wanting more
transparency and Google trying to protect against click fraud by giving too much
information away:

One of the complaints about Google?s investigation system that I keep hearing
is that Google is quite secretive and does not provide meaningful explanations
of the inspection results neither to the advertisers nor to the publishers.
After examining how their inspection systems work, I can understand this
secrecy. If Google provides such explanations, then the unethical users can gain
additional insights into how Google invalid click detection methods work and
would be able to ?game? their detection methods much better, thus creating a
possibility of massive click fraud. To avoid these problems, Google prefers to
be secretive rather than to risk compromising their detection systems and the
advertiser base.

And this interesting tidbit on how when someone gets kicked out of AdSense,
advertisers apparently get refunds:

Finally, I would like to point out that when Google terminates an AdSense
publisher, all the clicks generated at that publisher?s site over a certain time
period (valid and invalid) are credited to the advertisers whose ads were
clicked on that site….

How well are things going? That begins to be addressed at the bottom of page
41, and here’s a key statement from page 42:

The number of inquiries about invalid clicks for the Click Quality team
increased drastically since late 2004. However, the number of refunds for
invalid clicks provided by Google did not change significantly over the same
time period.
Therefore, the number of refunds per inquiry decreased
drastically since late 2004. Since each inquiry about invalid clicks leads to an
investigation, this means that significantly fewer investigations result in
refunds. This statistic can be interpreted in several ways. First, it can be an
indication that Google?s invalid click detection methods have significantly
improved over this time period and that reactive investigations do not find any
problems when searching for invalid clicks. Second, this statistic can mean that
Google tightened its refund policies and is less generous with its refunds than
it used to be. Third, this statistic can mean that more advertisers are looking
more carefully into their logs and are more suspicious about invalid clicks
since this problem received wide attention in the media and the public discourse
in general. Therefore, they may request Google to investigate suspicious
clicking activities even if nothing really happened. I examined investigative
activities of the Google Click Quality team and can attest that it consists of a
group of highly professional employees who do their investigations carefully and
professionally. Therefore, I do not believe in the second reason stated above.
The third reason is quite possible since advertisers are indeed concerned about
invalid clicks and request Google to investigate suspicious clicking activities
more frequently than before. However, the number of inquiries increased so
significantly that I would expect that the number of refunds would also increase
somewhat. Since this did not happen, I attribute this effect to the fact that
Google?s invalid click detection methods work reasonably well by now.

I’ve bolded the most important parts to me. The expert is saying that more
advertisers are raising inquiries, probably because of increased concerns (which
we know is the case from various surveys over the past two years) but that
Google isn’t refunding more. Nor is that Google just protecting itself, the
expert says. To him, it’s a case that the concerns aren’t matching the reality.
Click fraud — bad clicks getting past Google — do not appear to be on the

Nor is click fraud getting past filters a major problem compared to the
amount Google is proactively catching, the expert says:

The total amount of reactive refunds that Google provides to advertisers as a
result of their inquiries is miniscule in comparison to the potential revenues
that Google foregoes due to the removal of invalid clicks (and not charging
advertisers for them).

Another interesting part is how Google is comparing traffic across its
network to that from within, which is said to be a "gold standard" of
a pure site. The network is said to compare well:

Another indirect piece of evidence provided to me by Google is that
Conversions-Per- Dollar (CPD) rates on various partner sites of Google Network
are not significantly lower than on their ?flagship? site. CPD is the
statistic determining the number of conversions that occurred divided by the
dollar amount spent on advertising. This statistic shows how effective
advertising campaigns are for the advertisers. Since Google spent much effort
over the past 4.5 years to make sure that Google?s AdWords program works
reasonably well, it now serves as the ?golden standard? against which other
programs are compared at Google. Since CPD numbers for other parts of the Google
Network approach that of at, this is an indication that other
advertising programs work as well as AdWords works on Since other
parts of the Google Network are affected by invalid clicking activities
significantly more than, this is an indication to the Click Quality
team that their efforts to combat fraud on other parts of the Google Network are
as effective as on

At the bottom of page 43 is an overall conclusion about that Google’s doing a
reasonable job with detection, as best as this scientist can tell. It also takes
some slams at general reports of click fraud being widespread in the press as
not being proven true or false yet. I’ve bolded the key paragraph for all this

As a scientist, I am accustomed to seeing more direct, objective and
conclusive evidence that certain methods and approaches ?work.? Having said
this, I fully understand the difficulties of obtaining such measures for invalid
clicks by Google, as previously discussed in this report. Moreover, one can
challenge most of the reports pertaining to invalid clicking rates published in
the business press by questioning their methodologies and assumptions used for
calculating these rates. Most of these reports would not stand hard scientific

Still, as a scientist, it is hard for me to arrive at any definitive
conclusions beyond any reasonable doubt based on Points (1) ? (6) above that
Google?s invalid click detection methods ?work well? and remove ?most? of the
invalid clicks ? the provided evidence is simply not hard enough for me, and I
am used to dealing with much more conclusive evidence in my scientific work.

Having said this, the indirect evidence (1) ? (6) specified above,
nevertheless, provides a sufficient degree of comfort for me to conclude that
these filters work reasonably well. Finally, this statement should not be
interpreted as if I find Google?s effort to detect invalid clicks (a)
unreasonable, or (b) not working reasonably well. It only states that Google did
not provide a compelling amount of conclusive evidence demonstrating the
effectiveness of their approach that would satisfy me as a scientist.

Finally, the measures (1) ? (6) above are only statistical measures
providing some evidence that Google?s filters work reasonably well. This does
not mean, however, that any particular advertiser cannot be hurt badly by
fraudulent attacks, given the evidence that Google filters ?work.? Since Google
has a very large number of advertisers, one particular bad incident will be lost
in the overall statistics. Good performance measures indicative that filters
work well only mean that there will be ?relatively few? such bad cases.
Therefore, any reports published in the business press about particular
advertisers being hurt by particular fraudulent attacks do not mean that the
phenomenon is widespread. One simply should not generalize such incidents to
other cases and draw premature conclusions ? we simply do not have evidence for
or against this.

Page 44 has a section that restates conclusions in terms of economic aspects
— IE, any economic motivation for Google to hide or ignore click fraud:

First of all, most of the revenue that Google foregoes due to discarding
invalid clicks comes from the filters since they identify most of the invalid
clicks. The second source of the forgone revenues comes from the terminated
AdSense publishers (as stated before, all the clicks made on the terminated
publisher?s website generated over a certain time period are credited back to
the advertisers regardless of whether they are valid or invalid). However, this
second type of revenue is relatively small in comparison to the foregone
revenues due to filters. The third source of the foregone revenues comes from
the AdWords credits. However, these AdWord credits are miniscule in comparison
to the other sources of foregone revenues. In summary, the most significant
source of foregone revenues, by far, are Google filters. Hence their performance
is the most crucial factor for the whole invalid click detection program (note
that this observation does not mean that Google focuses mainly on this part of
the invalid click detection program since other parts are also important)….

It makes no business sense for Google to go after these extra revenues and
that the best long-term business policy for Google is to protect advertisers
against invalid clicks. Policy reversal on the doubleclick is a good example of
this. By not charging advertisers for the doubleclick since March 2005, Google
lost a ?noticeable? amount of revenues. However, the revenues lost as a result
of this action are insignificant in comparison to the revenues that Google risks
to lose if it loses trust of the advertisers. Therefore, reversing the
doubleclick policy makes sense not only from the legal, ethical and public
relations point of view, but it is also a sound economic decision.

Finally, the beginning of page 46 gives this overall conclusion:

Google has built the following four ?lines of defense? against invalid
clicks: pre-filtering, online filtering, automated offline detection and manual
offline detection, in that order. Google deploys different detection methods in
each of these stages: the rule-based and anomaly-based approaches in the
pre-filtering and the filtering stages, the combination of all the three
approaches in the automated offline detection stage, and the anomaly-based
approach in the offline manual inspection stage. This deployment of different
methods in different stages gives Google an opportunity to detect invalid clicks
using alternative techniques and thus increases their chances of detecting more
invalid clicks in one of these stages, preferably proactively in the early

Since its establishment in the Spring and Summer of 2003 the Click Quality
team has been developing an infrastructure for detecting and removing invalid
clicks and implementing various methods in the four detection stages described
above. Currently, they reached a consolidation phase in their efforts, when
their methods work reasonably well, the invalid click detection problem is
?under control,? and the Click Quality team is fine-tuning these methods. There
is no hard data that can actually prove this statement. However, indirect
evidence provided in this report supports this conclusion with a moderate degree
of certainty. The Click Quality team also realizes that battling click fraud is
an arms race, and it wants to stay ?ahead of the curve? and get ready for more
advanced forms of click fraud by developing the next generation of online

In summary, I have been asked to evaluate Google?s invalid click detection
efforts and to conclude whether these efforts are reasonable or not. Based on my
evaluation, I conclude that Google?s efforts to combat click fraud are

Related reading

google home
simpsons newsletter
Simple Share Buttons