Inside Google’s Search Office (hosted by the Churchill Club)

Inside Google’s Search Office (hosted by the Churchill Club)

pleasure to be here and doing a panel for the Churchill
Club. We’ve been talking about doing
some kind of a session about Search for just over
about a year. And when they got closer and we
came up with some ideas– this panel really came about
from Stephen Levy’s excellent book In the Plex. If you haven’t read it, my short
review is go read it. Not now, but maybe in about
an hour and a half. And engineer, David Bailey, is
quoted as being assigned to work in an office at Google. And he’s supposed to be working
in this office with Amit Singal, Ben Gomes,
and Matt Cutts. And he says, it’s definitely
the cool kids office. And that really struck me when
I read that, because I’d been to this office many times
over the years. I typically would go into Google
and have a day’s worth of briefings, and then at the
end of the day I’d usually end up talking to Matt on
webmaster issues. And we would be in his office
and it would kind of turn into this Search bowl session, which
is a lot of fun to hear all of them interacting about,
well maybe the results should be this way or this
particular issue. So this kind of session is kind
of bringing that office to life and some of the things
that go on with it. If I had to describe the areas
that each of these men oversee, when it comes to
Search, Amit would be the brains, and Ben would be
the looks, and Matt would be the brawn. [LAUGHTER] So I hope you’re happy with
your choices in life. [LAUGHTER] Amit oversees the ranking
algorithm at Google. How Google decides what content
should be shown in response to searches
that happen. Ben overseas features that help
you search better, as well as the user interface look
that lets you interact with Google Search. And Matt’s the bouncer. He’s the chief of police. The person in charge of keeping
the people who would spam and pollute and harm Google
Search results and bring disorder there
under control. And they’re each going to tell
you a little bit more about themselves. We’ll start with Amit because
he’s the baby of the group, having been at Google for only
ten and a half years. AMIT SINGAL: Thank you, Danny. So what Danny’s referring to is
the fact that of the three Googlers you are seeing up here,
I joined Google last. These two were already there,
and I’ve been here ten and a half years in a twelve and
a half-year-old Google. So that’s the composition
of the office. We have been there for ten and
a half years together, eleven and a half, and somewhere in
their for Matt, and we have worked together since
the day I arrived. I have had a fairly long
academic background before I arrived at Google. I got a Masters in Search and
then a PhD in Search. Yes, there is such a thing. And this is back when it was
the sleepy field that librarians used to study. And what computer scientists
thought that wasn’t really the prestigious thing to do
operating systems or compilers were. This is all geek talk. And I ended up getting a PhD
in Search and went to AT&T Bell Labs, which became AT&T
Labs, to pursue an academic career, and published a lot of
papers and so on about Search. And then in 1999, my good friend
Krishna Bharat, who is father of Google News, in a
conference over beers told me that he’s going to this company
court Google in 1999. And Krishna was single then,
and just to give you some color on the conversation. And I, and my lovely
wife, [? Shilpa ?] sitting there, we had
one child and the second one on the way. And I said, Krishna,
Google what? Like, you are single. You can afford to do this. Google shmoogle’s all
going to die. I have a family to feed. I work for AT&T. [LAUGHTER] And next year, I was here,
working with Krishna and these wonderful people. And the rest is, indeed,
history. Ben. BEN GOMES: So I think common
thread is Krishna, because I went to high school
with Krishna. We both had chemistry labs at
home and that was our common love, actually, doing chemistry
experiments. Now this is something that
both of us, being brown-skinned boys, don’t do
in this country anymore. [LAUGHTER] That was our common thread. That our common interest from a
long time ago and both of us went into computer science. He told me he’d interviewed at
Google, and a lot of sharp people there, and seemed like
a fun place to work. I was like, sure. Why no. I could be there, too. So I interviewed there. I got the job. And I remember my boss at Sun
asked me, he said, do you think Google is ever going
to be one tenth the size of Alta Vista? [LAUGHTER] I was like, I am not sure, but
you know it looks like a fun place to work. And his boss made fun of me in
the group meeting saying, they bring to work for a company that
has an exclamation mark in their name, because Google,
if you’ll remember, used to have an exclamation mark. Both of them now
work at Google. [LAUGHTER] And the scuttlebutt in the
valley at the time was Google had a secret plan, because
it clearly wasn’t Search. These are smart guys. there’s no business in Search. And so I remember when I first
joined Google, I asked Urs, who was my boss at the time– Urs Holzle, the VP
of Engineering– so what really is
Google’s plan? And when he told me, it’s
Search, I was so disappointed. [LAUGHTER] It turned out to work
out OK though. MATT CUTTS: Hey, everybody. My name’s Matt and
I was actually a computer graphics guy. I was working on my PhD at the
University of North Carolina at Chapel Hill– yeah, woohoo. Represent. A couple go Tar Heels– and got a job offer to go
work for Google in 1999. And my girlfriend at the time
said, well that sounds like a really good opportunity, but
I’m not willing to move all the way across the country
without a wedding ring. So we actually eloped and drove
across the country. Took a quick honeymoon. Showed up and the very first
project that I got a Google came about because my manager
stopped at my cubicle and said, Matt, how do you
feel about porn? [LAUGHTER] And I said, well it depends. Why are you asking, exactly? [LAUGHTER] And so they wanted me to write
a familY-safe version of Google, which is called
Safe Search. So I wrote the first
version of that. And in the process of doing
that, I found out that there are bad people on the Internet. [LAUGHTER] And so for the last 10 or 11
years, I’ve been dealing with bad people on the Internet. And of course, there’s a huge
number of good people as well, but that’s been a lot
of what I work on. DANNY SULLIVAN: And Matt has a
corn field that he can wish those bad people into. You don’t want to
be one of them. We’re going to dive into
some questions. Just a really bit of background
just to set the stage on some of the
technical stuff. It won’t be that much of
a technical issue. Search engines, the essential
pieces of them, there is a crawler that goes out and find
pages from all over the web and stores the content that this
crawler finds in what we call an index. It’s like a big book
of the web. The crawler’s constantly
updating this index, keeping all the pages fresh. Some of them it visits a lot,
maybe every few hours, maybe every few minutes depending
on how frequent that page. Some it goes back
as it needs to. Your personal blog, maybe
it only needs to drop by once a month. When all that stuff is put into
this index and we come along, we do our searches,
that’s where the search algorithm kicks in. And it goes and it flips through
that giant book, if you will, and it finds all the
pages that it thinks are most relevant and should be shown
in response to a search. Now, for me, and for others,
when you talk about what an ideal search engine
should be– and this is how the
big search engines are generally operated– they are akin to being
a newspaper. You have editorial content that
is supposed to represent what somebody feels as a
fair representation. Some of the best stuff that they
can put together that is being shown to you independent
of any advertising influences or any sponsorship that’s
going on out there. You can get ads but they go off
to the side, just like you might get in a regular
newspaper. If you follow through with the
newspaper metaphor, search engines have a much
more incredibly hard job than a newspaper. The newspaper will assemble
this content each day. They’re in their office
building. Maybe they’re getting phone
calls that come in. They’re getting story
tips or whatever. For a search engine, it’s as
if they’re in the middle of Times Square and they are
surrounded by people shouting things at them saying, I got
a great story for you. I got a great story for you. And some of those are great
ideas, but the people don’t shout very loudly. And you need to hear better
and bring them forward. Some of them shout very loud
and they’re terrible ideas. And some people are just
purposely being misleading. And in the midst of all this,
you have people who are literally just dumping
garbage all around where you’re working. Just junk all over the place. So go out there and come up with
some good search listings off of that. With that kind of background,
all these things going on, people actively trying to
mislead you, people who don’t even understand that when
they’ve rendered their entire web site built out of,
say, flash, it was kind of hard to read. How do you figure out who to
trust, who not to trust? How do you go about that core
part of the job, coming up with the right listings? AMIT SINGAL: That’s a great
summary of how search engines operate, Danny. Imagine you have a book with
billions and billions and billions and billions
of pages. You do have an index like you
have at the end of a book, where it would say which
word appears where. And that’s the index, a very
similar index that most search engines, including
ourselves, built. Now when you type a query into
Google, you give us a few words and we use that index to
find the few most relevant pages for your two, two and
a half, three word query. This is clearly a deeply
scientific process. It’s incredibly hard
as a science. Has been studied as an academic field for over 40 years. And at its heart, there
are simple heuristics. Simple heuristics like if you
said Danny as your query, then a page that contains the word
Danny many times is probably more relevant than the page that
contains Danny just once. That’s the first simple
principle of search. And then there are other
principals like words like the are not very important so don’t
use them as boldly as you would use a word
like Danny. And you take such simple
principles of how language behaves, and you build
algorithms based on those principles in an environment
that’s the modern web, and that’s with the basic description of a search algorithm. Now clearly, in the real world,
everyone wants to come up number for the query
Danny Sullivan. And therefore, they give us
pages which say Danny Sullivan one thousand times, where
as Danny is so polite. He only uses his name one
or twice on his website. So in the real world, all these
assumptions, all these principles are challenged like
we are fighting through all this complexity to get to
Danny’s page, which we do get our there. So that’s roughly
how it works. So we have taken simple
principles of how language behaves and coded it in into
algorithms to return to you what we believe are the most
relevant pages on the web. That’s the words of the web,
according to our algorithms. MATT CUTTS: And in the same way
that all good things in moderation, according to the
ancient proverb, the first time or two you see Danny
Sullivan, that’s great. But if you see it a whole
bunch then it starts to turn a lot worse. And so you have to realize that
the web is like a giant torture test. People will do
crazy things on the web. Any program you write to parse
a web page will break by the time you get web pages that
are long, that are complicated, people don’t close
their tables, and also malicious or deceptive people. And so you have to worry about
not just ambiguity, like there’s a Danny Sullivan who’s
a race car driver, turns out, but also just people who want
to rank for everything. And so it is a really
difficult problem. But through those sorts of
principles, those computer programs, those algorithms,
rather than having a single person saying I think this ought
to be number one or I like this result for that. It’s much more scalable
to have the computer programs and say, OK. They can work 24/7. They don’t have to sleep. It tends to work a lot better
in different languages. All the sorts of things. BEN GOMES: And I think one of
the early insights that Google had was that the link structure
of the web was going to be really useful for this. And so the page rank algorithm,
which was at the heart of Google so early is
such search was actually fundamentally leveraging the
link structure of the web to actually find out which is the
real Danny Sullivan, in addition to all the other things
that we were doing. AMIT SINGAL: So you can imagine
when you’re going on the web and someone says, click
here to go to for Danny Sullivan’s home page. That link– hyperlink as we call it– is a tiny recommendation
for Danny. And we take these people’s
voices which are encoded on these web pages as links and we
have built an algorithm to actually harness the
power of the web. These voices of authors who had
built this web to build whats Google today. DANNY SULLIVAN: Now the links
especially were the big claim to fame when Google
came along. People had been using links,
but you really took it up to a new level. And suddenly you could find
this needle in the million page haystack that
was happening. And over this period, if you
go back to say 2000, it’s largely what I’d say is
the Google decade. You were this unquestioned
leader when it came to search. That was the gold standard. And then towards the end of
last year and through this year, it’s been really amazing
the amount of attention that’s been focused on search
quality. We had the sunglasses merchant
who the New York Times profiled because he was
convinced that because he was such a rotten merchant that
all the bad reviews were giving him links that helped
him do better. We had complaints that people
were saying that, wow. I’ve written this content
somebody has simply copied it, or scraped it, and put it on
their own website and now they’re out ranking me. I’m the originator. You’re not finding my
original content. There were concerns that what’s
been dubbed content farms– and won’t get into the
debate about what those are– but people were concerned that
there were these content farm things and they were
flooding Google with low quality listings. And then suddenly it seemed like
there was this turn that oh, well Google is just
terrible and all your results are junk. And you’ve reacted to that with
a series of updates and a series of changes. The merchant prompted this
unprecedented five-day shift in your algorithm that I’d never
seen happen so quickly. You’ve had other updates
on the scrapers. Probably the most talked
about thing had been this Panda update. Google often gives nicknames
to when they make these algorithm changes. The most recent algorithm
change they had was nicknamed Panda. So you’re making
these changes. How do you decide what’s broken,
how to fix it, and assess whether or not your
fixes are working? AMIT SINGAL: This is a
very good question. And the heart of Danny’s
question is how do you decide if I should change my algorithm
in this direction and is that the right
change or not? And what we do at Google is that
have a few principles on which we operate this
search team. And the first and the foremost
principle that we have put in place is do what’s best
for the user. And once you use that principle
and develop science then to measure what’s best for
the user to the best to scientists’ ability– I’ve been a scientist in this
field for 20 years. Ben and Matt have worked here
for almost a dozen now. So what they have done
internally is developed very rigorous scientific methodology
to evaluate if turning of an algorithm right is
a good thing or turning it left is a good thing, and by how
much is it going to do a good thing for the
users or not. And it would be really
good to set up the context of the team. How we operate. Let me just use a very, very
simple example of how search progresses. And I’m going to use a dumb
example to just prove a point. Suppose one day an engineer
walks up to me and says, Amit. I’m going to promote all pages
that have pink background. Matt says, no. Blue backgrounds better, OK? [LAUGHTER] What does this engineer
do with this idea? The engineer goes and writes an
algorithm that incorporates his or her idea, which either
promotes pink pages or promotes blue pages, depending
on who won the argument in our office. That’s usually Matt. And then that algorithm is
unleashed in a sandbox, which we have of the entire web
inside our offices. Not really in our offices. They are in data centers,
but only available to our engineers. No users are subjected
to that sandbox. And the engineer runs this
blue-page promotion idea in the sandbox. So the new algorithm is blue
page ranked higher, other page raked lower. Now you have the new algorithm
and the current algorithm. The two ranking systems. We take
these two ranking systems and put them through amazingly
rigorous testing. Rigorous testing involves
things like we will take hundreds and thousands of
queries from our past logs, run the old algorithm, run
the new algorithm. We will show it to an
independent human being outside who doesn’t
even know what algorithm is being tested. And sometimes we do this on
them so that they don’t develop a favorite,
left or right. OK? And computers keep track of
whether left was the new or the right was the new. And statistically after you have
run thousands of queries through this left- right blind
test, first number emerges that says, no, promoting blue
pages is not such a great idea, Matt. MATT CUTTS: Sorry. It was a thought. AMIT SINGAL: So, there
we go, right? Now that’s the first test
you have to pass. Once you have passed that level
of testing, we take a tiny subset up over live traffic
and we take this algorithm and we put it outside
in a real data center where users are doing
their queries. And we take a very, very tiny
subset of over live traffic coming into that data center
and subject it to this new algorithm, if it was good. The blue one didn’t make
it to live testing. And if users are now liking the
new algorithm, where their liking is described by they are
clicking much higher in the ranks in that relevant
documents relevant documents are ranking higher, and they
are spending less time searching through results and
click on multiple results– they find the right result
quickly with speed– then the new algorithm’s
better. And with these tests in mind
and this much statistics behind every change and we make
about 500 changes to our algorithm every year, and we
run 20,000 plus such tests every year. With this level of rigorous
scientific testing, we put a statistical report together
put by a statistician, not limited to the engineer
or his or her team. That report is brought to a
group of people, all three of us included, to make a
decision whether this algorithm should be launched
to the world or not. And in that committee that you
may have about in Stephen’s books and articles, is where a
whole lot of debate happens. It’s not just these two numbers
that will force you to launch something. There are many more
considerations that go into launching a system like,
is it good for the web ecosystem at large? Would it benefit authors? Would it benefit high-quality
content? Would it keep our system simple
so that we can maintain it much longer? And with all these
considerations put together, that committee says, yes, we
should launch this or not. Hey, what did I miss? MATT CUTTS: Very good
description. And there’s a couple things
you might not realize. One is that there is no way to
change a computer program, to change how Google ranks and
scores things, that’s going to be perfect. That’s going to make every
single query better. Because we get over a billion
queries a day. And if you get hundreds of
millions of queries, you could make 500 of them great and only
one worse, but you’re never going to make all
of them better. So there’s always
this tension. There’s always this trade off
where you’re trying to have the goals of the best results
for users, the best quality, helping the web ecosystem. And then there has to be room
for intuition and experience. So, for example,
I work on spam. That’s people who
tried to cheat. And it’s definitely the case
that users often click on spam because they see something
that looks enticing. Oh, this is what I looked for. And so we’ve seen examples where
one person will click on the same spammer eight
times in a row. And you just want to– how can you do that? After seven times you couldn’t
tell this was not good stuff? And so by the looking at just
the raw clicks are just the raw statistics it might
look like this is a horrible change. So you have to take those
factors into account. But absolutely, the statistics
and those launch reports carry so much weight because they’ve
built up this intuition about what are good changes and what
are not as good for users. BEN GOMES: And some ways
this is essential for us moving faster. Because if you don’t know that
you’re making progress, you can constantly end up second
guessing yourself and so on. So being very rigorous is
essential to our speed of moving fast. So we are
constantly sure that we are doing the best thing we can for
users at any given point. And we go through the same
kind of rigor for our interface changes, too, because
those are sometimes even more complex to evaluate
because there are more moving parts in the system. So the live experimentation
and so on. There we augment it with user
studies where we have a usability lab when we watch
people use the product and make sure that it actually
works in practice by watching users. But they also go to the same
level of rigor for all the interface changes
that we make. AMIT SINGAL: So that’s a long
way of saying we are in the process of scientifically
determining what’s the best thing for the user. This measurement in itself
is a science. We are in the cutting edge
of that science. And that science is
evolving as well. But to the best of science’s
ability to predict whether it change is good for
a user or not. We use that science to make
changes to Google. DANNY SULLIVAN: I’ll come back
in a bit to the determining of the relevancy and if you’ve
got it right. But Matt, I want to pick
up on what you mentioned on the spamming. You are in this hostile
environment. What’s the conference
that you have? It’s like information
retrieval. Information retrieval, which
was a science, where people would be like, how do we get
our stuff out of the lexis nexis database nobody’s
mentioning. and then you have like these hostile information
retrieval– MATT CUTTS: Yes. There’s an entire conference
called adversarial information retrieval. It’s where the goals change
because the people are trying to cheat and deceive you. DANNY SULLIVAN: So you’re
in this environment. How do you measure
up what’s wrong? How do you prevent getting
the wrong person? You just said that you’ve got
people who will maybe click eight times on a spam thing. You have to use some
intuition. But then some people make
click eight times on the right thing. MATT CUTTS: Sure. DANNY SULLIVAN: So how
do you do that? MATT CUTTS: Absolutely. So we try to provide very
clear guidelines. If you search on Google for
quality guidelines, we actually have instructions for
publishers and webmasters about the sorts of things that
are good and the sorts of things that are not as good. And hopefully they make sense
because we want to judge the same page that a user sees. And so, by that principle, you
shouldn’t hide white text on a white background, you shouldn’t
show us a page about cartoons and then show hardcore
porn to users, things that you would think
would be intuitive. And so what’s good and bad,
what’s the curse, is that once you know how to see spam,
you will always see spam in any system. You’ll look at the cheaters,
you’ll find the people. You’ll recognize the
people who are trying to game the system. And the nice thing is
that most of the time it’s very clear. Spammers tend to be lazy, and
they tend to go all the way out and try to get as much
traffic as possible as quickly as possible. And that leaves some footprints
in some ways that you can spot. And what’s tricky is when
people get more towards that gray zone. So, for example, we have
this category of stuff that we call web spam. And we have very clear
guidelines. And some of the stuff that
happened in the last year regarding content farms was
stuff that you might consider just outside of the
guidelines. They didn’t necessarily
keyword stuff. They didn’t necessarily do
something horribly bad for users, but it was still really
low-quality content that regular people would
complain about. And so by just stepping barely
outside that zone, it fell between the cracks for
a little while. And the nice thing about Google
is that by being in the same office, you can turn around
and you can say, hey. is this your job or
is this my job? OK, we’ll tackle this. OK. And that has worked very well. At Google is also the concept
of a war room. We try not to go for big
war-like metaphors, but that one actually dates back
a long time ago. And whenever you have a
crisis you say, OK. Get everybody in
the same room. And that makes such a difference
for collaboration, such a difference for teamwork
because if somebody is a minute away, you might walk
three or four times a day to check in with them. But they are in the same room
or you can see through the glass walls, and our offices
have glass walls at Google, you can see if they’re
at their desk. You can walk right
over to them. You can see whether they’re
looking unhappy looking at their computer. And so you can say, hey,
is something broken? Do we need to fix something? And that really does make
a big difference in productivity as well. DANNY SULLIVAN: Mercury
news had had this article about, hey. I’ve been penalized by Google
and I’ve had this big network and I didn’t even know
it happened. And we’ve gone back and
forth on this before. I’m like why don’t you
just tell everybody if they have a penalty. You guys will tell people if
they have some penalties, but you won’t tell everybody. So why not just say, hey. You know what? You’re doing something bad and I
will report it to you in our Google webmaster
central system. MATT CUTTS: Absolutely. So Amit talked very well about
algorithmic search and the vast majority of what happens
is all involving computer programs. My team sometimes
has to take manual action. Because if you type in your
name and you get off-topic porn, so you write an angry
email to Google and say, I would not like this porn result
showing up for my name. I’ve never been in a porn
film in my life. And we write back and we say,
well it’s going to take us six to nine months and we think we
might have an algorithm that might help, that’s pretty
discouraging. So my group is one of the very
few where we actually are willing to take manual action. And so we’ve been trying a
communication experiment this past year where if we have taken
manual action, you can do is known as a reconsideration
request, so it’s basically an appeal. And we will tell you whether
we have taken manual action against your site are not. Now if an algorithm is
ranking your site lower, well I’m sorry. With over 200 million domains
there’s no way we can talk with every single web master
or publisher one-on-one. Literally everyone at
Google would have to do customer support. There would be no one left to
actually run the computer programs and write new
algorithms. So we think that that’s a relatively good
compromise in that if there has been manual action you can
now start to get information about that. You can say, here’s
what’s different. Here’s what’s new. Please let me back
into Google. I’ve taken off the hidden text
or whatever’s involved. DANNY SULLIVAN: And if you’re
logging in it’s become more broad, more of the thing. But there’s still some things
you just we’re not going to tell you. MATT CUTTS: Well previously we
hadn’t revealed everything we knew because there are some
really bad guys out there. Al Gore has had his web site– DANNY SULLIVAN: Al Gore’s bad. MATT CUTTS: Al Gore’s not bad,
but his web site has been. [LAUGHTER] Donald Trump has had
his website hacked. And so there are a lot of really
malicious people out there that will install malware,
viruses, spyware, trojans, whatever you want to
call them, stuff you don’t want on your computer. And you don’t want to
clue those folks in. So it is a tension. But we’ve absolutely been
moving more towards transparency. As much communication as we
can figure out how to do. In fact some Google employees
in this room have worked on trying to improve
that process. And have really done
a great job of it. DANNY SULLIVAN: Now people not
doing well in Google have sparked complaints. And in fact, people not doing
well in search engines have sparked complaints
since before we even had search engines. And there’s been various things
that have gone– the most common complaint that
I’ve heard in my time has been, well you’re not ranking me
well because you’re trying to get me to buy an ad. But lately, now it seems to be
that the reason that you’re not ranking people is
because they’re all competitors to you. And that to preserve the
Google monopoly, you’re blocking them off. So what’s the deal? You guys are wiping
off competitors? MATT CUTTS: So that the nice
thing about working in search quality is we don’t worry about
ads or revenue at all. We have a very clear mission
of doing what’s best for the user. So that’s not in our
area or scope of worrying about at all. AMIT SINGAL: And so there’s
a clear church and state separation between
search and adds. No matter how much money an
advertiser pays Google, and that kind of goes into
our revenue. They cannot improve
their ranking. OK. That’s fundamentally
how it works. And then the question that Danny
poses is hey, how are you now putting this stuff up? Because I’m competing
with you. And you have demoted me. This is stuff, your stuff. That’s what they call it. And I go back to our
first principle. Do what’s best for the user. Our job is to give users the
answers to their queries. What they ask for is what
we need to answer. Now in a most simplistic form,
if the user types the query two plus two. Should we return a list of pages
that have the words two plus two on it? Or should we say, four. What would you expect? If you are writing a search
engine, what would you do? Take the next example. When someone says the query
1600 Amphitheater Parkway, Mountain View California. Would you not show them a map
pinpointing exactly what they’re looking for? Our job is to answer
user queries. And that’s what we do. Everything that we do–
someone types GOOG. They’re looking for what’s the
stock price of our company today, at this point. And we return that value
right out there. When you work with our first
principle that it’s all about the user, and our job is
to answer your queries. Everything else falls
in place. BEN GOMES: I think we think
about it in terms of the time it takes you from the time you
enter your query to the time you get the information
you need. And it’s going to be a lot
faster for you to see the number four over there when
you typed in two plus two. So that is our goal. To get to the information,
the answer you need as quickly as possible. And that guides– we believe that’s what’s
best for the user. And that guides our
decision making. MATT CUTTS: And if you go to the
very extreme, if someone comes to Google, and types
in poison control. You really want them to get the
phone number for poison control as quickly
as possible. So you want to get
them that answer. Whatever it is they’re
looking for. DANNY SULLIVAN: So that
gives an answer. Some of the direct answers. And then you get into this issue
of people saying, well, I just did a shopping search. And instead of you showing me,
listing a bunch of shopping search engines, you’re sending
me into Google Shopping or Google Product Search. So now you’re just trying
to keep yourself there. AMIT SINGAL: This definition is
actually somewhat absurd. If you look at Google Product
Search, that you’re talking about Danny. It takes pages out
there on the web. It’s just a search
index system. It organizes that information
better related to your task. If your task is to figure out
how much does something cost? How well is it rated? Should I buy this? Is this merchant high quality? And Google Product Search takes
information that’s out there on the web and merchants
can feed it to us for free. For free. There’s no charge to
anyone who is in Google Product Search. They can feed us their prices,
their availability, and so on. And at the same time, they can
tell us what web pages they’re selling that item on. So Google Product Search is
just a different lens or a different interface search
that’s far more effective for query [? and ?] [UNINTELLIGIBLE]. And yes indeed we send users
to that interface. From there, they can do their
research, and go to that merchant to complete
their transactions. BEN GOMES: In the end they are
still going to Amazon or whatever that merchant is. AMIT SINGAL: I think you can
squint at it hard enough, and say this is Google’s
own stuff. But the truth is, it’s all the
pages out there on the web. And merchants out there
on the web, feeding us information for free. MATT CUTTS: Well and Danny, you
had actually made a really neat graphic a few years ago. Because at one point,
you might do a search for Tom Cruise. And then if you want a picture
of Tom Cruise, you had to click on a tab, images. And if someone types in sunset,
or daffodils, or roses, you might have learned
over time that people actually want pictures of a sunset,
or a daffodil, or roses. And so Danny had made up a cool
graphic that was like Google in 2015 with tabs
all over the place. Look for people, look
for whatever. And really what people want
is they just want to type something in and get something
useful back out. And they don’t want to have
which of the 32 different options of lenses do you
want to search through? DANNY SULLIVAN: Now if you go
to see the purest form of a search engine, to me, it’s
I’ve done the search. I’ve clicked. And I’ve gone outbound. And so from the search engine
to a destination. And that the search engine
itself is not a destination. And when you talk about like
with shopping search. When I’ve heard these arguments,
most of them don’t hold up with me because I think
ah yes, Google sent you from Google to Google
Shopping. Wherein you still left Google
Shopping and went to a destination merchant. But you get these tricky issues
where Google actually hosts content and indeed
becomes a destination. Cases like Google Books. Or Google Places, where you
are aggregating and consolidating a lot
of information. So if I do a search, rather
than going outbound to the merchant, I may go to Places. And perhaps the best example
of this is YouTube. Where I do a search, maybe I’m
going to get a YouTube video that’s coming up, and you
are a destination. So there’s inherent
conflicts in that. How do you deal with that? How do you deal with
those conflicts? AMIT SINGAL: So we deal with
our conflicts with the same first principle that
we have ever had. Test, test, test, experiment,
scientifically test, and make sure all your changes are
good for the users. And once you have that principle
in place, and you’re designing your result’s page,
because our job is to return a results page that’s really,
really valuable for the query. You’re designing your results’
page and testing it extensively. Then for the query evolution of
dance, you would see yes, evolution of dance site. And yes, you will see some
YouTube videos as well. Because that’s what the
users are looking for. So we keep going back to
our first principle. When ever a conflict arises,
we go back to our first principle: is this good
for our users? If it is, we’ll do it. MATT CUTTS: And at the same
time, it makes perfect sense. I remember AltaVista used to
be where you’d search for a person’s name like Jeff Dean. And it would say buy
Jeff Dean on eBay. Right? And you couldn’t really
buy people on eBay. So it wasn’t a very good thing
for the user experience. If we were always showing
something– like every single time we showed a result from
Google Books, that would be really annoying. So we know it’s not in Google’s
best interests to annoy users, to do things
that are bad for users. Because then they
get turned off. There’s plenty of other
places to go and get information on line. DANNY SULLIVAN: But as these
other things have expanded, have you ever thought, I wish
we didn’t have that? Because my life would
be simpler. Because then I wouldn’t
have to deal with these kinds of questions. AMIT SINGAL: We like your
questioning Danny. They’re fine. But fundamentally, right,
every time we build something– let’s say Maps. It was the most innovative
product when we launched Google Maps. The first product that allowed
you to scroll the map in using Ajax technology at the time. Everyone else followed suit. To the degree that now, if you
land at a page which has an embedded image of a map, you try
scrolling it right there. So we feel proud to build these
innovative products. And when we build these
innovative products then giving users answers right on
the result page is absolutely the right thing to do. So clearly our job is going back
to our first principle. Giving users answers to
what they asked for. And sometimes we have to lean
upon innovative products like Maps to answer a query. BEN GOMES: I think the core
part of our competency is actually ranking a
variety of these different information sources. I mean it started with PDF’s. When on the first
time, we started crawling PDF’s in 2000. PDF’s are long documents. And so they can begin to
dominate all the results. And so as soon as we started
crawling PDF’s, we saw results full of PDF’s. So we had to deal with this
issue of [UNINTELLIGIBLE] content there. And I remember. I was working on crawling and [UNINTELLIGIBLE], and also ranking. And then I realized, well
we just hired a world expert on ranking. Maybe I should ask him. And he has not yet even
come to Mountain View. I remember calling him up to
say, Amit, what you did your thesis on this stuff. What do you do? And so I remember talking
to him about this. AMIT SINGAL: So Ben calls me. And he says, hey, we just
started crawling PDF’s. And they’re long documents. They have the word over
and over again. For every query I’m seeing,
just PDF’s. For the query IBM, I’m seeing
PDF, PDF, PDF, PDF. Not And you have written a
dissertation on how to deal with varying document lengths
in such systems. And I had joined Google in its New York
office back in 2000. And Ben was in California. This was around late 2000. He calls me up. We were driving back home from
Jersey City from having dinner at a friend’s house with our
two kids in the back. Right? My wife was driving the van. And I’m talking to him over the
phone, solving, giving him formulas that basically solved
the document length problem. And at the same time like trying
to keep the kids quiet. You know, take it easy guys. It’ll be OK, fine. Right. To the degree that we didn’t
pay attention to what was happening to the car, and we ran
out of gas and had to call a tow truck. MATT CUTTS: So maybe focus
matters a lot. Whenever you’re trying to figure
out these apples and oranges and how to blend. DANNY SULLIVAN: Matt, you
alluded to this earlier. But you don’t manually
control the rankings. If there’s results that are
showing up, it is all down to the algorithm. With the exception of a tiny
little experiment, we won’t get into right now. But you just haven’t
done that. So things show up. And there have been
times when– the best example I think was
one of the first examples where you would do
a search for Jew. And you got this site called
Jew Watch that came up. It was a hate site. And people were like, how
can you allow this? And in the end, you
left it up. And this goes on all the time. There are these things here
that a lot of people would have a consensus of saying,
this sucks. You shouldn’t be listing that. Why don’t you take it out? So why don’t you take it out? AMIT SINGAL: Let me take that. Right Danny’s referring to a
principle that we have held dear to us in our search team. Which is that we would not
manually promote, demote, or remove results. Even if our judgment is saying
our algorithms are doing the wrong thing. The extreme case in point
was query Jew. When someone did that query, it
made all of us tremendously sad to see our algorithms fail
by putting an anti-semetic site at number one. We were in pain. We were really, really
hurt by this. This is not what our principles
are as individuals, as a team, as a company. And people said, why don’t you
just blacklist that site? And get rid of it? However, we said no. It’s our algorithms
going wrong. And we will find a solution to
this algorithmic problem by algorithms. Because this
judgment was so clear-cut in this case, that everyone’s
instinct would be to go blacklist that site. But in the real world,
not everything is such black and white. There are lots of
slippery slopes. There have been many
other queries. Where various interested parties
have wanted us to shoot results out of Google’s
search engine. One such query is Scientology. Where we have both perspectives
return in our top two or three results. And clearly one group doesn’t
want the other perspective to be there. If we start intervening
manually, first of all, we would make such arbitrary
judgments. Number two, we may paper over
one hole in our algorithm, which would not fix similar
problems in many more queries that our algorithm may be
causing, and in many more languages that we
don’t even read. So our principle that we have
held near and dear to our heart is we let our algorithms
reflect the voice of the web. And sometimes our algorithms
will be imperfect. And we work hard to make
them better and better. But our subjective judgment,
though mostly right, is not always the right thing. And that principle has actually
served us well over the last 10 years of
us doing that. And that is reflected in our
users coming back to Google and liking what we do. BEN GOMES: I think that’s a
principle that’s come down from Larry and Sergey. In fact, with the query “Jew,”
shortly after the problem arose, Sergey said, we will
not be changing this. And then the web changed,
and actually the ranking changed around. And he came back, and
he yelled at us like, why did you change? And he’s like, no, we didn’t
change anything. The web did change, briefly,
and then it flipped back again. But over time, the algorithms
have gradually improved to get rid of many or most of those
sorts of problems. And the side effect of that
actually, like you referred to briefly, is that we have been
very good in languages we don’t even speak, because
we’ve relied so much on algorithmic approaches. Because otherwise, we’d have to
do the same thing in every language in every country that
we’re in, and that’s an impossible task. A consequence of taking this
algorithmic approach is that we can then be excellent in
small languages, in languages that otherwise are not considered maybe that important. And it just works across
the world. AMIT SINGAL: Yeah, a normal
company would set their priorities based on revenue
in a language. We don’t pay attention
to that. We say, our job is to serve
every query everywhere, in every language, whether it’s in
Swahili or in Hindi or in English, of course. And this algorithmic approach
allows us to dish the best results for English. But the same algorithm, since
languages are not that different in general, when you
dive deeper– there are lots of differences that I’ll
not go into right now– but that algorithmic approach,
then, any algorithmic improvement made in English
not only serves English. It serves Japanese, Swahili,
Hindi, you name it. And the entire search system
in this world gets better. And we feel proud to have
had this principle. DANNY SULLIVAN: I’m sweating,
because I’ve got five questions, and 10 minutes
to get them in. So we’ll go into lightning
round. You’ve got, also, listings that
have impacts on people. Rick Santorum would probably
be very happy if you could make an algorithm change for
searches on his name. But you could argue, well,
he’s a public figure, did certain things, upset
other people. Maybe that’s just
what happens. But you’ve got people who are
not public figures, who are not known in any way, do
searches on, say, their names. And they go, I can’t believe
this thing’s coming up, and it needs to go away. This is terrible. So why not let them take
that stuff out? MATT CUTS: It’s a tough
call, because it’s a slippery slope, right? Whenever your faced with a
he said, she said sort of situation, you really don’t have
the time or the staffing or the judgment be able to make
the right call in every single situation. So normally, we say, look,
Google is attempting to be a reflection of the web. Right? If there’s someone who’s
important in real life, we want to reflect that
on the web. And if someone is libeling you
in real life, you can either send them a cease and desist,
you could get a court case. There’s lots of ways to
take care of that. And online, there’s lots
of great ways to get your message out. You can start a blog. You can tweet. You can be on Facebook. So there’s many, many ways
where you can have other search results that can show up,
rather than just this one negative search result. But at the same time, I
would say we feel the responsibility. Because we hear complaints, we
hear people who are unhappy. And it’s safe to say, with
so many users and so many queries, there’s always
someone who’s unhappy. Or there’s always someone who
you can point out a really bad search result. And so, I can’t speak for these
guys, but a lot of the times, I go to bed at night
thinking about how can I fix that problem. Right? Or you spend that time in the
shower thinking, OK, how is that going to get fixed. But as long as you have hundreds
of people behind the curtain who are thinking about
how to improve search quality on all those different
dimensions, then in general, it tends to go pretty well. AMIT SINGAL: Let me just briefly
add, we have spent a substantial amount of our
careers devoted to search, [INAUDIBLE] and we think day in and day out
about search and how to make it better. If someone has seen some content
about them that they don’t like, we hurt. But then, there is that case
when you are searching on a doctor, who you learn had his
license revoked three times. Surely that doctor doesn’t
like that result. But our responsibility’s
bigger than that. And that’s the principle
we operate at. MATT CUTS: Because it always
comes back to the user. What is in the best interests
of the user? DANNY SULLIVAN: Now, I talked
earlier that you’ve had these attacks on relevancy, and we’ve
got to improve things. And you’ve made these
sorts of changes. But one of the changes, and it’s
not even new, you’ve done it for a couple years, to try
to increase the relevancy of results has been to personalize
things. So where someone is based out
of, if you’re doing a search here versus down South, you’re
going to see maybe slightly different results. Your search history, the kinds
of things you searched on, sites you’ve been can come
in and influence it. So we have these personalized
results. And now, you’ve got people
like Eli Pariser and his Filter Bubble saying, well this
is terrible, because now you’re just showing us
things we all like. And we’re not getting
the diversity. So what’s your reaction
to that? Do we have this kind
of a bubble? Should there just be normal
results for everybody? Is this is a harmful
type of thing? AMIT SINGAL: So you can just
imagine, a normal result for searching for a restaurant in
Newport Beach would be all New York City restaurants. OK? That would be a normal result,
because by population and masses and consumption
and usage, that’s probably what wins out. So we do personalize results. And there are cases, like
restaurants, where we personalize them tremendously. Or weather, we personalize
weather results to where you are. And there are other cases,
like the query “banks.” Guess what? If I started returning Lloyds
Bank in the US, US wouldn’t be very happy. Or if I returned Bank of America
in the UK, they’ll say something snarky about us. So there is that level of
personalization which makes results tremendously relevant. And then the type of
personalization that Eli’s Fliter Bubble talks about is the
view personalization, that I’m just getting my view in
first. And the truth in the search is that we are returning
relevant results. But our algorithms are so finely
balanced to return some relevant results for you, and
some results that have the opposing point of view, as you
have seen time and time again in the controversies that we
have just talked about, that our algorithms are tremendously
balanced to give you a mix of what you want and
what the world voice says you should at least know. I can give a personal example,
the query “Lord’s.” You guys are thinking Dungeons & Dragons and swords and whatnot. But I’m an Indian who
loves cricket, OK? And there’s this cricket ground
in London, Lord’s. For me, that’s the
first result. However, all the swords and
Dungeons & Dragons are right there, number two and three. And that’s, I would say it’s
an ideal page for me. DANNY SULLIVAN: Cool. I’ll go with you on that. I speak a little cricket. I’m going through a Sophie’s
Choice of which question to throw away. You. You don’t. You all go. No, I’ve got two, and then we’ll
go into the Q and A. And we should make it fit. So we all not only have
personalization of results, we have socialization of results,
which is sort of a subset of that personal, who you know,
who you’re friends with, and so on. You’ve been using social search
for about two years now and having that influence
things. But now you have your
own social network. So how is that going
to have an impact? In particular, Google Plus and
the +1 system, on the search results that people
are seeing? AMIT SINGAL: That’s a
very good question. So Google was a pioneer in
launching social search. We were the first one to launch
it, where based upon who you know, we can bring their
recommendations to right top of your search results
sometimes, or we can just bring them to your attention. And this is great. Because if I’m searching for
a plumber or a locksmith, I would much rather have these
businesses reviewed by one of you, who I trust, as opposed to
the New York Times Square problem of everyone yelling,
I’m the best locksmith. Please hire me. So social search is a great
advance in search technology. And social search is based upon
who knows who, and who knows about what. And when you put these two
things together, a very powerful system emerges. You basically marry the real
world, how the real world functions, into search. And that search is more
relevant, because we can bring to you experts, and sometimes
experts and friends you may know, who may know something
about what you want to know. And that’s a much better system
than it is today. With Google Plus, we
would indeed– Google Plus has been a very
successful system. We are very excited about it. And with Google Plus, we have
much more data on who you know and what they know, so that
we can improve your search experience going forward. DANNY SULLIVAN: Have you notched
up the social dial a bit, since it’s launched? AMIT SINGAL: We experiment all
the time with numerous things. MATT CUTS: It’s early days,
but it’s pretty exciting– DANNY SULLIVAN: That’s
Google for you guys. MATT CUTS: –to look at the
potential of how well this might work over time, so. DANNY SULLIVAN: OK, I’ll squeeze
in this last thing, and then we’ll go to the Q and
A. And also, because we also want to talk about some of the
team building stuff, I want to catch you after the end of the
Q and A. Come back to you about some of the things
that have helped you. You started as a team
of 10 people around a Ping Pong table. And now you’ve got hundreds of
people all over the place, and yet you still make
it together. So we want some ideas on what
people can take from that. But Ben had– you had mentioned
a statistic when you were talking, something like
two humans only agree on relevancy 80% of the time. And that in fact– I think you had said this, maybe
[UNINTELLIGIBLE] said it, but– and that if it was a
year later, and I showed you the same set of results that a
year before, you had rated, you’d only agree with yourself
80% of the time. So how do you know if
you’re relevant? I know you mentioned the humans
and things that are out there, but can anybody know? And can we independently assess
whether or not you’re getting it right, when you
make these kinds of assumptions of, this
algo treat will make everything better? AMIT SINGAL: Yeah, no, let me
go to a sports analogy. In soccer, you have tens
of games in a season. And if you lose two,
you’re out. In baseball, you have 160-plus
games in a season. And winning or losing a single
game isn’t really going to determine whether you’ll make
it to the World Series. Just look at our Giants. And what we do is, by giving
thousands of queries, and each query to multiple human beings,
we can statistically measure the inter-human being
agreement on relevance that you mentioned, which is
about 80% in general. I say this is relevant, with
80% probability he will say this is relevant. So we can measure that and
incorporate that inter-human agreement into our deep
scientific measurement that I’ve talk about. And once you know that, then
you can rig up statistical tests to say, have I done enough
samples through to definitively say with 95%
confidence, things like t-tests and so on, that the new
system is indeed better. DANNY SULLIVAN: But does
it skew a bit? Because you ask these people,
who are not experts on the things that they’re searching
for, to tell you whether or not they think things
are relevant, so? AMIT SINGAL: But time and
time again, this 80% statistic holds up. So for your query,
I’m 80% accurate. Thus, for my query, you’re
80% accurate. And just knowing that and
incorporating it into our evaluation formula allows us
to do the right thing. BEN GOMES: And the live
data actually has– we have huge volumes of live
data, even when we do tiny experiments, right? And so there, you can really
make use of strong statistical techniques to make sure you’re
getting the answer right. And there, the person
who’s looking has a real information need. And so you get a good
determination of whether they are being satisfied. MATT CUTS: But I think the part
that might frustrate you is Google has spent 10 years
trying to figure out how to evaluate potential changes. But we haven’t talked as much
about that outside of Google. And so to have a third party to
say the relevancy is better of this search engine
or that search engine is really difficult. And unfortunately, a lot of
people will do three queries, and that’ll say oh, OK,
I found my homepage. Therefore, this is the
best search engine. And so there hasn’t been
as much rigor in the third-party analysis. A lot of it is reduced down to
metrics about who happens to have more growth and queries
this month, or something like that. DANNY SULLIVAN: OK, we’re going
to go to the open Q and A. And I’m also going to ask
you about where you think things will be in five years
from now when we get to the end of that, too. So I’ll save the last
five minutes there. If you have a problem with
ranking on Google, if you could move to this
side of the room. And we’ll get those
questions to you. So we have our first
question over here. AUDIENCE: OK. In the last year or two, I’ve
seen more and more of what I’ll call type-ahead, where
Google attempts to finish my typed query. And I think I’ve seen
search-ahead, where I actually get results before
I hit return. Can you comment on how that’s
worked from your perspective and where it’s going? BEN GOMES: Yeah, so we started
doing Google autocomplete about four or five years ago. And the idea there is that we
still think of the search process in terms of the time
from you have an information need to the time you’re seeing
the information you want. And so autocomplete actually
helps you formulate the query. Because if you get the query
formulation wrong, if you’ve got a spelling mistake, if
you’ve got the wrong words, it’s very hard for you to get
into the right information. So autocomplete was a first step
in helping you formulate your query and essentially
greasing the rails to your answer. Now the second step is, the best
way that you can evaluate whether this is the
right query, is if you can see the results. So the idea behind Google
Instant was to begin to show you the results as you type. So you can begin to evaluate
whether your partial query is what you wanted. And we find that people get
relevant results well before they are finished, several
seconds before they finish typing their query. And so, we are seeing a lot of
benefit to users from actually showing them results early. And we’ve been very, very
pleased with some of the impact it has on the time it
takes the users to get the information they want. AMIT SINGAL: Ben is being
modest. He led Google Instant. And Google Instant has been
a huge success for us. We are very proud of what
we have done there. MATT CUTTS: And in fact, Google
obsesses about speed to the point where we just launched
something where, if we really think it’s likely
that you’ll click on the number one result, then we
might pre-fetch that, pre-render it, so by the time
you click on that result, it shows up instantly. You don’t even have to wait
for that to download. So there’s hundreds of people at
Google thinking about every aspect of the search session. And how can we give you bionic
arms to search faster, and bionic brains, and all
that sort of stuff. And the goal is to get
you to what you want as quickly as possible. AMIT SINGAL: They didn’t buy
my Google helmet idea with electrodes. DANNY SULLIVAN: Next question. AUDIENCE: [INAUDIBLE] what
you’ve done to earn your success over the years,
day-to-day in your development [INAUDIBLE], how does it
affect you knowing that competition authorities around
the world, including the FTC, may well be [INAUDIBLE] looking over you shoulders,
oh and [INAUDIBLE] expensive and time-consuming to
have to keep up with other companies, [INAUDIBLE] reputation [INAUDIBLE] yet it’s always a roll
of the dice. [INAUDIBLE]. What is it you hestitate
to do? What is it you don’t, that
doesn’t get done? Because of this [INAUDIBLE]. MATT CUTTS: Yes, that’s
a great question. I think, having worked at the
company for 10, 11 plus years, the core essence, the principals
of the company have remained the same. Because it’s almost like
Spiderman, right? With great power comes
great responsibility. And I like to think that we’ve
operated with the idea that someone could be watching over
our shoulder, and tried to make the right decisions
all along. Now, you’re right. It can be a roll of the dice. And so, some of the things
that we’ve been doing are trying to educate people. How does search work? What are the different
policies? So for example, we’ve made over
450 different videos that talk about common questions
and answers. And if we can do a good job of
explaining, yes, there is some manual stuff, whenever
there’s spam. But the vast majority of
it is algorithmic. And explain how these
are best practices. And how every search engine
does them to some degree. Then hopefully, that decreases
the odds that there will be some miscommunication,
and some fluke will result in a problem. We have a very good legal team,
and they think a lot about all of these issues. But I like to think that the
core values at Google are such that, we try to behave as
if we were the underdog. We try to figure out, how can
you step into the other person’s shoes? So how can you have a
good appeal process? How can you scale up
on communicating? How can you scale up
on transparency and communication? But you have to be mindful
that, as an engineering company, we always have
to have the idea that we can be wrong. We are a very disruptive company
in a very disruptive space, which is technology. And so you have to be mindful
of that responsibility. And if there are people who have
suggestions for ways to improve, you absolutely want
to be open to that and listen to that. Whether that’s someone you meet
at a conference, someone who’s Tweeting to you,
or whether it’s someone who’s a regulator. DANNY SULLIVAN: Next question. AUDIENCE: Hi. I had a question about what
you do to deal with it the quote unquote bad guys. For instance, I understand
searches are generated as a result of an algorithm. But, for instance, J. C. Penney,
a couple of months ago, that New York Times
article about how it gamed the system. What did you do to affect
their rankings? And what can a company like J.
C. Penney do to gain in the rankings afterwards? MATT CUTTS: Yes, absolutely. So some of the Googlers who
worked on that exact thing are a couple tables away from
you, as a matter of fact, right now. So for people who didn’t
read the New York Times story, this was– J. C. Penney had engaged an SEO
company, a search engine optimization company. And there are certain things
that Google prefers not to see in our search results. So if you are buying links
that pass page rank, it’s almost like a form of payola. It’s like you’re paying someone
to say nice things about you, but you’re just
paying them to link to you. And so, that’s something that’s
against our guidelines. Just like you’re not allowed
to do payola on the radio. And so, whenever we see a
violation of our guidelines like that, we are willing
to take action. We absolutely try to write
algorithms to spot that, counteract those links, to
defuse and detect them so that they aren’t an issue
in the first place. And strangely enough, we’d
already detected those links several times. I think, if there was a failure
in that particular case, it was that we didn’t
escalate, and take stronger action earlier. But then the flip side is,
once the company tries to clean things up– and J. C.
Penney did a very good job of trying to make sure that they
were doing ethical search engine optimization after
that incident happened– then you have to make allowances
and say, OK, how do we let you back into
the search results? And at all times, the goal is
to try to make sure that you’re ranking appropriately
in the search results. So it is a tricky problem in
that, in some cases, judgment does come into play whenever
you’re looking at violations of our guidelines. But the vast, vast majority of
the time, you want those to be handled with the computer
programs, with the algorithms. So that you don’t have
to bring that judgement into play. BEN GOMES: Our goal, in general,
is to reflect the real authority of a web site. The goal of search in general. It’s not to be ahead
or behind it. And when somebody’s trying to
game the system to be ahead of their real authority in the
world, that’s when we have to take action. DANNY SULLIVAN: Next question. AUDIENCE: Thank you all for
a great panel so far. Ben Parr of Mashable. So you had this cool product
called Google Realtime Search, where you could be searching,
getting real-time updates and feeds through Twitter,
Facebook, et cetera. Then the Twitter deal expired,
and it disappeared. And this is a question
for my Google Plus audience, by the way. So the question is, when, or if,
is Google Realtime Search coming back? And do you need Twitter for
that, or is that going to come primarily from Google
Plus data? AMIT SINGAL: So what Ben is
referring to is, we launched Realtime Search based upon data
from various real-time data generation, information
generation systems like Twitter, Facebook fan
pages, and so on. And our deal with
Twitter expired. We didn’t come to
an agreement. And after that, we decided that
the value that product was providing was not enough
for our users, and we took it offline. And we are actively working
on– as we speak– figuring out, using our current
G+ data and other sources that are out there, to
revive the same functionality into Google search results. So I could say, stay tuned. We are working hard on it. DANNY SULLIVAN: Why
didn’t we get– and why don’t we have now–
just, the ability to search Google Plus? You’re a search company, and we
don’t have the ability to search your own social
network. AMIT SINGAL: Your feedback is
very well taken, Danny. Believe me. And again, we are on it. DANNY SULLIVAN: We are
always thinking. MATT CUTTS: It’s fair to assume
there’s always lots of things we want to do, and a
finite amount of people. So– DANNY SULLIVAN: Next question. AUDIENCE: With the addition of
Plus One search results, does this offer a new way to game the
system, where people can plus one, plus one, and get
their searches results to the top, or how does that work? DANNY SULLIVAN: You
can buy those now, I’ve read somewhere. MATT CUTTS: Not to be cynical,
but every change in search involves a potential way
to game the system. I’ve gotten a little jaded
over the years. But, the thing that’s actually
nice is, if you think about ranking, as it existed a few
years ago, it was primarily based on links, anchor text,
what’s on the page. So we have over 200
different signals. The idea that the web might move
from anonymous pages in some dark corner, where you
don’t really know who wrote what, to a web where you
actually do have some sort of annotation that says, this
person wrote this, or this person vouched for this page,
and you can have the reputation of individual
people or authors– so if someone who’s a New York
Times columnist shows up on a forum and leaves a two line
reply, that can be really important, even if nobody
links to it. So it’s absolutely the case that
people will try to game social signals, Google Plus,
all of these things. You already see people trying
to sell plus ones. But there’s different ways,
where you have new signals, and different ways to intersect
that, and hopefully prevent that. And the idea that you can get
a big win from all of these potential new signals
is absolutely worth giving it a try. So– AMIT SINGAL: And no signal is
used in its absolute form. Every signal in its absolute
form has its shortcomings, like you mentioned
for plus one. We crossed hundreds of signals
to build what’s Google today. MATT CUTTS: You can certainly
imagine having fun like, oh we’ll by some plus ones, and
then see what shows up. You can play games like that. DANNY SULLIVAN: The plus one
data right now– it is used as a ranking signal for when you’re
logged in, correct? AMIT SINGAL: When you’re logged
in, it is a social signal if someone you know has
vouched for something, indeed. It’s part of
logged out, is it being used as one of many signals? AMIT SINGAL: We are
experimenting with numerous things, always. DANNY SULLIVAN: Which is
Google speak for yes. AUDIENCE: So my question comes
from the standpoint of image and video searches. So if I was to search for a
moonlit beach, and if the images or the video wasn’t
tagged with those specific words, are you guys
doing some– advance the art of video and
image search, which can get me moonlit beaches, without those
words being tagged? AMIT SINGAL: So our image search
algorithm is far more sophisticated than just looking
for those words, either in the caption or
nearby on the web page. There’s a lot of computer vision
technology built into our image search algorithm. Some of the team members
are here today. Which, basically, use those
images to say, hey, this one looks like a beach, and
so does this one. And now, you have
a positive loop. You can find some great
images of beaches. And then you can find some other
images of beaches that didn’t really say beach, or
didn’t see beach with the same density as our algorithms
would have liked. So it’s a good question. We use a lot of that
technology. It works very well, and we are
constantly improving it, as you saw with our most recent
launch of Search by Image. An image comes to you, you
say oh, what’s that? You just drop it into Google
Image Search and, with very high likelihood, we’ll
find it for you, what you’re looking at. Using the same vision
algorithms. DANNY SULLIVAN: Next question. AUDIENCE: I come from
the mobile industry. And I’m sure that mobile phones
have different kinds of search queries. I’d be interested in
a comment on that. But the next thing that
we think is coming is machine-to-machine mobile
communication. And I’m wondering if you’re
preparing for machines to be generating queries
in that way. And how different do you think
machine queries will be then people-generated queries? BEN GOMES: Well, I think the
first point is not actually that accurate. The mobile query stream is
getting to be more and more like the web query stream. A while back, it
was different. When phones were really slow,
when interfaces were not full web browsers, the query stream
was very different. But today it’s gradually
approaching exactly the same distribution as the desktop
query stream. And so, to answer that part
of the question, I think, actually, that things are much
more similar that you think. AMIT SINGAL: And let me also
add, to that part of the question, with the innovations
that we have made at Google, with things like voice search,
where– it’s hard to type on a mobile phone, so we gave
you voice search. Amazingly accurate
voice search. And we start seeing human beings
behaving the same way as they behave on desktop. I would say that on mobile
devices, we still get somewhat more local oriented queries,
what’s near me. And for that, we have launched
numerous innovations. Like on Google’s homepage
itself, there’s a list of hot things you can do. There’s restaurants, coffee
shops and so on and so forth. So mobile query stream is
actually reflecting what users need, which means it’s coming
closer to the natural distribution of queries. And mobile has been a great
success for Google with all the innovations that the search team has made in mobile. Not only by voice search, but
things like, our buttons are more touchable on a
mobile interface. Our maps are far more designed
for this tiny interface. We have done immense amount of
work on mobile, and that reflects in our success
in mobile. Now, the question about
machine-to-machine searching. Right now, we haven’t observed
that much of it happening. And I think, once that system
picks up, we would have to analyze how that looks. But in an ideal world of search
for me, search would be so accurate that you can just
type a query, and the machine should assume that the search
engine, the other machine, would answer it correctly, and
then build a whole equal system on top of
that platform. And that’s our dream platform
that we are building together. DANNY SULLIVAN: Next question. AUDIENCE: This may be a
follow-up on the bionic arm question, or a comment there. Which is, I’ve heard people in
Google talk about search list search, which I interpret to
mean that, maybe you can even take the personalization to
the extent of anticipating what the user might want,
without even typing a query. So I’m curious about your points
of view on that, and what are the challenges
that poses to you? AMIT SINGAL: So it’s
a great question. Because this kind of technology
is what we, as kids, dream of, right? Computer would tell
you what to do. And– the truth is– MATT CUTTS: Maybe not
tell you what to do. But help you understand what
might be possible. AMIT SINGAL: And the truth is,
first of all, you can see that in a future that’s possible,
based on pieces that we already have, you can build
systems where a computer can help you tremendously in
making you a far more efficient human being. So my phone already
has my calendar. It knows when I’m free. It has my to-do list,
which says I have to buy a baseball mitt. And it has a map, based on
Google Local, of all the places that sell. And it knows where I am. So it’s not too far out there
that you can imagine that computer can gently prompt me,
hey, please do pick up baseball mitt. You are three minutes away from
Sports Authority right there and you have 30 minutes
free on your calendar. MATT CUTTS: And by the
way, it’s kind of annoying I have to have– yes, that would be nice. It’s kind of annoying that I
have to have a to-do list at all, right? Because Google has announced
something called Google Wallet. And wouldn’t it be great if you
could go to the grocery store and you could buy things
with your Google Wallet? All you do is, you tap to pay. And then, over time, if you
wanted to, and gave permission, Google could say,
oh, you haven’t bought cat food for six weeks. Normally you bought it
every four weeks. Do you want to just add
cat food to the list? And then, finally, I don’t have
to think about, oh, I need A1, or I’m out of salad
dressing, or whatever. All these little traces we
leave, if you’re willing to opt-in for those kinds of
things, Google could be the little tap on your shoulder
that’s like, hey, don’t forget to get wet cat food. AMIT SINGAL: And the key there
is that users have to opt-in to these things. It’s a critical aspect. I am a human being who deeply
cares about my privacy. And that’s the key part. Everything we do at Google,
we think about that. Sometimes we get it wrong. We stand up, apologize,
and move on. But that’s how these systems
and technology will evolve. If someone told me, 20 years
back, that you would type into some machine, what’s
the height of Mt. Everest, and it’ll spit out the
answer, I’ll go, you’re smoking crack, buddy. Go on. Right? But, see, what’s possible
with technology– we all have to dream it
together, and then build it in a privacy-preserving way. DANNY SULLIVAN: If only you guys
could answer our email, that would be a real solution. Next question. AUDIENCE: Brian Fox,
Western Union. First of all, Google has to
know that my cat died last week, because I blogged it. And you shouldn’t
send me that. That’ll be your job. I’m curious about
your vision– your image recognition
technology. Do you use the power of the
people doing the queries, when they confirm that the beach
really was what they were looking for? Does that loop back into your
algorithm and improve its intelligence? AMIT SINGAL: Numerous factors
go into deciding, in image search, what images are
most relevant, and most liked by users. And there is that positive loop
that, based on users’ choices of what we return,
we can improve the system going forward. So when you combine the powerful
research algorithm based on words with the power
of vision algorithm and this wonderful loop, that’s when you
get what’s Google create Image Search today. AUDIENCE: Lauren [? Chaude, ?]
also with Western Union. We’re a 160 year old company,
and I’m intrigued with the dichotomy of culture,
which I care about. So you’re obsessed with
statistics and algorithms, and yet you talk about glass walls,
and the power of being one minute away from each other,
and being physically co-located. Can you talk about that in this
global world, and web, and blah blah blah,
all that stuff? Thank you. MATT CUTTS: Yes, absolutely. It’s been enormously helpful
to be very close to the relevant people. And Google has offices
around the world. And so, for example, numerous
people in our office will have a video conference unit right
there, where the glass wall, they can look behind them, or
they can look through the glass screen and talk to
someone in New York. And so, with very little work,
it’s easy to bring up somebody and collaborate, from
Tel Aviv, from New York, from Zurich. And that makes a huge
difference. You always have time zones, but
just being able to have that face-to-face connection
makes a huge difference. Short-circuit an email
conversation. Hop into some sort of
face-to-face communication. It really saves a lot
of time and a lot of misunderstandings. AMIT SINGAL: And we have found
this proximity of teams– it makes us tremendously
efficient. We can churn out things, as
Danny pointed out, in one case, in five days, we could
launch an algorithmic change at this scale. And this proximity doesn’t
always have to be physical. And by video conferencing– and in our office, there
are two or three video conferencing units that
are open to the world, all the time– and people can just say,
hey, can you unmute? I need to talk to you. And there we go. We have a conversation. BEN GOMES: And I think, in the
history of Google, we went through periods where we were
very densely packed. But we also found– and that was
not by design, it was just that we hadn’t gotten
more space– but we also found those were
extremely efficient periods in the company. Where people were packed
into an office and they communicated a lot more than
they otherwise would have. It might not have been their
first choice, based on their background. Many people came from
backgrounds where you had your own office, and so on. But it created a kind of energy
that I don’t think arises otherwise, without
that kind of density. MATT CUTTS: There are some
companies where every developer has their
own office. Whereas Ben– there was one office as that
had three Bens in it. And so they called
it the Ben Pen. But it really does make a
difference to be able to just turn around. And we have all these
cues, right? You can do heavy-duty
video conferencing. You can do a hang out
in Google Plus now. You can do Google Chat. And there’s these cues that are
subtle, like, well, I’m red, but you can interrupt
me if it’s really, really important. All the way down to email,
and meetings, and those sorts of things. So having that spectrum to be
able to choose what’s best to get in touch with someone,
whether it’s something that’s a quick hit, 20 seconds, or a
half hour meeting, that really makes a big difference in
terms of being able to collaborate. AMIT SINGAL: In the early days,
we used to say, we pack them tight and give
them deodorant. And that’s how it works. DANNY SULLIVAN: And we’ve got
time for one more question. AUDIENCE: So Shailesh
from Citrix. So thanks for the panel
for wonderful insights into the search. So my question is about, when
I do my search, most of the times, I get results back saying
that, 20,000 results in two seconds or so. But thanks to your excellent
algorithms and principles, first two or three links I get
my results, many times. So why bother spending time for
finding and searching for the 19,000 plus results,
and giving them to us. Can’t you save time there? AMIT SINGAL: So you have seen
that number sometime appear on Google’s result page
that says we have 20,000 or 200,000 results. But the truth, indeed, is what
Shailesh said, that if you haven’t found what you’re
looking for in the top two or three results– which shouldn’t happen
that often, or you can send us mail– then really, going down further
is not that useful. It’s just that our algorithms
do compute numbers for all those results. And so we give the user
an indication of how much there is. But that doesn’t mean you have
to read 20,000 and feel pained about that. BEN GOMES: I think,
in some ways, it’s a historic artifact. There was a time when, when
you did a misspelling, you would get much fewer results
than the real query, right? And so people used to use
it as an indicator. It’s no longer true today. We correct your spelling,
and so on. But it’s a historic artifact
that has, I think, a little bit of nostalgic value
for us, too. MATT CUTTS: And by the way, I’ll
give you one tip, which is that, it is an estimate. It’s not an exact count. So if you ever notice, we only
give three significant digits when we guess how many
results there are. That’s a little cue to let you
know, it’s not really 982,000, it’s roughly 982,000. DANNY SULLIVAN: And I’m
grimacing, because I know I could still do a search where I
should be getting a smaller number, but I actually get
a bigger number, so– MATT CUTTS: It’s a
rough estimate. DANNY SULLIVAN: The
last two things. I want to come back to
the team aspect. So you’ve talked a little bit
about it already, you can network, but– MATT CUTTS: Yes, there is one
metaphor that Amit mentioned, which is the baseball
metaphor. Which is, when you’ve worked
together for so many years, it’s almost like having that
many games in a season. You don’t get that frustrated
if you lose one time, or if somebody tells you to go back
to the drawing board. Because I remember, whenever I
launched Safe Search, this porn filter, the first time,
I was all ready to go. I was already to flip
the switch. And two engineers tested it
out and said, you have too much stuff labeled as port
that’s not really porn. And I had to go back to the
drawing board and figure out how to make it better. And at the time, I
was [GROWLING]. I was kind of frustrated
about that. But over the course of doing
many, many, many, many, many launches, you build up that
trust to where you say, this person’s looking out for the
best interests of the user. This person’s looking out for
the best interests of Google. So take that as constructive
feedback. Don’t get so caught up in one
particular battle, one particular controversy, because
it’s guaranteed, tomorrow, there will be
a new controversy. A new point of discussion. And that’s helped a lot in
making things more collegial. AMIT SINGAL: Yes, and now I look
back at my 10 years of being part of this group– we
started around a Ping Pong table with 10 people. Now we have hundreds of
people in our group. And unwittingly, somewhere, we
developed the principle that has made this team out-innovate
every other team out there. And that principle was, I would
put leaders in place who I respect technically. So the entire management
hierarchy of the group is built of people like ourselves,
who were engineers, wrote code, can understand
what an engineer is going through. Their happiness, their pain. The entire group hierarchy is
built from people who have worked in the group for
many, many years. Hundreds of people report to
us now, but everyone who reports to us has managers who
have been there many years. The whole group leadership is
built off purely technical people with deep technical
knowledge about search, and deep understanding of what
goes into this innovation machine that is Google. And that, we just put in place
early on because I couldn’t find enough people to
manage people as the group was growing. So I said, why don’t– Matt, you manage 10 people. OK. So Matt got 10 people. And then, Ben, you
manage 20 people. And that’s how we
grew the group. And it has served
us very well. This is a lesson in leadership
that we have all learned in retrospect. We weren’t designing for this. But it so happened, that if
you are in the innovation space, you need to make sure
that your leaders are so technical, that everyone that
works for those leaders respects them as technical
people. And that has worked well. BEN GOMES: And I think, all
our leaders all stay with technical titles. And they think of themselves
as, first and foremost, engineers, as we do too. Even though they do a lot of
management and so on, their self-perception is as
technical engineers. DANNY SULLIVAN: Can I just
say, by the way– your title is– [INTERPOSING VOICES] are you senior, assistant
principle? AMIT SINGAL: Principal
engineer. DANNY SULLIVAN: And
your title is? BEN GOMES: Google fellow. DANNY SULLIVAN: And
you’re title is? AMIT SINGAL: My title is
also Google fellow. DANNY SULLIVAN: Which pretty
much has nothing to do with what you actually do. MATT CUTTS: But at Google, you
can get anything you want printed on your business card. Literally, they do not care. There have been some pretty
crazy business cards printed. Because, titles are– at least, at Google, they don’t
make as much sense to obsess about. And so, if you can get whatever
you want on your business card, then
you don’t fixate. You don’t obsess about it. You worry more about the
job at hand, and that tends to work well. DANNY SULLIVAN: And then, in a
Tweetable, or Google Plusable, or Facebookable short statement,
where are we at five years from now, other than
bionic arms doing all of our searches? AMIT SINGAL: So all of us have
our views of where we want to head five years from now. I was raised on a healthy
dose of Star Trek. And I want my Star
Trek computer. OK? That’s what I want in
five years from now. I should be able to talk to it,
ask it whatever, and it should be able to have a
conversation with me. And search is, of course,
fundamental to that. MATT CUTTS: I want Star Trek
in five years, but in one year, I want the ability to get
reminded that I need to get cat food while I’m
on my way home. And also the ability– Google Voice Recognition
has gotten very good. And it can’t be that much
harder to make a well-punctuated email. So I want to be able to do my
email while I’m driving home, I’m talking, maybe– at a stoplight, or whatever– and have it look as if
I’ve very carefully crafted it by typing. That can’t be that hard
of a problem. That shouldn’t even be
a three year problem. So I’ll go back and file
a bug when I get back. BEN GOMES: Yes, I think I share
Amit’s vision of the Star Trek computer. But I think, in the shorter time
frame, I want the search on my phone, which is with me
all the time, to work really, really smoothly and
effortlessly. And it’s not quite there yet. It’s gotten a whole
lot better. Voice recognition is
getting there. Transmission is getting there. You can now do these
amazing things. Like, you can take
a photograph– I was in South American
recently. I was in a restaurant
with Spanish menus. I took a photograph
of the menu. And with image recognition and
translation, I could then translate the menu
into English. I was like, wow. That’s amazing. And it’s all of– you can see where you’re
going to go. But you’re not there yet. And I want that reality to
become completely fluid, completely reliable, so that
people all over the world can actually communicate, and get
information really easily. And I think this matters, not
just here, but particularly in other parts of the world, where
people don’t have as easy access to information. Where people in India, and
people in Africa, and people in the Third World, really,
have access to this information the way that
we take for granted in many ways today. And we have access. And I think that will
empower their lives. And I think that that will
change the world in many ways. AMIT SINGAL: So let me
communicate our enthusiasm for search by saying, I feel like
a kid going to candy store every morning. And you haven’t seen
nothing yet. DANNY SULLIVAN: Well it sounds
like all search in the future will be mobile. And we won’t even
be searching. It will be happening for us. Thank you all very much
for being here. It’s really been a delight. I wish I had another three hours
to keep going at it. So–

2 Replies to “Inside Google’s Search Office (hosted by the Churchill Club)

  1. Em bỏ đăng kí đi ạ vì em làm dì cụng sai em biết là chếnh đi song mất icloud từ ngày đấy là thay đổi sao cụng không được,và bì thay oan dất khổ gđ chửi mắng bảo quét mã thẻ ngân hàng song cung biến luôn mây đỏ tu gồ+ và không còn điều dỳ em biết và em cứ vậy chỉ sớm muộn sẽ chết.mọi người hỏi điều em phải tôn chong mọi người.có luật sư từ hồi tổng thống chă gửi cả niềm tin và con em cùng di chúc hồ sơ tài sản sánh.ở ngân hàng quốc tế và yêu cầu bảo mật và làm theo di chúc mà chính luật sư yêu cầu.về lấy giấy khai sinh phải viết câu từ và chấm phẩy.hiện giờ kết nối mạng ai phai và nguồn điện và khi mất icloud là em không thể ạ. Trứ em không phải kẻ nhu nhược và nhất không làm sai.và phải hỏi và viết đơn sin trứ em không còn giầy vò nào hơn đâu ạ.ẢNH CỦA EM LẠI ĐĂNG TUẤN TÊN EM.faceebook bản quyền Tuấn lư lại đăng tuấn .mây meo và các cổng gêm là phải trực tiếp và sin phép đấy a.còn vòng đấu giải đấu HD của tốp đấu giả đấu vòng đấu như em không cố phát sex si cụng đổ tôi.em sin và cho em cấp phép em dùng họ lừa hay giả để làm vậy em cụng không hề chống hay cố.nhưng mỗi khi mọi người dúp cứ gửi và trực tiếp kêu cứu toàn cầu vì không có ai trả lời và bảo để em còn làm.nhờ dồi mà làm người dúp mình khổ em đau lắm ạ.hãy dúp em vì em chỷ cố găng cho chẻ và góp dúp người khổ toàn cầu vì em dất khổ.song con em bé sau này cháu sẽ gặp may.còn các khoản tiền các bên đã tim dúp cụng mất tính.phần quà cho gđ mọi người hiểu hết ngày này tháng khác.và lần báo thu bản quyền hoặc bảo có tội vi phạm cộng đồng mạng em hỏi và kiến nghị trưa vi phạm bao giờ ạ.

  2. Tổng thông luật sư mọi người em không tránh nhưng không thấy ai dúp em vui cả.còn một nghì không phần thưởng quà không.cúp nút các thứ chỉ làm khổ thêm vì mọi người báo nói linh tinh.bố mẹ em trửi này tổng thống và nước ngoài không trả mày làm sao .em đau lăm còn bom mìn em biết ai dúp.còn đúng da các bên cấp phép và mọi bên nhận phải bảo em ngay luật sư song cười hết.em làm sao

  3. Cảm ơn mọi người đã dúp đỡ em và ghim dúp em ai giả mạo anh sên sử.sâu các ban khóc em buồn.còn sai cụng nhiều phải Thông cảm ghim em sử ạ.

Leave a Reply

Your email address will not be published. Required fields are marked *