Site Logo

Search Engine Etiquette

When we first launched,  I read a few blog post that contemplated whether or not what we were doing is legal.  Not only do I firmly believe that it’s legal, I think the debate is somewhat moot.  We hold ourselves to a much higher standard. 

As a search engine, indexing the content on other sites is a privelege not a right.  Publishers should clearly see value in the relationship.  If they don’t, they can easily ask search engines (all or a selected few that are ill-behaved) to stay away.  doing this is as simple as adding a single file to their web site — robots.txt.

As I’ve stated in other posts, web publishers (including commerce sites, etc.) increasingly view search engines as playing an important role in their market ecology.  To play this role, however, i think companies need to follow certain unwritten rules (or net-etiquette):

1. Summarize and point, don’t aggregate

I’ve seen oodle referred to as a classifieds aggregator.  We are not an aggregator in that we don’t display listings.  Our job is to point our users to listings published on other sites — and in doing so, get users off our site as quickly as possible ( love-in not  lock-in).

To this end, we’ve provide (attributed) summaries that contain enough information for users to qualify their interest but  not enough for them to take action.  To respond to a listing, users need to click-through to the full listings on the publishers site.

2. Be a search engine not a publisher (or said another way, don’t compete with your content partners)

It’s a fundamental conflict of interest when a company attempts to both play the role of search engine and play the role of a publisher.  Why would publishers let their listings be indexed by a company building a competing  business off their back?  It will be interesting to see how this point plays out in the coming weeks.   Rumors are ablaze with Google’s entry into the classified listings business.  It’s also rumored to be adding classifieds listings into Froogle.  Will classified publishers stand for this?  I’m not sure why they would.

9 Responses to “Search Engine Etiquette”

  1. August 11th, 2005 at 3:12 am

    Karl says:

    I was thinking to write a meta search engine to crawl all your listing, plus some others not indexed by yours.Is it ok? If not, why?

  2. August 12th, 2005 at 6:41 am

    Cody says:

    I’m anxious for an answer to the previous comment…oodle?

  3. August 16th, 2005 at 10:35 am

    Craig says:

    Sorry for the delayed response. Have been traveling.

    Meta searching is fine as long as it’s handled appropriately (within the bounds of net-etiquette). Such is the case with other search engines such as Google and Yahoo…

  4. September 14th, 2005 at 11:58 am

    Dennis says:

    I noticed that the job section has monster, hotjobs, etc. Being a lawyer, I read their TOS and it specifically says that you cannot redisplay or manipulate their data w/o their consent. Do you have their written permission?

  5. September 17th, 2005 at 8:25 pm

    Dan says:

    Hey Dumb-Dumb,

    “We are not an aggregator in that we don’t display listings. Our job is to point our users to listings published on other sites.”

    An aggregator does not display content either. Therefore, you are an aggregator or a scraper.

  6. September 28th, 2005 at 1:07 pm

    mod says:

    Do you have their written permission? (
    monster, hotjobs)

    thanks.

  7. September 29th, 2005 at 9:36 am

    craig says:

    Dan,

    The only point I was trying to make is that people don’t think of Google as an aggregator. We act just like them (showing summaries to help people find the right listing).

    That said, if Google is an aggregator to you then please consider us an aggregator.

  8. September 29th, 2005 at 5:03 pm

    Craig says:

    With respect to the question(s) above — do we have written relationships with companies like Monster and do we check every terms of use agreement – the answer is no.

    There would be no billion page indexes if search engines needed to check every terms of use agreement on every website. Instead, search engines rely on a well established etiquette using robots.txt. We check robots.txt every time we visit a site. We also conform to the net-etiquette around search results lists (display summaries, attribute, etc).

  9. October 17th, 2005 at 6:53 pm

    Ray says:

    how, in what universe, can it occur to you that oodle does not compete with the content providers for viewers? why would a viewer go to a site that sells 10 widgets when it can select form a site that has 100? can you twist logic enough to answer that?

Leave a Reply