luckyrobot.com – Gerry Campbell header image 2
← →

Search is broken – really broken.

February 6th, 2009 · Comments

We are all really fortunate – in this age of the web, we have access to and can share ideas with the greatest minds of our time.

And, on top of this great connective technology, there’s an accompanying culture that values democracy over hierarchy, open sharing of ideas over closed and authenticity over hidden motives. This really is a great environment for recombination and synthesis – similar to the continental shelf in the Cambrian and Northern Italy at the earliest stages of the renaissance.

So, with great opportunity comes great responsibility…. If you’re reading this I consider you to be on the hook. ;-)

Personally, the responsibility I feel is to help people find information – which enhances the quality of their life, their station in life or simply saves time. I’ve said it before. Still true. I am really unsatisfied right now. We’ve been all distracted. Time to get busy.

In John Borthwick’s post this week, he brings up the idea that Google is soon to suffer from Creative Distruction… So true.

I can go beyond that and say that we – the people who make stuff on the web – are suffering from a bigger-than-big creative draught.

Tom cat starts chasing Jerry mouse. They run in a circle… and soon Jerry stops running and steps off to the side while Tom runs faster and faster in a circle completely forgetting what he was up to in the first place. They were both up to something when the games started and now they’re completely distracted.

That’s how we are with Search and information discovery – and possibly the entire web X.0 product. We’ve been running in circles for years and years, chasing index size, market share and revenue growth and have forgotten that the whole point in the beginning was to help people find information.

[sidebar]

There are exceptions – cases where real communities, rich with all of the facets of life exist together, but they are usually concentrated within small subsets of information, within limited groups of people or are only for the most technologically imaginative.

In fact, Howard Rheingold wrote about it in his insightful and brilliant book The Virtual Community way back in 1994. But it’s still elusive.

[/sidebar]

In the flurry of data, charts, investments revenue and metrics, reorgs and rounds of funding we’ve forgotten that we’re human, and need to build products that scratch the human itch. I’m not talking about the superficial Social thing we’re all a part of, where we connect to 500 of our best friends that we forgot we knew or never knew to begin with… I’m talking about an integrated lifestyle that includes our news, our entertainment, our communication, our shopping and transactions… our emotions and biases… into a coherent whole.

We have it in the physical world. For those of us who identify with a real-life community, it’s completely natural for a trip to the grocery to transform into a discussion about Lost when we bump into a friend. And it’s completely natural for us to skip conversing with the neighborhood acquaintances when we’re at the pool.

And it’s also completely natural for us to hear about a new clothing brand, a new Broadway show, a new car model or a new disease from friends at the coffee shop.

My point is this: in the real world entertainment, transactions, communication and information discovery are facets of a coherent, integrated whole. It’s the richness of life. This integration of things helps us prioritize our interests and time, helps us keep our social relationships in the right proportion and helps us to discover new things with surprising efficiency.

This is not the way it goes online… The web we have built is disassembling our lives and complicating them – not simplifying them.

Since web 1.0, when we took the cost of publishing to near-zero by putting a digital representation of the printed page online, our lives have been fragmenting. Since then, platforms, systems, meta systems and metrics have been built up to generate more content faster. More transaction opportunities. Web 2.0, with its platforms, feeds and social media have continued the trend. If you doubt me… How many email addresses do you have? How many ways can people message you online? Feel on top of it?

Don’t get me wrong – I am not discounting anything that’s been done to date. It’s all great stuff. Necessary for the next phase.

But let’s get on to the next phase, and remember what the potential is here… what history will measure and what our task is now:

Let’s use the building blocks we have in the form of social networks, search technology and semantic inference to RE-INTEGRATE our lives. To create the online replica of our real offline world. To use social inference to prioritize our information intake.

Enough fluff. The real point.

Our daily lives are rich with social inference, and they happen in real time. Search from Google, Yahoo… you name it – they are all based on published (e.g. considered, thought-through) documents that take minutes-to-weeks to update in the search index.

This is broken broken broken.

And blog search is not much better. Categorically identical.

Twitter search is great (go Summize!) as it is realtime and searches expressed information vs published. But the entire world’s entertainment, news, communication and transactions don’t happen solely there.

So there’s an opportunity. Realtime search, using social inference for discovery, ranking and prioritization.

My last major point here is that we need to think creatively about this. I believe that focusing on the query a user inserts into the box and the resulting blue links are WAY too limited. This is the embodiment of the brokenness.

I like what Kosmix is doing in delivering structured/federated results, it’s like AOL Fullview on steroids. And I like what Twitter is doing – there is a solid example in there for how engaging realtime expressed results can be.

I also love all of the approaches to apply “aboutness” and location to the Web – like OpenCalais, Glue, Outside.in, etc. These go a LONG way toward making the web more human.

These things are necessary but not sufficient.

If it’s not the query in the box and the blue links, what is it???

As humans, out life flows from mode to mode – topic to topic with very few breaks. The query doesn’t show up in the box on its own. It doesn’t materialize at the end of our fingers when they hit the keyboard. It comes from somewhere. The desire for information originates from the stimulus of our environment.

So, in this connected web-world full of smart people, I think this is the problem worth solving. I have been throwing the idea around with [insert pundit 1 here], [insert investor 2 here] and [insert super-clever folks N here] (sic.) and I think we are on to something. I’m getting pretty excited about it.

If you have ideas to share on the topic, or think you can contribute to the answer please let me know.

Tags: search

Viewing 26 Comments

    • ^
    • v
    Nice blog there.. a very interesting post.. I enjoy reading your post..
    • ^
    • v
    One of the huge emerging problems of the web is that as information moves faster and faster, it becomes stale quicker and quicker. Stale data isn't just less useful than than fresh data - it is potentially incorrect or misleading.

    Take the example of real estate listings in this down market. The prices are moving faster than the search index can reflect so 9 out of 10 search results on the page often have the wrong price.

    So how useful is search if what is returned is not "correct" ? The list price was correct at some point in time but is no longer valid. The heart of the problem is that search today lacks temporal semantics - i.e. the ability to represent the same piece of information along a timeline with different values. Making the problem worse is that current search engines have no way of reflecting temporal semantics and seeing the history of values - the changes along the timeline - might be of great interest. Stream search is a completely untapped facet of search.

    There are many aspects of search that remain unexplored. Search communities are a natural social pattern. Personalized search contexts are another opportunity - why after all these years should each search be completely divorced from the context of your prior search request ?

    Search is not getting better - it is getting worse. In part because information becomes stale more quickly and in part because we have not expanded on the search paradigm. Google dominates the space and is remarkably not very innovative when it comes to advancing Search. Everything else but Search.

    • ^
    • v
    Good writeup, Gerry. If you'd like to learn more about what I deem to be the future of the web, and how will we make it serve humans (instead, as you've pointed, us serving it), please read this: http://tinyurl.com/cqnylv
    • ^
    • v
    I would add for more on how f'd up search is my personal blog has a ton of opinion posts on it re: the future of search www.furrier.org
    • ^
    • v
    Just found this post after friending you on twitter. Totally right on. Search is so broken and I mean so 1999-2001.

    If you look at user behavior in the terms of "discovery and navigation" across today's apps then the notion of just an index seems silly
    • ^
    • v
    Gerry,
    Long time no see, and it's great to read your post. I agree that traditional search engines are oriented around authoritative content (where lots of links from authorities matter). This makes them less well-suited to ranking real-time content. I also have thought about the theme of fragmentation, though I think it applies to the real-world as well as to the web (you have soccer buddies but don't much discuss technology with them). (There's actually an inverse fragmentation happening with social networks, where you get to have just one persona to share with all your friends, instead of different personas you get to present in context of real conversations).

    It sure seems like knowledge of social graph and context should make search better, but real progress would come from examples of actual broken-ness. Hence: What are 10 actual queries, or tasks, for which search is broken today?
    • ^
    • v
    Please read my post about SDQ (Serendipitous Discovery Quotient). Basically, search has long been broken, it is autistic to the consequences of human cognition (we don't do singularity, everything has many facets).

    Links:

    1. http://bit.ly/3jZTWP - Post about SDQ with a BCG matrix
    2. http://www.openlinksw.com/weblog/public/search.vspx?blogid=127&q=sdq&type=text&output=html - collection of post related to SDQ

    Kingsley
    • ^
    • v
    Nice read. I've been thinking about this problem in a similar way. I think of these modes and contexts as analogous to the soft partitions we setup in our daily lives - professional, personal, familial, etc. If social tools such as Gmail, FaceBook, Flickr, MySpace et al. actually connect their graphs and data together then we can analyze our behaviors and automatically discern what our mode. Even cooler things happen with photo & video data. Think about searching across your entire graph for photos/scenes from a party you attended the previous night. You didn't post any but at least 10 other people did, some in your graph, maybe some just outside. You need only click on two or three related images and the system should be able to do the rest - determine by the EXIFs GeoCode and Date/Time what cluster you are requesting and farming your graph to figure out who else has data you need based on your mode. Think Pandora for searching via your graph - "Nope, Close but not it, Yes, that's it". In a few clicks the system can figure out the most logical metric for your responses and fetch data that will seem most natural to you.

    • ^
    • v
    Jerry - right on! But, in the end, it's pretty simple: search doesn't work because it's a paradigm that we avoid in our lives (looking for lost keys, needle-in-a-haystack, etc). When we pick up the newspaper we know that there is a limited data set available to review - our digital lives should be no different, but personalized by our preferences and desires. Serve me the information that I want (whether I know I want it or not) and you have a user-for-life.

    So, we'd like to introduce you to our solution: Kayanta. (http://www.kayanta.com) We'd love to give you a demo of the site/app and get your thoughts. We're ready for our Alpha and are about to start building our Alpha audience.

    You can reach me at hank@kayanta.com. Looking forward to the discussion.
    • ^
    • v
    Hank - would be happy to talk. I am crushed over the next week, but ping me on Twitter and we can set up a chance to talk.
    • ^
    • v
    Hey Gerry.

    Great post. Check out OneRiot.com. It's a real-time search engine, collecting data from people who opt-in to share their user activity. Our panel is now 2million strong and the results come from a fresh, real-time indexing of the web as it changes. It's early days, but we love where it's going.

    Hope you like it.

    Kimbal
    CEO, OneRiot.com
    • ^
    • v
    Kimbal - thanks for the comment. I have had a look... interesting approach. Sort of reminds me of what Seth Goldstein was doing with AttentionTrust before it evolved into SocialMedia. Sort of... but his was about profiles/privacy and I see you're building an index.

    BTW, I feel like we should meet - I heard a lot about you from Jay, Abdur and Greg over the summer...

    I will ping you some time soon to talk.
    • ^
    • v
    • ^
    • v
    in response to invitation to share ideas: social networks help search because they represent a valuable conversation involving language and values shared by an
    individual.

    another opportunity (as yet unrealized) is to understand and design for human conversations, not in the sense of talking or "natural-language processing" but in the sense of supporting an individual to use the sole means of understanding and generating knowledge: the internal conversation we all use to grok our world.

    our interfaces are still designed for "users", that is, humans who click buttons and receive information from machines. if instead we designed for "participants in conversation" in a strict sense---the development of points-of-view that hold beliefs that are subject to challenge and evolution---we might improve more than just search (at best we'd remove the false choices between searching and browsing, consuming and generating content).

    since "conversation works", that is, we are fined-tuned by evolution to converse effectively, it seems to me that the adoption of conversation as the means to design interfaces has a long runway and much to contribute to interface design. for further development of the approach, see http://pangaro.com/search/
    • ^
    • v
    Gerry. Nice article. Would love to catch up.
    • ^
    • v
    "Realtime search, using social inference for discovery, ranking and prioritization."

    It's an interesting idea. Unfortunately, I disagree with both parts. I think real time is way overdone. I'd much rather wait for the signal to noise ratio to straighten itself out, and then find out what really happened.

    The friends issue is trickier. By definition they are a subset of all the people you have met in the real world. But meeting people in the real world is limited by many things such a location, social status, occupation etc.The online world is more efficient in this respect. There is more scope for meeting people whose interests more closely match your own. And, so, when it comes to certain conversations, I'd much rather interact with my online contacts, rather than my 50 or so 'real' friends on FB.
    • ^
    • v
    David - I think you missed the point that the online social graph is a critical ingredient... the physical world is only an illustration...

    And getting the signal to noise relationship solved is the crux of the issue. In the simple query-response world we would simply call it "relevance" or "precision at X."
    • ^
    • v
    OK Gerry, point taken wrt social graph.

    My own approach is slightly different. I explicitly assign authority to people based on subject. So, for example, I give Tim O'Reilly and Scoble high marks for 'tech', and zero marks for 'soccer', whereas I give my friend Boz top marks for humour, because he is a really funny guy, but zero marks for everything else (because he ain't the sharpest knife in the drawer)

    I then break down a given stream (eg Twitter) into subject-specific virtual feeds which I then filter and throttle on the above and other criteria.

    The basic point is: Tim O'Reilly is in my tech stream (with high relevance) even though our social graphs don't overlap.
    • ^
    • v
    Gerry,

    Great post. I disagree with some points though and posted more thoughts here: http://blog.betterlabs.net/2009/02/09/why-google-wont-be-a-victim-of-twitter-why-johns-theory-may-be-flawed/
    • ^
    • v
    Nice post. Good clear thinking...

    A couple of quick responses:
    - Agreed: Twitter is by no means the only source of realtime-ness. No way.
    - Can Google (or Yahoo, or...)build a product that addresses this? Yes. No doubt.
    - Should we sit and wait until they do? Nope.
    - Don't miss my point that expressed information, vs published, benefits from a social graph for prioritization/interpretation
    • ^
    • v
    fascinating stuff - myself and a few other developer friends have been thinking a lot about this sort of thing as well lately...

    I think one of the important things that you touch on here is that to make search more relevant, it needs to know more/do more with context...and that's sort of the angle we're hacking the most on now...

    It really feels like it's at least the 'next' step to me...if you think about it, you are already spending a lot of time and energy revealing your interests and experience (via Twitter and blogs for example)...on a basic level, if you could take your recent tweet history and apply it as a lens on any searches you do...it seems that would provide you with some REALLY interesting and relevant (to you) results...and as a side effect, it would also help train you to share (ie. tweet and blog) more about the things you really do care about (so you could get better and better use from your searching systems)...

    It doesn't go all the way to taking us from here to where we need to be...but it seems like a good next step (and aside from the issues of scaling, it seems like the easiest next step too)...anyway - anyone interested in chatting more about this approach can ping me on Twitter @falicon ... I've love to hear more people openly discussing different approaches and ideas!
    • ^
    • v
    Gerry,

    You're wrong here...there is only one point and purpose...revenue generation and revenue growth, growth, growth.
    • ^
    • v
    Interesting point, but experience tells me otherwise.

    Trees don't grow to the sky. Revenue generation/growth is an endless cycle of innovation, discontinuity and cannibalism.

    It's the single-variable management mindset (e.g. revenue-only) that leads to strategically soulless companies - those are the companies that lose touch with their customers and open the door wide for the competition.
    • ^
    • v
    Pubsub did this back in 2005 but they were either too early or couldn't execute, and ever since we've been looking for a real-time notification and search system. I think one of the problems with delivering truly relevant information is that a lot of the content that's published doesn't come with relevant semantic information at the source, and I think that's preventing us from delivering really good results that can scale. Sure we can put information through semantic analysis machines, and crunch away at everything but at the volumes required to impact peoples lives... it just won't work. That being said, finding out about when things happen in real time is a good first step, and that's what summize does so well. I really liked your post and I'd love to continue this discussion.

    -Trevor
    • ^
    • v
    Trevor - thanks for the comment... ping me on twitter and we can talk directly. Would love to have a conversation about your thoughts/experience...

    G
    • ^
    • v
    Some good ideas and nice observations here, thanks. This post along with a number of others recently (by, eg, John Borthwick and Adam Greenfield) seem to be tending on a convergent course with some things I have been thinking lately; helpful and encouraging.

    In your last couple of paragraphs you go toward an idea I like: 'emergent search' by association or connotation; the kind of lateral jump we make during conversation from one thread to another, bound by metaphor as much as anything... I frequently get the feeling of this at work reading twitter.

    Concerning your remarks about fragmentation and complexity ("Since web 1.0...), the feeling that it is making life more complicated is a product of the extent to which we don't quite yet understand how the parts of the emerging system fit together. The incredible speed of popular adoption of social/communication technology tells us that more problems are being solved than created, I think. The fragmentation is necessary to discover the underlying potential for unity -- as you say, necessary for the next phase.
 

Trackbacks

(Trackback URL)

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus
viagra anxiety Order Viagra
taking viagra woman
"));

google

google

asus