Snag this widget and put it on your site

No doubt every generation views the changes occurring within their lifetime as historical and monumental. And we can debate where the real time web fits into communication’s evolutionary path that includes cave paintings, the printing press, and Radio Corporation of America (RCA). But, what’s not up for debate is that we’re seeing some real game-changing data and trends.

First, there’s the size of the web. Back in 1999 (when Google still had an exclamation mark after its name), Google had 24 million pages in its index. In 2003, Google broke the 3 billion page mark and then in 2008 the company announced it had 1 trillion unique URLs in its index. Applying these points on a curve shows that the web is growing by more than a billion new URLs every day. And the pace is only accelerating.

So, what’s driving this massive growth? In the early days of the web, content was created by PhDs, scientists, and institutions. We were dealing with highly researched, heavily edited content that inherently took time to create. Today, we’re seeing content explosion in the ‘expressed’ web — on Twitter, YouTube, blogs, Flickr…

For example, in August 2009, it was reported there are over 200,000,000 blogs.[1] In October, CNET reported that Twitter hit its 5 billionth tweet.[2] YouTube states that people upload hundreds of thousands of videos daily. And for some historical perspective, it took 38 years for radio and 13 years for TV to reach 50 million users. By contrast, Facebook added 100 million users in less than 9 months, the same amount of time it took iPhone apps to hit the one billion mark.

And as the web moves out from beyond the desktop, there are even greater opportunities for content creation. Armed with a mobile phone, we’re now able to publish content instantly, as we experience it — whether ‘it’ is something as mundane as our breakfast, hyper local as a weather event, or a tragedy or celebration with national even global implications. Mobility breaks down the traditional time barrier between the event and the dissemination of information about the event.

And the adoption of mobility is only growing. A recent report by the Pew Research Center[3] found that use of the Internet on mobile devices has grown sharply from the end of 2007 to the beginning of 2009. In 2009, 69% of adult Americans said they used their cell phone for a non-voice activity.

Mobile devices don’t just provide a new platform for content creation; there’s now information flowing to us, following us wherever we go. From news to fantasy sports, and shopping, content is continually streaming to us at home, at work, and everywhere in between.

This represents a major change. Think back to the days of AltaVista and Eudora. We experienced the Internet within discrete points of time. We logged into an account to retrieve email. We performed a static search for a piece of information and retrieved the results.

Today we have lifestreams and newsfeeds. Email effortlessly flows to our inbox. We are increasingly growing accustomed to experiencing the web without hitting refresh.

This concept has far-reaching significance. As content continually flows to desktops, laptops, phones, and other devices, the Internet has become a seamless part of our lives. The web is now a living, breathing organism. No longer a flat assembly of pages, it’s alive with meaning, personality, and social inference. And, the web is changing every second in this dynamic ecosystem.

This is where search needs to catch up. A traditional search engine crawls throughout the vastness of the web, takes a snapshot in time, and indexes each URL. When you perform a search, it returns results ranked 1-n about everything related to your search. All of these traditional services look at the web as an ocean — they want to gather it, process it, rank it, and then give users the best results for any given topic.

For a majority of searches, this approach works extremely well. Want to know who won the academy award in 1987? The size of Google’s index in 2003? How to replace the wiring in your house? Where’s a nearby hardware store and how to get there? Who’s the top rated electrician in your area? Traditional search has evolved to meet your needs with astonishing accuracy and relevancy.

Yet, these searches leave out the rivers of temporal information that are flowing throughout the web. The technology needed to bring the web to life in a streaming format is fundamentally different than the traditional approach of crawling, gathering, and indexing.

As a result, while people turn to search for transactional and navigational queries, they turn to social networks like Twitter and Facebook to capture the sentiment, expressions, and snippets of the moment. Yet, the limitations of these networks are clear — they provide only a small subset of what’s happening on the web as a whole, leaving out the full breadth of content published on blogs, news outlets, videos, pictures, and everything else on the web that can’t be condensed down to 140 characters.

True real-time search encompasses more than content on social sites. By viewing the web as a fast-flowing river rather than an ocean, a real time search can deliver a continually streaming results set that has the power to show the story as it unfolds. As a result, people don’t just retrieve facts and data, they become part of the story as it happens.

An obvious example is in the coverage of breaking news, such as a hurricane, an election, or a tragedy such as the shooting at Fort Hood. With access to streaming, real-time search, people can follow these stories as they happen — through first-hand Twitter posts and pictures captured from people on the scene, alongside breaking updates from credited news sources, overhead footage from news choppers, and people’s comments, reactions, and analysis from across the Web. This information valve can be vital to those in the immediate area, as well as help unfold the story for everyone else in real time.

Implications of the real-time web aren’t just limited to breaking news. Real-time search brings timely information for other types of queries. For example, a search for ‘cash for clunkers’ streams current references and content — such as an analysis of the released data from the program.

Understanding the opportunities before us is the name of the game now. We are standing at a crossroads and many have yet to accept there can be a different approach to information retrieval than crawling and indexing. As we move forward in the real-time web, there will be countless challenges, most importantly how to shape the experience so it resonates with users. The task before thought leaders and visionaries is to harness the power of the real-time web with the right set of applications. Without the proper focus on the end user experience, real time technology can overwhelm users with irrelevant and trivial data. Yet with the right approach and applications, real-time search further speeds the evolution of the Internet and helps people make better decisions, feel more connected, and have a deeper understanding of the world around them.

[1] http://socialnomics.net/2009/08/11/statistics-show-social-media-is-bigger-than-you-think/

[2] http://news.cnet.com/8301-13577_3-10378353-36.html

[3] http://www.pewinternet.org/Reports/2009/12-Wireless-Internet-Use.aspx?r=1

Repost from blog.collecta.com

—

Yesterday Twitter announced that it is making its stream available to both Bing and Google. And Bing will get Facebook, too.

Collecta is elated that these streams are now in the hands of the big guys.

The reason? Collecta has focused on doing something *different*. You could argue that the window for a startup entry in Twitter search ended when Summize was acquired by Twitter in July of 2008… an event Collecta CEO Gerry Campbell participated as an investor and advisor to Summize.

Since that time it has been clear to Collecta’s founders that real-time search is about not only Twitter, but the *rest* of the web.

You remember the rest of the web, right? The thing with 1.7 billion users and all the world’s published digital content… the thing that’s not constrained by 140 characters and shortened links?

That’s where Collecta is focused.

Twitter’s coming of age signals that the game is changing; the real-time web is here to stay, just as we predicted.

Collecta was founded to address an emerging market – one that is created based on the natural evolution of two trends:

Trend 1 – The web is accelerating. The entire pace of content generation is speeding up. And it’s not just user generated content. Examiner.com has assembled an army of journalists who are publishing high-value stories at a blistering rate. People and Entertainment Weekly blogs are on fire. There are thousands of examples. These are the rule – not the exception.

Trend 2 – Users have an insatiable appetite for information. Users expect timeliness. We’re now completely immersed in broadband, swimming in wireless access, connecting via our netbooks and mobile devices… the Internet’s information is an integrated part of our lives. As just as present as the air we breathe. This appetite for up-to-the-second information will grow for the foreseeable future.

In the confluence of these two trends lies an opportunity: Bring the vibrance and richness of the new “now web” to users. It shouldn’t be limited to a single social network, but reach across all media types and publishers to bring everything together. And it must be built upon technology capable of handling the skyrocketing volume at web scale.

At the heart of this opportunity sits Collecta -the world’s only real-time, streaming, push-based web search.

Throughout the excitement surrounding the Real-time search craze, Collecta has been diligently building revolutionary infrastructure — infrastructure that we expect to become the de-facto standard source for the Web’s timely content.

Yesterday Twitter hit the mainstream and time will show that phase 1 of real-time search has come to a close. The game went up by a few notches. Collecta has been waiting for this – bring on Round 2.

For more on the Google and Bing news, please read:
- ReadWriteWeb
- AdAge
- Washington Post

Yesterday I was on a panel at the AlwaysOn conference at Stanford University in Palo Alto. The conference itself is unusual – in a great way. It brings a wide range of investors and entrepreneurs together to talk about innovation and emerging trends. The 3 days are filled with notable people and great thinkers. It was a really great event.

The panel I was on related to the future of social media – specifically “Social Media Beyond Facebook.”I have to say it was one of the highlights of my conference participation over the years.

I say that because I really have respect for the entrepreneurs I shared the stage with, the intelligence and experience was landmark. Our discussion was really enlightening and i think we all came away feeling like we had illuminated some of the key trends that are influencing entrepreneurs who are building the next wave of businesses – post social enlightenment.

Bambi Francisco of Vator.tv led us through a series of questions that opened up several dimensions of discussion. Shervin Pishevar of Social Gaming Network, Max Ventilla of Aardvark and Clara Shih, author of The Facebook Era and founder of Heresay Labs and I threw the themes back and forth over about 45 minutes.

Key takeaways:

- Social networks are the new utility infrastructure for applications and products. All agreed that the traditional “ownership” of the customer is an outdated idea. Instead we are focused on offering value to users, seeking some of their attention in return.

- These networks serve a purpose that is fundamental and natural to the evolution of the Internet as a medium. The Internet has evolved from walled gardens (CompuServe & AOL) to destinations (Yahoo) to distributed atomic-level content and products, assembled on the fly at the user’s convenience in social networks.

- There is vast opportunity to create businesses that reach users within and across the networks. As usage patterns have migrated form destinations to self-focused sites, the opportunity has shifted. We now need to think of social networks as 1) a venue for distribution 2) a source of content. No solid business has a single distribution vehicle nor a single source of content. Nor a single venue for interacting with its users.

- There is a paradox – a low barrier to entry for new products, but a high standard for truly creating value. It’s easier than ever to create applications fo users, however sustainable companies must bring something unique to the picture that’s hard to replicate and valuable to the user. In most cases this is technology IP and infrastructure.

- Those companies that create real value need to be in the position to adapt in symbiosis with the networks.

- CRM is a critical dimension in the picture. Companies need to meet their customers where they are expressing themselves.

- Economic flows within the networks themselves are as of yet unsolidified – however there is full expectation that, when a solid economic construct emerges the participating companies will both benefit from and be subject to those economic flows. In fact, Collecta would be pleased to bear its share of the cost of infrastructure required to enable searching.

So, the future of Social Media? Social platforms are aggregators of users. To the user they provide an increasingly efficient way to stay in touch with the people and topics they care about. For the entrepreneur they are a rich opportunity to create products and services that are truly useful, entertaining or enlightening.

For everyone, they are evolving.

From my point of view, Collecta is in the position of finding the spontaneous, temporal value both within the social networks and in more traditional publishing venues and bringing that to users wherever they are. So Collecta’s strategy is simply that of a search engine: we want everything that’s relevant to our users. We also want to present that to them wherever it’s most useful.

We aspire to provide enough value that users will choose to spend some time with us.

Re-posted from MediaPost from Friday, July 17, 2009

The Death of One-To-Many by Gerry Campbell

Pure print-based businesses are in a spiral of decline that only a new interactive approach can remedy. Also dead is the time when entire communities were guided by their evening news or waited to learn key details about events from tomorrow’s morning headlines.We’ve all known this to some degree for some time. We’re witnessing an age where more than 100 million votes were cast on the final night of “American Idol,” there are 175 million users on Facebook, and the Internet finally overtook newspapers as American’s news outlet.

But what is really happening here?

Media has never been a static form — it has followed a trajectory from cave paintings and papyrus scrolls, to the advent of the printing press to the Radio Corporation of America (RCA) in 1919. Today, with blogs and a constantly updating social stream, we publish information faster than ever, changing our perception of “current” and fueling an insatiable demand for up-to-the-second information.

Beyond this accelerated pace, we’re witnessing a fundamental shift in casting roles. Media is no longer one (one reporter, one show, one company) talking at many. Rather, it’s become a cacophony of public conversations. In short, the audience now participates.

Some can argue that this isn’t a fundamental shift at all. We’re getting back to the oldest form of communication and commerce, the town bazaar, which was the central source of social interaction, news, communication and influence for centuries.

However, today the audience has the ability to interact in real time, and on a global scale like never before.

Take the standard prime-time show. Families no longer sit back in the living room to watch “I Love Lucy.” Now, we’re voting whether Spencer and Heidi should remain in the jungle; we’re voting for Adam Lambert or Kris Allen; and we’re deciding who goes home on “Dancing with the Stars.”

Participation isn’t just limited to voting on reality shows. All over the Web, conversations are taking place about everything from TV to politics. We no longer wait to talk to our colleagues at work the next day. These conversations are happening in real time.

Whether in the form of Twitter posts, status updates, board posts, or blog comments, the Internet is abuzz as the NBA finals enfold. Sports nuts dissect every possible trade permutation as the seconds tick down on the NY Jets’ draft pick. Fans try to make sense of Lost’s season finale, as it happens.

These conversations turn TV programming from passive entertainment to a vibrant community. The feedback loop is nearly synchronous. We want to participate. We want to be heard. We want to matter.

No one has been impacted by this new dynamic more than the news media. Print, local and cable news outlets have all been forced to adapt. Traditional newsprint is not an interactive format. Take issue with an article? Your options are limited to a letter to the editor, which might appear in print a few days later. It’s hard to envision anyone photocopying a newspaper clipping and mailing it anymore.

I believe it’s these core interactive needs of sharing and expressing (not an unwillingness to pay subscription fees) that have driven print media to adopt a digital presence alongside print — or in some drastic cases, drop the printing altogether.

On the Web, readers become participants. They can express their thoughts and opinions instantly, and share them with the entire community. Even those who don’t comment get a richer sense of involvement by simultaneously viewing the discussion threads of fellow readers alongside the original article.

Television news has also needed to open its doors and yield more control to the audience. Some programs give the audience a say in the editorial process by voting on the stories that should be featured on air. One can’t open a news channel’s Web site without seeing a poll. In short, our news is looking less like a college lecture with one professor and many students and more like an engaging seminar where participation isn’t just OK, it’s required.

Local and cable channels are even warming up to the concept of the “citizen journalist” – encouraging (with the necessary safety caveats from legal) people to send in footage of a storm surge, wildfire, mudslide or car chase. And for good reason. News is happening everywhere, all the time. And with today’s 24-hour news cycle, it’s virtually impossible to stay on top of everything, given the finite pool of professional reporters.

What’s the next generation of media going to be like?

Journalism will not be replaced altogether by Joe and his smartphone. Some topics require the credibility, gravitas, and due diligence of a credentialed reporter. However, it is true that big media will increasingly fall behind the microblogger and social network poster when it comes to delivering timely and localized information.

For the foreseeable future, the traditional reporter and microblogger will coexist in the news sphere — and the key will be to bring the two together for a better perspective on what’s happening.

Whether we’re talking about news or entertainment, there’s an explosion of new content in the digital universe. The next stage of media’s evolutionary path will be to collect all these conversations and content side by side, then giving the audience the new ability to see, and more importantly to participate in, all these threads simultaneously.

Future broadcast media programming will incorporate the audience into the story — and audience opinions will be expressed in many and varied venues.

Real-time search is the transformational force behind this next milestone and will impact the way we see breaking news, entertainment, brand mentions and more. For example, imagine the impact if a TV fan can find every discussion thread about his or her favorite show, while the show is unfolding, or a school administrator can follow updates on swine flu outbreaks as they happen.

Real-time search brings a new level of currency and context to events that will inevitably spark interaction at an even faster, broader, and more meaningful scale.

Good search market landscape article on Wired.com

I guest blogged today on Altsearchengines about realtime search, social search… differences and interrelationship.

Might be interesting to you if you seek clarification about what realtime means and how it all fits together.

Check it out at altsearchengines.com

Realtime search – you can discuss it, theorize about it, and sometimes it’s time to just show it.

I twittered earlier this week that I was watching the swine flu concept evolve in the Collecta realtime search internal product. I got a pretty significant amount of responses asking me to share, so why not.

But first, my reflection…

After spending hours entranced by the range of images, news stories, twitter messages and comments flowing from various points on the web, I became even more sure that realtime search is a thing we will rely on in the future.

Why? Because much of the relevant information being published on this super-timely topic is headed for one of the following places:

• A single site with limited readership
• A feed, to be experienced in a flow of other homogeneous feeds
• And worst of all, an index – to be searched later, when the full impact of the facts, opinions and impact has faded.

All 3 places are less than optimal. Readers interested in a topic deserve to have the best and most timely information come to them. Clean and focused.

Another thing occurred to me – and this is a HUGE point – the story I am assembling in my head about the topic is enlightened and in aggregate it’s editorially comprehensive!

Facts and information are forming before my eyes. Within a couple of minutes I literally watched the public awareness evolve. Can it be transferred by eating pork? No. Does Israel have the unilateral right to rename the disease? Maybe not.

It is amazing what the collective consciousness can share when viewed through this new Collecta window.

There’s also weirdness. Right-wing, Left-wing and everything in between. Did Obama ask media to overemphasize it to mobilize the country? Did Rumsfeld cause it to pump his portfolio? Maybe Bush caused it as an act of vengeance… ? It’s all in there.

My point is that for the first time I am in a position to aggregate the stories, blogs, photos and comments while they’re happening. I can get the whole picture right now. Then it’s up to me to sort out bias. To distill facts. To come to an enlightened opinion on the topic. This can take just a few minutes – the information is flowing right in front of me.

So, this site is just a sampler. There are bugs to work out, user experience and design dimensions to share. A long way to go. And when we launch in the near future with a full-fledged search, it won’t be a replacement for traditional web search.

It will, however, give users a new, more comprehensive view of what’s going on in the world right now.

Click the logo below to see realtime web results for swine flu.

Press Release here.

In the past several months I’ve written about the effects of splintering media, the amazing new medium exemplified in Twitter and how social networks are changing news.

Inherent in that has been the idea that the web – or what we know today as our online life – is in mid transformation and we’re in a unique punctuated time in the evolution of the Web. A point where the new winners are being defined and previous leaders risk becoming irrelevant.

I’ve also written about how the online world is operating on old ideas. Our previous approaches are not human enough, intuitive enough or timely enough to carry us into the next phase of the Internet.

It’s a world where infinite bandwidth and constant connection to the digital cloud provide us with news, entertainment, communication and commerce in a seamless flow that keeps pace with our human need for More, Faster and Better. Every depiction of the digital futute paints a picture of rapid-fire information intake and overload. Who doesn’t feel it?

Where is this leading? How is our reality going to be different?

We’ve got more stuff flying by than we know what to do with – we need Realtime search.

Media has splintered down to an individual level, where all of my acquaintances – as well as the traditional sources of information – are generating vast volumes of information every day. Information I may be interested in if I only had better tools to reach into the flow and collect things that are relevant to me. If I don’t tap into it effectively I’ll lose control and miss critical bits of info all together.

That’s a problem.

The answer is simple – but until now nearly impossible to create. I should be able to activate a query – set it out like a net to collect results on anything at all – and wait for the results to flow by.Some things like school lunch menus and research on mesothelioma may not be so timely, but it would be good to have an active query watch those things for me.

Other things like trade-able information on a stock in my portfolio, or details on a TV show while I am watching it, those things may very valuable to me in realtime.

This is the next frontier online, and it’s just beginning. Over time there will be the full range of experiences, business model modifications, and controversies to go with it.

There is an opportunity for a company or set of companies to blaze the trail and create the new, fast-twitch experience for realtime. This is not a small opportunity, John Battelle agrees.

So seeing the vision is the first part. The second is mobilizing to make it a reality.

In meeting up with Collecta (formerly Stanziq) I have found a team with the experience and a technology worthy of attacking to opportunity.

Today we are announcing the creation of Collecta and, with the support and excitement of founders Jack Moffitt, Patrick Mahoney and Brian Zisk as well as Jon Callaghan of True Ventures, I am joining as CEO and rolling in my pet project, Shrty.

Based on its roots in Stanziq, Collecta is already a strong search company driven by a truly innovative group of people: Jack Moffitt, Brian Zisk, Derek Powazek and a deep team of experts – focusing singularly on making a search of the web into a realtime experience. Check out the team bios on Collecta.com

Together we make a solid, rounded team for building a realtime search company.

Collecta has created a platform that promises to open up a new dimension in searching. It complements web search as we know it and add a new level of control and excitement to daily info-gathering.

The platform is in pre-beta, products will come very soon. Keep an eye out.