No doubt every generation views the changes occurring within their lifetime as historical and monumental. And we can debate where the real time web fits into communication’s evolutionary path that includes cave paintings, the printing press, and Radio Corporation of America (RCA). But, what’s not up for debate is that we’re seeing some real game-changing data and trends.

First, there’s the size of the web. Back in 1999 (when Google still had an exclamation mark after its name), Google had 24 million pages in its index. In 2003, Google broke the 3 billion page mark and then in 2008 the company announced it had 1 trillion unique URLs in its index. Applying these points on a curve shows that the web is growing by more than a billion new URLs every day. And the pace is only accelerating.

So, what’s driving this massive growth? In the early days of the web, content was created by PhDs, scientists, and institutions. We were dealing with highly researched, heavily edited content that inherently took time to create. Today, we’re seeing content explosion in the ‘expressed’ web — on Twitter, YouTube, blogs, Flickr…

For example, in August 2009, it was reported there are over 200,000,000 blogs.[1] In October, CNET reported that Twitter hit its 5 billionth tweet.[2] YouTube states that people upload hundreds of thousands of videos daily. And for some historical perspective, it took 38 years for radio and 13 years for TV to reach 50 million users. By contrast, Facebook added 100 million users in less than 9 months, the same amount of time it took iPhone apps to hit the one billion mark.

And as the web moves out from beyond the desktop, there are even greater opportunities for content creation. Armed with a mobile phone, we’re now able to publish content instantly, as we experience it — whether ‘it’ is something as mundane as our breakfast, hyper local as a weather event, or a tragedy or celebration with national even global implications. Mobility breaks down the traditional time barrier between the event and the dissemination of information about the event.

And the adoption of mobility is only growing. A recent report by the Pew Research Center[3] found that use of the Internet on mobile devices has grown sharply from the end of 2007 to the beginning of 2009. In 2009, 69% of adult Americans said they used their cell phone for a non-voice activity.

Mobile devices don’t just provide a new platform for content creation; there’s now information flowing to us, following us wherever we go. From news to fantasy sports, and shopping, content is continually streaming to us at home, at work, and everywhere in between.

This represents a major change. Think back to the days of AltaVista and Eudora. We experienced the Internet within discrete points of time. We logged into an account to retrieve email. We performed a static search for a piece of information and retrieved the results.

Today we have lifestreams and newsfeeds. Email effortlessly flows to our inbox. We are increasingly growing accustomed to experiencing the web without hitting refresh.

This concept has far-reaching significance. As content continually flows to desktops, laptops, phones, and other devices, the Internet has become a seamless part of our lives. The web is now a living, breathing organism. No longer a flat assembly of pages, it’s alive with meaning, personality, and social inference. And, the web is changing every second in this dynamic ecosystem.

This is where search needs to catch up. A traditional search engine crawls throughout the vastness of the web, takes a snapshot in time, and indexes each URL. When you perform a search, it returns results ranked 1-n about everything related to your search. All of these traditional services look at the web as an ocean — they want to gather it, process it, rank it, and then give users the best results for any given topic.

For a majority of searches, this approach works extremely well. Want to know who won the academy award in 1987? The size of Google’s index in 2003? How to replace the wiring in your house? Where’s a nearby hardware store and how to get there? Who’s the top rated electrician in your area? Traditional search has evolved to meet your needs with astonishing accuracy and relevancy.

Yet, these searches leave out the rivers of temporal information that are flowing throughout the web. The technology needed to bring the web to life in a streaming format is fundamentally different than the traditional approach of crawling, gathering, and indexing.

As a result, while people turn to search for transactional and navigational queries, they turn to social networks like Twitter and Facebook to capture the sentiment, expressions, and snippets of the moment. Yet, the limitations of these networks are clear — they provide only a small subset of what’s happening on the web as a whole, leaving out the full breadth of content published on blogs, news outlets, videos, pictures, and everything else on the web that can’t be condensed down to 140 characters.

True real-time search encompasses more than content on social sites. By viewing the web as a fast-flowing river rather than an ocean, a real time search can deliver a continually streaming results set that has the power to show the story as it unfolds. As a result, people don’t just retrieve facts and data, they become part of the story as it happens.

An obvious example is in the coverage of breaking news, such as a hurricane, an election, or a tragedy such as the shooting at Fort Hood. With access to streaming, real-time search, people can follow these stories as they happen — through first-hand Twitter posts and pictures captured from people on the scene, alongside breaking updates from credited news sources, overhead footage from news choppers, and people’s comments, reactions, and analysis from across the Web. This information valve can be vital to those in the immediate area, as well as help unfold the story for everyone else in real time.

Implications of the real-time web aren’t just limited to breaking news. Real-time search brings timely information for other types of queries. For example, a search for ‘cash for clunkers’ streams current references and content — such as an analysis of the released data from the program.

Understanding the opportunities before us is the name of the game now. We are standing at a crossroads and many have yet to accept there can be a different approach to information retrieval than crawling and indexing. As we move forward in the real-time web, there will be countless challenges, most importantly how to shape the experience so it resonates with users. The task before thought leaders and visionaries is to harness the power of the real-time web with the right set of applications. Without the proper focus on the end user experience, real time technology can overwhelm users with irrelevant and trivial data. Yet with the right approach and applications, real-time search further speeds the evolution of the Internet and helps people make better decisions, feel more connected, and have a deeper understanding of the world around them.

[1] http://socialnomics.net/2009/08/11/statistics-show-social-media-is-bigger-than-you-think/

[2] http://news.cnet.com/8301-13577_3-10378353-36.html

[3] http://www.pewinternet.org/Reports/2009/12-Wireless-Internet-Use.aspx?r=1