In terms of timing, the companies announced that the deal was signed. There is still the customary period before the deal is officially closed (at which point, I expect we're going to have a great party).

I'm including, below, the text of the announcements from the blogs of Powerset andMicrosoft.
I think these sum up pretty well the logic behind the acquisition on both sides.

It took a lot of work by many people to make this happen. Most significant, of course, was the entire team at Powerset, who executed so well to build and launch a wonderful product that showed the world what is now possible.

Immediately following the announcement, we had a day of calls with members of the press, which resulted in a lot of coverage. I'll try to post a collection of links next week.

One press meeting that I really enjoyed was a podcast with me, Ramez Naam (Group Program Manager for Microsoft Live Search), and Mike Arrington for TechCrunch. That link provides an article, transcript, and the full audio of the interview.

There is a lot more to say about Powerset, Microsoft, the acquisition, and what it means for the future of search, linguistic technology, semantic web, etc. I am excited to be staying on with Microsoft in a strategy and evangelist role and I am looking forward to the chance to talk and write a lot more about this, and from a whole new perspective, soon.

Here is the text of Powerset's blog announcement:

We’re excited to announce officially that Microsoft has signed an agreement to acquire Powerset.

Powerset has always been a small company with big dreams, with the ultimate goal of changing the way humans interact with computers through language. We set out to improve search by indexing Web pages based on the meaning expressed in them rather than just the literal words. Powerset licensed breakthrough technology from PARC, hired world-renowned computational linguists and search engineers, and recently released a search and discovery experience for Wikipedia articles. Our technology helps to improve search results and also makes new features possible, such as Factz, which aggregates information from many articles to summarize a topic.

With any startup, the challenge is to take the seeds of an idea and grow it into a viable company. At Powerset, we transformed our idea into a world-class semantic search platform, demonstrating the future of search with our Wikipedia search experience. But building a large-scale semantic search engine is expensive, requiring an engineering effort and computing resources beyond what most start-ups could ever imagine. Because our goals around improving search align so well, Powerset has decided to team up with Microsoft. We believe that this is the fastest way to bring our technology to market at a large scale.

Microsoft shares our goal to improve search through deeper analysis of queries and documents, and understands that our technology and expertise will play a key role in the evolution of search. With an existing search infrastructure, incredible capital resources, unlimited data, a leading search team, and clear mission to revolutionize the search landscape, Microsoft can rapidly accelerate our progress in building semantic search technology and bringing it to full Web scale. When we launched our first product, we heard: this is great, but when and how will we get Powerset to go beyond Wikpiedia? Microsoft accelerates our ability to move Powerset to the entire Web faster than anyone could have imagined.

Powerset will continue to operate much as we currently do, working in the same building, with the same organizational structure, and with the same uniquely talented and growing team (apply on our jobs page). We’ll continue to tackle the hardest problems in parsing, semantics, ranking, indexing, scalable computing, user experience and all of our other specialties. But now we’ll do it with the support of Microsoft and the vast resources of the entire Live Search team.

Over the past couple of years Powerset has made amazing progress. Starting with just a big idea, we licensed the best linguistic technology, recruited a top-notch team, built out our datacenter, engineered a world-class semantic search platform, tackled deep natural language issues, improved relevance, innovated an interface and launched a great product. So few start-ups ever tackle such deep, scientific problems successfully and create the kind of value we’ve delivered in such short order.

For now, Powerset.com will continue to host our Wikipedia Search & Discovery and we’ll be continuing to experiment with our product, based on user feedback. But, expect many announcements from us in the coming months about how we’re integrating our technology and features into Live Search.

And here's the text of Microsoft's blog announcement:

Powerset joins Live Search

We're excited to announce that we've reached an agreement to acquire Powerset, a San Francisco-based search and natural language company.

Powerset will join our core Search Relevance team, remaining intact in San Francisco. Powerset brings with it natural language technology that nicely complements other natural language processing technologies we have in Microsoft Research.

More importantly, Powerset brings to Live Search a set of talented engineers and computational linguists in downtown San Francisco. This is a great team with a wide range of experience from other search engines and research organizations like PARC (formerly Xerox PARC).

We're buying Powerset first and foremost because we're impressed with the people there. Powerset CTO and cofounder Barney Pell is a visionary and incredible evangelist. When he introduced our senior engineers to some of the most senior people at Powerset — Search engineers and computational linguists like Tim Converse, Chad Walters, Scott Prevost, Lorenzo Thione, and Ron Kaplan — we came away impressed by their smarts, their experience, their passion for search, and a shared vision.

That shared vision is to take Search to the next level by adding understanding of the intent and meaning behind the words in searches and webpages.

We know today that roughly a third of searches don't get answered on the first search and first click. Usually searchers find the information they want eventually, but that often requires multiple searches or clicks on multiple search results. Two specific problems are the most common reasons for this:

* Differences in phrasing or context between a user's search and the way the same information is expressed on webpages. Search engines don't understand today that "shrub" and "tree" are similar concepts. We don't understand that "cancer" sometimes refers to a disease and sometimes refers to a horoscope and when a query or a webpage refers to which.
* Lack of clarity in the descriptions for each webpage in the search results. Sometimes a result looks relevant from its short description on the results page but turns out to be not so relevant when you visit the actual page. As a result, searchers frequently click results and then rapidly click back when they realize they aren't what they're looking for.

These problems exist because search engines today primarily match words in a search to words on a webpage. We can solve these problems by working to understand the intent behind each search and the concepts and meaning embedded in a webpage. Doing so, we can innovate in the quality of the search results, in the flexibility with which searchers can phrase their queries, and in the search user experience. We will use knowledge extracted from webpages to improve the result descriptions and provide new tools to help customers search better.

Working with our existing Search team and other Microsoft teams that focus on natural language, Powerset will help us address all of those problems and opportunities.

We're looking to add even more talented engineers to the San Francisco team to accelerate our shared progress. If you're interested in joining the team, drop us a line.

We'll have more to say about the things we're doing in understanding searches and webpages through natural language technology in the coming months. In the meantime, please join me in welcoming Powerset to Microsoft!

Satya Nadella, Senior Vice President, Search, Portal, and Advertising

• $2000 to be spent on microfinance loans (each person would spend $100).
• $1000 on a program to stimulate Talin's dating life in the short and long term.
• $1200 donation for the Electronic Frontier Foundation (to be matched by Google).
• $400 to buy Talin a Kindle reader, including an e-book version of The Game, by Neil Strauss, which is required reading for Talin's dating life enhancement program.
• $400 to pay the caterers for who brought the yummy and nearly endless supply of dim sum.

Mike Arrington attended the party, his first time at Barney's house, and was surprised to find how few of Talin's friends read TechCrunch, but despite that he had a great time.

The whole photo set is here.

Our initial product offers users a whole new way to experience Wikipedia and Freebase content, based on our unique natural language understanding technology.

A write-up about Powersets Wikipedia product is available on the Powerset blog.

I will write more over the next few days about the product and it's role in the ecosystem of search, content, linguistics, and semantic technology, but for now I'm just incredibly excited. I'll just note a couple highlights from the evening.

We were planning to launch at 9pm PST. But in an unusual twist for a software company, one of our eager engineers actually flipped the switch to make everything live 15 minutes ahead of schedule. Since everything was working, we just decided to go with it!

Within the next couple of hours, the first press articles came out. Pretty much across the board, the journalists and bloggers captured the essence of our initial product. They got what was special about it, and also recognized it for the initial step that this represents (finally freeing us of the Google Killer hype that is impossible for a small startup to live up to).

Within 1 hour of launch, we received a note from a VC asking about possible investment in the company.

And 2 hours after we were live, we had our first denial of service attack. An automated script sent a never-ending sequence of bizarre queries at our system. Fortunately, our own engineers had been preparing for this kind of thing already and we managed to stay up and weather the storm.

The whole company was gathered in the office. We spent time alternating between: making speeches and toasts, reading press articles, looking at the traffic and load, and watching the initial queries float by. The last part was the most exciting: real users and real queries!

Since we launched on Sunday night on Mother's day (thanks, Mom!), we had it pretty easy with relatively light traffic. I think Monday is going to be an exciting day.

Alex Iskold wrote a nice article that provides an overview and categorization of semantic web approaches, technologies, and companies.

Here are a few key points from the article, interspersed with some of my own perspectives.

• The Semantic Web is now capturing broad attention, and has been called the number one trend in 2008 (by Richard MacManus, founder of ReadWriteWeb).
• Yahoo! recently announced that their search engine is going to support RDF and microformats. This will provide incentive for publishers to use semantic markup in their content. This echoes a point I made in my semantic web keynote talk last year (see below), that search engines would create incentives to drive the semantic web faster than people may have expected.
• Several companies are now offering web services to support or automate semantic markup. These include the Semantify web service from Dapper, the Open Calais web service from Reuters/ClearForest, and the Semantic Hacker API from TextWise.
• There are top-down and bottom-up approaches to the Semantic Web. Bottom-up approaches require people to enter semantic markup. This can be in strong semantic web formats using standards like RDF, or in lightweight markup formats, like Microformats.
• Search is potentially a killer app of semantic technologies. The author argues that semantic technologies alone are not enough to deliver better search, but when used in combination with the other search techniques they might be better. I agree that the combination is best. But I disagree with the statement that

  Google's algorithm, which is based on statistical analysis, deals just fine with semantic entities like people, cities, and companies.

  I think there is a significant gap today between what we are used to with search engines and what is possible with stronger semantic approaches, and this will become clearer over the next year. Contextual technologies use semantic markup within the page and combine that with external content and services. Thus a user does not have to search in order to benefit from the semantics. Examples include Snap, Yahoo Shortcuts, and SmartLinks. Such technologies are making their way into the browser, where they will have wider appeal and accelerate the trend toward the semantic web.
• Semantic databases focus on building and utilizing structure semantic information (as opposed to marking up unstructured content). Twine, by Radar Networks, and Freebase, by Metaweb, are two examples. (I am personally familiar with Freebase as we are integrating this within our offerings at Powerset.) Over time, we will see increasing synergies between the semantic technologies based on structured and unstructured data.

I highly recommend this article to people interested in semantic technologies and search. For my own perspective on the relationship between natural language, search, and the semantic web, you can see the video and presentation of my Keynote Talk at the 2007 International Semantic Web Conference, entitled Natural Language and the Semantic Web.

People will increasingly interact with computers using speech or touch screens rather than keyboards, Microsoft Corp. Chairman Bill Gates said.

“It’s one of the big bets we’re making,” he said during the final stop of a farewell tour before he withdraws from the company’s daily operations in July.

In five years, Microsoft expects more Internet searches to be done through speech than through typing on a keyboard, Gates told about 1,200 students and faculty members Thursday at Carnegie Mellon University.

David conjectures, as do I, that when people speak their searches they are more likely to use natural language than to use keywordese, and that this could change the game in search.

I personally can envision Microsoft trying to integrate speech based data entry as closely as possible with our normal style of speaking. Perhaps the phrase “Where can I buy a hd tv?” would be more natural for searchers when you take away the limitations of the keyboard.

Wide spread speech based data entry will almost certainly impact the way Microsoft and subsequently all other search engines deal with search queries.

It's interesting to see Bill Gates predicting this to happen within 5 years. In the blink of an eye, an entire industry is going to change dramatically.

While on the topic of predictions about voice and language, here's one of my predictions that I have been meaning to write up:

Within 8 years from now (2016), every category of consumer electronics will have some linguistic interface as a standard feature.

By "linguistic interface", I mean voice interactions or text-based interaction that is linguage-based. Not that these devices won't still have nonlinguistic interfaces too (e.g. there will still be buttons, most likely). And by "every category", I mean you will not find a category of consumer electronics that does not have some product in that category with that feature.

For example, users will expect to be able to talk to cameras, tvs, stereos, ipods, phones, watches, microwave ovens, refrigerators, cars, etc. There will still be some cameras that aren't language-enabled, but every category will have some products that are.

As my friends Cliff Nass and Scott Brave write in their book, Voice Activated, when people interact with devices using voice, it also invokes the rest of their social apparatus. You can't hear a voice without ascribing some kind of personality, gender, race, social status, etc to the source of the voice. So in addition to expecting linguistic capability, we're also going to start expecting personality within the next decade.

I'll stop here before I get carried away to the singularity...

The articles covers the Founders Brunch, a networking event for founders of companies that I attend regularly.

Many of my friends are quoted in the article, and there are photos of Auren Hoffman and Keith Rabois (our host this time). Peter Thiel expressed the networking aspect of this kind of event well:

Founders Brunch is important for the same reason Silicon Valley is important: There are all of these subtle network effects," said Peter Thiel, a 40-year-old former PayPal executive now bankrolling some of the hottest Internet companies. "Otherwise why wouldn't you start a tech company in Fresno where everything is cheaper? The advantage to being in Silicon Valley and the San Francisco area is that so many other people are doing the same thing."

Jessica noted that I had a new beard, and I explained my recent decision on growing it:

Barney Pell, the 39-year-old co-founder of Powerset, a natural-language search engine trying to challenge Google, sported a new beard he vowed not to shave until his San Francisco start-up launched its new product.

To be more accurate, I vowed not to shave off my beard until the launch, but I didn't vow that I wouldn't shave at all. I made that mistake during graduate school. I thought I was ready to submit my PhD thesis in about 3 months, and vowed not to shave or cut my hair until it was done. This was partly a way to motivate myself to finish, and partly a way to let my friends stop asking about my progress as they would clearly know when was done. As it turned out, my thesis advisor thought I had more work to do, and I wound up taking a full year before finishing. So by the time I was actually ready to submit my thesis, I had really long hair and a very full beard indeed. I'm not going to do risk that again...

Anyway, you might think I'm a maverick, but it turns out that most of Powerset is in on the gig. Almost all our employees are growing moustaches and/or beards in preparation for our upcoming launch. Even women who can't grow nearly as nice moustaches as the men have painted them on from time to time. And our folks even registered a domain name and created a website, PowerStache.com, featuring photos taken over time as people grow their beards and moustaches.

It's pretty silly and really wasn't initially a coordinated effort, but it's fun and reflects the excitement inside the company as we are nearing the time when the early version of our product will be available to the general public.

I'm happy that natural language and semantic search was included in the special issue. Andy Greenberg from Forbes.com published his piece on language and search engines devoting a good portion of the article to Powerset and Hakia, featuring interviews with me and with Hakia's founder Riza Berkan. The article, entitled "Language Web-lish" starts off with Andy using Powerset's metaphor comparing people's current use of search engines to communicating like cavemen:

A question in English, like "What year was Hillary Clinton born?" becomes what he calls a primitive "keywordese": "Hillary Clinton born year."

"We have this great gift of human intelligence based around language," says Pell, "and now we have to translate it into a grunting pidgin language to interact with machines."

Andy described an example I showed him from Powerset:

When a user enters the question, "In what year was Hillary Clinton born?," Powerset's algorithm doesn't simply scour the Web for this collection of words in close proximity. Instead, it looks at pages with an eye for their meaning. Reading the sentence "Born to Dorothy and Hugh Rodham in 1947, Hillary Clinton is a New York senator," Powerset will disassemble the sentence's grammar and extract the fact of Hillary Clinton's birth date. That fact is then connected with the user's question, even if the word order of the result and the query didn't originally match.

Andy also went through an example from Hakia:

Taking the question "What drug is best for treating a urinary tract infection?" Riza Berkan points to the word "drug." Hakia's algorithm, he says, understands that the word contains a massive subset of concepts including synonyms and specific names of medicines. When it spots a term that falls into that subset, like "Amoxicillin," Hakia can substitute the medicine's name for the word "drug" in the result.

"You don't want the word 'drug,' you want the name of the drug," says Berkan. "That's a hidden failure in search engines, and people don't even know what they're missing."

Other natural language and semantic search companies mentioned included Cognition Search and Lexxe.

As is typical, my friend Peter Norvig at Google gets the last word in the article:

Google's Peter Norvig, the search giant's director of research, knows just how complex semantic algorithms can be: His Berkeley Ph.D. thesis tried to develop one in 1978. Every sentence of text, he says, took weeks to analyze. "The result was kind of like a dancing bear," he says. "It was amazing that it could dance at all, but we didn't expect it to star in the Moscow Ballet."

But that doesn't mean Google's engineers are idly watching semantic search from a distance, says Norvig. The company's thousands of engineers are looking at how to incorporate semantic analysis into a search algorithm. But semantic analysis is just one of many directions that Google's teams are exploring... "Basically, we just do whatever works," says Norvig. "Instead of trying to understand everything, we're trying to understand something about billions of pages a week."

But does that pragmatic approach leave Google vulnerable to an innovative start-up willing to risk its fate on building meaning-based search from scratch?

"It's unlikely," says Norvig. "But even car companies have to worry about anti-gravity machines."

I think that analogy is quite a stretch. It's more like big car companies having to worry about smaller companies focused on electric cars. They don't have to worry about this immediately but, at some point, this is going to be the future of their industry.

Crunchies 2007 After Party, originally uploaded by Zivity.

Yesterday I attended The Crunchies, an award ceremony to honor innovation in the tech community. The event was organized by TechCrunch, GigaOm, ReadWriteWeb, and VentureBeat.

My personal highlights from the award ceremony were:

• Live performance by The Richter Scales, singing "Here Comes Another Bubble".

  In case you haven't seen the video before, it is must viewing. It combines melody from Billy Joel with acapella (my favorite kind of music) with technology startup themes and humor. The video opens with a line from my friend and Powerset investor and board member Peter Thiel stating there is absolutely not a bubble in technology. The song later features lyrics such as "Babies blogging in the womb" and "I sold my twenties for a worthless pile of tech stock". My friends in the group, Tom Shields and James Currier, invited me to come sing with them sometime, which could be a lot of fun.
  
  

• Fake Steve Jobs accepting the Crunchies award on behalf of Apple for the IPod.
  

  His speech is totally hilarious. The whole speech is like one big inside joke. I had previously read his book "Options: The Secret Life of Steve Jobs, a Parody", an autobiography of Steve Jobs as told by Fake Steve Jobs, and this video conveys the parody well.
  
  

• A video of my friend Nova Spivak (Founder of Radar Networks) answering the question about the most important technology innovation. Given the position he has taken in recent panels we have been on together, one might have thought he would talk about the Semantic Web, but instead Nova argued passionately about the virtues of Cool Whip!

  
  

  He illustrated many uses of the technology and had had the crowd rolling with laughter. This also inspired us to attempt to have a cool-whip afterparty, which fizzled out.
  

• Running video commentary by Sarah Meyers. Even without the platinum wig and corset she wore during her Party Crashers career (including crashing Powerset's Series A Funding Party), she's still adorable and very personable.
  
  
• Luke Nosek from The Founders Fund presenting the awards for "Best Business Model" and "Most Likely To Succeed". While many of the candidates were Founders Fund portfolio companies, I appreciated that he was wearing a Powerset t-shirt (with the grunting pigeon) under his jacket.
  
  

The After Party took place in the famous Green Room. My group had to wait a little while to get into the party, which exceeded the capacity of the room. The wait itself was fun because we were joined in line by MC Hammer. The party was enlivened by a photo activity sponsored by Zivity. In his award introduction Luke Nosek had described Zivity as "Myspace for Grownups." People took photos with props, costumes, and attitudes, accompanied by several Zivity models. I included a photo of me (in cowboy hat) with Pearl and Cyan in this post. The rest of the collection is fun.

Overall, while the event had its ups and downs, there was really a nice sense of community and cameraderie in both the presentations and in the audience. The award recipients made really brief and generally humble speeches (with the exception of Fake Steve Jobs, of course), most of them thanking their engineers and their moms. The videos shown during the ceremony were mostly sent in by nominated companies. Altogether it felt more like a summer camp show than the Oscars and it is good to see our community not taking itself too seriously. On that note, it was great to see Om Malik on stage at the event shortly after recovering from a heart attack that had left him hospitalized and the subject of much concern among his friends. When people saw him at the event there much applause and support.

I took some photos of the event myself and plan to post them here soon.

Social networks pose new social etiquette problems. Services like MySpace and Facebook make it very easy to request people to be your friends and there is high social motivation to generate large lists of friends so you feel popular. But having accepted many friends, it is easy to get swamped with viral marketing emails and invites to use new countless new applications. Moreover, having someone on your friend list makes a public statement that you are actually friends with them (more or less).

To address this, services like Facebook make it possible to remove someone as your friend. However, the dynamics of deletion can be quite dramatic, so for social engineering reasons, the systems notify users only when someone has added them as a friend -- no alerts or messages are sent when someone removes you as a friend. Thus a user has no easy way to find out if one of their "friends" has actually removed them. You can keep track of your friend count and try to remember who is no longer on your list, or periodically check your friends profile pages to see if you are still there, but all these approaches are generally frustrating and unmanageable. In thinking about this, I had an idea: to make an application that could tell you automatically when someone has removed you as a friend on a social networking service.

I was discussing this yesterday with Siqi Chen, a friend and colleague who recently left Powerset after his startup company, Fluid Play, received seed funding from YCombinator. Siqi is a strong engineer who is great with Ruby on Rails (he was a key developer on PowerLabs), and his new company specializes in building Facebook applications. Fluid Play already has several highly successful Facebook applications, including "Friends for Sale", which lets you "buy" friends on Facebook and send messages to other friends pretending they are doing your bidding. In the 4 weeks since "Friends for Sale" has been out, it has already generated over 450,000 users!

Siqi thought it would be a fun and straightforward project to implement my "Unfriender" application idea on the Facebook platform, and I was curious to learn about application development on the Facebook platform in general. So we got together on Saturday afternoon for a hack session (since I'm not a Ruby on Rails programmer, this generally means I get to kibitz and "product manage" as Siqi does all the programming work...). As a testament of the power of Facebook Platform and Siqi's coding prowess, we had the application built from scratch, debugged, and deployed in production within 8 hours wall-clock time.

The conference came at a busy time for me so, I had to keep my trip to Korea very short (less than 4 days on the ground!). This was sad for me because I have heard about Korea for such a long time, having grown up with Korean family (my step-father was Korean). I did manage to make a visit to two interesting Korean companies.

First was NHN, the parent company of Naver, Korea's dominant web portal / search engine. Naver pioneered human powered search, as now seen in Yahoo Answers, among others. It turns out that the technical folks at NHN are already collaborating with the tech folks at Powerset on Hbase, the Powerset-initiated open source project for big distributed tables, ala Google's BigTable. They are also working with a local university to apply Hbase to distributed storage of large RDF databases, a perfect tie in to the activity at the Semantic Web conference. NHN is a pretty amazing company, especially for one that most people in the USA have never heard of.

Second, I visited KT, formerly Korea Telecom. I met with there with SanKu Jo, Vice Present of New Business Group, and Dongmyun Lee, Vice President of BcN business Unit, who are also quite visionary in semantic technology.

While there was not much time for fun, my hosts at NHN (Paul Sung and Ed Yoon) picked me up at the airport and took me out to a meal at a traditional Korean restaurant. We sat on the floor which was heated from below. I was told this is traditional for Korea and it makes for a comfortable bed, without needing a mattress. It also helps relieve the fatigue of sitting cross legged on the floor while eating from the low table. The next day, during an hour between my company visits and catching the next flight, we went to a high-up mountain fortress in the clouds, from which we could see all of Seoul.

My only other time out in Korea was at a dinner in Busan with the conference organizers and other invited speakers. We went out to a Korean seafood restaurant. The highlight of this for me was eating some other-worldly sea creature that responded to prodding as though it didn't realize it was no longer alive.

Although I only got a taste of Korea this time, I look forward to visiting again soon and hopefully to have more time to see this amazing country.]]> http://www.barneypell.com/archives/2007/11/my_first_trip_t.html http://www.barneypell.com/archives/2007/11/my_first_trip_t.html Fun Mon, 19 Nov 2007 21:13:33 -0800 I gave an invited keynote talk last week at The 6th International Semantic Web Conference and the 2nd Asian Semantic Web Conference, 2007. The abstract for the talk is below. The image below links to the original video and presentation slides.

The live presentation (and video) contains technical demos that aren't in the slides. Some of the demos are already available inside Powerlabs (e.g. Powermouse, which lets you browse and query our semantic database of facts extracted from Wikipedia), while some of these are still internal (e.g. an open search box, and output of our natural language system on full sentences). I also gave some detailed walk-through showing how Powerset takes advantage of external semantic resources like Wordnet and Freebase.

For me, the most fun part of the talk was toward the end, where I got to speculate on how ecosystem effects can make natural language search and the semantic web become deeper and more powerful more quickly than people might expect. For example, advertisers, publishers, and vertical search sites will be able to contribute ontologies that enable them to get more users, better internal search, and more revenue, while having as a side effect that the broad search engines get more knowledgeable about different domains. The questions afterward were also challenging and interesting.

POWERSET - Natural Language and the Semantic Web