Barney Pell's Weblog

Robin Hanson (gmu) and David Pennock (yahoo): background on prediction markets

- problems
manipulation
sabotage
...
combinatorial

Robin's slides should be available online soon.

Bernardo Huberman (HP): predicting the future

o aggregating info
collective intelligences good at coop problem solving
how to agg ind skills to generate useful info?

o how do orgs predict?
ask experts and consultants
have meetings
designate someone as forecaster
take a vote (not very good)

o alternative: markets
markets aggregate and reveal info (hayek, lucas, etc)
to predict outcomes, use markets where the asset is info (rather than a physical good)
e.g. iowa electronic markets

o experiment: use markets in hp to predict printer sales etc
results weren't very good.
- markets within orgs: problematic because:
low participation
illiquidty
info traps
hard to motivate
easily manipulated

o a new mechanism:
identifies participants that have good predictive talents, and extracts their risk attitudes
induces them to be truthful
avoiding pitfalls of small groups
aggregates info in a nonlinear fashion

(two published papers on this topic)

what is it based on?
- people are not all the same
think of info in people's heads as the assets and use portfolio theory
use a market mechanism to determine an individual's risk attitudes and
performance
- then, ask people to forecast and performan a nonlinear aggregation of
their results taking into account their risk characteristics
info gathering process is simple, decentralized in time, and inexpensive to
implement

stages:
1. market for contingent securities
provides behavioral info, such as risk attitudes - synchronous
2. participants generate predictions no outcomes, which are then aggregated.
asynchronous

stage 1: lottery-like securities
marbles from an urn.
can buy and sell colors. things quickly converge to a price.
can extract risk attitudes: ratio of cash vs marbles people accumulated over
the game. risk averse people keep the cash, risk seeking over-invest in
certain colors.

stage 2: forecasting
participants given 100 tickets to be allocated among 10 securities
this determines probabilities
true state pays according to the number of tickets allocated to it

aggregating predictions: change exponent based on ind risk attitudes

result: just 9 players aggregated give a complete distribution
the info aggregation mechansism: group did better than the best individual.
IA also did better than the market.

possible contamination with external market info.

can you find an interesting and hard problem, where people are willing to
get an answer and devote resources to test it?

o predicting revenues and profits inside an organization
HP services: predict revenue month to month on first week of the month
did pilot for 5 months.
15 managers worldwide, part of hp services finance office.
have a tool that HP sends them where they can see the predicted revenue.
every first week they made a prediction, we got a number and compared with
official prediction.

actual value was straight on the mode of the prediction.
people placing forecast anonymously.
group prediction wasn't always below official projection, sometimes was above.

managers (esp good ones) said they'd be happy to have their names published.

now doing an internal one, to predict on volate commodities HP uses to make
computers.

o other examples:
- shell oil executive suggested running with top customers to predict demand
from refineries. a big problem is to decide production.
- prediction of success of products
must be very precise about measurable outcome of these projects.

1. we don't know how far into future a mechanism like this can predict.
10 years from now might not be interesting.

2. we don't know how it scales as the # of participants increase.

www.hpl.hp.com/research/idl

q: what markets failed and why?
lack of motivation
for our scenario, we don't pay that much. once you put a $ on the table, the
behavior is very different.

q: face to face interaction among participants?
only in the initial game

q: training procedure?
yes we have a protocol, and a test, until they pass.

q: rohit: agoric computation (markets to allocate computing resources). ecology of
computation book: what are predictions for prediction markets?
a: we are on threshold of utility computing (eg. Sun selling cpu for
$1/hour). transition will have to take place. like shit from internal
generators to electric utilities.
the difference is we're dealing with information that is valuable to
companies, so people get nervous.
but eventually believe it will have to happen.

q: learning over time, ability of highly informed traders to compensate for
uninformed traders. how robust to uninformed participants?

a: we are using methods to id people who have expertise (looking at emails exchanges,
etc).
the learning piece we measure: how is risk attitude changing over time.
haven't looked at other dynamical variables.

q: jp: used a 100 person forecasting center 15 years ago for allocation of
resources in systems integration. You shouldn't be able to see what someone
else is doing, as that influences you. This avoiding corruption and bias is
a different anonymity issue.
a: in our system you can't see what others are doing.

q: is HP actively using these predictions to make business decisions?
a: we are using it internally and with some customers. but there are issues.
in process or trying to spin out something.
question on how much you'd make if you did this.
might use this just as a product differentiator.

Emile-Servan Schrieber, CEO and president of Newsfutures: The business value of prediction markets

o quotes
1. thomas malone: There is an increase in human freedom in business that may, in the long
run, be as important a change for business as change to democracy was for
governments.

2. key value we see is prediction markets have the potential to extract the
best essence from group knowledge as an alterantive to majority decision.
(ie beyond democracy)

business application: forecasting and decision support through critical
knowledge aggregation
(unclear which apps will have the most value)

o markets as brains
intelligence emerges from interaction of many relatively dumb parts.
learning by reinforcement and prunjing improves performance

real money market: you don't control money flow. not as much correlation
between who is knowledgeable and who is rich.
play money market: everyone starts the same, accumulate money by being successful

o About Newsfutures company
global 1000 clients
over 40,000 markets operated since 2000
online properties:
world news exchange
innovation futures (tech review)
tech buzz game (yahoo)
patent applications on "multi-outcome" trading engine
market for each possible bin, each is alternative to every other

o Newsfutures Business model
tech licensing
public markets
advertising revenue
creates branded prediction trader panels (market research)
corporate solutions: use these branded trader panels
(not necessary to install systems inside the company)

prediction trader crowds
internal / external
public / restricted

internal restricted: frontline employees
external restricted: consumer panel, invited experts, supply chain
external public: innovation futures, tech buzzz, HSX, pharma traders

o HP experiment

o eLilly: beating official forecast
a dozen sales people, forecst monthly drug sales
usually get 10-15% participation
market accuracy advtange over official forecast: sometimes >75%

o beyond HP
18 companies (maybe 25 by end of this year) have implemented a prediction market (ramping fast)
(wisdom of crowds book got people very interested)

horizontal, across varied industries

o coprorate applications:
sales forecasting
demand forecasting
need to know 1.5 years in advance whether to create a bigger production
capacity. bestbuy is gauging early adopters. but expect retailers will
have to drop the price, so then what happens to the demand.
price forecasting
company mfg paper needs to forecast, as not traded in the market.
similar problem for chemicals like benzene, no organized market
companies are publishing the prices of deals in real-time. now we're using
their customers to create futures markets within buyer community.
project prioritization
product design, feature selection
campaign evaluation
candy company wants to figure out potential impact on sales of different
types of ads on tv. use marketing people to evaluate the campaigns.
regulatory monitoring
eli lilly wondered in 2003 what would happen in congress re: drug reimportation and benefit in medicare.

(Siemens recently sold its cellphone units to taiwanese company because their sales prediction problem was so bad...).

o some crowds:
sales people
marketing people
consumers (eg benzene) / readers (eg tech review)
supply-chain partners
external experts
general population (self-selection is very useful)
doesn't matter if trading population is small %

o beyond accuracy:
not just: accuracy, liquidity, money, incentives, sw, trading

mostly about:
business value
focusing attn on issues
accountability
haveing a voice / recognition
user-friendly design
dialogue

o challenges to internal markets
threat to traditional management (see Time magazine article listed above)

some questions are better left unanswered (unasked!)
e.g. when will this plant get built?
engaging busy professionals
getting the answer means publicizing the answer
must have, or nice to have?

conclusion: not one size fits all...

o state of the art
rounding the bend: many companies engaged, awaiting results
some early wins
some early losses (people just using sw didn't get very far)
a lot of learning
look for case studies to come out in 6 to 12 months

o example: sales forecasting (siemens)
agg opinions of 50 hq sales personnel in brazil to predict phone sales in
brazil. (25 trading)
very nice results so far

o public prediction trader panels
thinks that's where most of the business value will be

"will a new drug benefit be introduced under Meidcare?"
head lobbyist within eli lilly was telling bosses it would never happen but market was saying at least 40% chance it would happen, even at the worst of times

"how many new molecular entities will be approved by the FDA in 2003?"
people figured it out very accurately

"how much will access to physicians increase or decrease 2002 vs 2003)? spot on: 7.4%

innovation futures (launched fall 04, 10K traders)
sponsored market (by cisco)
"how many voip subscribers will the US have at the end of 2005?"

Google IPO market: both prediction sites were closer to actual result than google official estimates. real and play money did the same.

yahoo tech buzz

o take home:
if you own a comunity whose "brain" you'd like to leverage via a prediction market, Newsfutures can partner with you with operational experience, technology, and business models

q: how do you charge?
$25K up to 120K/year

q: assessing political risk in emerging markets?
(short Turkish Lira based on outcome of next parliament election)
no.

q: how to overcome cultural resistance for betting on the future?
a: we called a VP forecasting of a large car company. Told her people are investing in future outcomes, works well for elections.
she said: "It's like gambling". we said: "Yes, HP is using it to improve sales forceasts". She got it immediately. Businesses don't mind gambling as long as it produces results.

Walter Yuan, social science experimental lab at Caltech: jMarkets

o functional featues
multiple markets
continuous
anonymous
open-book market
order-driven markets with price priority and time precedence
role-assigned traders
constarint-regularted trading (short-sale and credit limit)

o non functional features:
fast (<30 traders, 20-4 markets)
trader friendly gui
easy to configure a marketplace sessino
easy to deploy
open source

trader friendly GUI demonstration

fast transaction and cooling

o advantages of experimental market software
flexible config, simple, abstract, strong domain expertise, open source
o disadvantages
security, scalability beyond 100+ participants, lack of business domain specific customization, clearing and settlement

o applications of jmarkets
1. president's day weekend ski equipment sales
rebalance inventory, stock up new equipment, adjust cash flow
- markets vs ebay:
two-sided
multiple units held, offered or put up for sale
market making opportunities
very hard to make a market in ebay, as you'd have to buy the products
and then try to sell them
efficiency widely established
price discovery

2. inter-outlet markets in sales personnel during busy holiday shopping
periods
sales personnel act as sellers of their service time; stores act as buyers
for competing limited sales service time
different approach: stores act as buyers and sellers of their salesperson time
similar problem: airline crew allocation

3. trading just-issued treasury bills among a group of insurance companies
ad-hoc, centralized market of forward contracts with designated participants
market is short-lived
opportunities for market makers
price discovery provides more info on later auction
dealers can be both buyers and sellers - market making; primary dealers
natural sellers; secondary dealers natural buyers, but not nec.
compare to present non-transparent, phone-based system

q: what institutional rules in these markets? keeping track of earnings?

right now only credit limit and short sell
have a module can view as a clustering engine.

q: disadvantages (security, settlement clearing, customization) seem
relatively trivial to solve in open source community. but scaling >100+
participants - some fundmental constraint that doesn't make it scale?
a: just haven't tested

q: really about markets, not predictions. (yes). did you try predicting
salespeople demand? (no)