Gamified Prediction Market: Assessing and Improving Research Accuracy at Blinc
17 November 2022

The MRS Census and GeoDems group champions new thinking and new talent; one area they have been particularly impressed with is the CDRC Masters Dissertation Scheme (MDS)

This programme offers an exciting opportunity to link students on Masters courses with leading retail companies on projects which are important to the retail industry. The scheme provides the opportunity to work directly with an industrial partner and to link students’ research to important retail and ‘open data’ sources. The project titles are devised by retailers and are open to students from a wide range of disciplines.

MRS CGG are proud to have been granted permission to publish abstracts from the dissertations and we are sure the students have a great future ahead of them.

This abstract is by Vanessa Sukanta

Title: Gamified Prediction Market: Assessing and Improving Research Accuracy at Blinc

Academic Institution: University College London

Industry Sponsor: Blinc Partnership

Background and Motivation
One of the research techniques used by Blinc to help the client, is the Gamified Prediction Market which is one of their product or research techniques that will be assessed in this paper.

Prediction Market is utilising the ‘wisdom of crowds’ by aggregating all the thoughts from individuals to predict future outcome. It was started in 1988 when the University of Iowa created the Iowa Electronic Market and compared the prediction with the actual result (Berg, Nelson and Rietz, 2008). This method has been widely used for predicting government election outcome, forecasting of infectious disease activity, and box office success of movies. The prediction market method is more engaging to participant than the traditional method, hence obtain more high-quality data. Prediction Market approach also shows better accuracy than traditional research techniques as it has less bias, since participants want to win and set aside their personal preference in choosing the selection (Williams, 2011). In addition, prediction market is proven to be effective even when using relatively small pool of participants (Kambil, 2003). This makes prediction market to be preferable and more cost effective than the traditional survey method.

However, albeit having this success with their clients Blinc have been struggling a bit to find ways to keep improving their research accuracy using this model. The problem they have is surrounding the participants that join the prediction markets. They are wondering if there are any characteristics or features from the individuals that have more accuracy in predicting correctly in the prediction market. With this information, they can choose the right crowds so that they can better serve their clients.

Data and Methods
In order to do that, this project is going to dive into individual’s participant data that join the predictive market betting. The actual result from the client is also needed to distinguish which participant predict accurately. From the betting data that is taken from Blinc’s internal database and the final actual result happening from the client, the individuals can be grouped into who are accurately and inaccurately predicted the prediction market. Afterwards, tying it back by identifying the characteristics and features from the survey data that is also collected from Blinc’s internal database.

Figure 1: Methodology Overview

The methodology is divided into 3 parts which are data preparation, grouping, and backward analysis. Data preparation includes data cleaning and aggregating betting values to a participant level data.

Next on the grouping part, participants will be divided into four groups to show the level of accuracy. While on participants that bet one time, we can directly label those that choose large audience as accurate and those that do not choose large audience as inaccurate. On participants that do multiple betting such as bet $500 on very large audience and $200 on large audience, will be needed to be ranked based on the highest betting amount. The participants that choose large audience as their highest amount will be labelled as group 1, participants choose large audience as the 2nd highest amount will be labelled as group 2, similarly with 3rd and 4th highest amount on group 3, and those who do not bet large audience at all on group 4.

Lastly, the backward analysis approach is then chosen to study the correlation between any character that every participant grouping has. The backward analysis part will be divided into two parts: betting behaviour and characteristic from survey. On betting behaviour analysis, some metrics that will be explored are whether the participants that predict accurately bet one time or multiple times, also if they bet in a large amount or small amount of money. While on the characteristic survey analysis, metrics will be explored based on the survey result such as if the accurate or inaccurate participants are coming from a specific gender or age group, if they have prior knowledge in the similar genre of drama being tested, if they have watched the specific TV channel where the drama is being shown, and the likelihood of each participants
watching the drama after seeing the concept and trailer.

Key Findings
There are two big findings that can be drawn from this study:
1. Prior knowledge and experience do not affect the accuracy of the prediction market
2. Rather than focusing about who the participants are, it is more important to focus on their interest on the concept being tested on the prediction market to increase the accuracy.

The first insight is supported by the analysis that the participant’s demographic such as age, gender, region, marital status, and income group, have no effects on the accuracy of each participant in the prediction market. The proportion of each demographic in each of group 1 to 4 do not differ, which shows that the spread of decision is consistent on those groups. Other than demographic, the participant’s relevant knowledge such as the TV channel, genre, and TV shows, do not have any impact in the accuracy of predicting the market. The distribution of the channel, genre, and TV shows preferences is consistent among group 1, 2, 3, and 4 such as having watched BBC One and enjoyment on similar genre which is fantasy drama. Supported by the study by Smith (2011) that expert will not be more accurate than the crowds in predicting the prediction market.

The second finding is supported by the point: participants that are not interested in both the concept and the trailer, they will tend to have a bad prediction on the prediction market and fall into group 4. It is supported by the insights where 90% participants that rate the concept as not very good or poor fall into group 4, which shows they have bad accuracy and judgement in predicting the viewership of the tv show. Similarly proven on analysis of likelihood to view after reading concept, rating trailer after watching the trailer, and likelihood to view after watching the trailer. It is very important to have pre-survey question for Blinc to filter out the participants that do not have any interest with the concept. As mentioned by Goldberg (2009) that the prediction market encourages the participants to gather information, reveal their true beliefs, and reveal the levels of confidence with which they hold those beliefs, to make decision on the most-likely outcome in the market. The participants’ interest and curiosity towards the concept tested becomes the strength of their prediction.

Finally, rather than experts, the interest in the concept or trailer will be more impactful on the accuracy on betting in the prediction market. The recommendation to Blinc is to have a pre-survey question in which when they have a low interest with the concept to be tested, those participants should not be chosen on predicting the prediction markets. It is very crucial on the participants selection step to improve the accuracy of the future prediction markets.

Value of the research
In overall, the research has provided an insightful recommendation to Blinc about the characteristic of accurate and inaccurate participants. These insights provide actionable strategy to improve the accuracy of Blinc’s prediction market by preselecting participants from the prediction market.
Since the data are limited to broadcasting media client in this research, the recommendation is limited for Blinc to their future broadcasting and media clients. Future work can be done in using the same method in another industry prediction market, to see if the current behaviour and characteristic of each accurate and inaccurate group persists. We hope that in the future, this project will not only be helping on increasing accuracy in the broadcasting and media clients, but also other industries.

References
Berg, J. E., Nelson, F. D. and Rietz, T. A. (2008) ‘Prediction market accuracy in the long run’, International Journal of Forecasting, 24(2), pp. 285–300. doi: 10.1016/J.IJFORECAST.2008.03.007.

Goldberg, S. H. (2009) ‘Prediction Markets’.

Kambil, A. (2003) ‘Betting on a new market’. Available at: http://hbswk.hbs.edu/archive/3808.html.

Smith, M. A. (2011) ‘Do experts know more than the crowd?: A case study’, Prediction Markets: Theory and Applications, pp. 235–255.

Williams, L. V. (2011) Prediction markets: Theory and Applications, Prediction Markets: Theory and Applications. doi: 10.4324/9780203815526