Recommender systems are the algorithms that determine what content we read, which products we buy, which movies we watch. “Customers who bought this also bought this” feature generates 35% of sales for Amazon, 2/3 of the movies watched on Netflix are recommended by internal algorithms, recommendations generate 38% more clickthrough on Google News [1], and Facebook Newsfeed shows the content that got the most likes from the user’s friends. Recommender systems are a booming market at the age of informational overload – it is estimated that in 2015 an average American will consume 15.5 of media content a day and about 74 gigabytes of information. Therefore, the demand for tools that can help cope with this overload is on the rise.

 

The Similarity Bubble of Collaborative Filtering and Content-Based Recommendations

There is, however, one big problem. Most recommender systems algorithms are based on similarity. The two main approaches – collaborative filtering and content-based recommendations – both assume that the users want to see the content that’s similar to what they already rate highly (or, in case of collaborative filtering, the content their friends rate highly).

For example, if somebody likes The Beatles they might be recommended Rolling Stones because many people would have high preferences for the both bands.

Often this is true, but what it produces is a filter bubble: the users are locked into clusters of similarity and their chances of discovering things that are truly novel are dramatically reduced.

For instance, in case of The Beatles, Yoko Ono made an album with Sonic Youth and while it may not appeal to the wider Beatles audience, it could be very interesting for some of the fans and even open up a whole new musical universe to them.

There are, of course, hybrid approaches in recommendation systems design that attempt to tackle this problem, but it rarely goes further than simply acknowledging that “serendipity” is an important factor in content discovery. Most likely there is a high pressure on audience retention and click-through rates (mainly short-term parameters) and offering a possibility to learn and discover something interesting and new is not really the top priority for online merchants.
However, it may be an important competitive advantage for specialized education resources, book shops, and MOOCs.

 

Serendipity, Unexpectedness, Diversity and Novelty in Recommender Systems

Serendipity is a hot topic in recommender systems today. The basic idea is that when the items recommended are too predictable for the user, she won’t be interested. So the intention is to find something the user would not expect, but still could find interesting. The most common approach to produce serendipity on an algorithmic level is to find a constellation of interconnected items (lists) that are outside of the user’s current scope of interest. The idea is that if the user is shown something that they don’t have any connection to, they will find it interesting [2]. The problem with this approach, however, is that when serendipity is understood in such a way it’s also quite likely that it will produce recommendations that the user will not at all find serendipitous [3]. The word “serendipity” in itself carries connotation of some sort of benefit, so “serendipitous” discovery should not only be unexpected and novel, but also somehow relatable to the person’s current range of interests, even if in a remote way. So the example of recommendation above – when something completely unrelated is offered to the user – has more to do with unexpectedness than with serendipity.

Some advances in this direction are made in the research that attempts to find a hybrid approach between serendipity and collaborative/content-based recommendations [4] using graph-based approach. The basic premise there is to recommend the items that may be connected to the user’s sphere of interests but have the lowest clustering (so they are not fully embedded in it).

For example, a standard approach would be to recommend somebody interested in recommender systems to read about machine-learning as those two topics are very often mentioned in the same context.
However, if we are to take serendipity into account, while keeping in mind (or in the algorithm’s program code) that the recommendation should still be connected to the current interests of the user, we might recommend Latent Dirichlet Allocation – a technique based on word co-occurrence normally used for topic modelling in natural language processing. It’s a less likely match, but the one that can still be very useful for studying recommender systems as is very well shown in [3] and [4].

The problem with introducing serendipity into recommendation algorithms is that the quality of predictions deteriorate, so it’s important to find the right balance between similarity and novelty, between the immediate surrounding and the periphery.

 

Divination: Finding Serendipity in Similarity

However surprising it may sound, the question of finding the right balance between similarity and serendipity has been studied since many centuries.

Various divinatory systems – from astrology to Tarot to I Ching – are based on providing a way of connecting the person’s present condition (or the condition determined by birth) to some way of predicting the future (or the unknown) tendencies. [5] If we analyze narrative structure of divinatory systems we’ll find a very specific graph topology, which has a distinct community structure and is still very well connected on the global level. That means that the theme present within those narratives are specific enough to be separated from the rest and yet they all connect easily on the global level (so that it’s easy to shift from one topic to another without losing the narrative). This is a great example of a narrative structure that incorporates both serendipity and similarity. They come with a set of efficient reading devices, which enable the reader to start from a certain theme, get a good overview of it, and then move on to the the other themes present within the narrative. This way the subjective experience is that of touching on what’s already known, while discovering something new.

Recommendation systems that aim to have a predictive power that go beyond simple product recommendations need to embrace methodologies used in divinatory narratives. The structural properties of such narratives from graph modelling perspective are close to that of small world networks. Therefore, the recommendations graphs should also be designed in a similar way: a multiplicity of distinct communities well integrated on the global level.

Recommended Reading

[1] Amatriain, X (2014), Recommender Systems (Machine Learning Summer School 2014 @ CMU) http://www.slideshare.net/xamat/recommender-systems-machine-learning-summer-school-2014-cmu

[2] Iaquinta, L. et al (2008) Introducing Serendipity in a Content-Based Recommender System. Hybrid Intelligent Systems

[3] Adamopoulos, P., and Tuzhilin, A. 2013. On Unexpectedness in Recommender Systems: Or How to Better Expect the Unexpected. ACM Trans. Intell. Syst. Technol. 1, 1, Article 1 (December 2013), 51 pages.

[4] YC Zhang et al (2012) Auralist: Introducing Serendipity into Music Recommendation. Proceedings of the 5th ACM Conference on Web Search and Data Mining (WSDM-12)

[5] Paranyushkin, D, Johnco, C (2014). The Divinational Network of Tarot.

 
The cover image is by Jeremy Hutchison, Objectless Expansion, 2014