A recent study “Cue Effectiveness in Communicatively Efficient Discourse Production” (PDF) published in Cognitive Science journal demonstrates how human communication is normally driven by efficiency – that is, we tend to distribute entropy equally over a discourse in order to be understood. When a topic shift occurs, the entropy is increased, so in order to compensate we tend to use the words, which make it much easier to predict the direction the discourse will follow.

Interestingly enough, when we analyzed various text structures using Textexture, creating network representation of texts, we found that structurally this was indeed the case for newspaper articles, dialogues, and recorded conversations. Such texts usually have several distinct interconnected clusters (topics) distributed across the graph, structurally very close to small-world  network. However, texts laden with ideological intent as well as poetry exhibited a very different network structure. Communist manifesto or quaran, for instance, did not have so many topical shifts and those that occurred would still refer the reader back to the central topic of the text. Poetry text network visualizations, on the other extreme, did have very decentralized graph structures, communicative efficiency was not the primary concern there and overall entropy is much higher. Read our paper on Text Polysingularity to learn more about the methodology we used.

9/11 article from The New York Times text network visualization

Lord Byron darkness poem text network visualization

Try InfraNodus Text Network Visualization Tool developed by Nodus Labs. You can use it to make sense of disjointed bits and pieces of information, get visual summaries for text documents, and generate insight for your research process: www.infranodus.com

Communist Manifesto text network visualization