Visualize, Analyze and Realize

I came today to talk about data visualization
But I got no slides for a presentation
So I’ll stand here and soliloquize
About the social media data I visualize

You got big data and I sympathize
Your data’s so big as to hypnotize
But there’s value in that data you want to realize
and you got no army of statisticians to analyze

You want to know what all that data implies?
You got to dig in and visualize!

Data so complex and rich makes you agonize?
Using links and nodes allows you to scrutinize
You can explore, interpret and categorize
As the relationships from your data clarifies.

Columns of digits and strings got you all slantwise?
Using colors and shapes I hope to symbolize
The patterns and structures you can emphasize
So that with a glance your plan can crystalize.

But if you want to know what all that data implies?
You got to dig in and visualize!

It’s not about pretty pictures to aggrandize
It’s about reducing the cost to recognize
Patterns in the data so you can theorize
And make decisions that are wise!

You want to grow your enterprise?
You want quick answers to your whys?
Avoid the complications from a surprise?
Got customers you hope to tantalize?
Maybe sing to your competitors their lullabies?

Then you NEED to know what all that data implies,
So you got to dig in and VISUALIZE!

The string “oo” can mean a lot of things.  When we first tossed a tweet we captured written in a language we didn’t understand into Google Translate, it suggested that “oo” was Portugese for “yes.”  Well, Google, right translation, wrong language: “oo” is actually yes in Tagalog.  As you might expect, there’s an awful lot of Tagalog in our dataset currently.  To begin my inquiry then, I asked our machines to return a dataset consisting wholly of tweets that contained “oo” in the text of the tweet, but did not use the hashtag “#oo.”

However, “oo” is actually a component of many different types of emoticons – those sideways smiley-faces that textually represent emotion.  “oO” denotes an expression of surprise for many of the few English tweets we acquired. “oO(bright idea)” signals a cartoon thought-bubble on the part of the reader.  And my personal favorite, “(_ _)oO” seems to be popular amongst the Japanese tweets we collected – it represents a person sleeping!

Before running a topic-model, we “tokenize” the tweet text strings.  By this, I mean that we remove any non-Roman alphabet characters and break the resulting strings into words according to the white space between them.  The results are called “tokens.”  However, this has two curious effects if used with default settings.  First, we remove all of the emoticons that signify an affective spin on the text.  Second, we remove all tweets that contain non-Roman alphabet characters – and I just lost most of the Asia Pacific in one blow.  Also, words with apostrophes are broken and take on different meanings.  In my first few attempts, I was left wondering who this “don” fellow was, and why he came up so frequently in my topic models.  Turns out “don” is a token “don’t.”

And while most of our “oo” topics are comprised of Tagalog that I can’t personally evaluate for topical cohesion, I did uncover one or two gems that I could read in English.  Justin Bieber and Drake remain popular artists the world over.  From both weeks, the only clearly English topics consisted mainly of casual netspeak: “oo, lmao, girl, ur, yea, baby, sh*t.”  But I admit to being personally affected by the topic that contained “ducktales oo.”  For those unfamiliar with the late 80’s cartoon named Ducktales, the chorus of the theme song is simply “Ducktales!  Woo-oo!”

And now that song’s been stuck in my head for the last week.

Want to explore the oo-but-not-#oo topic model results yourself?  I have two weeks available here: http://staff.washington.edu/jeckert1/d3/oo_1120-1126.html

You can change between weeks with the links at the bottom, hover over a bar to display the words that comprise a given topic, and view the topic’s weighting relative to one another.  Since this is d3, there’s probably a way I could make this prettier later….

edit:  Oh, and just in case I’ve got you jonsing for Ducktales now:

In the case of social media researchers, the situation today is that much of our research is like the flight data recorder: we collect, store, and report data and analyses, but we follow the dictum on the outside and “do not open” the box.

We’re discovering this is a mistake.

By keeping the black box closed, we can create a misleading impression when we report our research results.  We inhibit others from replicating our findings or testing the limits of our results if we do not fully disclose the details of our processes.  We may also miss the chance to ask research questions if we ignore the opportunities to explore the data by testing the sensitivity of our findings to changes in our research procedures.   There are some things we can do from the outside—approaches borrowed from systems theory and systems analysis approaches—but all of us will improve our research as we make our methods more visible…as we open up the black box.

Let’s look at some examples.  In conducting research with social media data, it’s helpful to think about the sequential ELT steps in data warehousing systems.  In following these steps, we:  Extract (data from streams or sources), Transform (the data by parsing it and including metadata that enable us to address our research questions), and Load (the transformed data into an accessible dataset).  And these are just the first steps—before we begin our analysis.  At each step, small variations in the procedures or rules we use can result in significant shifts to our later findings, to the questions we are capable of answering, and even to questions we can imagine asking.  For example, suppose we want to do an analysis of Twitter messages.  In extracting Twitter data, do we use the Twitter API?  If so, do we collect the data in real time (streaming API) or do we employ queries (search API), getting some retrospective tweets?  If we opt not to use the API, we could use one of several developer-based or commercial services (e.g., Gnip) to get our data, but can we afford it?  Each may have advantages, but the samples that result from each may be different.   If the samples differ, can we be confident in our research results in each case?

As it turns out, for some research questions, the answer is a qualified, “yes.”  Shawn Walker (SoMe Lab) and Kevin Driscoll (USC) have completed a study (the paper currently is under review) that suggests that different samples can provide consistent results.  For other questions, or for other data collection efforts, the limitations (filters) on what we can see can lead to different results.

When we study systems models, we typically think of a black box as something that transforms inputs to outputs, and we really only want to know the relationship of output to inputs.  How the transformation occurs is not always important, but there are some standard approaches to getting an idea of what is going on, and sometimes this is sufficient.  For example, we can perturb the input (“shock” the system with a sharp change in value, e.g., a step function) and observe the changes in output.  From this, we sometimes can infer characteristics of the transformations that are taking place inside the black box.  Another approach is to vary the input parameters (e.g., consider a range of values such as time, location, or length of a simulation run) and see how sensitive the output is to changes in these parameters or ranges.

Joe Eckert—the geographer in our SoMe Lab—has been using this approach.  Among other studies, he is examining the sensitivity of his findings to changes in geographic location.  He also argues that meaning can shift within the context of place.  From this perspective, we begin to think of a place as both digital and physical, thus requiring that we understand the code and algorithms by which it is constructed.  [See Joe’s discussion in the previous blog post of how these interactions become part of the lived experience of social media (Walking the Data with Certeau and Topic Modeling).]

Jeff Hemsley has been looking at the sensitivity of the models for the visualizations we use to explore the data.  (See his post  Network Visualization and Meaning Shifting Due to Algorithm Settings.)  He notes not only that the findings can be greatly affected by the model used for the visualization (we get different pictures depending on how we structure this model…not too surprising) but he also feels that by doing this sensitivity testing he gains a better feel for the data themselves.  He becomes more able to “make sense of” the data and thus ask more penetrating questions—he becomes a better researcher as he engages more deeply with the data themselves.

The SoMe Lab and our collaborators are working to ‘open up’ the black box of social media research so we have a better idea of what’s going on in the transformation of data from search APIs to research results that we can report.  And we are continuing to seek and find ways of connecting our methods and our data in ways that increase transparency and generate better ideas and postulates for research.

“The ordinary practitioners of the city live “down below,” below the thresholds at which visibility begins. They walk-an elementary form of this experience of the city; they are walkers, Wandersmanner, whose bodies follow the thicks and thins of an urban “text” they write without being able to read it. These practitioners make use of spaces that cannot be seen; their knowledge of them is as blind as that of lovers in each other’s arms. The paths that correspond in this intertwining, unrecog­nized poems in which each body is an element signed by many others, elude legibility. It is as though the practices organizing a bustling city were characterized by their blindness.” (Certeau 1984:93)

Certeau speaks here of the practice of living; a generative process that forms the interactions and concurrent generation of urban space.  One aspect of the space in which we live today is the ever-present hum of social media.  Our bodies are tied to these practices through the smartphones in our pockets and the computers at our tables.  Through practices of engagement, spaces of discourse emerge.

One way of attempting to discover the topography of such spaces is through textual analysis.  But how does one perform a close reading of 260 million texts, none longer than 140 characters?  Topic modeling offers one such avenue.  An oversimplified explanation of topic modeling is the use of Bayesian statistical packages to determine words that appear proximal to one another across a host of texts (here, each “text” is the aggregation of a day’s worth of tweets for a single user).  In this way, we computationally process the texts to determine which “topics” were most discussed on a given day, for a given hashtag’s users.  Since this is a computational process, the system returns an array of single words — not semantic topics as we’re used to thinking about them, like “food” or “sleep” — grouped into packages of 20.  Then the percentage of contributions that each user makes within their tweets is calculated.  Roughly speaking (I’m still new at this), this gives you sets of words that appear in proximity to one another across all of the users for a given hashtag.  Topic output looks like this:

11: pepper woman spray sprayed pregnant priest blind police year protesters including occupyarrests depolyed spd miscarried people maced blasted rich (#occupyseattle, 10/19/11)

This one’s fairly easy to interpret: folks speaking about practices of police with regard to protesters.  The “pregnant” and “priest” words refer to two protesters that were pepper sprayed following a march and demonstration in Seattle.  ”Occupyarrests” is a frequently used hashtag to pass information regarding the arrests of protesters.  ”Rich” is perhaps the only word that doesn’t make sense topically, but certainly makes sense in terms of proximity — the discursive division between rich and poor was a tool often employed by activists.

However, “walking the city” and experiencing Twitter as a user was the only way I would have identified this mish-mash of topic soup:

10: utc gmt s##### s#### obama occupyoakland occupyportland oct corpulace consrvblohrd fri p######### nov s############ oo f############# mon government f###### (#occupyseattle, 10/19/11)

Apologies for the #’s — it will make sense in a second.  ”UTC” and “GMT” are artifacts of a poorly engineered stopword list (like I said, still learning).  However, I immediately recognized those words that I’ve blocked out with #-marks: they were the names of particularly vocal accounts in the Twitter streams that I monitored while Occupy was still relatively active.  They looked like normal words, broken away from their context.  But I recognized P- in particular.  When I had less to do, like every other user name that I recognized, I also engaged P- in what might be described as a “flame war” surrounding his interjections of anti-protester vitriol into into the #occupyseattle stream.

For the users identified in this topic, this was the “topic” that made up the bulk of their contributions.  This series of interactions, large enough to be clustered together throughout the whole of the #occupyseattle stream, also made up a significant way in which users experienced and lived through the data that I now attempt to read.

Researchers should spend time in the communities which they research. For Certeau, that was the space of the city — for myself, this is the space of digital interaction.  Too many social media researchers only look at these events from afar.  While a distant reading, like topic modeling, can be useful in grasping which questions to ask, sometimes understanding the associations between people and concepts requires an act of interpretation on the part of the researcher.  And that researcher’s experience may be the difference in between ascertaining the meaning of output, or simply writing that output off as mere “topic soup.”

Workshop–Hawaii

Hawaii.

The name itself evokes images of tropical sun, warm waters, surfing, and relaxation.

So what are the people in this image doing inside, intent on looking at computer screens? Instead of savoring the sunshine and walking on the sand, here they sit.  Inside.  Hunched over laptops.  Interpreting a series of instructions to make sense of social media data.  Listening to the SoMe Lab team explain what they are seeing.  They are not behaving as you imagine Hawaiian visitors would behave.

These dedicated researchers are taking part in the workshop organized by the SoMe Lab team at HICSS46, held this past January in Wailea on Maui. As a part of the workshop, they were hearing from the SoMe team about lessons the team has learned in the past fifteen months.

In October 2011, the SoMe Lab team began collecting—and seeking to make sense of—the rising number of tweets related to activities oriented around the “Occupy Wall Street” movement. Since that time, we have accumulated about 200 million tweets. We have devoted hours to curating this collection, trying different visualizations and analytic tools, and working toward a set of tools that others can use to collect and analyze their own data from different social media sites. In pursuing these efforts, we felt that we were learning not only about the data collection but also about the process of making decisions on what data to collect; how to curate, process, and store the data; and even about some of the not-so-obvious ethical issues of doing social media research.
This HICSS workshop was intended to share what we had learned and to get feedback from participants on what they were facing and what they hoped to accomplish with their research.  [A background paper prepared for the participants is available here.]

This workshop is just one event in what could be an emerging set of events for a network of people interested in social media research.  The participants represent a small sample of the growing number of techies, social scientists, and socio-technical systems researchers seeking to understand the changes in communication patterns and social networks that emerge from the dynamic information ecosystems represented by the blogs, tweets, images, and videos that flow across the diverse—and constantly changing—range of social media platforms. They are helping define the core and the edges of the complex interdisciplinary space represented by social media. They are struggling to establish transparent and reproducible approaches and methods. To help in communications, we have created a mailing list. This mailing list will be the starting point for a listserv and other communications channels. If you did not receive an email from me after HICSS (confirming your participation in the mailing list) and would like to be included, send an email to somelab@uw.edu and we’ll add your address to our list. (We envision only occasional communications about news and events, you won’t receive a steady stream of emails!)

click for larger view

As a researcher interested in information flows in digital environments I’m often interested in finding patterns in social trace data. For this discussion we can think of digital social trace data as the text that people post into threaded topics on forums, like on Reddit or a Wiki Talk page on Wikipedia. One way to find patterns in this kind of data is to make visualizations based on different quantifiable dimensions in the data, for example, total topic volume per day, volume per thread per day, and, possibly, the intensity of the discussion (as interpreted by qualitative researchers). In the remainder of this post I will note what we can learn from our visualization as well as its limitations and then post the R code I used to make the plot.

This type of plot is called a stacked spline plot. The curved lines are known as “splines” (don’t know why) and each one encloses an area, in this case shaded with some value between red and white. The total number of posts on a given day is represented by top spline in the plot. The volume for a given thread is the distance (y-axis) between its spline and the spline below it (the top of a different thread). So each topic is stacked on the one below it. If thread X had 0 posts on a given day, and thread Y had 3 posts on that day, we would see thread Y mound upward, but all we would see for thread X is its grey spline. In fact, the first limitation to this kind of plot is differentiating between specific threads. You can work around this by assigning topics specific colors. We use colors differently here.

In this plot I have shaded threads by how intense the discussion in the thread was. How do you do that? You could have qualitative researchers read each post against some agreed upon criteria for the vehemence of the discussion. In fact, I include this to show that we can include qualitatively coded data with our quantitative representation of an information flow to give the data visualization more depth and meaning. It doesn’t have to be intensity. It could be any particular measure that researchers are interested in.

click for larger view

One serious limitation of using splines is that data points are targets for the lines; the spline may not actually go all the way up or down to the point, so the exact point may not be faithfully represented. It does this because the spline algorithm is attempting to smooth the points out. I’ve made a second version with polygons instead of splines to highlight this: compare October 24^th and 25^th in the first (red) and second (blue) plots. The data is the same, but the spline fails to capture the depth of the dip or the height of the second peak.

Still, this kind of visualization is good for giving a sense of the patterns of exchange in online forums as well as comparing one topic to another in the same forum. We may not be interested in exact numbers, but rather in exploring the data and making some initial observations. The third plot (orange) shows an alternate data set over the same time frame. Why does this second topic have such a different signature and much lower overall volume? Why the large volume right at the start of the first plot? Which are we more likely to see in most topics? Will it differ based on some sort of category of topics? If we knew the topics of discussion for these plots we could generate some observations and explanations. In this case, both data sets were generated for this example (the first is based on a real, controversial Wikipedia Talk page data).

click for larger view

How did I make the plot? Below is the code I created in R to make the plot. Instead of writing about the code, I have inserted comments so that if you download to script you will have the explanations handy and modify with an understanding of what I had intended. Let me know if you come up something cool, and please acknowledge the Social Media Lab at UW if you use it as is.

First, the plotting function, then below, the calls to it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125

# Author: Jeff Hemsley jhemsley at uw dot edu
# Twitter: @JeffHemsley
# Created: Sometime in 2010
# 
# Function creates a stacked spline from a matrix of data. Can also use polygones
#   for straight lines instead of splines. The function is not mature in that it 
#   hasn't been tested (or even written) to deal a broad range of cases. I have
#   used it to plot a number of threads over time (days), but other applications
#   are certainly possible.
# 
#  mat.in: matrix where rows are x-axis and columns are y-axis. Uses rownames
#          for x-axis lables. The cell values indicate the y value for a given
#          column (thread) at a point in time (day).
#  seriesborders: (FALSE) to print borders around the spline or just the fill.
#          default is NO borders.
#  splineseries: (TRUE) if true then spline else polygone
#  col.vec: vector of colors. If null, random colors are assigned to the (columns) 
#          threads. 
#  col.fg: The color for the spline border. Default is grey.
#  weekend: a vector with the same length as the number of rows in mat.in. Function
#          expects a 0 (no shading) or 1 (vertical shaded block). If the i-th value 
#          in the vector is 1, then were x == 1 the function will shade. Used to 
#          note weekends. Alternate colors currently not supported.
#
plotstackedseries<-function(mat.in, seriesborders=FALSE, splineseries=TRUE
      , col.vec=NULL, col.fg=NULL, weekend=NULL
			, f.main="Stacked Spline Plot", f.ylab="Counts", f.xlab="Days"){ 
 
	myfg <- par("fg")
	mybg <- par("bg")
	rows <- dim(mat.in)[1]
	cols <- dim(mat.in)[2]
	# Since we are plotting a shape instead of a line, we need an additonal
	# start and end point for each thread. This allows us to close the shape
	new.rows <- rows + 2
	a <- 2
	b <- rows + 1
	# we can adjust how curved the spline is relative to the original points. 
	# At 0 there is no cruve, at -1 it overshoots the points, I like .9.
	spline.shape <- c(0,rep(.9,rows),0)
 
    # new matrix with a first and last row of zeros to close the splines
    zip.vec <- rep(0, cols)
    m <- rbind(zip.vec, mat.in, zip.vec)
 
	# Now. The way to do this is to over plot each thread on top of the last.
    # So the thread at the top has to be shaped according to its own cell 
    # values as well as all the cells below it. In otherwords, we need to 
    # do a cummulative sum on the rows (we expect these to represent days).
	m <- t(apply(m, 1, cumsum))
 
	# now we plot an empty plot that we'll put splines on later
	y.max <- max(m)
	x <- seq(1:length(mat.in[,1]))
	# all of our splines are x y plots, so we need xs.
    x.vec <- c(0,x,max(x)) 
 
	plot(x, mat.in[,1], type="n", ylim=c(0,y.max)
		  , xaxt="n"
	    , yaxt="n"
			, main=f.main
			, ylab=f.ylab
			, xlab=f.xlab
			, bty="n"
	)
 
    # custom axis stuff.
    my.at <- axTicks(1)
	if (my.at[1] == 0) {
	  my.at[1] <- 1
	} 
    # bottom axis lables are the row names of the matrix
	axis(1, at=my.at, label=rownames(raw.thread.mat)[my.at], line=-.7
	     , las=0, tck=-.02, cex.axis=1.1, lwd=1) # las=2 for perpindicular
    # left had y axis 
	axis(2, at=axTicks(2), label=axTicks(2), line=-.7
	     , las=2, tck=0, cex.axis=1.1, lwd=0) 
 
    # if no color vector is passed in, use random heat colors
	if (is.null(col.vec)) {
	  col.vec <- sample(heat.colors(cols), size=cols, replace=F)
	}
 
    # deal with borders.
	if (seriesborders==FALSE) {
		col.fg <- col.vec
	} else if (is.null(col.fg)) { 
		col.fg <- myfg
	} 
 
	#print weekends?
	if (is.null(weekend)==FALSE) {
	  y.lable.max <- max(axTicks(2)) 
		for (i in 1:rows) {
			if (weekend[i]>0) {
				x.weekend<-c(i,i,i+1,i+1)
				y.weekend<-c(0,y.lable.max,y.lable.max,0)
				polygon(x.weekend, y.weekend, col=rgb(0,0,1,.1), border=NA)
			}
		}
	}
 
	# for the series, we work from back to frount. We plot the 
    # tallest filled in spline shape first, followed by the next
    # tallest and so on. So each spline series is plotted over the
    # last one. Nifty.
	for (i in cols:1) {
 
		if (length(col.fg)==1) {
			local.fg <- col.fg
		} else {
			local.fg <- col.fg[i]
		}
 
        # we can plot a spline or a polygon
		if (splineseries==TRUE) {
		  xspline(x.vec, m[,i], col=col.vec[i], shape=spline.shape, open=F
              , border=local.fg, lty=1, lwd=.5)
		} else {
			polygon(x.vec, m[,i], col=col.vec[i], border=local.fg)
		}
	}
 
	par(fg=myfg, bg=mybg)
}

Below is the code I used to call the function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

# Author: Jeff Hemsley jhemsley at uw dot edu
# Twitter: @JeffHemsley
# Created: Sometime in 2010
# 
# the location of the files I used are at:
# http://somelab.net/wp-content/uploads/2013/01/thread_data_v7.csv
# http://somelab.net/wp-content/uploads/2013/01/thread_data_v6.csv
# note that rows are dates and cols are threads: series in cols.
# Grab these files or make your own and fix the path info below
#
f.path <- "c:/r/RUserGrp/"
f.name <- "thread_data_v7.csv"
f.raw.thread.mat <- paste(f.path, f.name, sep="")
plot.file.name <- paste(f.path, "topic5.png", sep="")
 
raw.thread.mat <- as.data.frame(read.csv(file=f.raw.thread.mat
                                         , header=TRUE, row.names=1))
num.conversations <- dim(raw.thread.mat) [2]
num.posts <- sum(raw.thread.mat)
 
# and, I already checked when the weekend dates are, so this vector
# will highlight the weekends in the plot: 1: time off, 0: work
week.end <- c(1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1
              , 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0)
 
# for these examples, the intensity is random binomial distributed
# becuase it seems to loosly resemble actual data
conversasion.intensity <- rbinom(num.conversations, 10, 0.3)
# since we are going to color the threads based on this, lets
# normalize the values 0 - 1
conversasion.intensity <- conversasion.intensity/max(conversasion.intensity)
 
# I usually use rgb colors, but hsv allows me to work in monochrome 
# h=0: red, .1: gold/yellow, .3: green, .6: blue, .7: purpleish
# V:  1 is brightest
# s (saturation) conversasion.intensity. 0 to 1, with 0 being none
conversasion.intensity.color <- hsv(h=.1, s=conversasion.intensity, v=1, alpha=1)
 
# I sent this out to a png, but it looks pretty nifty sent to pdf or svg
png(filename=plot.file.name, width=1024, height=768)
plotstackedseries(raw.thread.mat, seriesborders=T, splineseries=T,
                  col.vec=conversasion.intensity.color,
                  col.fg=rgb(.5,.5,.5,.5), weekend=week.end, 
                  f.main="Talk threads by volume and intensity",
                  f.ylab="Posts", f.xlab="Days"
)
 
sub.title <- paste("1 topic, ", num.conversations, " threads, ", num.posts, " posts")
mtext(sub.title, side=3)
dev.off()

A couple of weeks ago Bob wrote about a post about a research note that was recently accepted to the iConference. In it we outline the beginnings of a research project where we look at the interaction of different media platforms (Twitter and Blogs) with more traditional sources. In this post I go through the R code we used to plot, and visually compare, the volume of different information sources.

The data for this example is randomly drawn along a Pareto distribution so anyone should be able to just open the file, run it and have plots. Like I did in the last R example, I have used comments in the code to explain what I’m doing in the creation of these plots. After the code I give a brief introduction on the tool I use to select colors.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89

#
# Synchronized volume over time with events for different sources
#
 
# used to adjust the alpha (transparency) of rgb colors below.
library(VGAM)
 
# set up some params for later use. I could have surfaced more plot 
# params but for this plot these are things I was fiddling with
plot.type <- "h"
text.adjust <- c(.5,.3) # left-right & up-down with .5 being center
 
# this is volume of data flow over hours, so how many hours. Normally
# you have this or will calculate it, but I am making random data
num.hours <- 300
my.local.x <- 1:num.hours
 
# points in time when events happened. These are entirely arbitrary.
event.x.vec <- c(45, 75, 275) 
 
# for this plot we are comparing three different information flow
# volumes: tweets, blog posts and news. Yes, blogs and news can be
# a fuzzy distinction, but this is about the R code.
# I use a random Pareto distribution because it is a reasonable
# approximation for some types of information flows/attention 
# (and if you want a reference supporting that let me know).
news.text <- "news"
tweets.counts <- rparetoI(num.hours, scale=10000, shape=3)
news.color <- rgb(128,30,128,alpha=255,maxColorValue=255) 
 
blogs.text <- "blogs"
blogs.counts <- rparetoI(num.hours, scale=100, shape=2)
blogs.color <- rgb(32,135,91,alpha=255,maxColorValue=255)
 
tweets.text <- "tweets"
news.counts <- rparetoI(num.hours, scale=1, shape=1.5)
tweets.color <- rgb(196,159,47,alpha=255,maxColorValue=255)
 
# probably not the best practice here, but I take a vector
# of x values for events at hour X and a vector of y min 
# and max. So for an x, draw a dotted line from y.min to
# y.max at x[i]
plot.events <- function(x.vec, y.min.max) {
  for (i in 1:length(x.vec)) {
    event.x <- x.vec[i]
    lines(c(event.x, event.x), y.min.max, col="black", lty=2)    
  }
}
 
# work horse function. Again, I could surface more plotting
# parameters, but I am focused on a specific kind of comparison
# I set a few defaults. X is expected to be a sequence, but it
# doesn't have to start at 0. I'm using hours, but it could be
# any units.
plot.vol <- function(x, y, event.x.vec, caption="", plot.type="h", col="black", adj=c(.5,.5)) {
 
  # after the basic plot we want to add the events and then the 
  # caption. Simple calcs to find where to put 'em:
  my.local.y.lim <- c(0, max(y)) 
  # I tried c(min(y), max(y)) but like the above better for
  # histogram type plots
  my.local.y.mid <- round(mean(my.local.y.lim), 0)
  my.local.x.mid <- round(median(x), 0)
 
  plot(x, y, ylim=my.local.y.lim, yaxt="n"
       , type=plot.type, lwd=2, col=col, bty="n"
       , main="", xlab="", ylab="")
 
  # I keep the captions simple and I put them in the
  # plot so that the data set plots are close enough together
  # for comparison
  plot.events(event.x.vec, my.local.y.lim)
  text(my.local.x.mid, my.local.y.mid, labels=caption, col=adjustcolor(col, alpha.f = .5), cex=5, adj=text.adjust)
  mtext("hour", side=1, line=1.5, cex=.8, adj=.1)
  mtext("volume", side=2, line=3.5, cex=.8) # #padj=-5,     ) #adj=c(0,1), cex=.8, outer=T)
  axis(2, at=axTicks(2), labels=format(axTicks(2), scientific=F, big.mark=","), las=2, tick=F, line=-2.2)
}
 
# now plot out each set. I set mfrow to get all three on the 
# same screen. The xpd param controls clipping for plot area.
# I only seem to need to set this for histogram type plots
#par(mar=c(5,4.5,2,2), mfrow=c(3,1), xpd=F) # c(bottom, left, top, right) c(5, 4, 4, 2) + 0.1.
par(mar=c(3.5, 5.5, 2, 2), mfrow=c(3,1), xpd=F) # c(bottom, left, top, right) c(5, 4, 4, 2) + 0.1.
plot.vol (my.local.x, tweets.counts, event.x.vec, tweets.text, plot.type, tweets.color, text.adjust)
plot.vol (my.local.x, blogs.counts, event.x.vec, blogs.text, plot.type, blogs.color, text.adjust)
plot.vol (my.local.x, news.counts, event.x.vec, news.text, plot.type, news.color, text.adjust)
 
# gimmy back my defaults.
par(mar=c(5.1,5.1,4.1,2.1), mfrow=c(1,1), xpd=NA)

So how do I choose my colors? I use Color Scheme Designer which allows you to move a slider around a color wheel to pick your main color and then select if you want monochrome, complement, triad and other color schemes that work with your main color. You can use the tab selection on the bottom to alter saturation and brightness of the scheme as well. Once you have the colors you like, you can export in different formats. I usually export to rgb for R code.

We recently held a workshop that dealt with our lessons learned in working with our corpus of data collected in reference to the Occupy Wall Street movement.  We took folks through a mock research project and one approach to how researchers might “do” social media research, with hands on examples.  You’ll find our “Working Document” that details these learned lessons, along with the slides presented at the workshop.  You can find them by clicking “read more” below!

Download the document by clicking here.: HICSS_workshop_final.pdf

Download Joe’s slides (“prepping the data”) here: eckert_hicss_final.pptx

Download Shawn’s slides (“collecting the data”) here: HICSS Workshop – Collecting Data

Wayne Gretsky is quoted as saying that a great hockey player plays where “the puck is going to be,” not where it is.  Gretsky, like the great NBA point guards (think Magic Johnson or Mark Price), was quick to detect emerging patterns in movement and flows–then take advantage of what was about to happen.  In our research efforts, we often try to detect patterns in order to explore what these patterns may tell us about underlying processes.

The SoMe Lab is examining patterns in the movement and flows of information between and among social media platforms.  We observe that traditional media news may inform or trigger information exchanges in the blogosphere or Twitter; and vice versa.  We want to look closely at these patterns to gain insights into phenomena such as virality, the birth and life cycle of interest networks, and the dynamics of a fluid cast of gatekeepers.   The accompanying image illustrates the patterns that distinguish the volume of tweets, blog posts, and traditional news items following the pepper spraying incident at UC-Davis November 18, 2011.

We’re far from developing a comprehensive view of these flows, but we have been able to visualize and understand what seem to be distinguishing frequency patterns of news and conversations about distinctive types of events.

We wrote a research note on this topic that will appear in the proceedings of the 2013 iConference (to be held in Fort Worth in February; prepublication draft here).  In the note, we illustrate and discuss the numbers of tweets, blog posts, and news stories that focused on specific events related to the Occupy movement during the period from October 2011 through November 2012.  (The above illustration is taken from this research note.)  We looked at the timing and volume of stories and tweets at specific planned events and serendipitous events.

We observe that different types of events appear to exhibit particular patterns of activity (similar to what Sharon Meraz and Zizi Papacharissi refer to as “rhythms” in their IR13 presentation).   We found a contrast among events that were planned—by civilian authorities or by protesters—and those that were unexpected and serendipitous.  For example, the number of traditional media stories (newspapers, TV) in advance of authority-planned events is much greater than the number for in advance of protester-planned events.  The number of stories, blog posts, and tweets about serendipitous events (such as the pepper spraying of students at UC-Davis) peaked soon after the events, but often had longer periods of activity.  Some events exhibited secondary peaks appearing as a consequence of follow-up stories (e.g., investigation reports).  Other events and their stories exhibited durability as they evolved into memes that represented more than the events themselves (e.g, the pepper-spraying officer).

The evidence in our initial efforts suggest that patterns—visualized using computer-based algorithms but observed and interpreted by researchers—may be a productive approach to exploring the evolutionary dynamics of our emerging information ecosystem.

The patterns we observe are relatively simple—almost obvious once they have been pointed out—but we may use these simple patterns to point us to more complex relationships.  As researchers, we won’t pretend to be Wayne Gretsky or Magic Johnson, but perhaps we can improve the tools that enable us to see patterns in our data and use these tools to help us probe the mechanisms that produce these patterns.

If you have similar observations about information flow patterns in social media, let us hear from you.  And if you have techniques for visualizing these patterns, let’s compare notes.