Sometimes I feel like I have a slightly unfair advantage with this project since I work at a research institution. Twice now I’ve brought up AR to faculty members I’m meeting with. The first time I was meeting with a faculty member in the College of Education and I noticed she had a book about Action Research on her bookshelf. I told her that was my research methodology and that led to a thirty minute conversation with her picking apart my project. Which was awesome and terrifying but really more awesome. She led me to some excellent research about structuring comments in group blog posts. Which I will be adding to my Lit Review shortly and most likely bringing into the course structure next week.
The second was with a faculty member in the College of Business. We are good friends and I remember him telling me about his latest research project last year. I’m cautious to offer up his research idea before its published, but its basically looking at specific websites, the text they use, and seeing if you can correlate product success to words used to describe it. This requires scraping the text from the sites and running it through an analysis software. I had a similar idea to potentially use a similar technique for my students blogs which I syndicate back to the course hub so I have easy access to the raw data with WordPress’s export functionality. He tipped me off to LIWC (Linguistic Inquiry and Word Count) which has several dictionaries that can code your data for simple things like pronouns, first person, third personage usage to very specific use cases like death, sexuality, anxiety, and other emotions.
LIWC calculates the degree to which people use different categories of words across a wide array of texts, including emails, speeches, poems, or transcribed daily speech. With a click of a button, you can determine the degree any text uses positive or negative emotions, self-references, causal words, and 70 other language dimensions. – LIWC.net
They have a free online version that’s slightly limited so I ran one of my own blog posts through it and it was fascinating to see how it quantified the words I chose to use in the post. The free taste had me sold and so I quickly grabbed the software.
My aim isn’t to use it as my sole interpreter of students posts, but rather to give myself a quick barometer on how each course is doing. Most weeks require students to do a “reflection post” which is structure suspiciously similar to those in our program: what were you asked to do / how did you react / and why do you believe you reacted that way? This week was the first week where I was able to get a first true read from the blogs on how the course was doing. I chose a few different dictionaries to assess different aspects which I am particularly interested in. The first is I’s, we’s, pronouns, first person, third person, and social processes. I want to see how often they are talking about themselves and how often they are talking about other students, people, resources, etc. My goal, particularly of the online class, is to try to cultivate an environment that is fertile for social engagement. The second bucket that I’m looking at are attitudes (positive, negative, anxiety, anger, and sadness). This is just to help gauge student health. The last major bucket is cognitive (insight, causation, tentativeness, certainty, inhibition, inclusion, and exclusion). This is a big one because I want to be able to see if through the blog posts I can infer cognitive growth. So here’s a look at this weeks reflection broken down by section:
Sect 002 is my face-to-face and 995 is my online. I’ve highlighted in red where you see a somewhat significant difference in the two sections. Not incredibly surprisingly, face-to-face uses words that are more associated with they, social processes, positive emotions, and causation. This makes sense as they work in a classroom where I can work them through what they are being asked to do and why (causation). As for online, there’s a jump in anxiety and sadness. As I had thought earlier in the week, this week’s worth would be a little tougher on the online students. One interesting outlier was tentativeness which was more significant in face-to-face. I don’t want to make too large of an assumption here, but I think there’s something to be said about having to perform the work (which is computer-based) in a social setting when you aren’t comfortable. There’s a nervous apprehensiveness that comes along with something like that and so, to that end, I think online student’s have the upper hand as they can work with the comfort of knowing no one is watching them make mistakes. The other interesting metric was that “we” wasn’t more significant in the face-to-face, but that makes a little sense knowing that students are being to asked to reflect on their learning experience.
I’ve also got the data coded on a student-by-student level. One thing I explored earlier was “insight.” I wanted to know who got the highest score on “insight” and did it correlate to any other metric like, say, word count for instance. As it turned out, I had one student who had nearly double the score of the next closest student:
Is this student the leader in word count? Nope, not even close. It’s actually below average. But the student does also have the highest score in “certainty” and “pronouns.” Interestingly the student is also above average in anxiety. Now I’m not ready to call any of them correlated with such a small sample size but it will be interesting to track long term. Does “certainty” seem to lead to high “insight?” It certainly makes sense. If so, how can I promote writing that seems insightful? One way would be give the students the blog posts as an example. Here’s the blog post but coded in such a way that you see all of the words that trigger the insight dictionary:
So you’ll see the pattern : learn, understand, felt, grasp, feeling, feel. For an educator, this is powerful stuff to see the learning process clearly articulated. It’s not a long post but it’s to the point. And, again, I’m not wanting to look at it purely quantitatively but always value it in its entirety as well. And it brings up good points. Does the student feel like this was their best week? Nah. But they felt like they got something out of it and they are willing to work towards the next goal. This seems positive.
My goal is to do this process each cycle and see if I can affect any movement, particularly on the social and insight end. Hopefully I can make a tweak here or there that will trigger students to begin to see their learning experience within the context of a social space or better yet use their reflection to give advice to other students. We shall see!