Thursday, 30 April 2009

Bron's feedback on my draft plan

Hi everybody, if you would like to see Bronwyn's feedback on my draft plan follow this link http://docs.google.com/Doc?id=dgg6nrz2_155fxpfbmdx. It's really useful.

Cheers

Adrienne

Tuesday, 21 April 2009

Draft evaluation plan

Hi all, this is the draft plan. It is still quite vague. I think the next thing I need to do is think very carefully about my questions, then all should become clearer. Feedback, questions and requests for clarification much appreciated.

Thanks, Adrienne

Draft Evaluation Plan

Introduction:
This evaluation investigates online discussion and the ways in which communities can be used to improve learning and construct knowledge. The evaluation is for assessment for the Evaluation of eLearning for Best Practice course. The primary evaluator is Adrienne Moyle.

Background:
The media that is being evaluated is an online course called Flexible Learning Course run by Bronwyn Hegarty and Leigh Blackall at Otago University. The course makes can be accessed at http://flexible-learning-course.blogspot.com/ and makes use of wiki educator, google groups and email.

Participants have set up individual blogs on which they post their thoughts and writings.

This course is a level 7 course registered with the New Zealand Qualifications Authority. It has been developed by staff in the Educational Development Centre of Otago Polytechnic and is designed to help you access and interpret information about flexible learning, as well as engage in professional dialog about flexible learning, and present a plan for developing flexible learning options in a course of your choosing.How do you join this course? This course follows a schedule and can be undertaken self paced, or with others in 15 weeks facilitated (5 hrs per week). The next facilitated course starts 16 March 2009 ends on 5 July 2009. To join the facilitated session click the "join" tab at the top of this website.Facilitators: Guidance, assessment and certification services are available at any time by contacting the course facilitators:

Purposes:
The purpose of my evaluation is to look at the effectiveness of the online interaction in this course.

I will be using the following elearning guidelines which have been taken from http://elg.massey.ac.nz.

TD4 What makes an effective online discussion?
TD9 What strategies in online discussions will build rapport with students?
TO11 Are methods in place to measure staff and student perceptions of the online discussions?


Audiences:
The primary and secondary audiences of this evaluation will be the staff and students involved in the Flexible Learning Course. A tertiary audience will be the staff and students of the Evaluation of eLearning for Best Practice course, to which the evaluation will be submitted for assessment. If required the evaluation will be available to other staff and students of Otago University and Manukau Institute of Technology.

Decisions:
Decisions will be made as to the quality of the evaluation undertaken when it is assessed for the Evaluation course. Decisions will be made as to the design of online interaction in the Flexible Learning course once the effectiveness of the interactions has been evaluated.

Questions:

Methods:
Mixed Method, Eclectic Pragmatic

Interviews
Questionnaires
Confidence Log
Focus Group
Discussion content analysis

Sample:
Ideally all staff and students in the Flexible Learning course will participate in the evaluation. It is quite likely that some will choose not to for various reasons.

Instrumentation:
Gunawardena method – content analysis
Survey monkey – online questionnaires, online confidence log
Elluminate – focus group

Limitations:
This is my first evaluation. The sample size may be small. The questionnaires may not be that sound. I would have liked to have organized the evaluation before the course started.

Logistics:
It will all be done by me.
Implementation – questionnaires – online, content analysis – online, focus group – illuminate, confidence log – online, interviews – telephone/online

Time Line:

Implementation
The Flexible Learning course runs from 16 March to 5 July. The Evaluation course runs until June 29 with implementation scheduled for May 11-26, analysis scheduled for May 25 – June 8 and reporting scheduled for June 8 – 29.

That schedule has been set in accordance with the assessment requirements for the Evaluation course.

I may need a longer implementation time as I want to try out a variety of methods, plus a summative assessment will need to be done at the end of the FL course. It looks as though I may need to ask for an extension for my Eval course!

Focus groups and confidence log – May 11
Interviews
Questionnaires
Content analysis – up to 5 July
Reporting – 10 July


Budget:
My time: $25 hour
Approx 25 hours total


Things to do next:

Questions for Focus Group
Questions for Questionnaire
Format for Confidence Log
Format for Content Analysis

Monday, 20 April 2009

Catch up

Catch up
Hi everyone, I’ve decided to write a bit more about where I’m at now because my posts so far have been a bit rushed and I hadn’t really done all of the reading I needed to understand the content.

Week 3 Guidelines

Have finally had some time to really sit and look at the guidelines. I looked through the PDF (the cube hides a lot!) and have decided that what I am really interested in is online interaction. With this topic in mind I have narrowed it down to 3 guideline questions. TD4 What makes for an effective online discussion? TD9 What strategies in online discussions will build rapport with students?, and TO11 Are methods in place to measure staff and student perceptions of the online discussions?

Week 4 Paradigms

Have had another look at Reeves and he is actually really onto it and quite wry once you figure out what he is talking about. Basically there are different ways in which people have approached educational research and evaluation (paradigms). Paradigm 1 is quite analytical and scientific and relies on quantitative methods such as those used in statistical evaluation. This type of evaluation is probably done on a large scale where you have large amounts of data to analyse mathematically.

Paradigm 2 is more qualititative, meaning that human variance is taken into account, and takes into account how humans construct “reality”. Using this type of paradigm involves being an anthropologist or a human researcher. Paradigm 3 looks at deconstruction, or uncovering “hidden” structures and assumptions in education. Paradigm 4 is a mix of the other paradigms, where you use what you think will be useful for your particular context.

I will be using Paradigm 4 for my evaluation. I suspect I may have constructivist leanings as I am interested in how students collaborate and construct knowledge through online discussion. The methods I use will be mainly qualitative – questionnaires and interviews, with some quantitative evaluation of the number of interactions undertaken. I think deconstruction would be very interesting, but it is probably beyond the scope of this assignment for me.

Weeks 5&6 Methods

The purpose of my evaluation will be to investigate how to improve online discussions for better learning. I will be using the Eclectic-Mixed Methods-Pragmatic Paradigm, and conducting a summative, effectiveness evaluation. In line with the multiple methods evaluation (Shotland) model, I would like to use questionnaires, confidence logs and focus groups. The variety of methods will provide triangulation to test adequacy. The questions will need to be designed with bracketing in mind to measure range. I will also use quantitative information about the number of interactions that take place online.

One thing I am a bit concerned about is that I have not planned my evaluation before the course started, so have not introduced myself to the students yet …

Chapter 8 of Interactive Learning Systems Evaluation has provided me with some good questions about learner reactions to online discussions. What knowledge, skills and attitudes do learners derive from online discussions? It suggests using a data collection matrix and very specific and carefully worded evaluation questions, so I will have to put a lot of thought into those.

I read Interaction online:Above and beyond requirements of assessment by Meg O’Reilly and Diane Newton. This article used action research, which fits well with the iterative process of refining a course as you go. They analysed how students used the discussion forums ie for planning, encouraging, clarifying, summarizing, and the percentages were quite interesting. They found that students valued the online learning environment and used the discussions to assist them with their learning.

In Knowledge Building Community: Keys for using online forums, Quing Li found that “successful employment of online forums can foster knowledge building community in which desired student qualities are cultivated”. While a deconstructivist would probably look at “what are desired student qualities?”, I will be looking at what knowledge is constructed.

The students and course that Tuncay Saritas studied in The construction of knowledge through social interaction via computer mediated communication, was similar to mine in that they were studying the learning principles, technologies and techniques for teaching and learning in a distance education system.

Saritas investigated interaction patterns; did the interaction support the construction of knowledge?; and factors that influenced the level of knowledge construction.

Saritas used an online survey, an interview with the instructor and content analysis of the discussions. He found that the quality and quantity of interaction was influenced by the structuring of the discussions. He recommends establishing common rules for discussions, and that instructors are active participants in the discussions as they play a significant role in guiding students towards higher levels of learning.

Saritas study explored the dynamics of learning community. It used the Interaction Analysis Model of Gunawardena et al (1997) to investigate an evidence of knowledge construction. I want to find out more about this model as it may be useful for my evaluation. Has anyone ever heard of it, or used it? http://www.alnresearch.org/HTML/AssessmentTutorial/ExampleResearch/GunawardenaLoweAnderson.html
http://www.alnresearch.org/HTML/AssessmentTutorial/Measures/GunawardenaLA_Measure1.html

OK, now I have updated where I am at, it is now time to start on the negotiation and writing of an evaluation plan for Wk 7. Catch you soon.

Adrienne

Tuesday, 7 April 2009

Type of evaluation best suited to my project

Week 6

I’ve been reading abstracts about studies of interaction online, but haven’t really connected up to evaluation paradigms yet.

I’ve looked at enhancing learning through student-centred social learning environments[1], and how interaction promotes critical thinking and knowledge construction[2][3][4]. One study considers Gilly Salmon’s work and some unintended consequences of a practical (pragmatic?) approach.[5]
One study that looked at ways of recording and studying online interactions was useful as a reminder that how you study or analyse interactions can influence your results.[6]

I had a discussion with Bronwyn and we decided that an eclectic mixed-method pragmatic evaluation is probably what I am looking at. This will allow me to pick and choose a variety of research methods based on what will fit in with what I want to study.

I need to find some time to do more reading and thinking before I start on my plan. I will be doing a summative evaluation of the Flexible Learning course http://flexible-learning-course.blogspot.com/. Bronwyn is including some student-centred learning outcomes/goals in the course. I will probably use surveys, focus groups and some type of analysis of the interactions between students, teachers and content. I was thinking of looking at technological difficulties, but maybe I should just leave that and not let this get too big.

[1] Interaction online: Above and beyond requirements of assessment Meg O'Reilly and Diane Newton Australian Journal of Educational Technology2002, 18(1), 57-70.
[2] Investigating critical thinking and knowledge construction in an interactive learning environment.Wang, Qiyun, Woo, Huay Lit, Zhao, Jianhua. Interactive Learning Environments; Apr2009, Vol. 17 Issue 1, p95-104.
[3] Knowledge Building in an Online Environment: A Design-Based Research Study.
Qing Li, Journal of Educational Technology Systems; 2008-2009, Vol. 37 Issue 2, p195-216.
[4] The construction of knowledge through social interaction via computer-mediated communication. Saritas, Tuncay. Quarterly Review of Distance Education; Spring2008, Vol. 9 Issue 1, p35-49.
[5] E-learning and the transformation of social interaction in higher education.
Slevin, James, Learning, Media, & Technology; Jun2008, Vol. 33 Issue 2, p115-126.
[6] Coding procedures to analyse interaction patterns in educational web forums.Manca, S.,Delfino, M.,Mazzoni, E. Journal of Computer Assisted Learning; Apr2009, Vol. 25 Issue 2, p189-200.