Hi all, this is the draft plan. It is still quite vague. I think the next thing I need to do is think very carefully about my questions, then all should become clearer. Feedback, questions and requests for clarification much appreciated.
Thanks, Adrienne
Draft Evaluation Plan
Introduction:
This evaluation investigates online discussion and the ways in which communities can be used to improve learning and construct knowledge. The evaluation is for assessment for the Evaluation of eLearning for Best Practice course. The primary evaluator is Adrienne Moyle.
Background:
The media that is being evaluated is an online course called Flexible Learning Course run by Bronwyn Hegarty and Leigh Blackall at Otago University. The course makes can be accessed at http://flexible-learning-course.blogspot.com/ and makes use of wiki educator, google groups and email.
Participants have set up individual blogs on which they post their thoughts and writings.
This course is a level 7 course registered with the New Zealand Qualifications Authority. It has been developed by staff in the Educational Development Centre of Otago Polytechnic and is designed to help you access and interpret information about flexible learning, as well as engage in professional dialog about flexible learning, and present a plan for developing flexible learning options in a course of your choosing.How do you join this course? This course follows a schedule and can be undertaken self paced, or with others in 15 weeks facilitated (5 hrs per week). The next facilitated course starts 16 March 2009 ends on 5 July 2009. To join the facilitated session click the "join" tab at the top of this website.Facilitators: Guidance, assessment and certification services are available at any time by contacting the course facilitators:
Purposes:
The purpose of my evaluation is to look at the effectiveness of the online interaction in this course.
I will be using the following elearning guidelines which have been taken from http://elg.massey.ac.nz.
TD4 What makes an effective online discussion?
TD9 What strategies in online discussions will build rapport with students?
TO11 Are methods in place to measure staff and student perceptions of the online discussions?
Audiences:
The primary and secondary audiences of this evaluation will be the staff and students involved in the Flexible Learning Course. A tertiary audience will be the staff and students of the Evaluation of eLearning for Best Practice course, to which the evaluation will be submitted for assessment. If required the evaluation will be available to other staff and students of Otago University and Manukau Institute of Technology.
Decisions:
Decisions will be made as to the quality of the evaluation undertaken when it is assessed for the Evaluation course. Decisions will be made as to the design of online interaction in the Flexible Learning course once the effectiveness of the interactions has been evaluated.
Questions:
Methods:
Mixed Method, Eclectic Pragmatic
Interviews
Questionnaires
Confidence Log
Focus Group
Discussion content analysis
Sample:
Ideally all staff and students in the Flexible Learning course will participate in the evaluation. It is quite likely that some will choose not to for various reasons.
Instrumentation:
Gunawardena method – content analysis
Survey monkey – online questionnaires, online confidence log
Elluminate – focus group
Limitations:
This is my first evaluation. The sample size may be small. The questionnaires may not be that sound. I would have liked to have organized the evaluation before the course started.
Logistics:
It will all be done by me.
Implementation – questionnaires – online, content analysis – online, focus group – illuminate, confidence log – online, interviews – telephone/online
Time Line:
Implementation
The Flexible Learning course runs from 16 March to 5 July. The Evaluation course runs until June 29 with implementation scheduled for May 11-26, analysis scheduled for May 25 – June 8 and reporting scheduled for June 8 – 29.
That schedule has been set in accordance with the assessment requirements for the Evaluation course.
I may need a longer implementation time as I want to try out a variety of methods, plus a summative assessment will need to be done at the end of the FL course. It looks as though I may need to ask for an extension for my Eval course!
Focus groups and confidence log – May 11
Interviews
Questionnaires
Content analysis – up to 5 July
Reporting – 10 July
Budget:
My time: $25 hour
Approx 25 hours total
Things to do next:
Questions for Focus Group
Questions for Questionnaire
Format for Confidence Log
Format for Content Analysis
Subscribe to:
Post Comments (Atom)
Hi Adrienne
ReplyDeleteA very comprehensive draft - looks to me like you're very much on track. Now I better get stuck into mine!
Great stuff Adrienne
ReplyDeleteyour plan is shaping up nicely. As we discussed on email, I have put more detailed feedback on the plan in Google documents and you can put this link in your blog. See: http://docs.google.com/Doc?id=dgg6nrz2_155fxpfbmdx
Your guidelines are excellent and just need some sub-questions. For example for TD4 - What makes an effective online discussion?
Example sub-questions:
1. Which types of online discussion promote learning? [This will enable you to look at the email discussion forum, discussion on the course blog and participant blogs. Discussion on WikiEducator was not really used apart from an initial optional introduction.]
2. what are the features of an engaging interaction?
Also decisions must be directly related to the outcomes of the evaluation, that is the answers to your questions. Say one answer is that the use of participant blogs inhibits regular interaction between course participants because they prefer to have face-to-face or email discussions, then you have found out that blogs may not be a good way to promote discussion.]
Hi Adrienne,
ReplyDeleteIt looks like you are really well focused in your evaluation plan. Very interesting.
Regarding your purpose "The purpose of my evaluation is to look at the effectiveness of the online interaction in this course" what sort of interaction will you be concentrating on.
Student - content interaction
Student - tutor interaction
Student - student interaction.
As sugested by your blog - you would investigate on online discussion - is it mainly peer disccusion or will you be looking at the overall interaction.
It seems like it will end up being a quite extensive evaluation and you may need more time to complete it. :)
Good luck with your evaluation.
Regards
Krishan Mani
Hi Adrienne,
ReplyDeleteI like your evaluation plan. Quite pro and very thorough thought. Online interaction is really important in online learning / teaching. I am keen to know your evaluation outcomes :-)
Talking about your sample, I noticed the course you evaluate is currently half way through. So students started the course just about one month ago. Maybe for some of them this is their first time using blog, google group and WE as communication platforms. I remembered when I had my last paper, which is similar to the one you are evaluate now, at the beginning I really struggled to manage all sorts of info from differenct sources - WE, blogs, google group... At the end of the course amazingly I found I am used to it and started love it. So I suppose maybe for the course you are evaluating now, there mite be several students like me before, struggling using online tools and hardly to say any efficient interaction.
Therefore, to get the whole picture, do you think it's better if you can get any feedback from former students who finished the course(I know it might be very hard...) ? They might give more info from different perspectives. Or, as you mentioned, have some feedback at the end of the course, will be very helpful too.
I wonder where you will put your evaluation questionnaire to - WE? Google group? Blog? Guess you might already think about this question. I think we need to be very careful about it. For example, if you publish it only on google group, then those students are very active in blogs but seldom join the google group (like I was for my former paper) may lose the chance to provide their feedbacks. So the feedbacks you get probably are mainly from those who prefer google discussion group. That means the data you collect is likely based on biased group.
Cheers,
Joy
Hey, thankyou very much for your constructive feedback.
ReplyDeleteKrishan you are right, I have just been drafting my sub-questions and it is getting huge! I think I am interested in all the interactions, and how the design of the interactions affects student learning. I had thought to ask the students which type of interaction they valued more.
Thanks Joy, I hadn't thought about where to post the questions, I would probably like a link from the course blog, and will need to ask the facilitators if that is OK. I am planning to ask for confidence ratings before starting the course and then at the end. I should have done it at the start of the course, but will ask them if they can remember how it was at the start and if it has changed.
Cheers,
Adrienne