Translate

Friday, August 22, 2014

Evaluation & Innovation: Michael's Perspective

I recently returned from my regular trip to the Distance Teaching and Learning Conference in Madison, WI. One of my highlights was a pre-conference workshop on developmental evaluation with Michael Q. Patton. I asked Michael to lend his voice to our series.

I am the Evaluator- Patton
Michael Q. Patton, Ph.D.


Michael Quinn Patton is an independent evaluation consultant based in Minnesota. He is former President of the American Evaluation Association (AEA).  He has authored six evaluation books including a 4th edition of Utilization-Focused Evaluation and 3rd edition of Qualitative Research and Evaluation Methods. He is recipient of the Alva and Gunnar Myrdal Award for "outstanding contributions to evaluation use and practice" and the Paul F. Lazarsfeld Award for lifetime contributions to evaluation theory, both from AEA.  His latest book is Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use.  
http://www.guilford.com/books/Developmental-Evaluation/Michael-Quinn-Patton/9781606238721

     

How do we recognize innovation in teaching and learning with technology?
Innovation is contextual not absolute.  What is accepted practice in one context may be different and innovative in another.  Thus, innovation is a socially constructed notion which has to be understood and interpreted within a context. 


What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges?
The challenge is that traditional ways of doing evaluation (logic models, SMART goals, predetermined outcome indicators, mandated rubrics and metrics) inhibit innovation.  The solution is developmental evaluation which accompanies innovators on the emergent journey of innovation, adapting evaluation to emergent issues.  For details about developmental evaluation see: http://www.guilford.com/books/Developmental-Evaluation/Michael-Quinn-Patton/9781606238721

Can you point to some promising innovations in teaching and learning?
Digital portfolios for online learners to archive their learning and products of their learning. These are accessible to the learners on a cumulative, long-term basis and can be shared for special purposes (e.g., a job opportunity).

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?
Online learning, like all effective interventions, is most effective when geared to specific subgroups with identifiable needs.  For example, homeless youth have special needs.  Here is a study that identifies how to work effectively with homeless youth: http://www.terralunacollaborative.com/wp-content/uploads/2014/03/9-Evidence-Based-Principles-to-Help-Youth-Overcome-Homelessness-Webpublish.pdf

Previous Post in Series: Steve Ehrmann's Perspective

Thursday, July 24, 2014

Evaluation & Innovation: Steve's Perspective

Over the next several weeks, I will be inviting several educational researchers and practitioners to offer their perspectives on the role of evaluation in teaching, learning, and technology innovation. Each is asked to freely respond to the same four questions. My hope is to foster discussions that enrich understandings, challenge assumptions, and strengthen processes for those who evaluate proposals or design studies. The goal is to compile a useful reference here.

Steve Ehrmann photo
Stephen C. Ehrmann, Ph.D.
Most recently, Stephen C. Ehrmann served as Vice Provost for Teaching & Learning at the George Washington University.  Previously he was founding Vice President of the Teaching, Learning, and Technology Group (TLT Group) and Director of the Flashlight Program for the Evaluation of Educational Uses of Technoloy; Senior Program Officer with the Annenberg/CPB Projects; Program Officer with the Fund for the Improvement of Postsecondary Education (FIPSE), and Director of Educational Research and Assistance at The Evergreen State College. He has a Ph.D. in Management and Higher Education from MIT.

Blog and bio: http://sehrmann.blogspot.com 


How do we recognize innovation in teaching and learning with technology?
It combines something unfamiliar with something promising. 


What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges? 
  • Evaluation is one of those things that everyone assumes he or she can do; the biggest challenge is not realizing there is history here – methods, previous findings, etc.
  • Because innovation is by definition somewhat unfamiliar, to some degree you don’t know what you’re looking for.
  • Those in charge of the evaluation are often advocates of the project. This role often leads them to counter-productive actions such as:

o   Delaying the creation of an evaluation plan until all the available money has been committed to other purposes and, if it’s a grant proposal, the proposal is due in a day or two.  Two major reasons why evaluation plans for grant-funded edtech innovations are often so weak.
o   Another reason: Delaying the evaluation until ‘all the kinks have been worked out’.  This means the evaluation isn’t being used to identify the kinks or to help figure out how to work them out.  It also often means that the evaluation is delayed until a time when no one remembers to do it, or has any commitment to do it.  The initiative can end (as an initiative) with stacks of data sitting somewhere, unanalyzed.
  • Rapture of the technology: paying too much attention to the technology itself as though, by itself it drives change.  That’s almost never true, so techno-centric evaluations can easily produce puzzling, misleading results.  Imagine two institutions, each using a different brand of learning management system.  Institution A uses LMS-A, while B uses LMS-B, a competitor.  Institution A has a long tradition of learning communities, seminars, group projects while institution B, valuing access highly, historically has asked little group work of its commuting students and distant learners.  An evaluation is done of online communication among students – how frequent? How productive?  A beats B. A technocentric evaluation is designed on the assumption that it’s the technology that determines the use of the technology, so the author concludes that LMS-A is better for online learning communities than LMS-B.  

In an unpublished article, I’ve described 10 principles for doing a better job evaluating innovations in  education, especially innovations using technology. Briefly:
  1. Above all, do no harm. (For example, don’t ask for data if you don’t have a firm idea of how you can satisfy your survey respondents and other informants that helping you was, indeed, worth their valuable time.)
  2. Design the evaluation so that, no matter what you find, each stakeholder is somehow better off.
  3. Be ready to compare apples with oranges because that kind of comparison is almost inevitable in a study of an educational innovation.
  4. Focus on what people DO most often with the program (their ‘activities’) rather than looking only at the tools with which they do it.
  5. Study why those people do those things in those ways.
  6. Compare the program with the most valuable alternative, not with doing nothing.
  7. Study costs, not just benefits.
  8. Remember that education does not work like a well-oiled machine – doing the ‘same thing’ often does not produce the ‘same results.’
  9. Recognize that different people using the program often have different motives, perceptions, circumstances, and, therefore, outcomes.  Realizing this leads to a dramatically different approach to assessment and evaluation – the students are doing different things (#4 above) because they have different reasons and different influences acting on them (#5 above). So the outcomes will differ from student to student.
  10. Start now!  It’s never too early in the life of innovation, even going back to before it’s introduced, to do useful studies.
Can you point to some promising innovations in teaching and learning?
      The use of questions designed to make students think with polling techniques (such as clickers or cell phones), and peer instruction.  See here for a great video on this: https://www.youtube.com/watch?v=WwslBPj8GgI
      
     Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?
      This is a provocative study, done over 15 years ago:  J. C. Wright, S. B. Millar, S. A. Kosciuk, D. L. Penberthy, P. H. Williams, and B. E. Wampold (1998) “A Novel Strategy for Assessing the Effects of Curriculum Reform on Student Competence,” Journal of Chemical Education, v. LXXV, pp. 986-992 (August).







Thursday, July 10, 2014

New Interview Series on Evaluation Coming Soon!

I am lining up a series of researchers, practitioners, and leaders to share their thoughts on various types of evaluation. The first series of interviews will focus on how one evaluates innovation in teaching, learning and/or research. So be sure to check back in a couple of weeks for the first post. I'll also be lining folks up for the blog in Madison at the 30th Annual Conference on Distance Teaching & Learning being held August 12-14 in Madison, Wisconsin. I'm presenting at the conference as well.

More about the Conference!

Sponsored by the University of Wisconsin-Madison, this event is a great place to hear leading experts and share best practices with colleagues from around the world in the field of online education and training. Plus, Madison is a beautiful place to visit in the summer. If you’re not familiar with this event, and want to find out more, I encourage you to visit their website at www.uwex.edu/disted/conference. Hope to see you there.

Saturday, May 10, 2014

Opportunity for Graduate Students with AEA

I still draw from my experiences as a graduate student participating in large-scale technology program evaluations. I could not function effectively in my current administrative role if I did not know how to use various types of evaluations to make decisions. Please encourage your graduate students to take advantage of opportunities such as this one by the American Evaluation Association.
CALL FOR APPLICATIONS
AEA GRADUATE EDUCATION DIVERSITY INTERNSHIP PROGRAM (GEDI) 
DEADLINE: Friday, June 6, 2014 
The American Evaluation Association welcomes applications for its Graduate Education Diversity Internship Program that provides paid internship and training opportunities during the academic year. The GEDI program works to engage and support students from groups traditionally under-represented in the field of evaluation. The goals of the GEDI Program are to:
  • Expand the pool of graduate students of color and from other under-represented groups who have extended their research capacities to evaluation.
  • Stimulate evaluation thinking concerning under-represented communities and culturally responsive evaluation.
  • Deepen the evaluation profession's capacity to work in racially, ethnically and culturally diverse settings.
Interns may come from a variety of disciplines, including public health, education, political science, anthropology, psychology, sociology, social work, and the natural sciences. Their commonality is a strong background in research skills, an interest in extending their capacities to the field of evaluation, and a commitment to thinking deeply about culturally responsive evaluation practice.

The Internship: Building on the training content described below, the interns work the equivalent of approximately two days per week at an internship site near their home institutions from approximately September 1 to July 1. The interns may work on a single evaluation project or multiple projects at the site, but all internship work is focused on building skills and confidence in real-world evaluation practices. Interns receive a stipend of $8,000 in recognition of their internship work based on completion of the internship and satisfactory finalization of program requirements, including any deliverables due to the host agency, progress reports, and reflections on the internship experience.

Training and Networking Components: It is assumed that students come to the program with basic qualitative and quantitative research skills. The GEDI Program then works to extend those skills to evaluation through multiple activities:
  
Fall Seminar. A five-day intensive seminar, held in Claremont, California, provides an orientation that expands the student's knowledge and understanding of critical issues in evaluation, including thinking about building evaluation capacities to work across cultures and diverse groups. The interns complete a self-assessment in the Fall, clarifying their own goals during program participation.

AEA Annual Conference. Interns will spend a week at the American Evaluation Association annual conference. While there, they attend  (a) pre-conference workshops selected to fill gaps in their knowledge and skills, (b) conference sessions exploring the breadth and depth of the field, and (c) multiple networking events to connect them with senior colleagues. The interns also conduct a small-service learning project in the form of an evaluation of one component of the conference.

Winter Seminar. A three-day seminar, held in January or February, provides the students with additional training, coaching on their evaluation projects, and panel discussions with evaluation practitioners working in a range of contexts.

Evaluation Project. Interns will have the opportunity to provide support to an agency's evaluation activities in close proximity to their graduate institution. Interns will provide three updates on their evaluation project activities as part of the internship program, describing and reflecting on the application of their evaluation knowledge to the actual project activities.

Monthly Webinars. The students gather each month for a two-hour webinar to check in on evaluation projects and site placements, add to existing skill-sets, and learn from invited guest speakers.

AEA/CDC Summer Evaluation Institute. The program ends with attendance at the Summer Evaluation Institute held in Atlanta each June. There, students once again connect and finalize project reporting, attend training workshops, and participate in a graduation ceremony.

Specific Support Mechanisms: Interns are supported by colleagues at school, at their site placements, and within the sponsoring association:

An Academic Advisor. The academic advisor at the Intern's home institution supports and coordinates coursework and other activities, while helping to integrate the internship program with the student's plan of study.

A Sponsoring Agency. Students generally are matched with sponsoring agencies near their graduate institution that provide the opportunity to perform evaluation activities compatible with students' research interests and skills.

Supervising Mentor. A colleague at the host site with evaluation experience acts as a guide and mentor throughout the program.

GEDI Program Leadership. GEDI Program Director and AEA President-Elect (2015) Dr. Stewart Donaldson is an experienced evaluator. Working with a cadre of colleagues, he, Co-Director Dr. Ashaki M. Jackson, and Program Liaison John LaVelle oversee the curriculum and site placements. Throughout the internship the leadership are available to guide, advise, and support the interns in achieving their professional goals and the goals of the program.

AEA Staff Support. AEA staff provides logistical support throughout the internship. Post-internship, they work to connect program graduates with opportunities for leadership, participation, and networking within the association.

Online Community. The GEDI cohort uses an online community space for checking in, turning in updates, asking questions, and informal networking.

Student Benefits: Interns receive support from advisors and mentors, quality training focused on evaluation, real-world work experience, registration waivers and guidance at two professional evaluation conferences, and multiple opportunities for professional networking. In recognition of the time involved in the program (approximately 2 days per week), each intern also receives a stipend and is reimbursed for major travel expenses related to the program (airfare and shared hotel specifically), but is responsible for travel incidentals (to and from home/airport, to/from hotels, meals not taken together, etc.).

Eligibility: We seek students who are not already enrolled in an evaluation program/specialization or pursuing an evaluation degree who:

  • Are enrolled in a masters or doctoral-level program in the United States and have completed the equivalent of one full year of graduate level coursework;
  • Are residing in the United States;
  • Have already been exposed to research methods and substantive issues in their field of expertise;
  • Demonstrate via written essays the relevance of evaluation training to their career plans and their commitment to culturally responsive practice;
  • Are eligible to work for pay in the United States outside of an academic environment (non-U.S. citizens will be asked to provide documentation of current eligibility); and
  • Have support from his/her academic advisor.
Criteria for Selection: The interns will be selected based on their completed applications, materials provided, and subsequent finalist interviews focusing on:
  • Their thinking around and commitment to culturally responsive evaluation practice;
  • The alignment between their skills, aspirations, locale, and internship site placement needs;
  • The quality of their academic, extracurricular, and personal experiences as preparation for GEDI; and
  • Their capacity to carry out and complete the program, including support from an academic advisor
To applyDownload the GEDI Application and return all requested materials via email as described on that document on or before Friday, June 6, 2014. Please note that it may take a few weeks to compile the requested information and thus we recommend that you begin as soon as possible before the deadline.

Questions: We recommend beginning by reviewing our Frequently Asked Questions (FAQ) page. Should you have further questions, please contact Program Liaison John LaVelle via email gedi@eval.org with questions about the program.

More about the program: Go to the GEDI homepage

Sunday, April 13, 2014

Awarding Badges of Honor


Last fall of 2013, Clif Mims asked members via Google Plus about their uses of badges. I posted that I would be experimenting with badges in my upcoming graduate course. This blog post is the first installment of my reporting on the badge experience. I am inviting others to share their stories. 
Image from Mozilla OpenBadges site
I'm teaching an instructional technology evaluation course in Moodle this spring. Among other things, this graduate education course is designed to help learners:
  • Generate a technology program evaluation proposal for stakeholders. 
  • Plan and manage technology program evaluations. 
  • Acquire and employ effective data collection instruments and techniques. 
  • Analyze and draw conclusions from evaluation data. 
  • Report the results of technology program evaluation.
My goal in implementing badges was to start to establish a means by which researchers at Lehigh would be able to view students' skills across several courses in order to help match students with various technology evaluation or research opportunities. I chose four skills related for which I would award badges. The badges were awarded for formulating research questions; choosing and utilizing reliable, valid instrumentation; sampling techniques; and formative evaluation design. It was important to me that learners had ample opportunities to demonstrate mastery of key concepts and the application of these skills. I had hoped to use Mozilla's Open Badges but didn't find getting started as straightforward as I had thought.
 With a busy schedule, I found it easier to slightly modify graphics to quickly design my own badges. In Moodle, I found that the automatic awarding of badges based on criteria didn't always work. We're still investigating why. In the future, I think I will opt for the manual awarding for all four.

The good news is that I was able to actually use the badge data when approached to recommend students for a software research study. Overall, I find badges worthwhile and plan to look into OpenBadges for my next iteration. The other important outcome is that the graduate students seemed genuinely interested in earning them and helping me implement them effectively.

I'd love to hear how others have implemented badges to help assemble project or research teams.