Translate

Saturday, September 13, 2014

Evaluation & Innovation: Rick's Perspective

Drs. Shearer and Amankwatia at DTL
Dr. Rick Shearer and I at DTL
    

     
      Penn State University has been a good PA neighbor and friend to me  and other practitioners in the distance education field. At the Distance  Teaching and Learning Conference, I ran into Rick Shearer and asked  him to lend his voice to our series. Rick is Director of Penn State's  World Campus Learning Design. Dr. Shearer has been involved in the  distance education field for over 25 years.
 

      How do we recognize innovation in teaching and learning with technology?
I believe the best description of innovation in teaching and learning is related to the idea that innovation is a process where we use new technologies to solve existing problems in our learning environments. So in essence we likely have innovation happening everywhere and we want it to happen organically and not in a managed way. Thus instructional designers daily are using new tools to solve instructional problems in our online courses. What we need is a way to capture these innovations and better disseminate information so folks do not have to re-invent the wheel.

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges?
The biggest challenge is finding a way to pilot an innovation and then to also be able to walk away from it if it doesn’t work. Thus, how can you run small pilot studies on technology innovations in courses and then evaluate the effectiveness without getting into long-term contractual relationships with vendors. Often it is easy to get an innovative idea into a course, but much harder to stop using it if it does not meet the need in helping students reach the stated learning outcomes. Also, in piloting these innovations and technologies we must allow time over several semesters to evaluate the impact and we cannot get caught up in running out to adopt the next shinny object that has come along. We must take a more measured approach to our testing and evaluation of the innovations.

Can you point to some promising innovations in teaching and learning?
If we know of innovative solutions they probably are not innovative anymore, thus a better question may be about problems where we need technologies to address them.

Key areas where I see a lot of innovative approaches happening in the future will be around learning analytics (not predictive analytics) where we explore ideas of personalized learning paths and mastery-based approaches. What do these look like in today’s connected world, where we value the social aspect of learning.

Also, how will the distance education community move in order to provide verification of our students for the DOE and possibly for upcoming requirements of the reauthorization of the higher education act.

Another area that is already emerging, but needs more work is the integration of social presence type tools in our courses that are seamless, but also protect the privacy and rights of our students.

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?
Unfortunately there are few places for practitioners to publish their work. Most journals are research oriented and require full studies that are difficult for practicing IDs to take on.
Although AECT has recently provided a venue for practical type research/reports, we need more options available for these types of practical studies. 

Previous posts in the series:

Friday, August 22, 2014

Evaluation & Innovation: Michael's Perspective

I recently returned from my regular trip to the Distance Teaching and Learning Conference in Madison, WI. One of my highlights was a pre-conference workshop on developmental evaluation with Michael Q. Patton. I asked Michael to lend his voice to our series.

I am the Evaluator- Patton
Michael Q. Patton, Ph.D.


Michael Quinn Patton is an independent evaluation consultant based in Minnesota. He is former President of the American Evaluation Association (AEA).  He has authored six evaluation books including a 4th edition of Utilization-Focused Evaluation and 3rd edition of Qualitative Research and Evaluation Methods. He is recipient of the Alva and Gunnar Myrdal Award for "outstanding contributions to evaluation use and practice" and the Paul F. Lazarsfeld Award for lifetime contributions to evaluation theory, both from AEA.  His latest book is Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use.  
http://www.guilford.com/books/Developmental-Evaluation/Michael-Quinn-Patton/9781606238721

     

How do we recognize innovation in teaching and learning with technology?
Innovation is contextual not absolute.  What is accepted practice in one context may be different and innovative in another.  Thus, innovation is a socially constructed notion which has to be understood and interpreted within a context. 


What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges?
The challenge is that traditional ways of doing evaluation (logic models, SMART goals, predetermined outcome indicators, mandated rubrics and metrics) inhibit innovation.  The solution is developmental evaluation which accompanies innovators on the emergent journey of innovation, adapting evaluation to emergent issues.  For details about developmental evaluation see: http://www.guilford.com/books/Developmental-Evaluation/Michael-Quinn-Patton/9781606238721

Can you point to some promising innovations in teaching and learning?
Digital portfolios for online learners to archive their learning and products of their learning. These are accessible to the learners on a cumulative, long-term basis and can be shared for special purposes (e.g., a job opportunity).

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?
Online learning, like all effective interventions, is most effective when geared to specific subgroups with identifiable needs.  For example, homeless youth have special needs.  Here is a study that identifies how to work effectively with homeless youth: http://www.terralunacollaborative.com/wp-content/uploads/2014/03/9-Evidence-Based-Principles-to-Help-Youth-Overcome-Homelessness-Webpublish.pdf

Previous Post in Series: Steve Ehrmann's Perspective

Thursday, July 24, 2014

Evaluation & Innovation: Steve's Perspective

Over the next several weeks, I will be inviting several educational researchers and practitioners to offer their perspectives on the role of evaluation in teaching, learning, and technology innovation. Each is asked to freely respond to the same four questions. My hope is to foster discussions that enrich understandings, challenge assumptions, and strengthen processes for those who evaluate proposals or design studies. The goal is to compile a useful reference here.

Steve Ehrmann photo
Stephen C. Ehrmann, Ph.D.
Most recently, Stephen C. Ehrmann served as Vice Provost for Teaching & Learning at the George Washington University.  Previously he was founding Vice President of the Teaching, Learning, and Technology Group (TLT Group) and Director of the Flashlight Program for the Evaluation of Educational Uses of Technoloy; Senior Program Officer with the Annenberg/CPB Projects; Program Officer with the Fund for the Improvement of Postsecondary Education (FIPSE), and Director of Educational Research and Assistance at The Evergreen State College. He has a Ph.D. in Management and Higher Education from MIT.

Blog and bio: http://sehrmann.blogspot.com 


How do we recognize innovation in teaching and learning with technology?
It combines something unfamiliar with something promising. 


What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges? 
  • Evaluation is one of those things that everyone assumes he or she can do; the biggest challenge is not realizing there is history here – methods, previous findings, etc.
  • Because innovation is by definition somewhat unfamiliar, to some degree you don’t know what you’re looking for.
  • Those in charge of the evaluation are often advocates of the project. This role often leads them to counter-productive actions such as:

o   Delaying the creation of an evaluation plan until all the available money has been committed to other purposes and, if it’s a grant proposal, the proposal is due in a day or two.  Two major reasons why evaluation plans for grant-funded edtech innovations are often so weak.
o   Another reason: Delaying the evaluation until ‘all the kinks have been worked out’.  This means the evaluation isn’t being used to identify the kinks or to help figure out how to work them out.  It also often means that the evaluation is delayed until a time when no one remembers to do it, or has any commitment to do it.  The initiative can end (as an initiative) with stacks of data sitting somewhere, unanalyzed.
  • Rapture of the technology: paying too much attention to the technology itself as though, by itself it drives change.  That’s almost never true, so techno-centric evaluations can easily produce puzzling, misleading results.  Imagine two institutions, each using a different brand of learning management system.  Institution A uses LMS-A, while B uses LMS-B, a competitor.  Institution A has a long tradition of learning communities, seminars, group projects while institution B, valuing access highly, historically has asked little group work of its commuting students and distant learners.  An evaluation is done of online communication among students – how frequent? How productive?  A beats B. A technocentric evaluation is designed on the assumption that it’s the technology that determines the use of the technology, so the author concludes that LMS-A is better for online learning communities than LMS-B.  

In an unpublished article, I’ve described 10 principles for doing a better job evaluating innovations in  education, especially innovations using technology. Briefly:
  1. Above all, do no harm. (For example, don’t ask for data if you don’t have a firm idea of how you can satisfy your survey respondents and other informants that helping you was, indeed, worth their valuable time.)
  2. Design the evaluation so that, no matter what you find, each stakeholder is somehow better off.
  3. Be ready to compare apples with oranges because that kind of comparison is almost inevitable in a study of an educational innovation.
  4. Focus on what people DO most often with the program (their ‘activities’) rather than looking only at the tools with which they do it.
  5. Study why those people do those things in those ways.
  6. Compare the program with the most valuable alternative, not with doing nothing.
  7. Study costs, not just benefits.
  8. Remember that education does not work like a well-oiled machine – doing the ‘same thing’ often does not produce the ‘same results.’
  9. Recognize that different people using the program often have different motives, perceptions, circumstances, and, therefore, outcomes.  Realizing this leads to a dramatically different approach to assessment and evaluation – the students are doing different things (#4 above) because they have different reasons and different influences acting on them (#5 above). So the outcomes will differ from student to student.
  10. Start now!  It’s never too early in the life of innovation, even going back to before it’s introduced, to do useful studies.
Can you point to some promising innovations in teaching and learning?
      The use of questions designed to make students think with polling techniques (such as clickers or cell phones), and peer instruction.  See here for a great video on this: https://www.youtube.com/watch?v=WwslBPj8GgI
      
     Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?
      This is a provocative study, done over 15 years ago:  J. C. Wright, S. B. Millar, S. A. Kosciuk, D. L. Penberthy, P. H. Williams, and B. E. Wampold (1998) “A Novel Strategy for Assessing the Effects of Curriculum Reform on Student Competence,” Journal of Chemical Education, v. LXXV, pp. 986-992 (August).







Thursday, July 10, 2014

New Interview Series on Evaluation Coming Soon!

I am lining up a series of researchers, practitioners, and leaders to share their thoughts on various types of evaluation. The first series of interviews will focus on how one evaluates innovation in teaching, learning and/or research. So be sure to check back in a couple of weeks for the first post. I'll also be lining folks up for the blog in Madison at the 30th Annual Conference on Distance Teaching & Learning being held August 12-14 in Madison, Wisconsin. I'm presenting at the conference as well.

More about the Conference!

Sponsored by the University of Wisconsin-Madison, this event is a great place to hear leading experts and share best practices with colleagues from around the world in the field of online education and training. Plus, Madison is a beautiful place to visit in the summer. If you’re not familiar with this event, and want to find out more, I encourage you to visit their website at www.uwex.edu/disted/conference. Hope to see you there.

Saturday, May 10, 2014

Opportunity for Graduate Students with AEA

I still draw from my experiences as a graduate student participating in large-scale technology program evaluations. I could not function effectively in my current administrative role if I did not know how to use various types of evaluations to make decisions. Please encourage your graduate students to take advantage of opportunities such as this one by the American Evaluation Association.
CALL FOR APPLICATIONS
AEA GRADUATE EDUCATION DIVERSITY INTERNSHIP PROGRAM (GEDI) 
DEADLINE: Friday, June 6, 2014 
The American Evaluation Association welcomes applications for its Graduate Education Diversity Internship Program that provides paid internship and training opportunities during the academic year. The GEDI program works to engage and support students from groups traditionally under-represented in the field of evaluation. The goals of the GEDI Program are to:
  • Expand the pool of graduate students of color and from other under-represented groups who have extended their research capacities to evaluation.
  • Stimulate evaluation thinking concerning under-represented communities and culturally responsive evaluation.
  • Deepen the evaluation profession's capacity to work in racially, ethnically and culturally diverse settings.
Interns may come from a variety of disciplines, including public health, education, political science, anthropology, psychology, sociology, social work, and the natural sciences. Their commonality is a strong background in research skills, an interest in extending their capacities to the field of evaluation, and a commitment to thinking deeply about culturally responsive evaluation practice.

The Internship: Building on the training content described below, the interns work the equivalent of approximately two days per week at an internship site near their home institutions from approximately September 1 to July 1. The interns may work on a single evaluation project or multiple projects at the site, but all internship work is focused on building skills and confidence in real-world evaluation practices. Interns receive a stipend of $8,000 in recognition of their internship work based on completion of the internship and satisfactory finalization of program requirements, including any deliverables due to the host agency, progress reports, and reflections on the internship experience.

Training and Networking Components: It is assumed that students come to the program with basic qualitative and quantitative research skills. The GEDI Program then works to extend those skills to evaluation through multiple activities:
  
Fall Seminar. A five-day intensive seminar, held in Claremont, California, provides an orientation that expands the student's knowledge and understanding of critical issues in evaluation, including thinking about building evaluation capacities to work across cultures and diverse groups. The interns complete a self-assessment in the Fall, clarifying their own goals during program participation.

AEA Annual Conference. Interns will spend a week at the American Evaluation Association annual conference. While there, they attend  (a) pre-conference workshops selected to fill gaps in their knowledge and skills, (b) conference sessions exploring the breadth and depth of the field, and (c) multiple networking events to connect them with senior colleagues. The interns also conduct a small-service learning project in the form of an evaluation of one component of the conference.

Winter Seminar. A three-day seminar, held in January or February, provides the students with additional training, coaching on their evaluation projects, and panel discussions with evaluation practitioners working in a range of contexts.

Evaluation Project. Interns will have the opportunity to provide support to an agency's evaluation activities in close proximity to their graduate institution. Interns will provide three updates on their evaluation project activities as part of the internship program, describing and reflecting on the application of their evaluation knowledge to the actual project activities.

Monthly Webinars. The students gather each month for a two-hour webinar to check in on evaluation projects and site placements, add to existing skill-sets, and learn from invited guest speakers.

AEA/CDC Summer Evaluation Institute. The program ends with attendance at the Summer Evaluation Institute held in Atlanta each June. There, students once again connect and finalize project reporting, attend training workshops, and participate in a graduation ceremony.

Specific Support Mechanisms: Interns are supported by colleagues at school, at their site placements, and within the sponsoring association:

An Academic Advisor. The academic advisor at the Intern's home institution supports and coordinates coursework and other activities, while helping to integrate the internship program with the student's plan of study.

A Sponsoring Agency. Students generally are matched with sponsoring agencies near their graduate institution that provide the opportunity to perform evaluation activities compatible with students' research interests and skills.

Supervising Mentor. A colleague at the host site with evaluation experience acts as a guide and mentor throughout the program.

GEDI Program Leadership. GEDI Program Director and AEA President-Elect (2015) Dr. Stewart Donaldson is an experienced evaluator. Working with a cadre of colleagues, he, Co-Director Dr. Ashaki M. Jackson, and Program Liaison John LaVelle oversee the curriculum and site placements. Throughout the internship the leadership are available to guide, advise, and support the interns in achieving their professional goals and the goals of the program.

AEA Staff Support. AEA staff provides logistical support throughout the internship. Post-internship, they work to connect program graduates with opportunities for leadership, participation, and networking within the association.

Online Community. The GEDI cohort uses an online community space for checking in, turning in updates, asking questions, and informal networking.

Student Benefits: Interns receive support from advisors and mentors, quality training focused on evaluation, real-world work experience, registration waivers and guidance at two professional evaluation conferences, and multiple opportunities for professional networking. In recognition of the time involved in the program (approximately 2 days per week), each intern also receives a stipend and is reimbursed for major travel expenses related to the program (airfare and shared hotel specifically), but is responsible for travel incidentals (to and from home/airport, to/from hotels, meals not taken together, etc.).

Eligibility: We seek students who are not already enrolled in an evaluation program/specialization or pursuing an evaluation degree who:

  • Are enrolled in a masters or doctoral-level program in the United States and have completed the equivalent of one full year of graduate level coursework;
  • Are residing in the United States;
  • Have already been exposed to research methods and substantive issues in their field of expertise;
  • Demonstrate via written essays the relevance of evaluation training to their career plans and their commitment to culturally responsive practice;
  • Are eligible to work for pay in the United States outside of an academic environment (non-U.S. citizens will be asked to provide documentation of current eligibility); and
  • Have support from his/her academic advisor.
Criteria for Selection: The interns will be selected based on their completed applications, materials provided, and subsequent finalist interviews focusing on:
  • Their thinking around and commitment to culturally responsive evaluation practice;
  • The alignment between their skills, aspirations, locale, and internship site placement needs;
  • The quality of their academic, extracurricular, and personal experiences as preparation for GEDI; and
  • Their capacity to carry out and complete the program, including support from an academic advisor
To applyDownload the GEDI Application and return all requested materials via email as described on that document on or before Friday, June 6, 2014. Please note that it may take a few weeks to compile the requested information and thus we recommend that you begin as soon as possible before the deadline.

Questions: We recommend beginning by reviewing our Frequently Asked Questions (FAQ) page. Should you have further questions, please contact Program Liaison John LaVelle via email gedi@eval.org with questions about the program.

More about the program: Go to the GEDI homepage