Evaluation & Innovation: Anissa's View

Thanks to an ITForum nomination, I introduce to some and present to others this month's post by 

Dr. Anissa Likey-Vega. As faculty at Kennesaw State University, Anissa serves as the coordinator for the Online Teaching Endorsement and Certificate programs for the Department of Instructional Technology.  She is also founder and lead consultant at Curriculum Research and Development (CR&D), an educational technology program evaluation firm that specializes in serving the K12 sector. She has her Ph.D. in Instructional Design and Technology from Georgia State University. Curriculum Research & Development: http://www.curriculumrd.com/

Department of Instructional Technology at Kennesaw State University http://bagwell.kennesaw.edu/departments/itec

How do we recognize innovation in teaching and learning with technology?

I’m sure you’ve heard this before: If you look for a problem, you will find one; however, near that problem you will also likely find innovation. Whether K12, higher education, or industry training, the teaching and learning process is in itself an ill-structured problem complicated by unique contexts, limited resources, and (those pesky) humans. People will be innovating to solve the problems that serve as the biggest barriers to them in their context. This doesn’t mean innovation will always work or will always be adopted to saturation, but each innovation will stir debate and nay-sayers. This tension occurs because once an innovation has proven itself effective enough to be widely adopted in the environment, it is no longer innovative. It becomes the new status quo.

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges?

Each evaluation is unique. The combination of context, time, problem, and innovation is unique every single time. Yes, there are general principles and models that can guide an evaluation plan; however, flexibility is critically important to the process and knowing when it is appropriate to change course can be challenging. Evaluation expertise is not always valued as much as it should be and this can result in evaluation outcomes that offer little direction to organizational managers. Practitioners should read up on evaluation practices in their fields and knowledgably look for evaluation expertise within their organization. If such expertise is not readily available, then it is wise to consider outsourcing the process. While it is not always possible, the best evaluations are pursued early in the innovation process, or maybe even before. This allows for the data to drive the change process and optimize the outcomes.

Can you point to some promising innovations in teaching and learning?

I’m probably a bit biased on this one since I coordinate the online teaching certificate at Kennesaw State, but I think blended and online learning practices and tools have the most potential to disrupt teaching and learning in a positive and meaningful way.  Of course, if you teach at a fully-online school, these tools and practices are now the status quo; however, many settings particularly in the K12 sector are not there, yet. Online instruction grants the learner more power in the differentiation of instruction and it puts a teacher’s instructional design on display for self-reflection. This self-review is not as easily achieved in a real-time unrecorded class session. Also, their best designs are more easily replicated when stored in a digital format rather than gone with the passing of time. This allows for teachers to put their best foot forward even on an off day.

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?

Well, this is a bit challenging given that evaluation is context and problem-type specific.  This means it would be difficult to curate a research journal on the topic of innovation evaluation alone.  The audience would likely be scattered in context and problem-type, resulting in few articles to warrant devoted readership for each individual. I read about evaluation in journals that cross all disciplines. In fact, when I wrote my dissertation there was no published method to evaluate one or more intended curricula. I had to develop that process on my own. That said, my evaluation-related books that are resting closest to me right now are also the ones that show the most wear. These are the works of Thomas Guskey and Joellen Killion. I have to modify their designs as expected for each situation, but they both promote excellent planning and structure in the evaluation process.  Each time I start to wrap my brain around a new program or innovation that I need to evaluate, their works serve as my evaluation muses.

Previous posts in the series:

Bryan Alexander's Perspective

Rick Shearer's Perspective

Michael Q. Patton's Perspective

Steve Ehrmann's Perspective

Evaluation & Innovation: Bryan's Perspective

Dr. Bryan Alexander

I met Bryan in Atlanta at a NITLE symposium in April of 2013, but as those who know Bryan understand, it seems like I've known him forever.  I am a regular reader of his posts and enjoy his sense of humor. He was the first to volunteer to help with this blog. Although I started this series with my Lehigh students and AECT in mind, I'd like to hear from my fellow evaluation instructors and professors about how this series helps you with your students. 
Bryan Alexander is a futurist, researcher, writer, speaker, consultant, and teacher, working in the field of how technology transforms education. He completed his English language and literature PhD at the University of Michigan in 1997, with a dissertation on doppelgangers in Romantic-era fiction and poetry. Learn more about this NITLE fellow here http://bryanalexander.org/bio/

How do we recognize innovation in teaching and learning with technology? 

Not well, and that's a problem.  Teaching in general is something difficult to track for a variety of reasons, including the way course management systems lock away classes from observers..  For now we hear about innovative teaching through publications (statistically very rare), social media (a bit better), and personal contact.  Social media may be the best route for discovery innovation, as its ease of use lets practitioners share thoughts, reactions, observations on the fly.

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges? 

There are many challenges, depending on the background of would-be evaluators.  Professional training can make it difficult to appreciate innovation, either when trying to perceive new work in someone else's field, or in seeking to understand changes to one's own.  Observers can also focus too much on the first iteration of a project, rather than looking to its development and maturation over time.  It is also important to distinguish between achievement and learning versus other student responses (i.e., approval of change).

Overcoming these challenges is implicit in each one - pay more attention to innovation over time, pay less attention to non-learning responses.  Additionally, evaluators would do well to use social media to develop their reflections.

Can you point to some promising innovations in teaching and learning? 

The flipped classroom is perhaps the most notable.  It depends on no specific technology (usually Web video or podcasting), and is clearly focused on improving learning through enhancing the classroom experience.

Open education is growing steadily, as the amount of open material builds and the number of practitioners increases.

Mobile learning continues to transform teaching and learning, rendering nearly all locations potential connections to vast amounts of content and collaboration.

Gaming offers potentially huge and deep gains for learning.

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case? 

Columbia's Teachers College keeps publishing useful material.

There are a good number of books on mobile technology worth reading, even as that tech's rapid advances dates them quickly: Rheingold's Smartmobs, Katz and Aakhus's Perpetual Contact, Ito'sPersonal, Pedestrian, Portable.  

Let me return to social media.  Readers should curate sources on Twitter, the blogosphere, podcasting, etc,, as innovators often inhabit this space to share their thoughts.

Previous posts in the series:

Rick Shearer's Perspective

Michael Q. Patton's Perspective

Steve Ehrmann's Perspective

Evaluation & Innovation: Rick's Perspective

Dr. Rick Shearer and I at DTL

    

     

      Penn State University has been a good PA neighbor and friend to me  and other practitioners in the distance education field. At the Distance  Teaching and Learning Conference, I ran into Rick Shearer and asked  him to lend his voice to our series. Rick is Director of Penn State's  World Campus Learning Design. Dr. Shearer has been involved in the  distance education field for over 25 years.

.  

       http://learningdesign.psu.edu/index.php/section/staff

      How do we recognize innovation in teaching and learning with technology?

I believe the best description of innovation in teaching and learning is related to the idea that innovation is a process where we use new technologies to solve existing problems in our learning environments. So in essence we likely have innovation happening everywhere and we want it to happen organically and not in a managed way. Thus instructional designers daily are using new tools to solve instructional problems in our online courses. What we need is a way to capture these innovations and better disseminate information so folks do not have to re-invent the wheel.

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges?

The biggest challenge is finding a way to pilot an innovation and then to also be able to walk away from it if it doesn’t work. Thus, how can you run small pilot studies on technology innovations in courses and then evaluate the effectiveness without getting into long-term contractual relationships with vendors. Often it is easy to get an innovative idea into a course, but much harder to stop using it if it does not meet the need in helping students reach the stated learning outcomes. Also, in piloting these innovations and technologies we must allow time over several semesters to evaluate the impact and we cannot get caught up in running out to adopt the next shinny object that has come along. We must take a more measured approach to our testing and evaluation of the innovations.

Can you point to some promising innovations in teaching and learning?

If we know of innovative solutions they probably are not innovative anymore, thus a better question may be about problems where we need technologies to address them.

Key areas where I see a lot of innovative approaches happening in the future will be around learning analytics (not predictive analytics) where we explore ideas of personalized learning paths and mastery-based approaches. What do these look like in today’s connected world, where we value the social aspect of learning.

Also, how will the distance education community move in order to provide verification of our students for the DOE and possibly for upcoming requirements of the reauthorization of the higher education act.

Another area that is already emerging, but needs more work is the integration of social presence type tools in our courses that are seamless, but also protect the privacy and rights of our students.

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?

Unfortunately there are few places for practitioners to publish their work. Most journals are research oriented and require full studies that are difficult for practicing IDs to take on.

Although AECT has recently provided a venue for practical type research/reports, we need more options available for these types of practical studies. 

Previous posts in the series:

Michael Q. Patton's Perspective

Steve Ehrmann's Perspective

Evaluation & Innovation: Michael's Perspective

I recently returned from my regular trip to the Distance Teaching and Learning Conference in Madison, WI. One of my highlights was a pre-conference workshop on developmental evaluation with Michael Q. Patton. I asked Michael to lend his voice to our series.

Michael Q. Patton, Ph.D.

Michael Quinn Patton is an independent evaluation consultant based in Minnesota. He is former President of the American Evaluation Association (AEA).  He has authored six evaluation books including a 4th edition of Utilization-Focused Evaluation and 3^rd edition of Qualitative Research and Evaluation Methods. He is recipient of the Alva and Gunnar Myrdal Award for "outstanding contributions to evaluation use and practice" and the Paul F. Lazarsfeld Award for lifetime contributions to evaluation theory, both from AEA.  His latest book is Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use.  
http://www.guilford.com/books/Developmental-Evaluation/Michael-Quinn-Patton/9781606238721

     

How do we recognize innovation in teaching and learning with technology?

Innovation is contextual not absolute.  What is accepted practice in one context may be different and innovative in another.  Thus, innovation is a socially constructed notion which has to be understood and interpreted within a context. 

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges?

The challenge is that traditional ways of doing evaluation (logic models, SMART goals, predetermined outcome indicators, mandated rubrics and metrics) inhibit innovation.  The solution is developmental evaluation which accompanies innovators on the emergent journey of innovation, adapting evaluation to emergent issues.  For details about developmental evaluation see: http://www.guilford.com/books/Developmental-Evaluation/Michael-Quinn-Patton/9781606238721

Can you point to some promising innovations in teaching and learning?

Digital portfolios for online learners to archive their learning and products of their learning. These are accessible to the learners on a cumulative, long-term basis and can be shared for special purposes (e.g., a job opportunity).

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?

Online learning, like all effective interventions, is most effective when geared to specific subgroups with identifiable needs.  For example, homeless youth have special needs.  Here is a study that identifies how to work effectively with homeless youth: http://www.terralunacollaborative.com/wp-content/uploads/2014/03/9-Evidence-Based-Principles-to-Help-Youth-Overcome-Homelessness-Webpublish.pdf

Previous Post in Series: Steve Ehrmann's Perspective

Evaluation & Innovation: Steve's Perspective

Over the next several weeks, I will be inviting several educational researchers and practitioners to offer their perspectives on the role of evaluation in teaching, learning, and technology innovation. Each is asked to freely respond to the same four questions. My hope is to foster discussions that enrich understandings, challenge assumptions, and strengthen processes for those who evaluate proposals or design studies. The goal is to compile a useful reference here.

Stephen C. Ehrmann, Ph.D.

Most recently, Stephen C. Ehrmann served as Vice Provost for Teaching & Learning at the George Washington University.  Previously he was founding Vice President of the Teaching, Learning, and Technology Group (TLT Group) and Director of the Flashlight Program for the Evaluation of Educational Uses of Technoloy; Senior Program Officer with the Annenberg/CPB Projects; Program Officer with the Fund for the Improvement of Postsecondary Education (FIPSE), and Director of Educational Research and Assistance at The Evergreen State College. He has a Ph.D. in Management and Higher Education from MIT.

Blog and bio: http://sehrmann.blogspot.com 


How do we recognize innovation in teaching and learning with technology?
It combines something unfamiliar with something promising. 

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges? 

• Evaluation is one of those things that everyone assumes he or she can do; the biggest challenge is not realizing there is history here – methods, previous findings, etc.
• Because innovation is by definition somewhat unfamiliar, to some degree you don’t know what you’re looking for.
• Those in charge of the evaluation are often advocates of the project. This role often leads them to counter-productive actions such as:

o   Delaying the creation of an evaluation plan until all the available money has been committed to other purposes and, if it’s a grant proposal, the proposal is due in a day or two.  Two major reasons why evaluation plans for grant-funded edtech innovations are often so weak.

o   Another reason: Delaying the evaluation until ‘all the kinks have been worked out’.  This means the evaluation isn’t being used to identify the kinks or to help figure out how to work them out.  It also often means that the evaluation is delayed until a time when no one remembers to do it, or has any commitment to do it.  The initiative can end (as an initiative) with stacks of data sitting somewhere, unanalyzed.

• Rapture of the technology: paying too much attention to the technology itself as though, by itself it drives change.  That’s almost never true, so techno-centric evaluations can easily produce puzzling, misleading results.  Imagine two institutions, each using a different brand of learning management system.  Institution A uses LMS-A, while B uses LMS-B, a competitor.  Institution A has a long tradition of learning communities, seminars, group projects while institution B, valuing access highly, historically has asked little group work of its commuting students and distant learners.  An evaluation is done of online communication among students – how frequent? How productive?  A beats B. A technocentric evaluation is designed on the assumption that it’s the technology that determines the use of the technology, so the author concludes that LMS-A is better for online learning communities than LMS-B.  

In an unpublished article, I’ve described 10 principles for doing a better job evaluating innovations in  education, especially innovations using technology. Briefly:

1. Above all, do no harm. (For example, don’t ask for data if you don’t have a firm idea of how you can satisfy your survey respondents and other informants that helping you was, indeed, worth their valuable time.)
2. Design the evaluation so that, no matter what you find, each stakeholder is somehow better off.
3. Be ready to compare apples with oranges because that kind of comparison is almost inevitable in a study of an educational innovation.
4. Focus on what people DO most often with the program (their ‘activities’) rather than looking only at the tools with which they do it.
5. Study why those people do those things in those ways.
6. Compare the program with the most valuable alternative, not with doing nothing.
7. Study costs, not just benefits.
8. Remember that education does not work like a well-oiled machine – doing the ‘same thing’ often does not produce the ‘same results.’
9. Recognize that different people using the program often have different motives, perceptions, circumstances, and, therefore, outcomes.  Realizing this leads to a dramatically different approach to assessment and evaluation – the students are doing different things (#4 above) because they have different reasons and different influences acting on them (#5 above). So the outcomes will differ from student to student.
10. Start now!  It’s never too early in the life of innovation, even going back to before it’s introduced, to do useful studies.

Can you point to some promising innovations in teaching and learning?

      The use of questions designed to make students think with polling techniques (such as clickers or cell phones), and peer instruction.  See here for a great video on this: https://www.youtube.com/watch?v=WwslBPj8GgI

      

     Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?

      This is a provocative study, done over 15 years ago:  J. C. Wright, S. B. Millar, S. A. Kosciuk, D. L. Penberthy, P. H. Williams, and B. E. Wampold (1998) “A Novel Strategy for Assessing the Effects of Curriculum Reform on Student Competence,” Journal of Chemical Education, v. LXXV, pp. 986-992 (August).