Crowdsourcing and Evaluation: Addressing Issues of Security

I look forward to talking to AECT colleagues on Thursday, 11/9/17 about accommodating crowdsourced data in social research and program evaluation. This conversation complements my future research agenda on using specialized crowds for e-learning evaluation.

Steve King aka Dr. Security

For issues about managing data and cybersecurity, I typically turn to the expertise of Steve King, Netswitch COO/CTO.  Learn more about Steve and read what I learned about security and engaging crowds in e-learning evaluation. Steve is happy to advise evaluators on issues of architectural requirements and cloud data security. Reach out to him on Twitter and LinkedIn @sking1145.

Platform Considerations for Human Intelligence Tasks

A secured Web conferencing platform or a MOOC separate from a school’s network should support crowdsourced evaluation activities. A secure web-conferencing platform that imposes a bunch of restrictions is necessary to assure there is no easy avenue to compromise crowd evaluation tasks.

Assuming the crowd will be operating on an external host (cloud provider like AWS or Azure), the platform ought to be able to offer several services. Cloud providers typically offer content monitoring services so that up and downloads can be erased/destroyed at discretion. They also offer meta-data backup of the event itself so key information can be retained without leaving the crowd initiator or evaluator vulnerable to actual content being compromised.

Data Processing Considerations

The cloud service provider should be able to process large volumes of high velocity, structured (spreadsheets, databases) and unstructured (videos, images, audio) information as well as secure all of it. Even if an evaluator choses to run evaluation tasks in a local private cloud, any provider will offer this level of security.

But it is CRUCIAL to separate the cloud from the school’s network so that the crowd members can’t gain access to the school’s information or administrative network. If an evaluator can only run off the school’s system/network itself, she will be forced to set-up a sub-net at least and buy some hardware which could get more expensive than running the whole thing in a cloud. I would NOT advise an evaluator to try the latter option.

Credentials Considerations

In order to have a clearly defined crowd, crowd initiators must assign every member a credential and use whatever vetting process the evaluator would normally use to determine their appropriateness for participation. A SSN or EIN number matched with key locator information or student ID that can be validated could work.

Other Key Platform, Data, and Credential Considerations:

The Federal Risk and Authorization Management Program (FedRAMP), while specifically related to government security requirements, provides compliance guidelines that will guarantee a secure web conferencing platform.

FedRAMP standards meet the baseline security controls set out by the National Institute of Standards.

A web conferencing platform should be FedRAMP compliant, and if not, should employ a layered security model of some sort with the following characteristics:

·         Gated access is one important characteristic. 

Gated access refers to the security options that manage entrance to and usage of virtual rooms employed by a web conferencing platform.  Gated access also helps prevent DDoS attacks

·         Platform, data, and credential restriction settings are important.

The ability to set restrictions on the hours a virtual room can be accessed minimizes the time in which sensitive and vulnerable information can be viewed and compromised. Crowdsource HITs initiators will want to be able to monitor remote users. The platform should allow the encryption of all information in transit. It should also allow session locks and the ability to terminate credentials so the crowdsource initiator can manage the platform access. Session locks allow evaluators to control who can enter a room at what time. Credential termination should be both manual and automatic based on a member leaving a room or meeting space where their credentials will no longer work for re-entry without a new login.

The platform should also let evaluators encrypt the event recordings, both at rest and in transit so that when a crowd initiator shares the contents with evaluation members unable to participate synchronously, only authorized members will have the encryption keys that will allow them access.

Initiators will want to be able to control and define roles and access privileges for the crowd which will establish the specific conditions by which a member can interact with a group or room. Role-based access control and dynamic privilege management are keys to this next layer. It is the evaluator’s ultimate control over who gets to enter which rooms, so she can decide that a member who needs to share information with a key group but should not be allowed direct access to that group can be assigned to a sub-conference room where a primary conference member can meet them, gain the information and return to the primary meeting.

Dynamic privilege management allows an evaluator to enable the retention of a member’s virtual identity while suspending their access privileges, so a member could have their privileges upgraded temporarily for a one-time event and then returned to their prior status. This could also facilitate evaluation requirements for working on individual or group e-learning tasks, and protecting small human intelligence tasks or HITs.

The conferencing platform should also allow a way to pair a person with unique authenticators that can customize their privileges.  This is done through individual access codes which are essentially the member’s fingerprints. Depending upon the privileges granted, individual access controls keep track of access rules and determine which sessions each member will be allowed to enter. This comes in handy for the identification of suspects following an information leak or inappropriate sharing of sensitive material.

Is there a Viable Role for Crowdsourced Data in Evaluation Studies? Steve’s Thoughts

Dr. Steve Ehrmann

I’m starting to engage researchers in a conversation about crowdsourced data. In more formal ways, I started with the publication of a chapter in Revolutionizing Modern Education through Meaningful E-learning (Khan, B. ed., 2016). I will continue in Jacksonville, at a Research & Theory AECT round table. Over the next four months, I’ll also post the thoughts of various instructional design or distance learning researchers, leaders, and practitioners on the potential role in crowdsourced data. We’ll look at e-learning, innovation, and technology program evaluations in particular. There may be some who have some experience and feedback. I’m looking forward to the exchanges.

For our first installment, we’ll hear from Dr. Steve Ehrmann. Steve and I met when he was at George Washington University and later we exchanged ideas around a couple of redesign initiatives that started with his time at the Kirwan Center for Academic Innovation at the University System of Maryland and continue until today.

Steve shared thoughts below about how we might approach crowdsourcing and innovation. First take a look at his unpublished paper attached here and his description of the uniform impact and unique uses perspectives (pp. 9-11).  

The uniform impact perspective focuses on the same outcome for each user of the program (how well did they learn X, for example).  In contrast, the unique uses perspective assumes that each user may interpret the program differently, use it differently, and experience different results; this perspective assumes that results differ qualitatively from one user to the next. 

Crowdsourcing would be especially appropriate for a unique uses study of an innovation.  An evaluator or researcher might take the following steps:

1. Identify a crowd of users and figure out how to reach them online. 

2. Explain to them why it's in their interest to contribute their time to your inquiry (i.e., to respond thoughtfully to your message).  Intrinsically motivating your crowd produces more valid feedback than extrinsic rewards (e.g., entering them in a lottery if they contribute.) 

3. Ask each user to consider what's been most valuable for them about their use of the program.  What's been most burdensome, frustrating, or limiting about their use?  Explain that you need the crowd to produce a responses that are (a) each quite important to the person involved and (b) qualitatively different from other benefits and problems suggested by others.  (In 1990, if you were studying uses of word processing with personal computers, the first answers might have to do with the benefits of multiple fonts and the ease of editing. But, after enough brainstorming, eventually someone might mention that their whole approach to rethinking has changed because rewriting is so easy.)

4.  Starting with these two long lists, begin a second round of inquiry about each item. Are there patterns? Are there connections? Do they suggest new ways of understanding the program itself?

The closest I’ve seen to this approach occurred in a face-to-face discussion of perhaps 20 first generation users of an educational innovation: the use of chat rooms by students in f2f composition courses (as I recall, this was in the early 1990s). The classes met in computer labs. Instead of talking, faculty and students would type to one another.  A couple of months into the first term, faculty from perhaps ten institutions met to discuss what was happening in their courses. The first 45 minutes focused on technical issues and on what the faculty liked.  Then one faculty member, ashamed, admitted that students in his course had erupted into a barrage of profanity and obscenity.  A long pause. Then two or three others said something similar had happened to them, too.  Cutting to the end of the discussion, one faculty member remarked, “Think about the French revolution. Think about what happens when powerless people get power. Some windows get broken. But they’re investing energy into writing. That’s what you need most in a writing class. So the trick is not to crush the revolution but to figure out how to channel the energy!”  Later conversations revealed two additional, different ways to interpret this innovation, each with different insights about how to make more intentional, effective use of such chat rooms. 

Michael Scriven calls this a goal-free evaluation. What I’d emphasize is creating a process through which users can provide quite different pictures of what the program is for, how it can be used, what benefits can be created and at what cost.   Those perspectives may be incompatible with each other and at least some may come as a surprise to the people who created the program.

Thank you, Steve, for starting off our new discussion!

Fall Series: Evaluation and the Wisdom of the Crowd

New opportunities exist for including stakeholders' and others' input in evaluation research. Can you imagine the nature of the improvements we might front load in e-learning and instructional designs if we're able to incorporate qualitative data seamlessly into the decision making process? I'd like to explore those possibilities this fall. 

Crowdsourcing involves gathering and processing large volumes of high velocity, structured (spreadsheets, databases) and unstructured (videos, images, audio) information. Qualitative research also involves large data sets.  We can now quickly process large data sets in ways we could not in the past. For example, some helpful applications for possible qualitative research and e-learning program evaluation include “big data” techniques for information retrieval (IR), audio analytics, and video analytics (Gandomi & Haider, 2014). The techniques to acquire, clean, aggregate, represent, and analyze data are many and help justify the re-conceptualization of our evaluation paradigms and models--especially our interpretivist and postmodernist ones. Should we use crowdsourced data? If so, under what conditions given some of the challenges of internet security and crowds. I've done some research and thinking about this over the last year. Let's talk.

Defining and Framing

Estelles-Arolas & González-Ladrón-de-Guevara’s 2012 literature review on an integrated definition of crowdsourcing included 10 with a problem-solving purpose. Their proposed definition is as follows: 

Crowdsourcing is a type of participative online activity in which an individual, an institution, a non-profit organization, or company proposes to a group of individuals of varying knowledge, heterogeneity, and number,via a flexible open call, the voluntary undertaking of a task.These should be useful to our discussion since the use of evaluation data in this context facilitates problem solving.

Crowdsourcing can change the way evaluators work, giving them access to just-in-time assistance in performing evaluation tasks. Researchers and practitioners will need more information to help determine if existing paradigms, approaches, and methods should be reassessed to accommodate crowdsourcing. Evaluators should also think carefully about what tasks might be appropriate for various program evaluation approaches, given some of the problems with crowdsourcing.

Additional questions to frame this blog feature: Does e-learning program evaluation need to develop its own definition of crowdsourcing, or merely validate Estelles-Arolas and González-Ladrón-de-Guevara’s (2012) definition? Do members of the American Evaluation Association task force want to consider conducting a needs-assessment for whether crowdsourced input will impact its guiding principles for evaluators (American Evaluation Association, 1995)?

Sources cited:

Estelles Arolas, E, & González-Ladrón-de-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information Science, 38(2), 189-200.

Gandomi, A., & Haider, M. (2014). Beyond the hype: Big data concepts, methods, and analytics. International Journal of Information Management, 137-144.

Evaluating Innovation and the Wisdom of the Crowd

I've had the privilege of collecting the thoughts from various people on this blog about how we might approach evaluating innovation. Many regard e-learning as disruptive with the potential to innovate teaching and learning. This year, I will start a new series on the potential role of crowd sourced data in evaluation: innovation, instructional design, e-learning, and technology program. As before, I'm looking for candidates to interview over the next five to six months. Please send me nominations.

In the meantime, I'm looking forward to a discussion this fall with NATO E-Learning (August) and AECT conference (November) goers. I posit that there may be opportunities for crowd data to inform our instructional designs. The wisdom of a defined crowd can be beneficial during instructional design and redesign processes. For organizations such as NATO that has tremendous human capital, the crowd can help its members solve their unique design problems and make decisions about the unknown or unfamiliar in ways essential to their goals of promoting stability, security, and prosperity. I maintain Given the complexity of developing programs, services, policies, and support for e-learning, leaders may find it challenging to regularly evaluate programs to improve quality. It's worth a conversation in a world of lifelong learners and MOOCs. 

Do you agree? Let's talk in Jacksonville, AECT. Others, let's talk here. Read more about my thoughts on the topic in my latest book chapter Massive Open Program Evaluation: Crowdsourcing’s Potential to Improve E-Learning Quality in the book pictured below.

Program Evaluation Resources

I haven't fallen off of the face of the evaluation map or of social media. My activity level needed to decrease. Not only are we finalizing the details of our move from PA to VA, but I've been working also in the building you see in this picture. What is she doing? I'm writing a book chapter: Massive Open Evaluation: The Potential Role of Crowd Sourced Input to Improve E-learning.

During this journey, I've come across several valuable resources. Two were passed on to me by Tom Reeves, PhD Professor Emeritus of Learning, Design, and Technology, UGA:

http://www.sagepub.com/upm-data/3988_Chapter_1_overview.pdf 

http://www.evaluativethinking.org/docs/SEC_Session1/HistoryofEvaluation.pdf​ 

Here's Anita Baker's Evaluation Services website. It has excellent resources and tools.

http://www.evaluationservices.co/Home_Page.html 

Have a great summer!

  

Characteristics of an Effective Evaluator

                  Who wants to be an evaluator?

I'm fond of the saying, never take advice from someone you're not willing to trade places with in that area. Some of Whom would you consult before your next evaluation project?

Why would you take technology program or e-learning evaluation advice from this source? What characteristics makes that person or that evaluation resource valuable to you?

Share your thoughts here. 

Evaluation & Innovation: Ward's View

Dr. Ward Mitchell Cates

To wrap up the series, I'm starting 2015 with an interview with one of my favorite instructional design thinkers and my personal evaluation mentor, Ward M. Cates. As you probably guessed, our connection started at LehighU. Let's see what he has to say about innovation and evaluation. Looking forward to some thoughtful blog interactions. I know instructional designers and graduate students are pros at blog interactions. Right AECT graduate student assembly?

Dr. Cates is a past International President of the Association for Educational Communications and Technology, which awarded him their Distinguished Service Award in 2008, and is the current President of the ECT Foundation Board, the non-profit wing of AECT. He has just stepped down after reviewing for ETR&D for 20 years.  His research focuses on instructional and interface design for educational software.  Longtime associate dean for the College of Education at Lehigh University, Ward is just completing his 40^th year as a tenure track faculty member.

For more info, visit www.lehigh.edu/~wmc0

How do we recognize innovation in teaching and learning with technology?

“Innovation” simply means, “new, not previously tried.”  When we talk about “innovation in teaching and learning,” we typically mean more than that, however.  We use the phrase to mean trying new delivery systems, new instructional and learning methods and assessments and new approaches to learners that are both new and effective. Innovation without success is seldom worth talking about, except for purposes of case learning.  Thus, what we really mean is (effective) innovation.  I think the only real way to recognize (effective) innovation is to look at outcomes.  That is, (effective) innovation produces enhanced outcomes that take the learner closer to what is desired, both by the instructor and the learner.  Technology is the tool here, and using technology to do things that either have been done before or have failed to be effective is not (effective) innovation.  Technology offers affordances and opportunities.  If, however, the design of the teaching and learning episode(s) is not, by itself innovative, adding technology will not make it so.  (Effective) innovation in teaching and learning with technology seems to be about the confluence of the innovation of the design with the effects of the opportunities and affordance of the technology employed to produce enhanced outcomes.

How we evaluate outcomes has to be based on what is desired by both the instructor and the learner.  Thus, we must begin with what each seeks and how the design and delivery help attain the desired outcomes. Secondarily, we should consider cost-effectiveness.  That is, how has the (effective) innovation increased scalability or reduced staffing or affected other similar financial limitations that hamper current practice.  But we must be cautious not to sacrifice desired outcomes to cost analyses.  Cost-effective, but lower-expectations, learning can sacrifice teaching and learning on a golden altar..

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges?

The greatest threat, it seems to me, is that one might mistake the creativity and power of the technology for an innovation in teaching and learning.  We technophiles tend to be easily enthralled by new technologies’ ability to do things we could not have even imagined a few years earlier.  Thus, we can be wowed by the flash of the technology and not recognize that a technology actually enables nothing that has not been done before the introduction of that technology, or it simply adds a new toy without enhancing outcomes meaningfully.  

“Meaningful” is a key criterion here.  I often cite what I call the technology coefficient.  It is a fraction with the numerator made up of the need for the new technology in terms of gains in teaching and learning and the denominator made up with how much the technology costs to implement and how difficult it is to learn and use. When (need/gain) is great and (cost/difficulty) is low, the resultant coefficient is large and the use of the technology is likely to be warranted.  When the inverse is true, the coefficient is small and the “true cost” of the use of the technology exceeds its meaningful contribution.  Think of the coefficient this way: Educational need over technology.

The way to overcome this threat is to focus on outcomes and not technology.  One should look for what an (effective) innovation contributes and why.  If one cannot explain WHY something is an innovation, one might be led to suspect the influence of the novelty effect.  Further, we need to avoid being persuaded by champions or affiliating too strongly with the technology.  Focus on design and desired outcomes.  Even if the innovation produces new and interesting outcomes, we need to remember that we seek not just new outcomes, but DESIRED outcomes.

Can you point to some promising innovations in teaching and learning?

I am encouraged by the focus on depth versus breadth and by the periodic re-emphasis on “critical thinking.”  I am not sure these are innovations but they represent foci that could lead to (effective) innovations.  I fear the dictatorship of standardized testing acts to kill such innovation, however.  While I am not in any way opposed to standardized testing, I have little regard for testing that measures low-level learning, memorization and simple calculation.  If testing is to be a major measure of outcome, it must measure something we desire to see accomplished.

I recall something I overhead over the fence in my yard many, many years ago.  A 12-year old neighbor was talking with her 8-year-old brother about a standardized test they had both taken that day in school.  The younger brother asked his sister what she had answered to the question, “How many feet are in a yard?”  She told him, “3.” He then said, “Darn, I missed it.  I wrote, ‘Depends on how many animals are standing in the yard.’”  So, was he wrong?  Was her answer a better answer?  It seems to me, we need to decide what the desired outcome is and to design tests that allow us to measure those outcomes, not just the ones that are easy to ask and easy to score.

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?

I am disinclined to name some initiatives/studies as “effective” and, by omission, imply others are not.  Let me suggest instead that we not become so caught up in “presentism” that we fail to examine and explore the research done over the last 50+ years.  I constantly encounter researchers who state as their topic of research something like “online video” but clearly have done no reading in the long history of research on related areas, such as film use.  The medium/delivery method is less important than the affordances.  We need to consider broadly, rather than narrowly define our topic to minimize what we need to read.  Much excellent research has been done over the years, not just in the last 10 years.  I suspect “presentism” is aided and abetted by online searching.  Sometimes, one just has to go into the stacks and handle a paper source.  Not everything worth reading has been scanned or converted.

Lastly, I think reading broadly to get context is crucial to true scholarship.  An evaluator/researcher needs to have a grasp on a wide range of literature and previous work. Each thing one reads adds to one’s understanding and may enhance one’s arsenal of evaluation weapons.

Previous posts in the series:

Dirk Ifenthaler's Perspective

Anissa Likey-Vega's Perspective

Bryan Alexander's Perspective

Rick Shearer's Perspective

Michael Q. Patton's Perspective

Steve Ehrmann's Perspective

Evaluation & Innovation: Dirk's View

The last post for 2014 belongs to Dirk Ifenthaler. Dirk and I ran into each other in Jacksonville, FL at AECT. In the midst of a hectic schedule, he took the time to share his thoughts. Although he would have liked to provide more insights, he says he does have many arguments ready for us, as necessary. Ready to hear from Dirk?

Dr. Dirk Ifenthaler

Dirk Ifenthaler’s previous roles include Professor and Director, Centre for Research in Digital Learning at Deakin University, Australia, Manager of Applied Research and Learning Analytics at Open Universities Australia, Professor for Education and Interim Department Chair at the University of Mannheim, Germany.

He was a 2012 Fulbright Scholar-in-Residence at the Jeannine Rainbolt College of Education, at the University of Oklahoma, USA. Dirk’s research outcomes include numerous co-authored books, book series, book chapters, journal articles, and international conference papers as well as successful grant funding in Australia, Germany, and USA . He is the Editor-in-Chief of the Springer journal Technology, Knowledge and Learning.
Website: http://www.ifenthaler.info
Technology, Knowledge and Learning: http://www.springer.com/10758/

How do we recognize innovation in teaching and learning with technology?
The major lesson to be learned from fifty years of research in the area of teaching and learning with technology is that what happens in learning environments is quite complex and multi-faceted. Students cannot learn from technology, rather, technology is a vehicle to support the processes of learning.
Due to the often very slow publication process, reports and results of innovative projects are published years later which does not facilitate the often discussed link between research and practice. The recent activities of researchers in social media help to disseminate some aspects of innovation, however, important results are often held back for publications which again slows down the innovation process.

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges?
There is a clear lack of empirical evidence focusing on innovation in teaching and learning with technology. We need to conduct multidisciplinary and multimethods research within innovative learning environments involving researchers, education providers, and international institutions. Clearly, practitioners and researchers need to talk to each other. One place is AECT (www.aect.org) which brings together these two groups at their annual International Convention.

Can you point to some promising innovations in teaching and learning?
The current debate focusing on learning analytics is very promising. However, most often learning analytics is misunderstood as being a tool for reporting educational data. The real innovation of learning analytics is the real-time support and feedback for all stakeholders involved in teaching and learning with technology. This will result in facilitation of learning processes, optimization of teaching practices and materials, as well as improved curricular designs and administrative processes.

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?
Check out Technology, Knowledge and Learning (www.springer.com/10758). There, I am supporting Work-in-Progress Studies and Emerging Technology Reports. These two publication types shall help to disseminate results from innovative projects already in their early phase.

Previous posts in the series:

Anissa Likey-Vega's Perspective

Bryan Alexander's Perspective

Rick Shearer's Perspective

Michael Q. Patton's Perspective

Steve Ehrmann's Perspective

Evaluation & Innovation: Anissa's View

Thanks to an ITForum nomination, I introduce to some and present to others this month's post by 

Dr. Anissa Likey-Vega. As faculty at Kennesaw State University, Anissa serves as the coordinator for the Online Teaching Endorsement and Certificate programs for the Department of Instructional Technology.  She is also founder and lead consultant at Curriculum Research and Development (CR&D), an educational technology program evaluation firm that specializes in serving the K12 sector. She has her Ph.D. in Instructional Design and Technology from Georgia State University. Curriculum Research & Development: http://www.curriculumrd.com/

Department of Instructional Technology at Kennesaw State University http://bagwell.kennesaw.edu/departments/itec

How do we recognize innovation in teaching and learning with technology?

I’m sure you’ve heard this before: If you look for a problem, you will find one; however, near that problem you will also likely find innovation. Whether K12, higher education, or industry training, the teaching and learning process is in itself an ill-structured problem complicated by unique contexts, limited resources, and (those pesky) humans. People will be innovating to solve the problems that serve as the biggest barriers to them in their context. This doesn’t mean innovation will always work or will always be adopted to saturation, but each innovation will stir debate and nay-sayers. This tension occurs because once an innovation has proven itself effective enough to be widely adopted in the environment, it is no longer innovative. It becomes the new status quo.

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges?

Each evaluation is unique. The combination of context, time, problem, and innovation is unique every single time. Yes, there are general principles and models that can guide an evaluation plan; however, flexibility is critically important to the process and knowing when it is appropriate to change course can be challenging. Evaluation expertise is not always valued as much as it should be and this can result in evaluation outcomes that offer little direction to organizational managers. Practitioners should read up on evaluation practices in their fields and knowledgably look for evaluation expertise within their organization. If such expertise is not readily available, then it is wise to consider outsourcing the process. While it is not always possible, the best evaluations are pursued early in the innovation process, or maybe even before. This allows for the data to drive the change process and optimize the outcomes.

Can you point to some promising innovations in teaching and learning?

I’m probably a bit biased on this one since I coordinate the online teaching certificate at Kennesaw State, but I think blended and online learning practices and tools have the most potential to disrupt teaching and learning in a positive and meaningful way.  Of course, if you teach at a fully-online school, these tools and practices are now the status quo; however, many settings particularly in the K12 sector are not there, yet. Online instruction grants the learner more power in the differentiation of instruction and it puts a teacher’s instructional design on display for self-reflection. This self-review is not as easily achieved in a real-time unrecorded class session. Also, their best designs are more easily replicated when stored in a digital format rather than gone with the passing of time. This allows for teachers to put their best foot forward even on an off day.

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?

Well, this is a bit challenging given that evaluation is context and problem-type specific.  This means it would be difficult to curate a research journal on the topic of innovation evaluation alone.  The audience would likely be scattered in context and problem-type, resulting in few articles to warrant devoted readership for each individual. I read about evaluation in journals that cross all disciplines. In fact, when I wrote my dissertation there was no published method to evaluate one or more intended curricula. I had to develop that process on my own. That said, my evaluation-related books that are resting closest to me right now are also the ones that show the most wear. These are the works of Thomas Guskey and Joellen Killion. I have to modify their designs as expected for each situation, but they both promote excellent planning and structure in the evaluation process.  Each time I start to wrap my brain around a new program or innovation that I need to evaluate, their works serve as my evaluation muses.

Previous posts in the series:

Bryan Alexander's Perspective

Rick Shearer's Perspective

Michael Q. Patton's Perspective

Steve Ehrmann's Perspective

Evaluation & Innovation: Bryan's Perspective

Dr. Bryan Alexander

I met Bryan in Atlanta at a NITLE symposium in April of 2013, but as those who know Bryan understand, it seems like I've known him forever.  I am a regular reader of his posts and enjoy his sense of humor. He was the first to volunteer to help with this blog. Although I started this series with my Lehigh students and AECT in mind, I'd like to hear from my fellow evaluation instructors and professors about how this series helps you with your students. 
Bryan Alexander is a futurist, researcher, writer, speaker, consultant, and teacher, working in the field of how technology transforms education. He completed his English language and literature PhD at the University of Michigan in 1997, with a dissertation on doppelgangers in Romantic-era fiction and poetry. Learn more about this NITLE fellow here http://bryanalexander.org/bio/

How do we recognize innovation in teaching and learning with technology? 

Not well, and that's a problem.  Teaching in general is something difficult to track for a variety of reasons, including the way course management systems lock away classes from observers..  For now we hear about innovative teaching through publications (statistically very rare), social media (a bit better), and personal contact.  Social media may be the best route for discovery innovation, as its ease of use lets practitioners share thoughts, reactions, observations on the fly.

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges? 

There are many challenges, depending on the background of would-be evaluators.  Professional training can make it difficult to appreciate innovation, either when trying to perceive new work in someone else's field, or in seeking to understand changes to one's own.  Observers can also focus too much on the first iteration of a project, rather than looking to its development and maturation over time.  It is also important to distinguish between achievement and learning versus other student responses (i.e., approval of change).

Overcoming these challenges is implicit in each one - pay more attention to innovation over time, pay less attention to non-learning responses.  Additionally, evaluators would do well to use social media to develop their reflections.

Can you point to some promising innovations in teaching and learning? 

The flipped classroom is perhaps the most notable.  It depends on no specific technology (usually Web video or podcasting), and is clearly focused on improving learning through enhancing the classroom experience.

Open education is growing steadily, as the amount of open material builds and the number of practitioners increases.

Mobile learning continues to transform teaching and learning, rendering nearly all locations potential connections to vast amounts of content and collaboration.

Gaming offers potentially huge and deep gains for learning.

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case? 

Columbia's Teachers College keeps publishing useful material.

There are a good number of books on mobile technology worth reading, even as that tech's rapid advances dates them quickly: Rheingold's Smartmobs, Katz and Aakhus's Perpetual Contact, Ito'sPersonal, Pedestrian, Portable.  

Let me return to social media.  Readers should curate sources on Twitter, the blogosphere, podcasting, etc,, as innovators often inhabit this space to share their thoughts.

Previous posts in the series:

Rick Shearer's Perspective

Michael Q. Patton's Perspective

Steve Ehrmann's Perspective

Evaluation & Innovation: Rick's Perspective

Dr. Rick Shearer and I at DTL

    

     

      Penn State University has been a good PA neighbor and friend to me  and other practitioners in the distance education field. At the Distance  Teaching and Learning Conference, I ran into Rick Shearer and asked  him to lend his voice to our series. Rick is Director of Penn State's  World Campus Learning Design. Dr. Shearer has been involved in the  distance education field for over 25 years.

.  

       http://learningdesign.psu.edu/index.php/section/staff

      How do we recognize innovation in teaching and learning with technology?

I believe the best description of innovation in teaching and learning is related to the idea that innovation is a process where we use new technologies to solve existing problems in our learning environments. So in essence we likely have innovation happening everywhere and we want it to happen organically and not in a managed way. Thus instructional designers daily are using new tools to solve instructional problems in our online courses. What we need is a way to capture these innovations and better disseminate information so folks do not have to re-invent the wheel.

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges?

The biggest challenge is finding a way to pilot an innovation and then to also be able to walk away from it if it doesn’t work. Thus, how can you run small pilot studies on technology innovations in courses and then evaluate the effectiveness without getting into long-term contractual relationships with vendors. Often it is easy to get an innovative idea into a course, but much harder to stop using it if it does not meet the need in helping students reach the stated learning outcomes. Also, in piloting these innovations and technologies we must allow time over several semesters to evaluate the impact and we cannot get caught up in running out to adopt the next shinny object that has come along. We must take a more measured approach to our testing and evaluation of the innovations.

Can you point to some promising innovations in teaching and learning?

If we know of innovative solutions they probably are not innovative anymore, thus a better question may be about problems where we need technologies to address them.

Key areas where I see a lot of innovative approaches happening in the future will be around learning analytics (not predictive analytics) where we explore ideas of personalized learning paths and mastery-based approaches. What do these look like in today’s connected world, where we value the social aspect of learning.

Also, how will the distance education community move in order to provide verification of our students for the DOE and possibly for upcoming requirements of the reauthorization of the higher education act.

Another area that is already emerging, but needs more work is the integration of social presence type tools in our courses that are seamless, but also protect the privacy and rights of our students.

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?

Unfortunately there are few places for practitioners to publish their work. Most journals are research oriented and require full studies that are difficult for practicing IDs to take on.

Although AECT has recently provided a venue for practical type research/reports, we need more options available for these types of practical studies. 

Previous posts in the series:

Michael Q. Patton's Perspective

Steve Ehrmann's Perspective

Evaluation & Innovation: Michael's Perspective

I recently returned from my regular trip to the Distance Teaching and Learning Conference in Madison, WI. One of my highlights was a pre-conference workshop on developmental evaluation with Michael Q. Patton. I asked Michael to lend his voice to our series.

Michael Q. Patton, Ph.D.

Michael Quinn Patton is an independent evaluation consultant based in Minnesota. He is former President of the American Evaluation Association (AEA).  He has authored six evaluation books including a 4th edition of Utilization-Focused Evaluation and 3^rd edition of Qualitative Research and Evaluation Methods. He is recipient of the Alva and Gunnar Myrdal Award for "outstanding contributions to evaluation use and practice" and the Paul F. Lazarsfeld Award for lifetime contributions to evaluation theory, both from AEA.  His latest book is Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use.  
http://www.guilford.com/books/Developmental-Evaluation/Michael-Quinn-Patton/9781606238721

     

How do we recognize innovation in teaching and learning with technology?

Innovation is contextual not absolute.  What is accepted practice in one context may be different and innovative in another.  Thus, innovation is a socially constructed notion which has to be understood and interpreted within a context. 

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges?

The challenge is that traditional ways of doing evaluation (logic models, SMART goals, predetermined outcome indicators, mandated rubrics and metrics) inhibit innovation.  The solution is developmental evaluation which accompanies innovators on the emergent journey of innovation, adapting evaluation to emergent issues.  For details about developmental evaluation see: http://www.guilford.com/books/Developmental-Evaluation/Michael-Quinn-Patton/9781606238721

Can you point to some promising innovations in teaching and learning?

Digital portfolios for online learners to archive their learning and products of their learning. These are accessible to the learners on a cumulative, long-term basis and can be shared for special purposes (e.g., a job opportunity).

Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?

Online learning, like all effective interventions, is most effective when geared to specific subgroups with identifiable needs.  For example, homeless youth have special needs.  Here is a study that identifies how to work effectively with homeless youth: http://www.terralunacollaborative.com/wp-content/uploads/2014/03/9-Evidence-Based-Principles-to-Help-Youth-Overcome-Homelessness-Webpublish.pdf

Previous Post in Series: Steve Ehrmann's Perspective

Evaluation & Innovation: Steve's Perspective

Over the next several weeks, I will be inviting several educational researchers and practitioners to offer their perspectives on the role of evaluation in teaching, learning, and technology innovation. Each is asked to freely respond to the same four questions. My hope is to foster discussions that enrich understandings, challenge assumptions, and strengthen processes for those who evaluate proposals or design studies. The goal is to compile a useful reference here.

Stephen C. Ehrmann, Ph.D.

Most recently, Stephen C. Ehrmann served as Vice Provost for Teaching & Learning at the George Washington University.  Previously he was founding Vice President of the Teaching, Learning, and Technology Group (TLT Group) and Director of the Flashlight Program for the Evaluation of Educational Uses of Technoloy; Senior Program Officer with the Annenberg/CPB Projects; Program Officer with the Fund for the Improvement of Postsecondary Education (FIPSE), and Director of Educational Research and Assistance at The Evergreen State College. He has a Ph.D. in Management and Higher Education from MIT.

Blog and bio: http://sehrmann.blogspot.com 


How do we recognize innovation in teaching and learning with technology?
It combines something unfamiliar with something promising. 

What challenges might practitioners and researchers face when evaluating innovation? Do you have ideas for how to overcome these challenges? 

• Evaluation is one of those things that everyone assumes he or she can do; the biggest challenge is not realizing there is history here – methods, previous findings, etc.
• Because innovation is by definition somewhat unfamiliar, to some degree you don’t know what you’re looking for.
• Those in charge of the evaluation are often advocates of the project. This role often leads them to counter-productive actions such as:

o   Delaying the creation of an evaluation plan until all the available money has been committed to other purposes and, if it’s a grant proposal, the proposal is due in a day or two.  Two major reasons why evaluation plans for grant-funded edtech innovations are often so weak.

o   Another reason: Delaying the evaluation until ‘all the kinks have been worked out’.  This means the evaluation isn’t being used to identify the kinks or to help figure out how to work them out.  It also often means that the evaluation is delayed until a time when no one remembers to do it, or has any commitment to do it.  The initiative can end (as an initiative) with stacks of data sitting somewhere, unanalyzed.

• Rapture of the technology: paying too much attention to the technology itself as though, by itself it drives change.  That’s almost never true, so techno-centric evaluations can easily produce puzzling, misleading results.  Imagine two institutions, each using a different brand of learning management system.  Institution A uses LMS-A, while B uses LMS-B, a competitor.  Institution A has a long tradition of learning communities, seminars, group projects while institution B, valuing access highly, historically has asked little group work of its commuting students and distant learners.  An evaluation is done of online communication among students – how frequent? How productive?  A beats B. A technocentric evaluation is designed on the assumption that it’s the technology that determines the use of the technology, so the author concludes that LMS-A is better for online learning communities than LMS-B.  

In an unpublished article, I’ve described 10 principles for doing a better job evaluating innovations in  education, especially innovations using technology. Briefly:

1. Above all, do no harm. (For example, don’t ask for data if you don’t have a firm idea of how you can satisfy your survey respondents and other informants that helping you was, indeed, worth their valuable time.)
2. Design the evaluation so that, no matter what you find, each stakeholder is somehow better off.
3. Be ready to compare apples with oranges because that kind of comparison is almost inevitable in a study of an educational innovation.
4. Focus on what people DO most often with the program (their ‘activities’) rather than looking only at the tools with which they do it.
5. Study why those people do those things in those ways.
6. Compare the program with the most valuable alternative, not with doing nothing.
7. Study costs, not just benefits.
8. Remember that education does not work like a well-oiled machine – doing the ‘same thing’ often does not produce the ‘same results.’
9. Recognize that different people using the program often have different motives, perceptions, circumstances, and, therefore, outcomes.  Realizing this leads to a dramatically different approach to assessment and evaluation – the students are doing different things (#4 above) because they have different reasons and different influences acting on them (#5 above). So the outcomes will differ from student to student.
10. Start now!  It’s never too early in the life of innovation, even going back to before it’s introduced, to do useful studies.

Can you point to some promising innovations in teaching and learning?

      The use of questions designed to make students think with polling techniques (such as clickers or cell phones), and peer instruction.  See here for a great video on this: https://www.youtube.com/watch?v=WwslBPj8GgI

      

     Are there some effective research initiatives or studies our readers should examine? If not, why do you think that is the case?

      This is a provocative study, done over 15 years ago:  J. C. Wright, S. B. Millar, S. A. Kosciuk, D. L. Penberthy, P. H. Williams, and B. E. Wampold (1998) “A Novel Strategy for Assessing the Effects of Curriculum Reform on Student Competence,” Journal of Chemical Education, v. LXXV, pp. 986-992 (August).