Main Body

Assessment Instruments for Teaching With Rich Media

How can instructor competence be assessed?

There is no singularly accurate method to assess instructor competence for teaching with rich media. A training manager may choose to assess competence using whichever educational philosophy or instructional theory desired.

We offer two approaches to assessment that reflect typical approaches to competency assessment in-practice: the Key Phrase method based on objective measures of proficiency, and the Practitioner method which his based on the candidate’s value to a community of practice.

The Key Phrase method of designing assessment criteria

In the Key Phrase method, each Performance Statement is deconstructed down to its key word or phrase and then surrounded by qualifiers that are observed in a candidate’s work. Variations in the qualifiers serve as the levels of differentiation across the assessment instrument. This method was developed by and is used as a foundation for developing assessment instruments in the CWPro eportfolio assessment system (Chalk and Wire Learning Assessment, Inc., 2013).

The basic principles of the Key Phrase assessment instrument framework are:

  • “Proficient” is designated as the target level of performance. A level beyond Proficient, such as “Exemplary”, is not a relevant level of assessment in this context because we are only interested in whether the candidate has achieved proficiency. Anything beyond attaining proficiency is extraneous, from a performance assessment perspective.
  • The key phrase is retained throughout all levels of assessment. Progressing levels of assessment are differentiated according to the observable presence or absence of a set of qualifiers or indicators.

Below is an example of the Key Phrase method for constructing a proficiency statement:

Step 1:  Select a Performance Statement:

“Write assignment briefs or prompts that draw from rich media engagement experiences that are relevant to instructional objectives.”

Step 2:  Identify the key phrase embedded within the Performance Statement:

Key phrase: “Write assignment briefs…”

Step 3:  Surround the key phrase with qualifiers or indicators that can be observed as a basis of determining proficiency. In this case, we will use “consistent and independent performance” as the observable criteria. This statement will become the Proficient level of assessment.

Proficient:Able to consistently and independently write assignment briefs or prompts that draw from learners’ rich media engagement experiences that are relevant to instructional objectives.

Step 4: Generate the other levels of assessment using variations of consistency and independence that can be observed in the performance.  Additional assessment levels are provided below to demonstrate the range of granularity to describe performance.

Novice: “Able to identify instances for the need to write assignment briefs or prompts that draw from learners’ rich media engagement experiences that are relevant to instructional objectives.” (This statement presumes that the learner’s skill is limited only to being able to identify when the skill is needed but is unable to perform the process due to lack of skill or knowledge).

Emerging: “With significant support, able to write assignment briefs or prompts that draw from learners’ rich media engagement experiences that are relevant to instructional objectives.”

Basic:  “With minimal support, able to write assignment briefs or prompts that draw from learners’ rich media engagement experiences that are relevant to instructional objectives.”

Proficient:  “Able to consistently and independently write assignment briefs or prompts that draw from learners’ rich media engagement experiences that are relevant to instructional objectives.”

 

Several factors support this method for producing valid and reliable results (Chalk and Wire Learning Assessment, Inc., 2013):

  • The key phrase in the assessed performance remains consistent and transparent throughout each level of performance. There is no “moving the goalposts” across different assessment levels by using vague subjective terms such as “good”, “strong”, or “excellent”.
  • Assessment across levels is keyed strictly to observable performance in terms of the pegged level of proficiency. This enables the candidate to gain a meaningful sense of their progress based solely on the words used in the rubric.
  • Using this method system-wide supports inter-rater reliability by minimizing subjective or abstract interpretations of performance.

The example above uses four levels of assessment, but alternative labels can be used:

Performance Statement Benchmark Milestone 1 Milestone 2
Performance Statement Novice Emerging Proficient
Performance Statement Unsatisfactory Marginal Proficient

What is (intentionally) missing from the Performance Statements?

The Performance Statements provided in the proposed set of Competencies are purposefully descriptive. They do not articulate all of the possible details related to a candidate’s expected performance.  For example, the Performance Statements do not prescribe:

  • Which devices, software, hardware, or methods of computing to use.
  • Which forms of social communication to use in the community of practice.
  • Which channels of internal communication to use.
  • Which personnel or offices to collaborate with.
  • Which specific standards of institutional ethical practice, UDL, or ADA compliance to adhere to.
  • Which theoretical underpinnings for teaching, learning, or instruction to align with.
  • Which modes of instructional program delivery to which performance must be applied, i.e. face-to-face, blended/hybrid, fully online, professional development training, etc.

This design leaves space for the Competencies and Performance Statements to be used as a general set of standards upon which context-specific demonstrations of proficiency can be added by or removed by Training Managers as needed. The instructor-in-training would be advised in advance as to the specific outcomes of their training that are relevant to their institution’s system and practice.

Example Matrix of Competencies and Performance Statements for Teaching with Rich Media

Technical Competency Performance Statement Emerging Basic Proficient
Locate online rich media resources relevant to subject matter. Use Internet search skills to locate resource repositories, Open Educational Resources (OER), social media based communities, and Web tools. With significant support, able to use Internet search skills to locate resource repositories, Open Educational Resources (OER), social media based communities, and Web tools. With minimal support, able to use Internet search skills to locate resource repositories, Open Educational Resources (OER), social media based communities, and Web tools. Able to use Internet search skills to locate resource repositories, Open Educational Resources (OER), social media based communities, and Web tools.

Pedagogical Competency

Performance Statement Emerging Basic Proficient
Demonstrate knowledge of applicable theories and research that describe the advantages of rich media in online learning. Recognize opportunities for using rich media in online course design which offer an advantage for learners in comparison to other communication methods. With significant support, able to recognize opportunities for using rich media in online course design that offer an advantage for learners in comparison to other communication methods. With minimal support, able to recognize opportunities for using rich media in online course design that offer an advantage for learners in comparison to other communication methods. Able to consistently and independently recognize opportunities for using rich media in online course design that offer an advantage for learners in comparison to other communication methods.
Leadership Competency Performance Statement Emerging Basic Proficient
Participate in the community of practice related to teaching with rich media in online instruction. Participate in periodic professional community-based gatherings for the purpose of professional growth. Minimally able to participate in periodic professional community-based gatherings for the purpose of professional growth. With some assistance, able to participate in periodic professional community-based gatherings for the purpose of professional growth. Able to fully participate in periodic professional community-based gatherings for the purpose of professional growth.

The Practitioner method of designing assessment criteria

Wenger’s Community of Practice model (2011) describes the characteristics of a group of people who “… share a concern or a passion for something they do and learn how to do it better as they interact regularly (p. 1).” A community of online instructors would determine what is knowable, relevant, and observably present in the performance of teaching online with rich media from the perspective of an exemplary practitioner of the craft.

To apply structure to this approach, we adapt a framework used by Aric Mayer, Western Washington University, as the basis of their MBA graduate program (Mayer, 2014). This framework originates from and is endorsed by the Institute of Leadership & Management (UK).

Competencies are observable at three tiers of proficiency: Knowing, Doing, and Being.

Applied to teaching with rich media, three tiers of proficiency are set as follows:

Knowing: Candidate demonstrates theoretical comprehension of the body of knowledge and the ability to connect knowledge to situational needs within the online teaching and learning environment.

Doing: Candidate demonstrates the ability to apply skills and theories in praxis.

Being: Candidate demonstrates the character of professional participation in the community of practice to promote exemplary work and share knowledge.

 

As stated in Mayer’s program and adapted for our purposes, “Goals represent what we want our instructors to be. Objectives describe what we want our instructors to do.” Thus, the goal in the Practitioner approach is to nurture online instructors’ proficiencies so that they more than merely practitioners – they are change agents who are able to lead by example and influence their colleagues’ practice. Competencies, as an expression of a Community of Practice, represent what practitioners would do to embody this goal.

Our matrix for Knowing, Doing, and Being are projected across three sets of proficiency: Technical, Pedagogical, and Leadership (see table 1).

TECHNICAL PEDAGOGICAL LEADERSHIP
BEING
DOING
KNOWING

Table 1: Matrix of Rich Media Competencies across CoP Proficiencies

Here is an example of how a Performance Statement can be implemented in the CoP model:

 

TECHNICAL PEDAGOGICAL LEADERSHIP
BEING Teach others how to use Internet search skills to locate resource repositories, Open Educational Resources (OER), social media based communities, and Web tools. Teach others about the purpose of and how to evaluate multimedia resources for their suitability to use in instruction. Create an online community for the purpose of providing professional growth for other members.
DOING Use Internet search skills to locate resource repositories, Open Educational Resources (OER), social media based communities, and Web tools. Evaluate multimedia resources for their suitability to use in instruction. Participate in periodic professional community-based gatherings for the purpose of professional growth.
KNOWING Describe how to use Internet search skills to locate resource repositories, Open Educational Resources (OER), social media based communities, and Web tools. Explain the purpose of evaluating multimedia resources for their suitability to use in instruction. Identify online communities for participating in professional gatherings for the purpose of professional growth.

Table 1: Matrix of Rich Media Competencies across CoP Proficiencies

 

Some areas of the matrix can be assessed with one-time demonstrations or with examples of proficiency; others will require more longitudinal observation to assess, such as those associated with leadership.

Training program designers and their stakeholders must consider these factors when planning how to certify instructors for specific qualifications.

License

Icon for the Creative Commons Attribution-ShareAlike 4.0 International License

Teaching With Rich Media by Steve Covello is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License, except where otherwise noted.

Share This Book