Research on Digital Literacy assessment instruments – fulltext

The following paper was presented as a research project for IDE-712 Analysis for Human Performance Technology Decisions, (Dr. Jing Lei, professor). It is not an exhaustive survey of every existing Digital Literacy assessment instrument, but it is a start. Here is the Fulltext: Digital Literacy Assessment Instruments (1780 downloads)

A Review of Digital Literacy Assessment Instruments

Instructional Designers rely on tools that bear useful measurements for assessing learner characteristics in front-end analysis settings. In the digital network environment, we find learners possess a wide variety of skills, experiences, interests, attitudes, and comprehension about digital tools, information systems, and content. Instructional designers, as well as learning institutions, may desire information from normative or criterion based assessment instruments to measure learner competence in the areas of Digital Literacy prior to engaging them in instructional activity.

As a starting point, however, we encounter a disparate field of research making the process of locating assessment instruments difficult and time consuming. There are a number of broadly defined territories within the geography of digital literacy, e.g. economics, politics, education, communication media (Gapski, 2007), each with their own interpretation of Digital Literacy. There are discussions on whether the framework of digital literacy should be pedagogical or functional (Pietrass, 2007; Calvani, Cartelli, Fini and Ranieri, 2008), questions about gender bias in assessment questionnaires (Pietrass, 2007), and discord over certain assessment instrument methods, such as the validity of self-reporting, narrowness of results due to subject specificity, or pre-/post-testing inconsistencies (Baird, 1973, in LeBold et al, 1998; Sieber, V., 2009; Davies, Szabo, and Montgomerie, 2002; O’Connor, Radliff, and Gedeon, 2001; Kvakiv, 2005). Further, there are various terms to describe similar or overlapping research, such as information literacy, ICT competence, web literacy, 21st century skills, new literacy practices, information fluency, tech literacy, information competence, digital competence, computer literacy, media literacy, eCompetence, media competence, and others.

Thus, selecting a means for measuring Digital Literacy requires understanding the construct of the research, and then answering some critical questions about assessment context and approaches (Gapski, 2007):

  • Which level of analysis is relevant (the individual digital literate student, groups and/or a digital literate school as a social system)?
  • Which context of usage is relevant with regard to the “myriad of digital literacies”?
  • What is the object of measuring (processes or structures)?
  • Which perspective method is applied (self-observation, external observation, qualitative and quantitative approaches)?

In this paper we review literature concerning the basis of assessing student Digital Literacy, and the instruments used in the process. Although not an exhaustive document, this report may assist those who seek orientation on the basics of Digital Literacy, and assessment instruments used to measure it.

Digital Literacy Defined. Digital Literacy has been defined as an umbrella framework for a number of complex and integrated sub-disciplines – or “literacies” – comprised of skill, knowledge, ethics and creative output in the digital network environment (Calvani, Cartelli, Fini and Ranieri, 2008). For each component, there are distinct bodies of research with similar sounding names, many of which pre-date the “digital era”. Thus, the definitions of certain literacies have undergone transformation or expansion as innovation and research redefine them. “Computer Literacy” in the 1960s, for example, had originally connotated proficiency in programming, while contemporary definitions include no such proficiency (Martin and Grudziecki, 2006). “Computer Literacy” has since evolved to include information technology, or, “IT Literacy”, then later, information and communication technologies, or,  “ICT Literacy” (Martin and Grudziecki, 2006).

Further, changing definitions of literacy often converge or overlap with emerging new literacies. The emergence of Web 2.0, or online social media applications, introduces the additional dimensions of comprehending authorship, privacy and plagiarism (Anderson, 2007) – a mixture of Information Literacy, Technology Literacy, creativity and ethics. Table 1 below represents some sub-disciplines that comprise the domain of Digital Literacy:

Table 1

Sub-Disciplines of Digital Literacy

Sub-Discipline Definition
Information Literacy Finding and locating sources, analyzing and synthesizing the material, evaluating the credibility of the source, using and citing ethically and legally, focusing topics and formulating research questions in an accurate, effective, and efficient manner (Eisenberg, Lowe, and Spitzer, in Meyer et al, 2008, p. 2).
Computer Literacy An understanding of how to use computers and application software for practical purposes (Martin and Grudziecki, 2006).
Media Literacy A series of communication competencies, including the ability to access, analyze, evaluate and communicate information in a variety of forms including print and non-print messages (Alliance for a Media Literate America, 2010)
Communication Literacy Learners must be able to communicate effectively as individuals and work collaboratively in groups, using publishing technologies (word processor, database, spreadsheet, drawing tools…), the Internet, as well as other electronic and telecommunication tools (Winnepeg School Division, 2010)
Visual Literacy The ability to ‘read,’ interpret, and understand information presented in pictorial or graphic images; the ability to turn information of all types into pictures, graphics, or forms that help communicate the information; a group of competencies that allows humans to discriminate and interpret the visible action, objects, and/or symbols, natural or constructed, that they encounter in the environment (Stokes, 2002)
Technology Literacy Computer skills and the ability to use computers and other technology to improve learning, productivity, and performance (U.S. Department of Education, 1996).

The Educational Testing Service (ETS), a leader in the development of assessment instruments, defines ICT Literacy based on the findings of the Report of the International ICT Literacy Panel (2002) (see Table 2). In this overarching description, “ICT Literacy” might easily be used interchangeably with “Digital Literacy”.

Table 2

Components of ICT Literacy

Proficiency Definition
Define Using ICT tools to identify and appropriately represent an information need.
Access Collecting and/or retrieving information in digital environments.
Manage Using ICT tools to apply an existing organizational or classification scheme for information.
Integrate Interpreting and representing information, such as by using ICT tools to synthesize, summarize, compare and contrast information from multiple sources.
Evaluate Judging the degree to which information satisfies the needs of the task in ICT environments, including determining authority, bias and timeliness of materials.
Create Adapting, applying, designing or inventing information in ICT environments.
Communicate Communicating information properly in its context (audience, media) in ICT environments.

Note: Reproduced from Katz (2005)

Gapski sub-divides Digital Literacy into two main strands: “instrumental-technological” and “normative media-educational,” corresponding respectively to usage/functionality, and pedagogical aims (Gapski, 2001, in Pietrass, 2007; Gapski, 2007). These strands are operationalized into three forms of competence: (a) Interpreting messages; (b) Choosing messages; and (c) Articulating messages. In turn, these competences inform objectives and measures of functional, cognitive and ethical proficiencies.

Calvani, Fini and Ranieri (2009) summarize digital literacy as a combination of concrete and unquantifiable skills (see Figure 1):

[Digital Literacy] is being able to explore and face new technological situations in a flexible way, to analyze, select and critically evaluate data and information, to exploit technological potentials in order to represent and solve problems and build shared and collaborative knowledge, while fostering awareness of one’s own personal responsibilities and the respect of reciprocal rights/obligations (p. 60-61).

Figure 1 – Intersecting Areas of Digital Literacy

Martin and Grudziecki (2006) propose a model of individual development between levels of Digital Literacy disciplines (see Figure 2):

Figure 2 – Three Levels of Digital Literacy Development

The model reflects the fluidity between the cognitive, functional and creative aspects of Digital Literacy suggesting that literacy development is not necessarily a linear process, nor rooted only in specific tasks.

From a more philosophical approach, Digital Literacy introduces questions about concepts of individualism and media psychology. Gapski (2007) describes Digital Literacy as having subjective interpretations across various global cultural systems, suggesting that the values used in assessment must consider a wide set of socially based strategies for measurement.

Aviram and Eshet-Alkalai (2006) distinguish digital literacy as an integration of five separate but interrelating literacy skills: (a) photo-visual literacy; (b) reproduction literacy; (c) information literacy; (d) branching literacy; and (e) socio-emotional literacy. These skills are presented as an expression of culturally distinct epistomologies, with sets of skills constituting distinct learning styles and personality types. Further, they contrast the traditional “linear/industrial/individualistic/independent” literacy skill sets, against the contemporary “lateral (non-linear)/branching/shared/relational” literacy skill sets – with each “society” opposing the other in their epistemological approach to information.

Rationale for Assessment and Measurement. Integration of digital technology devices and network applications into the learning environment has been supported by numerous government, education and advocacy organizations for practical and pedagogical reasons, such as fostering global competitiveness, functioning in the labor market and in succeeding in post-secondary education. Thus, assessment follows out of necessity.

Katz (2005) decribes the following imperitives for integrating Digital Literacy assessment into the educational framework:

  • To support institutional ICT literacy initiatives
  • To guide curricula innovations and evaluate curicula changes
  • To guide individual learning
  • To establish a clear definition of skills and knowledge

In K-12 education, an effort has been made by the Obama administration to place “Technological Literacy” at the forefront of national education priorities. Defined broadly as “the capability to use, understand, and evaluate technology as well as to apply technological concepts and processes to solve problems and reach one’s goals,” Technological Literacy has been mandated to become a part of the assessment of educational progress of pre-college level students, and will become a formal part of the National Assessment of Educational Progress (NAEP), also known as the ‘Nation’s Report Card,’ which gauges the educational progress of elementary and secondary students. The assessment will be instituted nationwide in 2012 (Perez and Murray, 2010). The California Emerging Technology Fund, in their 21st Century Learning and Workforce matrix of best practice, supports the use of ICT assessment as a high school exit requirement (CETF, 2008).

But while emphasis on the value and integration of Digital Literacy has gathered strength in K-12 (Manzo, 2009; Davies, Szabo, and Montgomerie, 2002), higher education institutions are confronted with the realization that universally adopted normative testing on admissions-critical academic instruments, such as the SAT and ACT, have not adopted nor codified Digital Literacy (Perez and Murray, 2010 pp. 131-132). Despite its apparent importance, it has only been relatively recently that Digital Literacy assessment has been implemented in post-secondary education as a large-scale pre-test – and then only after matriculation. In 2005, Purdue University, Portland State University, and the California State University system participated in the launch of an assessment program using the ETS ICT Literary Assessment test (later renamed iSkills, then iCritical Thinking)(Business Wire, 2005). Lehigh University implemented their own campus-wide Research Skills Assessment in 2007 to be administered prior to incoming students arriving on campus to understand the perceptions and skills with which Lehigh students arrive (Bowerman, 2007).

In 2008, Simmons College launched the Information Technology and Information Literacy Competencies, or “iComps Exam”, for use on all incoming freshman. It is a 50-question multiple-choice and true/false proctored exam taken online covering information in five broad areas:

  • Information Access – library resources and services, foundational research skills
  • Ethics – copyright issues, Simmons Acceptable Use Policy, ethical and legal use of information
  • Personal Computing – connectivity, operating systems, security
  • Using Simmons Systems
  • Applications – Microsoft Office, Browsers, iTunes

Students must pass the test by their third semester (Simmons College, 2008), and if a student fails the iComps test three times, he or she is automatically placed into a remedial course (Matthews-DeNatale, 2009).

Implementation of Digital Literacy assessment at the secondary and post-secondary levels offers a starting point for classifying learners and for establishing a practical entry point for instruction, and perhaps for maintaining longitudinal tracking of students throughout an academic career for institutional evaluation and employment (O’Connor, 2005).

Instrument Selection:

Key factors in selecting an instrument include examination of the instrument’s output and its approach. Lynch and Swing (2006) describe the following key features of an ideal assessment instrument:

  1. Validity of the assessment approach.
  2. Reliability of the data collected.
  3. Feasibility of implementation (time and training needed for implementation).
  4. Alignment with assessment context.
  5. Usefulness of information.
  6. Consistency with curriculum/program objectives.
  7. Feasible representation of key areas of knowledge/skill (rather than measuring all objectives in all contexts)
  8. Multiple assessment approaches to measure competence in multiple dimensions.
  9. Multiple observations to improve reliability.
  10. Multiple observers to improve precision.
  11. Fairness in opportunity for participants to perform.
  12. Assessment according to pre-specified standards or criteria.

In specifying assessment standards for Digital Literacy, developers have relied on the International Society for Technology in Education (ISTE) NETS-S, Educational Technology Standards for Students, to indicate ideal performance outcomes. NETS-S is composed of the following categories (International Society for Technology in Education, 2007):

  1. Creativity and Innovation
  2. Communication and Collaboration
  3. Research and Information Fluency
  4. Critical Thinking, Problem Solving, and Decision Making
  5. Digital Citizenship
  6. Technology Operations and Concepts

These categories are elaborated upon in detailed descriptions of actions and objectives, and serve as a launching point for creating items in assessment instruments used in several states (Manzo, 2009) (See Appendix B).

In the development of assessment content, items and activities must be constructed to produce responses in the form of measurable data. For example, in the design of the Project SAILS (Standardized Assessment of Information Literacy Skills), the designers relied upon Item Response Theory (IRT) (see Appendix A) to create a normative, flexible exam that can measure the underlying traits of the examinee, even if the items are different on several tests (O’Connor, Radcliff, and Gedeon, 2001). O’Connor et al (2001) describe the criteria for producing validity in the scaling of data:

  1. Is a discernible line of increasing intensity defined by the data?
  2. Is item placement along this line reasonable?
  3. Do the items work together to define a single variable?
  4. Are persons adequately separated along the line defined by the items?
  5. Do individual placements along the variable make sense?
  6. How valid is each person’s measure?

These criteria help form distinct boundaries for classifying students’ literacy levels. In practice, it has also been recommended to include an additional practical examination to demonstrate realistic skills (O’Connor, Radcliff, and Gedeon, 2001). For example, Pereira, et al., (2009) suggest that assessment include a practical examination in the form of an eFolio – an electronic document demonstrating student output – where rubrics can measure observation or frequency of certain behaviors and skills.

Instruments used in Digital Literacy Assessment. The following instruments represent only a sampling of many Digital Literacy-related assessment instruments available for use with high school students, higher education students, and with pre- and in-service teachers. Refer to Appendix A for Internet links to online demonstrations, sample items, and case studies of other assessment instruments related to Digital Literacy and sub-component disciplines.

Instrument: ETS iCritical Thinking (formerly iSkills, formerly ICT Literacy Assessment)

Orientation: Digital Literacy (comprehensive)

Availability: Commercial

Website: http://www.ets.org

Intended audience: Grades 10 through college (core audience), teachers, employers, adult employees

Standards: Association of College and Research Libraries (ACRL) Information Literacy Competency Standards for Higher Education: Standards, Performance Indicators, and Outcomes; and Objectives for Information Literacy Instruction: A Model Statement for Academic Librarians

Methods: Online exam; simulated situations. Administered online at a certified testing center (12,000). Duration is approximately 75 minutes.

Background: Developed in response to a need for large-scale institutional assessment of Information Literacy and technical skills founded on cognitive and problem-solving skills. The iSkills/iCritical Thinking test instrument is the product of the findings of the International ICT Literacy Panel (2001). The panel sought to determine the need for assessment of ICT Literacy across countries and within specific organizations, such as schools and businesses, and secondly, to develop a workable Framework for ICT Literacy. This framework provided a foundation for the design of instruments, including large-scale assessments intended to inform public policy and diagnostic measures to test an individual’s ICT skills.

Features: Both the Core and Advanced iSkills assessments consist of approximately 60 items derived from performance on 15 interactive, performance-based tasks. The scoring of the items follow rubrics that specify the nature of responses needed to gain full credit (1), partial credit (0.5), or no credit (0). The overall raw score on the assessment is the sum of all item scores. The exam features realtime, simulated, scenario-based task items designed to measure the ability to navigate, and critically evaluate information. Covers E-mail, Instant Messaging, Bulletin Board Postings, Browser Use, Search Engines, Data Searches, File Management, Word Processing, Spreadsheets, Presentations, Graphics.

Reporting: Includes individual, group and institutional comparisons. Testing software includes the ability to track a participant’s paths leading to a solution, i.e. tracking mouse clicks on web page objects in seeking information.

Sample: Representative test items, retrieved online April 30, 2010.

http://sjsu-dspace.calstate.edu/xmlui/bitstream/handle/10211/73/TEL_Somerville_Smith_Macklin_PDFA.pdf?sequence=6

Sample: ETS iSkills Aggregate Feedback Report, retrieved online April 29, 2010:

http://www.pc.maricopa.edu/data/GlobalFiles/file/committees/assessment/informationliteracy/Results%20and%20Recommendations/2008%20Spring%20iSkills%20Performance%20Report.pdf

Sample: ETS iSkills pilot test participant pre-test survey methods and statistics (Katz and Macklin, 2007). Retrieved online April 29, 2010:

http://www.iiisci.org/Journal/CV$/sci/pdfs/P890541.pdf

Research: See Chapter 9: The ICT Literacy Framework. Retrieved online May2, 2010 from: http://www.nald.ca/fulltext/measlit/Part3.pdf

Instrument: Project SAILS

Orientation: Information Literacy (library, research and information evaluation)

Availability: Commercial

Website: https://www.projectsails.org/

Intended audience: Higher education students

Standards: Association of College and Research Libraries (ACRL): Information Literacy Competency Standards for Higher Education: Standards, Performance Indicators, and Outcomes; and Objectives for Information Literacy Instruction: A Model Statement for Academic Librarians.

Methods: 45 “forced answer” multiple-choice items, administered through the Project SAILS website. Duration is about 35 minutes.

Background: Project SAILS (Standardized Assessment of Information Literacy Skills) was initiated in 2002 out of Kent State University, Kent, Ohio, by Carolyn Radcliff, Julie Gedeon and Lisa O’Connor. Intended to answer: Does information literacy make a difference to student success? Does the library contribute to information literacy? How do we know if a student is information literate? See also Project Trails (Tools for Real-time Assessment of Information Literacy Skills) in Appendix A.

Features: A large-scale, knowledge-based multiple-choice test, featuring a variety of basic and advanced information literacy skills and concepts, such as research strategies; selecting sources; understanding and using finding tools; developing and revising search strategies; evaluating results; retrieving materials; documenting sources; and legal and social issues related to ethical and effective use of information.

Reporting: Reports detail the performance of students organized by ACRL standards and by skill sets based on the ACRL document, “Objectives for Information Literacy Instruction: A Model Statement for Academic Libraries.” Within each skill set, the overall average student performance is presented along with breakouts by class standing, major, and any special groups designated by the institution, and also compared to performance at groups of other schools. It shows which information literacy objectives are the most difficult and which are the easiest in graphical presentation of data and explanatory text. Includes documents about the test and the testing session in which the school participates, along with the test questions, the skill sets, and demographic profiles of participating schools.

Sample: Representative test items, retrieved April 30, 2010 from:

https://www.projectsails.org/abouttest/samples.php

Research: Instrument development, retrieved February 22, 2010 from:

http://www.pla.org/ala/mgrps/divs/acrl/events/pdf/oconnor.pdf

Research: Item development, based on Systems Design and Item Response theories. Retrieved February 22, 2010 from:

http://www.lib.uoguelph.ca/services/information_literacy_&_instruction/information_literacy_testing/components/documents/applying_system_design.pdf

Research: Case study with University of Guelph, 2008, retrieved February 22, 2010 from:

http://www.lib.uoguelph.ca/services/information_literacy_&_instruction/information_literacy_testing/SAILS_results.cfm

Instrument: iDCA (Digital Competence Assessment)

Orientation: Digital Literacy (comprehensive)

Websites: http://www.digitalcompetence.org/

Availability: Open source

Intended audience: High school students, age 13-18 (core: 15-16)

Standards: Association of College and Research Libraries (ACRL), Programme for International Student Assessment (PISA)

Methods: Multiple-choice, situated response, and simulation. Administered online through Moodle.

Background: The DCA (Digital Competence Assessment) research group was formed within the National Research Project in 2006 by Prof. Antonio Calvani (University of Florence, Italy). The group is composed of four local research units from the Universities of Florence, Turin, Cassino and Salerno, Italy. The groups set out to respond to both the International ICT Literacy Panel (2001), and to the assessment tests being carried out by the PISA organization. The PISA exams are conducted every three years to 15-16 year olds as a comprehensive end-of-compulsory education assessment battery. It is in use in 65 countries, as of 2009.

Features: The iDCA is composed of three separate tests: the Instant DCA, Situated DCA, and the Projective DCA. As of May 2, 2010, only the Instant DCA is available for use.

The Instant DCA is intended to be a rapid, broad-ranging test of 85 items: multiple-choice, matching and short answer, covering three sections: technological, cognitive and ethical issues (see Figure 3).

  • In the technological section, the questions are focused on the understanding of common everyday situations working directly at the computer.
  • The cognitive section is more identified with Information Literacy, such as solving problems involving linguistic or logic-linguistic competences.
  • The ethics section concerns suitable behaviors in the use of technologies and general behavior on the Internet.

The Situated DCA is intended to be used as an instrument in the short/medium run, and consists in four test typologies:

  • The first typology (Technological Exploration) asks students to deal with an unknown technological interface and learn how to use and master it.
  • In the second typology (Simulation) data must be empirically processed and hypotheses formulated based on possible relationships.
  • In the third typology (Inquiry) information pertaining to a predefined subject must be critically selected and gathered.
  • Lastly, in the fourth typology (Collaborative Wiki) student groups draft a document together following criteria of collaborative activity management.

The Projective DCA is conceived as a means to meaningfully assess students’ attitudes in the medium run. It consists of a set of drawings aimed at exploring the awareness the participant has of the emotional and social impact of the use of ICTs, especially with children and in intercultural contexts. The drawings come with structured assignments to produce answers within a range of predefined concepts.

Reporting: Not stated.

Sample: http://www.digitalcompetence.org/moodle/

Research: http://je-lks.maieutiche.economia.unitn.it/en/08_03/ap_calvani.pdf

Figure 3 – Component Map of Instant DCA

A Case Study. In 2001, Educational Testing Service, inc. (ETS) convened an international panel of experts, chaired by Barbara O’Connor, Communications professor at California State University/Sacramento to assess the need for a large-scale ICT Literacy assessment instrument, and to develop a framework from which assessment instruments may be developed. Based on the findings of the panel, a team of librarians, classroom faculty, education administrators, assessment specialists, researchers, user-interface and graphic designers, and systems developers combined to develop the iSkills test. In 2004, the ETS iSkills (later renamed iCritical Thinking) began beta testing. Table 3 represents a chronology of field trials and tests.

Table 3

Chronology of ETS iSkills field trials and test administrations

Date Administration Appx. # of students Appx. # of participating institutions
July–September 2004 Field trials for institutional assessment 1,000 40
January–April 2005 Institutional assessment 5,000 30
May 2005 Field trials for alternative individual assessment structures 400 25
November 2005 Field trials for advanced level individual assessment 700 25
January–May 2006 Advanced level individual assessment 2,000 25
February 2006 Field trials for core level individual assessment 700 30
April–May 2006 Core level individual assessment 4,500 45
August–Dec. 2006 Core level: Continuous administration 2,100 20
August–Dec. 2006 Advanced level: Continuous administration 1,400 10

Note: Reproduced from Katz (2007)

In the field trial period between January and May, 2006, over 6,000 participants were tested in core and advanced skill levels. Test-takers consisted of 1,016 high-school students, 753 community college students, and 4,585 four-year college and university students. Results are reproduced below (Katz, 2007).

Field trials revealed the following trends in student competency:

  • Overall, students performed poorly on both the core and advanced level, achieving only about half of the possible points on the tests.
  • Students generally do not consider the needs of an audience when communicating information.
  • Students tend not to check the “fair use” policies of information on the assessment’s simulated Web sites.
  • Test-takers appeared to recognize that .edu and .gov sites are less likely to contain biased material than .com sites.
  • Eighty percent of test-takers correctly completed an organizational chart based on e-mailed personnel information.

Web site skills and evaluation:

  • Only 52 percent judged the objectivity of the sites correctly.
  • Sixty-five percent judged the authority correctly.
  • Seventy-two percent judged the timeliness correctly.
  • Overall, only 49 percent of test-takers uniquely identified the one Web site that met all criteria.

When selecting a research statement for a class assignment:

  • Only 44 percent identified a statement that captured the demands of the assignment.
  • Forty-eight percent picked a reasonable but too broad statement.
  • Eight percent picked statements that did not address the assignment.

When asked to narrow an overly broad search:

  • Only 35 percent selected the correct revision.
  • Thirty-five percent selected a revision that only marginally narrowed the search results.

Other results suggest that these students’ ICT literacy needs further development:

  • In a Web search task, only 40 percent entered multiple search terms to narrow the results.
  • When constructing a presentation slide designed to persuade, 12 percent used only those points directly related to the argument.
  • Only a few test-takers accurately adapted existing material for a new audience.
  • When searching a large database, only 50 percent of test-takers used a strategy that minimized irrelevant results.

In a post-test survey, 94 percent of the students said that to perform well on the test required thinking skills, as well as technical skills; 90 percent said that the assessment was appropriately challenging; and 75 percent indicated that the tasks they were asked to perform on the assessment reflected activities they did at school or work.

Among the validity measures presented in the literature, the following passage is offered (Katz, 2007, p. 7):

Students’ self-assessments of their ICT literacy skills align with their scores on the iSkills assessment (Katz and Macklin 2006). The self-assessment measures were gathered via a survey administered before the 2005 assessment. Interestingly, although students’ confidence in their ICT literacy skills aligned with their iSkills scores, iSkills scores did not correlate with the frequency with which students reported performing ICT literacy activities. This result supports librarians’ claims that mere frequency of use does not translate to good ICT literacy skills, and points to the need for ICT literacy instruction.

This information conflicts with other studies that indicate an opposite phenomenon – that higher education students who take self-assessment tests tend to overrate their skills; freshman college students more so than upper-level students, men more so than women (Kvakiv, 2005; Ivanitskaya, 2010; Sieber, 2009) – attributed, in part, by students realizing that more in-depth usage of ICT over several years in higher education reveals greater awareness of software features. It is difficult to resolve these two claims, though it is plausible that frequency of use implies usage in a narrow area of ICT, suggesting “high experience” but “low overall competence”.

An additional case study of iSkills use at a large Midwestern university (Hignite, Margavio, and Margavio, 2009) revealed stratified trends among 600+ undergraduate students: with a score of 164 points indicating “proficiency”, Caucasians in this sample scored a mean of 157.08 (r=86), while Non-Caucasian scored a mean of 135.58 (r=526); females scored slightly higher than males 156.45 – 150.89, respectively; and higher ACT scores correlated positively with iSkill scores. Among Non-Caucasians, the study did not take into account the possible effect of language comprehension of non-native English speaking participants.

iSkills (now iCritical Thinking) continues to be developed. The most recent literature describes the need to measure the effectiveness of ICT literacy instructional methods by comparing student performance before and after instruction. The goals are to understand how first-year students acquire information-processing skills, identify best practices for integrating information literacy into the curriculum, and assess the impact of skill acquisition on overall academic achievement (Katz and Macklin, 2007).

Conclusion

One might someday assess Digital Literacy in a student much like assessing the various “tools” said to comprise an ideal athlete, e.g. aptitudes in physical, cognitive, ethical, and social aspects of being – each competence expressed by various statistical implications. In conducting front-end analysis for instructional systems, the Instructional Designer would be well-served by instruments for measuring Digital Literacy that produce reliable values for making decisions, much like performance statistics are used to assess the value of athletes in intra- and inter-organizational settings.

Selecting the best assessment instrument for Digital Literacy involves consideration of many factors, including approach, feasibility, implementation, scope, reporting structure, and cost, as well as consideration of output needs and social context. As the population of students and their respective Digital Literacy skills evolve – as will the network systems upon which they operate – we may also expect that Digital Literacy assessment instruments will continue to develop to measure the competences needed to succeed in educational environments.

References

Alliance for a Media Literate America (2010). Media Literacy – NAMLE – National Association for Media Literacy Education – Advancing Media Literacy Education in America. Retrieved online May 1, 2020 from: http://www.amlainfo.org/home/media-literacy

Anderson, P. (2007). What is Web 2.0? Ideas, technologies and implications for education. JISC Technology Standards Watch, February, 2007. Retrieved onine February 22, 2010 from: http://www.jisc.ac.uk/media/documents/techwatch/tsw0701b.pdf

Aviram, A., and Eshet-Alkalai, Y. (2006). Towards a Theory of Digital Literacy: Three Scenarios for the Next Steps. European Journal of Open, Distance and E-Learning. Retrieved online February 23, 2010 from: http://www.eurodl.org/index.php?p=archives&year=2006&halfyear=1&article=223

Bowerman, R. (2007). Assessing Student Information Literacy Levels using the Banner Survey Tool and the Luminis Portal at Lehigh University, Educause. Retrieved online February 22, 2010 from: http://net.educause.edu/ir/library/pdf/MAC07038.pdf

California ICT Digital Literacy Policy and California Basic Elements of ICT Digital Literacy Continuum of Assessment Skills, California Emerging Technology Fund (November 17, 2008). Retrieved online February 23, 2010 from: http://www.ictliteracy.info/rf.pdf/California%20ICT%20Policy%20and%20Assessment%20Report.pdf

California State University Students Take New ETS ICT Literacy Assessment (April 4, 2005). Business Wire. Retrieved online April 29, 2010 from HighBeam Research: http://www.highbeam.com/doc/1G1-131135522.html

Calvani, A., Cartelli, A., Fini, A., and Ranieri, M. (2008). Models and Instruments for Assessing Digital Competence at School. Journal of e-Learning and Knowledge Society, Vol. 4, n. 3, September 2008 (pp. 183 – 193).

Calvani, A., Fini, A., and Ranieri, M. (2009). Assessing Digital Competence in Secondary Education – Issues, Models and Instruments. (M. Leaning, Ed.) Issues in Information and Media Literacy: Education, Practice and Pedagogy , 153-172.

Davies, J. E., Szabo, M., and Montgomerie, C. (2002). Assessing Information and Communication Technology Literacy of Education Undergraduates: Instrument Development. Retrieved February 22, 2010, from Association for the Advancement of Computing in Education: http://www.aace.org/DL/

International ICT Literacy Panel (2002). Digital transformation: A framework for ICT literacy –A report of the International ICT Literacy Panel. Princeton, NJ: Educational Testing Service. Retrieved online February 21, 2010 from: http://www.ets.org/Media/Tests/Information_and_Communication_Technology_Literacy/ictreport.pdf

Matthews-DeNatale, G. (2009). Information Literacy and IT Fluency, Conference panel presented a the EDUCAUSE Learning Initiative (ELI) Annual Conference, January 2009, Orlando, Florida. EDUCAUSE Review, vol. 44, no. 3 (May/June 2009): 8–9.

Fahser-Herro, D., and Steinkuehler, C. (2009). Web 2.0 Literacy and Secondary Teacher Education. Journal of Computing in Teacher Education , 26 (2), 55-62.

Gapski, H. (2007). Some Reflections on Digital Literacy. Proceedings of the 3rd International Workshop on Digital Literacy (pp. 49-55). Crete, Greece: CEUR-WS.org. Retrieved online February 22, 2010 from: http://ceur-ws.org/Vol-310/paper05.pdf

Hignite, M., Margavio, T., and Margavio, G. (2009). Information literacy assessment: moving beyond computer literacy. College Student Journal , 43 (3), 812-821.

International Society for Technology in Education (ISTE). (2007). ISTE | NETS for Students. Retrieved February 21, 2010, from ISTE: http://www.iste.org/Content/NavigationMenu/NETS/ForStudents/NETS_for_Students.htm

Ivanitskaya, L (2010). Research Readiness Self-Assessment (The RRSA) Overview and FAQ. Retrieved April 29, 2010 from: http://researchinstruction.wikispaces.com/Info+Lit+Assessment

Katz, I., (2005). Beyond Technical Competence: Literacy in Information and Communication Technology, Educational Technology v. 45 no. 6 (November/December 2005) p. 44-7

Katz, I., (2007). Testing Information Literacy in Digital Environments: ETS’s iSkills Assessment. Information Technology and Libraries. Chicago: Sep 2007. Vol. 26, Iss. 3; pg. 3, 10 pgs. Retrieved April 29, 2010 from: http://library.ias.edu/hs/ssstheme/20081218_ICT%20.pdf

Katz, I. and Macklin, A. S. (2007). Information and communication technology (ICT) literacy: Integration and assessment in higher education. Systemics, Cybernetics and Informatics, 5(4), 50-55.

Kvakiv, R. (2005). Convenience, Communications, and Control: How Students Use Technology. In D. Oblinger, and J. Oblinger (Eds.), Educating the Net Generation. Educause. Retrieved online Febrary 21, 2010 from: http://www.educause.edu/Resources/EducatingtheNetGeneration/ConvenienceCommunicationsandCo/6070

LeBold, W., Montgomery, L., Budny, D., Bjedov G., and Ward, S. (1998). Assessing Computer Literacy and Achievement Using Self-Concepts, Frontiers in Education Conference, Tempe, AZ, 4-7 Nov 1998 Volume: 1, pp 1 – 7.

Lynch, D., and Swing, S. P. (2006). ACGME Outcome Project. Retrieved February 22, 2010, from ACGME – The Outcome Project: http://www.acgme.org/outcome/assess/keyConsider.asp

Manzo, K. K. (2009). Draft Unveiled on Technological Literacy for NEAP. Education Week, August 26, 2009, p. 9.

Martin, A., Grudziecki, J. (2006). DigEuLit: Concepts and Tools for Digital Literacy Development, University of Glasgow, Scotland. Retrieved online April 30, 2010 from: ics.heacademy.ac.uk/italics/vol5iss4/martin-grudziecki.pdf

Massachusetts Department of Education. (2009). Technology Standards – Education Technology. Retrieved February 22, 2010, from Massachusetts Department of Elementary and Secondary Education: http://www.doe.mass.edu/edtech/standards/sa_tool.html

Meyer, K., Hunt, S., Hopper, K. M., Thakkar, K., Tsoubakopoulos, V., and Van Hoose, K., (2010). Assessing Information Literacy Instruction in the Basic Communication Course, Communication Teacher, 22:1, 22 – 34

National Association for Media Literacy Education. Retrieved online May 1, 2010 from: http://www.amlainfo.org/media-literacy/definitions

Oblinger, D. G., and J. L. Oblinger, eds. (2005). Educating the Net Generation. Washington, D.C.: EDUCAUSE. Retrieved online February 22, 2010 from:  http://www.educause.edu/educatingthenetgen

O’Connor, B. (2005). Digital Literacy. University Journal News, 8 (8), pp. 1-2, California State University, Fresno. Retrived online February 23, 2010 from: http://www.csufresno.edu/journal/vol8/08/news/digital.shtml

O’Connor, L. G., Radcliff, C. J., and Gedeon, J. A. (2001). Assessing Information Literacy Skills: Developing a Standardized Instrument for Institutional and Longitudinal Measurement. ACRL Tenth National Conference. Denver, CO.

Pereira, A., Oliveira, I., Tinoca, L., Amante, L., de Jesus Relvas, M., do Carmo Teixeira Pinto, M., (2009). Evaluating continuous assessment quality in competence-based education online: the case of the e-folio. Retrieved online February 20, 2010, from European Journal of Open, Distance and E-Learning: http://www.eurodl.org/?article=373

Pérez, J., Coffin-Murray, M. (2010). Generativity: The New Frontier for Information and Communication Technology Literacy, Interdisciplinary Journal of Information, Knowledge, and Management, Volume 5, 2010.

Pietrass, M. (2007) Digital Literacy Research from an International and Comparative Point of View, Research in Comparative and International Education, 2(1), 1-12. Retrieved February 23, 2010 from: http://dx.doi.org/10.2304/rcie.2007.2.1.1

Pruitt-Mentle, D. (2000). The C3 framework: Cyberethics, cybersafety and cybersecurity implications for the educational setting. Retrieved online February 23, 2010, from Know Where They Go: http://knowwheretheygo.org/static/content/MATRIX.pdf

Purdue University Students Take New ETS ICT Literacy Assessment; New Test Measures Students’ Ability to Process and Communicate Information in a Technological Environment. Business Wire (March 16, 2005). Retrieved online April 29, 2010 from: http://www.highbeam.com/doc/1G1-130343304.html

Sieber, V. (2009). Diagnostic online assessment of basic IT skills in 1st-year undergraduates in the Medical Sciences Division, University of Oxford, British Journal of Educational Technology 40 no. 2 215-26 Mr 2009

Simmons College Placement Examinations, Center for Academic Achievement. Retrieved online April 29, 2010 from: http://my.simmons.edu/academics/caa/placementexams.php

Stokes, Suzanne (2002). Visual Literacy in Teaching and Learning: A Literature Perspective, Electronic Journal for the Integration of Technology in Education, Idaho State University. Volume 1, No. 1, Spring 2002.

U.S. Department of Education. (1996). Getting America’s students ready for the 21st century: Meeting the technology literacy challenge. Retrieved online May 1, 2010 from: http://www2.ed.gov/updates/PresEDPlan/part11.html

Van Joolingen, W. (2004). The PISA framework for assessment of ICT literacy, (Powerpoint presentation). Retrieved online May 1, 2010 from: http://www.ictliteracy.info/rf.pdf/PISA%20framework.ppt

Winnepeg School Division (2010). Winnipeg School Division Technology Outcomes Continuum Guide (K-S1). Retrieved online May 1, 2010 from: http://www.wsd1.org/techcont/communication_literacy.htm

Appendix A – Online Resources Related to Digital Literacy

Instruments and Resources for Digital Literacy and ICT Literacy:

A Web portal for information and research on ICT Literacy. Retrieved May 1, 2010 from: http://www.ictliteracy.info/

2012 National Assessment of Educational Progress (NAEP) technological literacy framework – 11/04/09 discussion draft. Intended for K-12 assessment on a national scale, in trials for national implementation in 2012. Retrieved April 30, 2010: http://www.edgateway.net/cs/naepsci/view/naep_nav/9

Arizona TechLiteracy Assessment (TLA), by Learning.com. A large-scale online instrument used to measure middle-school students for proficiency in practical application and problem-solving skills, using authentic simulation. Intended to take only the duration of a singe class period. Not intended to be high-stakes, though offers individual student results. Piloted in 2006. Retrieved February 23, 2010 from: http://www.learning.com/casestudies/arizona.htm

An ICT Literacy development project from University of Alberta, CA, measuring undergraduate participants – 2000. Retrieved February 23, 2010 from:

http://www.editlib.org/d/9843/proceeding_9843.pdf

http://www.quasar.ualberta.ca/it/research/Szabo/Edmedia02.pdf

Syracuse University Center for Digital Literacy: http://digital-literacy.syr.edu/site/view/80

Australian national ICT Literacy assessment program in years 6 and 10, from Australian Ministerial Council on Education, Employment, Training and Youth Affairs. Measures information and communication technologies literacy, based on the International ICT Literacy Panel report, 2003 (ETS), and the Italian OECD PISA ICT Literacy Feasibility Study, 2002. Retrieved April 30, 2010, from:

http://www.mceecdya.edu.au/mceecdya/nap_ictl_2005_years_6_and_10_report-press_release,22065.html

http://www.mceetya.edu.au/verve/_resources/ICTL2005_Assessment_Exemplars.pdf

http://www.mceecdya.edu.au/mceecdya/nap_ictl_2008_report_press_release,31023.html

A historical description and analysis of Digital Literacy, and component literacies from a European perspective. Retrieved online April 30, 2010 from: http://www.flacso.edu.mx/competencias/index2.php?option=com_docman&task=doc_view&gid=167&Itemid=9

Resources for Information Literacy (library and research related skills via ICT):

 

The Bay Area Community Colleges Information Competency Assessment Project, a collaborative project among faculty librarians in the San Francisco Bay Area. Retrieved February 23, 2010 from:

http://www.topsy.org/ICAP/ICAProject.html

Network of Illinois Learning Resources in Community Colleges, Information Literacy Toolkit. The Toolkit for Success is designed to help teachers/faculty and librarians work together to address the information literacy needs of their at-risk high school and community college students. Retrieved February 23, 2010 from: http://www.nilrc.org/IMLS/default.asp

Information Seeking Skills Test (ISST), developed by James Madison University. Retrieved February 21, 2010 from: http://muse.jhu.edu/journals/journal_of_general_education/summary/v052/52.4demars.html

The Information Literacy Test (ILT) is a computerized, multiple-choice test developed collaboratively by the James Madison University (JMU) Center for Assessment and Research Studies (CARS) and JMU Libraries. It is designed to assess the ACRL Information Literacy Competency Standards for Higher Education. Retrieved February 21, 2010 from: http://www.madisonassessment.com/assessment-testing/information-literacy-test/

A comprehensive collection of assessment instruments and rubrics compiled by Jon Mueller, Professor of Psychology at North Central College, and author of “The Authentic Assessment Toolbox”. Retrieved April 27, 2010 from: http://jonathan.mueller.faculty.noctrl.edu/infolitassessments.htm

 

Various resources, blogs, aggregates, wikis on Information Literacy:

http://en.wikipedia.org/wiki/Information_literacy

http://information-literacy.blogspot.com/

http://www.pageflakes.com/informationliteracy/

http://www.jmu.edu/assessment/resources/prodserv/instruments_ilt.htm

Project TRAILS, (also developed by Kent State) a self-guided, self-administered assessment tool designed for use by library media specialists and teachers to determine the information literacy competencies of their high school students. Librarians and teachers at other grade levels may find it of use as well. Once an assessment is selected, the library media specialist or teacher can choose how to use it to serve local needs. Retrieved February 23, 2010 from: http://www.trails-9.org/index.php?page=home

Research Readiness Self-Assessment (RRSA), from Central Michigan University. Retrieved February 23, 2010 from: http://rrsa.cmich.edu/twiki/bin/view/RRSA/Versions

RRSA Sample (by request): http://rrsa.cmich.edu/cgi-bin/rrsahp.cgi

Lehigh University’s Banner Survey Tool for Information Literacy. Retrieved February 23, 2010 from:

http://net.educause.edu/ir/library/pdf/MAC07038.pdf

http://www.educause.edu/Resources/AssessingStudentInformationLit/159073

Technology Assessment tools:

Simple K’s student technology assessment. Retrieved February 22, 2010 from:

http://www.simplek12.com/student-technology-assessments-are-now-simple-finally-0

Massachusetts Technology Self-Assessment Tool: A technology self-assessment tool for teachers, K-12 school districts, or statewide measurement. Retrieved online February 22, 2010 from:

http://www.doe.mass.edu/edtech/standards/tsat_revisions.pdf

http://www.doe.mass.edu/edtech/standards/sa_tool.html

iKeep Safe Digital Citizenship Matrix – the C3 Matrix. Used as a basis for assisting educators in integrating concepts of cyber-security, cyber-ethics and cyber-safety. Retrieved online February 22, 2010 from: http://www.doe.mass.edu/edtech/standards/tsat_revisions.pdf

Digital Citizenship, addressing appropriate technology behavior – NETS-S, NETS-T standards based strategies. Retrieved online February 19, 2010 from: http://teacherline.pbs.org/teacherline/courses/TECH340/docs/TECH340_bailey.pdf

European Computer Driving Licence, the certifying body for computer literacy in Europe. Focuses on practical computer skills.

http://www.ecdl.com/publisher/index.jsp

http://www.icdlus.com/

Organisation for Economic Co-operation and Development (OECD), Programme for International Student Assessment (PISA): http://www.oecd.org/pages/0,3417,en_32252351_32235731_1_1_1_1_1,00.html

Media Literacy/Education/Studies:

An article about defining and distinguishing between Media Literacy, Media Education, and Media Studies, featuring opinions of global leaders in media theory and pedagogical practice. Alexander Fedorov – Media Education: Sociology Surveys. Taganrog: Kuchma Publishing House, 2007.  228 p. Retrieved online February 23, 2010 from: http://www.edutubeplus.info/resources/media-education-sociology-surveys

Item Response Theory:

http://en.wikipedia.org/wiki/Item_response_theory

http://echo.edres.org:8080/irt/baker/

Appendix B

National Educational Technology Standards for Students 2007: http://www.iste.org/nets/