From EvaluationWiki

Contents 1 EvaluationWiki 1.1 EvaluationWiki Needs YOU! 1.2 EvaluationWiki Skeleton 1.2.1 Evaluation Theory 1.2.2 Evaluation Practice 1.2.3 Evaluation Procedures 1.2.4 History of Evaluation 1.2.5 Evaluation Education 1.3 EvaluationWiki Resources

EvaluationWiki

EvaluationWiki was founded in September of 2006 by the non-profit organization Evaluation Resource Institute (ERI). The mission of EvaluationWiki is to make freely available a compendium of up-to-date information and resources to everyone involved in or interested in the science and practice of evaluation. This compendium will be a continually growing and evolving representation of evaluation knowledge.

Like the very successful and popular Wikipedia, EvaluationWiki allows everyone to contribute. Whereas the mission of Wikipedia is to create a compendium of all knowledge, the mission of EvaluationWiki is to create a compendium of knowledge within the domain of evaluation. We have a long way to go before EvaluationWiki will become a valuable resource that people around the world will turn to first for evaluation information and resources. But as the Chinese Proverb says, "A journey of a thousand miles begins with a single step." We ask that you take that first step with us by creating or editing the information contained here on EvaluationWiki.

EvaluationWiki Needs YOU!

EvaluationWiki will become a valuable resource for the evaluation community if professional evaluators, students, professors, educators, administrators, and anyone interested in any of topics falling under the broad heading of evaluation contributes regularly.

How do you get started?

First, you will need to Create an Account. Unlike some wikis such as Wikipedia, EvaluationWiki will maintain the quality of the information contributed by requiring users to identify themselves -- there will be no anonymous editing of EvaluationWiki. In addition, each user will be encouraged to complete their profile (simply click on your user name to edit your profile). What you put there is entirely up to you. Posting information about yourself allows the community to get to know you as an individual and understand the perspective that you bring to your contributions. For example, simply placing your curriculum vita there would be a great way to do this. That way, users of EvaluationWiki can assess the information that individuals contribute both on the merits of the contributed information, but also based on the credentials of the contributor.

You've created an account. What can you do to contribute to EvaluationWiki?

Currently (or in the very near future), your contributions to EvaluationWiki can be in one of four functions:

1. Adding to or editing the Evaluation Textbook.
2. Posting opinions about evaluation topics using the Discussion Tab.
3. Contributing to EvaluationWiki Publications in one of three ways:
   1. submitting original evaluation-related articles for peer review and publication;
   2. assisting in the activities related to posting and linking existing full-text articles for which we have permission to include on the site;
   3. contributing to the EvaluationWiki Bibliography.
4. Posting information you wish to share about past or current projects in EvaluationWiki Projects.

The EvaluationWiki Textbook

In the EvaluationWiki Textbook the community posts and edits entries about the various topics that we collectively think comprise the science and practice of evaluation. We envision the entries written in a textbook or encyclopedia-like style. The entries should be a straightforward, scholarly discussion of a given topic.

This is not the place to comment or to post opinions about a topic, but it is the place to edit, or expand the factual information. I know that this can be a fine line and I think we will just have to see how it develops. For example, take an entry on qualitative methods. Someone may see a factual error or omission and edit or expand the entry. If, however, they disagree with the use of qualitative methods they should not post their “opinion” in the encyclopedia. Opinions about evaluation should be posted in the Discussion Tab.

EvaluationWiki Discussion Tab

Using the EvaluationWiki Discussion Tab, the community posts opinions about evaluation topics. Continuing with the example above, if someone thinks qualitative approaches are too “soft” and don’t result in the scientific, quantitative-type information that comprise quality evaluation, they would post that opinion here in the “commentary room”. Others could disagree and carry on a spirited debate, but not in the entry about “qualitative methods” in the textbook or encyclopedia. I think we should have a link or some indication from the entry in the EvaluationWiki Textbook to indicate that there are opinions and/or a continuing or archived debate on the topic.

EvaluationWiki Publications

EvaluationWiki Publications is for publication of evaluation resources EvaluationWiki Publications is divided into three parts. The first part is for original, peer-reviewed articles about evaluation-related topics submitted for peer-review by individuals in the EvaluationWiki community. This process should be rigorous enough to allow an individual to validly list the publication on their vita. For our colleagues in the academic world, publication of the article on EvaluationWiki could be credited toward their tenure and promotion process.

The second part of EvaluationWiki Publications is for posting full-text articles that can be searched and linked to other areas of EvaluationWiki or to the worldwide web. For example, we may get permission from an author to post a full-text article about meta-evaluation. That article could be linked to the encyclopedia entry, to the biography of the author, and to their homepage at their non-profit institution. We would carefully vet articles posted here for the proper legal permission to post.

The third part of EvaluationWiki Publications is for a bibliography of categorized, useful articles on various topics for which we cannot get permission to download the full text. These articles could be linked to other EvaluationWiki resources.

EvaluationWiki Projects

EvaluationWiki Projects will be used for posting information about evaluation projects (either current or past) about which our community members have knowledge. This area would contain summaries of evaluation projects that the author or contributor chose not to publish or could not get published in another venue for any number of reasons. Standard information would be requested on each project like: 1) topic area; 2) sample size; 3) evaluation design; 4) types of data collected; 5) evaluation questions; 6) grant program; 7) funding agency; 8) funding amount; 9) amount of evaluation budget, etc. [Note: this is not fully developed and we would like help on getting it right.] The project data files could also be posted and linked to the project description. Of course, identifying information at all levels could or would be stripped out of the files.

This area could develop into a valuable resource for research. For example, there are numerous Teaching American History projects across the country that were funded through the U.S. Department of Education. We could post information about these individual projects in EvaluationWiki Projects as could other community members. Eventually, the site could contain a rich, extensive database of contributions about Teaching American History projects that could, for example, serve as a basis for a meta-analysis of knowledge gains of teachers involved in the project.

You could just start writing (or adding to) articles on evaluation topics with which you are most familiar. Or you may have some relevant text for which you hold the rights; that book chapter you never got around to publishing might as well get some use. Share your thoughts and ideas with the world. Placing that text in the appropriate articles within EvaluationWiki would be a great service to the project.

There are also many public domain documents regarding evaluation that have been published and are available online. Because these documents are public domain, they can be "harvested" for text that can be placed within EvaluationWiki. An EvaluationWiki article has been created to list these public domain documents. We strongly encourage the participation of students enrolled in evaluation degree programs to contribute. Students - ask your university faculty members about giving you program credit for participating in the development of EvaluationWiki. After all, what better way to learn about evaluation?

As you browse through the existing articles in EvaluationWiki you may find information that you believe to be incorrect, outdated, or incomplete. This is because EvaluationWiki is new and very reliant upon public domain resources at this stage. Please help us to make the content better.

EvaluationWiki Skeleton

Below we present a very rough "skeleton" of the domain of evaluation. This skeleton was developed by referencing the organizational structure of commonly used evaluation textbooks (Worthen, Sanders, Fitzpatrick, 1997). We present this skeleton below to make it easy for you to locate where your contributions are needed. Please click anywhere on the skeleton below to access the relevant article. If the article for that topic does not yet exist - great! You'll be presented with the opportunity to create the article yourself. By no means do we think that this skeleton is comprehensive or definitive; we present it merely as a jumping off point. If you think you can do better, you're probably right. You can even edit the skeleton. Just click the 'edit' button to the right to edit the skeleton.

Evaluation Theory

• • Informal vs. Formal Evaluation
  • Purposes of Evaluation
  • Uses of Evaluation
  • Essential Activities of Evaluation
  • Formative and Summative Evaluation
    • Summative evaluation
    • Formative evaluation
  • Types of Evaluation
    • Program Evaluation
      • Outcome-based Evaluation
      • Objectives-Oriented Evaluation
      • Management-Oriented Evaluation
      • Consumer-Oriented Evaluation
      • Expertise-Oriented Evaluation
      • Adversary-Oriented Evaluation
      • Participant-Oriented Evaluation
    • Product Evaluation
    • Personnel Evaluation
    • Self Evaluation
    • Advocacy Evaluation
    • Policy Evaluation
    • Organizational Evaluation
    • Cluster Evaluation
    • Meta Evaluation
      • Joint Committee Standards for Program Evaluation
      • AEA Guiding Principles for Evaluators

Evaluation Practice

• • Proposal Solicitation and review
  • On-Site Evaluations
  • Technical Assistance to Programs and Projects
  • Data
    • Internal validity
      • Threats to Internal Validity
    • Collecting Data
      • Instruments
        • Pilot testing
        • Surveys
          • Survey Sampling
          • Steps in Surveying
          • Survey Development Timetables
          • How to Use a Random Number Table
          • Rating or Intensity Scales
          • Sample Sets of Response Alternatives
          • Questionnaire
        • Assessments
        • Sample Instruments
          • Data Collection Instruments for Local Use
          • Higher Education Center's Instruments Database
      • Quantitative Methods
        • Sampling
        • Cost Analysis
        • Data Analysis
      • Qualitative Methods
        • Observations
        • Qualitative Data Analysis
      • Mixed-Methods
      • Planning and Organizing Data Collection
      • Data Collection Designs
        • Causal Data
        • Descriptive Data
    • Managing Data
    • Interpreting Data
    • Reporting Data
      • Purposes of Evaluation Reports
      • Reporting the Results
  • Clients
  • Evaluation Planning
    • Logic Model
    • Evaluability Assessments
    • Selecting an Evaluator
    • Identifying Stakeholders
      • Internal Evaluation
      • External Evaluation
    • Evaluation Contracts and Agreements
  • Practical Guidelines for Conducting and Using Evaluations
    • Political, Ethical and Interpersonal Considerations in Evaluations
      • Ethical Standards
        • Joint Committee on Standards for Educational Evaluation
      • Evaluation Certification
      • Conflicts of Interest
  • Evaluators
  • Evaluation Glossery Terms
    • Frequenty Used Words in Evaluation
      • List of Evaluation Glossery Terms
  • Evaluation in developing countries

Evaluation Procedures

Legend: (m) = meeting; (d) = decision point; (i) = independent work

• • Pre-Proposal Development
    • Derive the purpose for doing the evaluation (m)
    • Decide whether to pursue the project (d)
    • Determine Responsibilities (stakeholder signoff) (m)
    • Develop Logic Model draft (m). Logic Modeling Resources
    • Derive the evaluation questions from the logic model
    • Create a measures chart draft by linking the evaluation questions to the appropriate method (see list of methods). (i,m)
    • Ensure that the methods satisfy the following criteria. (i)
    • Complete the evaluation plan by modifying the proposal text. (Sample)
    • Submit this evaluation plan as part of the proposal.
  • Plan
    • Finalize the Logic Model and Measures Chart (m)
    • Determine Desired Transfers (what does the program expect to transfer from instruction into practice?) (m)
    • Develop a Draft Timeline (i,m)
    • Agree on expectations
    • Complete IRB
    • Arrive at a Contractual Agreement with an account to charge to.
    • Potential Methods
  • Identify & Implement Methods
    • Develop instruments as necessary
    • Gain approval for instruments from stakeholders
    • Potential Methods
  • Gather & Analyze Data
    • Establish who will collect the data (e.g. who will proctor exams, administer survey, lead interview, etc.?). Final Timeline
    • Ensure study participants have the right instruments/forms (including consent forms)
    • Collect instruments/forms
    • Analyze Data to Derive Findings (e.g. scan bubble forms, categorize and count themes from focus groups, etc.)
  • Report
    • Create a master report document (update with findings from each new data source. Gary Miron's Report Checklist.)
    • Report initial findings from each data source
    • Add findings to the master report (alter findings and recommendations as necessary).
    • Interpret Data (ask the stakeholder what it really mean from the their perspectives)
    • Conclusions (what logical explanations can we draw?)
    • Recommendations for action (what do we do about it?). Negotiate this with stakeholders
    • Check your report for quality
    • AEA Framework
    • AEA Guidelines

History of Evaluation

• • Biographies of Prominent Evaluators
    • Michael Quinn Patton
  • Origins of the Discipline of Evaluation
  • Recent Trends in Evaluation

Evaluation Education

• • Careers in Evaluation
  • Jobs in Evaluation
  • Evaluation Preparation
    • Evaluation Conferences
      • Consortium for Research on Educational Accountability and Teacher Evaluation
    • Evaluation Degree Programs
      • Interdisciplinary Ph.D. in Evaluation
      • Masters or Doctoral Degree in Evaluation
    • Evaluation Training
      • Accredited
        • Certificate of Advanced Study in Evaluation
      • Non-Accredited
        • Certified Grants Evaluator
  • Evaluation Journals
  • Evaluation Professional Organizations
  • Evaluation Discussion Boards or Listservs
    • EvalTalk
    • Evaluation Cafe

EvaluationWiki Resources

Consult the User's Guide for information on using the wiki software.

• How to Use the Wiki - Manual
• Free resources for methods in evaluation
• Public Domain Evaluation Resources