Validity and Reliability

Reliability is the consistency of a test in measuring something.

If you give the same test to the same student three times and the results are 80 - 65 - 90, the test is not consistent in measuring the students' ability. The test has low reliability.
If you give the same test to the same student three times and the results are 85 - 85 - 85, the test is consistent in measuring the students' ability. The test has perfect reliability.

Reliability is not black and white, you cannot say the test is either reliable or not reliable. Instead you have to express the reliability in the form of degrees: very low, low, moderate, high, very high, perfect. (Remember about this when we talked about correlation in the statistics classes?)

Methods of estimating reliability are:

1. Test-retest: one test, administered twice.
2. Equivalent form: two similar tests, administered once.
3. Split-half: one test, administered once, split into two parts (odd & even numbers)
4. Internal consistency: one test, administered once, apply KR 21 formula.
5. Interrater reliability: one test, administered once, scored by 2 people.

Which method should you use?
If you use an objective test in your skripsi, I suggest internal consistency (KR21).
If you use an essay test, I suggest internal consistency (Cronbach Alpha).
If you use a writing test, I suggest interrater reliability (Pearson r).

Validity refers to the relevance of the content of test in measuring something.

If you use a math test to measure the students' English proficiency, the test is not valid.
If you use a reading test to measure the students' writing ability, the test has very low validity.
If you use a reading test to measure the students reading ability, the test has very high validity.

Methods of estimating validity:

1. Content validity: the test should representatively contain the items that are supposed to be measured. Map the content of your test against the content of the curriculum/syllabus/lesson plans. If they match, your test has high content validity.
2. Criterion-related validity: the content of a test is related with another test (a criterion). Correlate the results of your test and the results of a standardized test (the criterion). If the correlation is high, your test has high validity, too.
3. Construct validity: the test should measure the construct well, e.g. a reading test should measure the construct 'reading comprehension', a grammar test should measure the construct 'grammar ability', etc.

Criterion-related validity can be measures in two ways:

1. Concurrent validity: Administer your test and the standardized test at the same time, e.g. administer your test and the standardized test today.
2. Predictive validity: Administer your test, then administer the standardized test after several months have passed, e.g. administer your test today, administer the standardized test next semester.

Construct validity can also be measured in two ways:

1. Pilot study: administer the test to two groups of subjects, one group has the construct while the other does not have the construct. If the former scores higher than the latter, the test has construct validity.
2. Intervention study: administer the test to a group who does not have the construct as a pretest. Give treatment to the group by teaching the construct, then give them a posttest. If there's a significant difference between the results of pretest & posttest, the test has construct validity.

Which method should you use?
If you use an objective test in your skripsi, I suggest content or concurrent validity.
If you use an essay or writing test, I suggest content validity.

Research Instruments

Research instruments are the tools that you use to get data (words or numbers).

The instruments commonly used in educational research are: tests, questionnaires, interviews and observations.

Tests are categorized into two types:

1. Objective tests, such as true-false, multiple choice, matching, completion, etc.
2. Subjective tests, such as essay and writing tests.

If you use a test as the research instrument, you need to estimate the reliability and validity of the test. These characteristics of the test will be elaborated in a different entry.

Questionnaires are categorized into two types:

1. Structured: It has a fixed number of questions and a limited choice of answers. The respondents should choose one or more suitable answers. The aim of administering this questionnaire is to get facts.
2. Unstructured: It consists a number of questions. The answers are in the form of essay and the content may vary depending on the respondents. The aim of administering this questionnaire is to explore something.

Interviews are categorized into three types:

1. Structured: It has a fixed number of questions and limited choice of answers. It's like a structured questionnaire, but the questions are spoken by the researcher to the respondents.
   
2. Semi-structured: It has a few questions as guidelines, and can be added when necessary.
3. Unstructured: There are no questions at all in the beginning. The researcher determines the questions after immersing in the field for some time. The things to be asked will appear after the immersion.

Observations are categorized into two types:

1. Participatory: The researcher participates in the activities of the subjects, e.g. You conduct a study in SMA 1 and replace the teacher in the classroom. You teach and observe the students.
2. Non-participatory: The researcher does not participate in the activities of the subjects, e.g. You conduct a study in SMA 1. You only sit in the back of the classroom and observe the teaching-learning process.

When you observe your research subjects, you need one of these two things to help you collect data:

• Checklist: a list of things that you look at when observing something/someone.
• Field notes: anything you write when observing something/someone.

Some issues about research

In the last session you asked me lots of good questions about research. I need to post the issues you addressed here.

1. Terms
These terms are synonymous: research, study, investigation, inquiry.
So in this context, study is not the same as learn. Rather, it refers to research.

2. Quantitative vs. Qualitative
Which one is better? Good question! The answer is: both.
The best research is the one that can provide the most appropriate answer to your research questions (problems). If you are wondering whether your teaching technique is effective and want to find out how many students can succeed when taught using that technique, then you must conduct quantitative research. If you conduct qualitative research to find out about this, then you're heading for the wrong direction. In this case, quantitative research is better than qualitative one.
But if you want to know how to apply a certain teaching technique in the classroom, then you must conduct qualitative research. In this case qualitative research is better than quantitative one.

3. Quantitative and Qualitative
One of you asked whether you could use both the quantitative and qualitative approaches in a single study. Good news, the answer is: yes. It's called mixed method. You need it when you have, for example, the following research questions:
1. How can songs be used to teach English to the 5th graders at Mbebekan elementary school? (qual.)
2. Can the use of songs improve the 5th graders' vocabulary size? (quan.)

4. Observation
The example of quantitative & qualitative observation can be downloaded here.

5. Questionnaire
The example of quantitative & qualitative questionnaire can be downloaded here.