Introduction Research, according toHatch and Lazaraton (1991), means trying to provide answers to important questionsor problems in an organized, systematic way that generally leads to discoveryor interpretation of facts. Research is therefore believed to contributeprofoundly to knowledge in particular fields. In order to do successfulresearch which yields valid and reliable results, methodologies need to becarefully chosen and should reflect the researcher’s overall orientation totheir research. In this essay, I will give a commentary on two most populardata collection methods – questionnaires and observation. I will begin with abrief definition of two methods, explaining when and why each is used incertain context and drawing on my own personal experience as a researcher.
Theirstrengths and weaknesses will then be discussed and put in comparison when necessary.Lastly, I will identify any potential problems when using these two methods. 1. When to use and why to use? 1.1 QuestionnaireDefined by Brown (2001:6)as written instruments that give participants structured, predefined questionsor statements “to which they are to react either by writing out their answersor selecting them among existing answers”, questionnaire is possibly the mostcommonly used method in surveys (Dörnyei, 2007). According to Young (2016),questionnaires are mostly used to report participants’ background, behavioursand demographic information. Wallace (1998) adds that questionnaires can beused to elicit various kinds of data, such as personal perceptions, experiences,or attitudes.
In a classroom context, Wallace argues that questionnaires mightbe favourable when the researcher wants to touch upon “knowledge, opinions,ideas, and experiences of learners, fellow teachers or parents” (1998:124). Thismethod tends to produce typically quantitative data. 1.2 Observation Rather than collectingself-report data like questionnaires, which primarily bases on what respondentstell the researcher (Dörnyei, 2007), observation is a direct method that “drawson the direct evidence of the eye to witness events first hand” (Denscombe: 2007:192).This calls for a more active role of researchers who need to go in quest ofinformation and observe what actually happens.
Observation is particularlyappropriate for classroom research, in which many aspects of teaching,learning, or interaction could be usefully investigated. According to Hopkins (1993),observation plays a vital role in teachers’ professional development. This isbecause this method may attest how fixed values, beliefs or opinions have beenapplied in actual setting; or how teaching approaches are put into practice withtheir real effectiveness (Heighham & Croker, 2009). In agreement with thisviewpoint, Anderson, Herr and Nihlen (1994) claims that observation can be auseful tool to “help demystify what is actually going on as opposed to what onemight hope or assume is happening” (p.
129). It should also be notedthat observation might be used in conjunction with other methods to add moreevidence and increase the validity of findings (Heighham and Croker, 2009).Followings are two Bachelor’ theses in my university. a.
Ways teachers deal with students’ oral errors at the production stageand their effects – a case study in APAX English center. b. The application of task-basedapproach to English Speaking activities for first year fast track students bythe teachers of English in FELTE, ULIS, VNU.Rather than being usedexclusively on its own, observation was employed along with interviews and/orquestionnaires in these theses. This method offered preliminary informationabout the respondents’ external behaviours and was followed up by surveys whichinvestigate their inner beliefs, gaining more insights into the issues. 2. Strengths and Weaknesses 2.1 Questionnaires 2.
1.1 Strengths:Manyresearchers reach a consensus on the wide coverage of questionnaires, which givesit an edge over other methods (Denscombe, 2007; McDonough & McDonough, 1997).It helps gathering considerable amount of information, from a large number ofrespondents and by different means of collecting data in a cost-effective way. Additionally,Dornyei (2007) claims that questionnaires are a highly versatile tool of surveywhich allows the use of both closed-ended and open-ended questions, coveringvarious topics. Secondly, the value of data is at its best thanks to the use of”pre-coded answers” from participants, which “fit into a range of optionsoffered by the researcher” (Denscombe, 2007:159).
These data can later be usedto identify any major differences among different research’s findings, toconfirm, reject or form new hypothesis (Kelley et al, 2003). Lastly, Denscombeemphasizes that not only having advantage for the researchers, questionnairesare also respondent-friendly since they only need to choose from existingoptions which are “spelt out for them” (2007:159) without much time or effortinvested in thinking of the answers. 2.1.2 Weaknesses There aresome main criticisms regarding the validity of questionnaires.
First, Dornyei (2007:115) argues thatquestionnaires could only provide “thin description of the target phenomena”,which may lead to “superficial data” and limits in-depth investigation. Inaccordance with Dornyei’s perspective, Young (2016) shows that questionnairesonly present the issue on the surface level; for example, the ranking system ina questionnaire may reflect more or less of different categories but neitherindicates nor explains the extent of how much more or less, in which differentresearchers may have different interpretations. Another weakness ofquestionnaire, pointed out by Wallace (1998:124) is that this “introspective”method only provides what the respondents choose to tell. Consequently,checking the reliability and truthfulness of responses remains a constantchallenge. Furthermore,heavily relying on self-report, questionnaires may face the presence ofresponse bias such as the social desirability bias (or prestige bias) whichshows the tendency of people to choose acceptable or desirable answers which donot tell “what they actually feel or believe” (Dornyei, 2003:12). Another biasdetected by Paulhaus (1991) is acquiescence bias, indicating respondents whoare more inclined to positive connotation and reluctant to provide strongnegative responses.
Lastly, contributing to response bias, the halo effectconcerns the human tendency of overgeneralization, showing how the overallimpression of something or somebody may influence respondent’s opinions orattitudes when it comes to specific details (Oppenheim, 2000). These bias,mainly coming from the respondents, may heavily affect the validity of thequestionnaire. Denscombe (2007) also points out another potential for thepresence of bias which creeps in the pre-coded questions. He argues thatquestionnaires may reflect the researcher’s mindset rather than respondents’ thinkingsince pre-coded questions “shape the nature of the responses”, resulting inanswers being directed away from the respondents’ “perceptions of matters tofit in with a line of thinking established by the researcher”(p.160). 2.2 Observation2.2.
1 StrengthsSimilar to questionnaires, observationalso offers a large amount of information regarding participants’ behaviour ina specific setting (Mackey & Gass, 2005). However, if questionnaires tendto miss the insights that could be provided by the participants, the mainstrength of observational data is that it can “allow the researcher to gaindeeper and more multi-layered understanding of the participants and theircontext” under investigation (Mackey & Gass, 2005:176). Moreover, Dornyei(2007) believes that allowing researchers to immerse themselves in the settingcould present more objective data than “second-hand self-report” one (p.185).Interestingly, he claims that two approaches of collecting data, namely theunstructured and highly structured approach, might sometimes be combined toreach the best effect. The former, often associated with participantobservation and believed to generate qualitative data, tries to find out “notjust what happens but why it happens” (Dunkerton, 1981:145). However, sincethis depends predominantly on the subjective view of the researcher, thereliability of data may be doubted. On the other hand, a highly structuredapproach, which usually refers to systematic observation, decides the focus ofresearch beforehand through the use of observation scheme.
Though the focus ison overt behaviours, not on why they happen, the data associated with”selective perception of observers” may generate objective observations,reducing observer’s bias, in terms of emotions or personal background.(Denscombe, 2007:199). In line with Dornyei’s (2007), Dunkin and Biddle (1974)proposes a combination of both approaches, which was effectively applied in Casea (part 1.2). For the first time, an unstructured approach was adopted; I actedas a teaching assistant being involved in the classroom’s activity. For thesecond attempt, I then constructed a data-collection scheme on the premise ofthe collected information and acted as a complete observer. This proved to bean effective blend as the chance to miss interesting events is likely todecrease, with a more focus on what is of importance of the research.
2.2.2 Weaknesses Mackey and Gass (2005) assumes thatobservation does not reveal motivation behind participants’ particular actionsor behaviours, in contrast with questionnaires which score high in terms ofclear indication of motivational sources. Another weakness of observation,identified by Denscombe (2007:193), relates to the issue of observer’s”psychology of memory and perception”. To be more specific, he mentions threepsychological factors that may greatly influence observation’s direction:”selective recall” relates to the limited capacity of memory, deterringresearcher from remembering all situations that are being observed; “selectiveperception” explains how some specific information is retained why some isrejected; and “accentuated perception” which shows how physical and mentalcondition and past experiences may affect the observation process. These mightbe legitimate reasons why findings are interpreted differently amongresearchers, which may decrease the research’s reliability.
3. Problems/issues that need to be addressed3.1 QuestionnairesDesigningwell-constructed questionnaires is a process that requires careful attentionsince researchers barely have opportunities to make adjustments once thequestionnaire has been distributed to the participants.
There exists problemsduring the designing process and after the questionnaires are collected to be analysed.As for thedesigning process, Wallace (1998) and Nunan (1992) emphasizes that theambiguity of the questions may cause misunderstandings and misinterpretationfor participants. Sharing the same viewpoint, Young (2016) claims that itemswording in the questions can assume “an unexpected importance”, in which justminor differences can trigger “different levels of agreement or disagreement”(p.
103), particularly in the case when non-factual data such as attitudinaldata is assessed. Another problem lies in the use of Likert scale. Worcesterand Burns (1975) found that a balanced four-point Likert scale without midpointpushed respondents, especially those who slightly lean towards either agreementor disagreement to the positive end of scale. However, McDonough and McDonough(1997) assumes that the existence of the neutral response might be hard tointerpret. Likewise, the no-opinion option should also be used with caution. Onthe one hand, Schuman and Pressure (1981) advocates the use of this optionsince it may reduce respondents’ pressure who have no relevant knowledge toanswer the question.
On the other hand, opponents of no-opinion option arguethat it may not enhance data quality and is likely to “preclude measurement ofsome meaningful opinions” (Krosnick et al, 2002:371). The researcher then needsto be careful when deciding whether or not to offer this type of option in thequestionnaire. Whenquestionnaires are collected for analysis, the issue of the comparability ofthe translated version should also be taken into consideration.
This “back-translation”process (McDonough & McDonough, 1997) must ensure to have closest meaning to theoriginal version to guarantee reliability. 3.2 ObservationThere aretwo major concerns while undertaking this kind of method. The first issue relates to thetranscription process. Different from data of questionnaires which can berelatively quickly processed, by either the researcher or the use of any dataanalysis software, presenting observational data remains a challenge. Bailey(2008) has listed several difficulties such as how to choose contextual datathat needs to be interpreted or how data should be presented, since it is notan easy task to “represent the full complexity of human interaction” on awritten transcript (p.
130). Secondly, as naturalsetting is highlighted as a condition for observation to take place (Denscombe,2007), it is advisable to avoid disrupting the naturalness of the setting tosee things as they normally occur. Denscombe (2007) notices the observer effect– the presence of the observer in the context- may make participants “aware of being studied- conscious of it- and then react in a way that isnot usual”, or “embarrassed, or disguise normal practice” (p.66). Regardingthis effect as “observer paradox”, Richards (2003:108) suggests that the use ofnote-taking should not be made apparent to the participant. He also affirmsthat the relationship between the observer and the participant should also begiven more thoughts, considering how it might affect individual and group’sperformances, and ultimately the research’s findings.
ConclusionIn this essay, I have discussed severalstrengths, weaknesses and relevant issues of two research methodologies:questionnaires and observation, each of which is a very common research toolthat is specifically designed for certain purposes. In many cases one datacollection method might be complemented by the use of other methods fortriangulation – a powerful technique to add depth to the data analysis as wellas improve the validity and reliability of the research. Both aforementionedmethods demand great attention and subtlety in many aspects of designingprocess.
Being aware of the potential problems researchers might face whenadopting these methods is then essential to raise the standards of goodresearch practice.
