Introduction Research, according to Hatch andLazaraton (1991), means trying to provide answers to important questions orproblems in an organized, systematic way that generally leads to discovery orinterpretation of facts.
Research is therefore believed to contribute profoundlyto knowledge in particular fields. In order to do successful research whichyield valid and reliable results, methodologies need to be carefully chosen andshould reflect the researcher’s overall orientation to their research. In thisessay, I will give a commentary on two most popular data collection methods -questionnaires and observation. I will begin with a brief definition of twomethods, explaining when and why each is used in certain context and drawing onmy own personal experience as a researcher. Their strengths and weaknesses willthen be discussed and put in comparison when necessary. Lastly, I will identifyany potential problems when using these two methods. 1.When to use and why to use? 1.
1QuestionnaireDefined by Brown (2001:6) as writteninstruments that give participants structured, predefined questions orstatements “to which they are to react either by writing out theiranswers or selecting them among existing answers”, questionnaire is possiblythe most commonly used method in surveys (Dörnyei, 2007). According to Young(2016), questionnaires are mostly used to report participants’ background, behavioursand demographic information. Wallace (1998) adds that questionnaires can beused to elicit various kinds of data, such as personal perceptions, experiences,or attitudes. In a classroom context, Wallace argues that questionnaires mightbe favourable when the researcher wants to touch upon “knowledge, opinions,ideas, and experiences of learners, fellow teachers or parents” (1998:124). Thismethod tends to produce typically quantitative data. 1.2Observation Rather than collecting self-reportdata like questionnaires, which primarily bases on what respondents tell theresearcher (Dörnyei, 2007), observation is a direct method that “draws on thedirect evidence of the eye to witness events first hand” (Denscombe: 2007:192).
This calls for a more active role of researchers who need to go in quest ofinformation and observe what actually happens. Observation is particularlyappropriate for classroom research, in which many aspects of teaching,learning, or interaction could be usefully investigated. According to Hopkins (1993),observation plays a vital role in teachers’ professional development. This isbecause this method may attest how fixed values, beliefs or opinions have beenapplied in actual setting; or how teaching approaches are put into practice withtheir real effectiveness (Heighham & Croker, 2009). In agreement with thisviewpoint, Anderson, Herr and Nihlen (1994) claims that observation can be auseful tool to “help demystify what is actually going on as opposed to what onemight hope or assume is happening” (p.129). It should also be noted thatobservation might be used in conjunction with other methods to add moreevidence and increase the validity of findings (Heighham and Croker, 2009).Followings are two Bachelor’ theses in my university.
a. Waysteachers deal with students’ oral errors at the production stage and theireffects – a case study in APAX English center. b. Theapplication of task-based approach to English Speaking activities for firstyear fast track students by the teachers of English in FELTE, ULIS, VNURather than being used exclusively onits own, observation was employed along with interviews and/or questionnairesin these research. This method offered preliminary information about the respondents’external behaviours and was followed up by surveys which investigate theirinner beliefs, gaining more insights into the issues. 2.Strengths and Weaknesses 2.1Questionnaires 2.
1.1Strengths:Many researchers reach aconsensus on the wide coverage of questionnaires, which gives it an edge overother methods (Denscombe, 2007; McDonough & McDonough, 1997). It helpsgathering considerable amount of information, from a large number ofrespondents and by different means of collecting data in a cost-effective way. Additionally,Dornyei (2007) claims that questionnaires are a highly versatile tool of surveywhich allows the use of both closed-ended and open-ended questions, coveringvarious topics. Secondly, the value of data is at its best thanks to the use of”pre-coded answers” from participants, which “fit into a range of optionsoffered by the researcher” (Denscombe, 2007:159). These data can later be usedto identify any major differences among different research’s findings, toconfirm, reject or form new hypothesis (Kelley et al, 2003). Lastly, Denscombeemphasizes that not only having advantage for the researchers, questionnairesare also respondent-friendly since they only need to choose from existingoptions which are “spelt out for them” (2007:159) without much time or effortinvested in thinking of the answers.
2.1.2Weaknesses There are some main criticismsregarding the validity of questionnaires. First, Dornyei (2007:115) argues that questionnaires could only provide”thin description of the target phenomena”, which may lead to “superficialdata” and limits in-depth investigation. In accordance with Dornyei’s perspective,Young (2016) shows that questionnaires only present the issue on the surfacelevel; for example, the ranking system in a questionnaire may reflect more orless of different categories but neither indicates nor explains the extent ofhow much more or less, in which different researchers may have differentinterpretations. Another weakness of questionnaire, pointed out by Wallace(1998:124) is that this “introspective” method only provides what therespondents choose to tell. Consequently, checking the reliability andtruthfulness of responses remains a constant challenge.
Furthermore, heavilyrelying on self-report, questionnaires may face the presence of response biassuch as the social desirability bias (or prestige bias) which shows thetendency of people to choose acceptable or desirable answers which do not tell”what they actually feel or believe” (Dornyei, 2003:12). Another bias detectedby Paulhaus (1991) is acquiescence bias, indicating respondents who are moreinclined to positive connotation and reluctant to provide strong negativeresponses. Lastly, contributing to response bias, the halo effect concerns thehuman tendency of overgeneralization, showing how the overall impression ofsomething or somebody may influence respondent’s opinions or attitudes when itcomes to specific details (Oppenheim, 2000). These bias, mainly coming from therespondents, may heavily affect the validity of the questionnaire. Denscombe(2007) also points out another potential for the presence of bias which creepsin the pre-coded questions. He argues that questionnaires may reflect theresearcher’s mindset rather than respondents’ thinking since pre-codedquestions “shape the nature of the responses”, resulting in answers being directedaway from the respondents’ “perceptions of matters to fit in with a line ofthinking established by the researcher”(p.160). 2.
2 Observation2.2.1 StrengthsSimilar to questionnaires, observationalso offer a large amount of information regarding participants’ behaviour in aspecific setting (Mackey & Gass, 2005). However, if questionnaires tend to miss the insights that could beprovided by the participants, the main strength of observational data is thatit can “allow the researcher to gain deeper and more multi-layeredunderstanding of the participants and their context” under investigation(Mackey & Gass, 2005:176).
Moreover, Dornyei (2007) believes that allowingresearchers to immerse themselves in the setting could present more objectivedata than “second-hand self-report” one (p.185). Interestingly, he claims thattwo approaches of collecting data, namely the unstructured and highlystructured approach, might sometimes be combined to reach the best effect. Theformer, often associated with participant observation and believed to generatequalitative data, tries to find out “not just what happens but why it happens”(Dunkerton, 1981:145). However, since this depends primarily on the subjectiveview of the researcher, the reliability of data may be doubted.
On the otherhand, a highly structured approach, which usually refers to systematicobservation, decides the focus of research beforehand through the use ofobservation scheme. Though the focus is on overt behaviours, not on why theyhappen, the data associated with “selective perception of observers” maygenerate objective observations, reducing observer’s bias, for example,emotions or personal background. (Denscombe, 2007:199). In line with Dornyei’s (2007),Dunkin and Biddle (1974) proposes a combination of both approaches, which waseffectively applied in Case a (part 1.
2). For the first time, an unstructuredapproach was adopted; I acted as a teaching assistant being involved in theclassroom’s activity. For the second attempt, I then constructed adata-collection scheme on the premise of the collected information and acted asa complete observer.
This proved to be an effective blend as the chance to missinteresting events is likely to decrease, with a more focus on what is ofimportance of the research. 2.2.2Weaknesses Mackey and Gass (2005) assumes that observationdoes not reveal motivation behind participants’ particular actions or behaviours,in contrast with questionnaires which score high in terms of clear indicationof motivational sources.
Another weakness of observation, identified byDenscombe (2007:193), relates to the issue of observer’s “psychology of memoryand perception”. To be more specific, he mentions three psychological factorsthat may greatly influence observation’s direction: “selective recall” relatesto the limited capacity of memory, deterring researcher from remembering allsituations that are being observed; “selective perception” explains how somespecific information is retained why some is rejected; and “accentuatedperception” which shows how physical and mental condition and past experiencesmay affect the observation process. These might be legitimate reasons whyfindings are interpreted differently among researchers, which may somehowdecrease the research’s reliability.3.Problems/issues that need to be addressed3.1QuestionnairesDesigningwell-constructed questionnaires is a process that requires careful attentionsince researchers barely have opportunities to make adjustments once thequestionnaire has been distributed to the participants. There exists problemsduring the designing process and after the questionnaires are collected to be analysed.
As for the designingprocess, Wallace (1998) and Nunan (1992) emphasizes that the ambiguity of thequestions may cause misunderstanding and misinterpretation for participants. Sharingthe same viewpoint, Young (2016) claims that items wording in the questions canassume “an unexpected importance”, in which just minor differences can trigger”different levels of agreement or disagreement” (p.103), particularly in thecase when non-factual data such as attitudinal data is assessed.
Another problemlies in the use of Likert scale. Worcester and Burns (1975) found that abalanced four-point Likert scale without midpoint pushed respondents,especially those who slightly lean towards either agreement or disagreement tothe positive end of scale. However, McDonough (1997) assumes that the existenceof the neutral response might be hard to interpret. Likewise, the no-opinionoption should also be used with caution. On the one hand, Schuman and Pressure(1981) advocates the use of this option since it may reduce respondents’pressure who have no relevant knowledge to answer the question.
On the otherhand, opponents of no-opinion option argue that it may not enhance data qualityand is likely to “preclude measurement of some meaningful opinions” (Krosnicket al, 2002:371). The researcher then needs to be careful when deciding whetheror not to offer this type of option in the questionnaire. When questionnaires arecollected for analysis, the issue of the comparability of the translatedversion should also be taken into consideration. This back-translation process,according to McDonough (1997) must ensure to have closest meaning to theoriginal version to guarantee reliability. 3.2ObservationThere are two majorconcerns while undertaking this kind of method. The first issue relates to the transcriptionprocess. Different from data of questionnaires which can be relatively quicklyprocessed, by either the researcher or the use of any data analysis software, presentingobservational data remains a challenge.
Bailey (2008) has listed severaldifficulties such as how to choose contextual data that needs to be interpretedor how data should be presented, since it is not an easy task to “represent thefull complexity of human interaction” on a written transcript (p.130). Secondly, as natural setting ishighlighted as a condition for observation to take place (Denscombe, 2007), itis advisable to avoid disrupting the naturalness of the setting to see thingsas they normally occur. Denscombe (2007) notices the observer effect – thepresence of the observer in the context- may make participants “aware of being studied- conscious of it- and then react in a way that isnot usual”, or “embarrassed, or disguise normal practice” (p.66). Regardingthis effect as “observer paradox”, Richards (2003:108) suggests that the use ofnote-taking should not be made apparent to the participant. He also affirmsthat the relationship between the observer and the participant should also begiven more thoughts, considering how it might affect individual and group’sperformances, and ultimately the research’s findings.
ConclusionIn this essay, I have discussed several strengths,weaknesses and relevant issues of two research methodologies: questionnairesand observation, each of which is a very common research tool that isspecifically designed for certain purposes. In many cases one data collectionmethod might be complemented by the use of other methods for triangulation – apowerful technique to add depth to the data analysis as well as improve thevalidity and reliability of the research. Both aforementioned methods demandgreat attention and subtlety in many aspects of designing process.
Being awareof the potential problems researchers might face when adopting these methods isthen essential to raise the standards of good research practice.