IDENTIFYING QUALITY INDICATORS OF SAE AND FFA: A

Jenkins & KitchelIdentifying Quality Indicators based on proportion of gender in each of thecategories to taken into account what hastraditionally been a male-dominatedprofession.This study utilized the DelphiConference form. The researcher verballyinvited the experts to participate in thisstudy via telephone. Following the phoneinvitation, experts received a letter thankingthem for participating and summarizing thephone invitation. A prenotice e-mail wassent three days prior to each questionnairereminding the participants about theupcoming round. Panel members receivedan e-mail from the researcher containing ahyperlink to access the questionnaire foreach round. The initial questionnaire wasdeveloped by the researcher and wasconstructed in Web format. Both face andcontent validity were established by a panelof experts of agricultural education andrelated faculty from two universities. Interrater reliability was addressed in developingthe items from round one to round two. Tworaters developed themes from the itemsindependently and a low (below 40%consistency in all areas) was found. Whenconferring on the items, the ratersdetermined that one rater was groupingitems more broadly than the other. The tworaters then conferred on the grouping tocreate the final list of items used in roundtwo and subsequent rounds. In addition, toassist with reliability, the raters alsodeveloped topic areas to also assist withclarity of item interpretation from expert toexpert.Three open-ended questions weredeveloped for round one and were stated as―what are specific indicators of quality[instruction, SAE or FFA] in a school-basedagricultural education program?‖ This studyutilized the SAE and FFA versions of thequestion. The responses from round onewere categorized using a modified versionof the open-ended question coding techniquedeveloped by Montgomery and Crittenden(1977). The modification was that topicareas were created after the items wereselected because of the lack of consistentliterature to define specific topic areas. Afterthe responses to round one (n 31; 86.11%response rate) were categorized, the roundtwo questionnaire was developed andJournal of Agricultural Educationdistributed. The round two questionnaireasked participants, ―to what extent do youagree that the item is an indicator of qualitySAE (or FFA)?‖ using a five-point Likerttype scale: 1 strongly disagree, 2 disagree, 3 uncertain, 4 agree, and 5 strongly agree. Round two had a responserate of 86.11%.Items from round two that received ascore of ―4‖ (agree) or ―5‖ (strongly agree)by 100% of the respondents reachedconsensus and were identified as qualityindicators. Items from round two thatreceived less than 75% of the respondentsscoring the item as a ―4‖ or ―5‖ wererejected as indicators and were thereforeremoved from the study. Literature isunclear on a proper cutoff for consensus.The researchers concluded the likelihood ofagreement being reached with 25% or morebeing neutral or disagreeing would be slim.Therefore, the items on the round twoquestionnaire that did not reach consensus,but had more than 75% of the respondentsscoring the items as a ―4‖ or ―5‖ were usedin round three. Round three had a responserate of 83.33% and sought to determineconsensus. Round three had participantsindicate either agree or disagree for eachitem. The round three questionnaire wasdeveloped and included the individual‘sscore, the group‘s mean score, and thestandard deviation for each item.Participants were merely asked if theyagreed or disagreed that an item should be aquality indicator. Round three used similarbenchmarks for consensus. If an itemreached 100% agreement, it was included asa quality indicator. If only 75% or lessagreement from the panel was reached forany particular item, then that item wasdiscarded as a possible quality indicator andnot included into the next round.Round four had a response rate of85.71% and sought to determine ifsemantics contributed to disagreement onround three statements. Only participantswho disagreed with the inclusion of an itemfrom round three participated in round four.Participants were asked if changing thewording of the item would change theiragreement on inclusion as a qualityindicator. If they agreed that they wouldinclude the indicator if a change were made,35Volume 50, Number 3, 2009

Jenkins & KitchelIdentifying Quality Indicators a ―5‖ (strongly agree). The area wasundecided on the remaining 27 qualitySAE statements, meaning 99.9% to 75% ofthe respondents marked either a ―4‖(agree) or a ―5‖ (strongly agree).Therefore, those items went to round three.As illustrated in Table 1, four of the26 SAE statements in round three reachedconsensus. Of those, two (50%) items camefrom the SAE characteristics area, one(25%) item came from the records area, andone (25%) item came from the supervisionarea. In addition, 1 of the 26 SAE statementswas determined not to be a quality indicatorof SAE, meaning less than 75% of theparticipants marked an ―agree‖ for that item.The participants who disagreed on theremaining 21 SAE statements received thestatementsontheirroundfourquestionnaires.Round Four sought todetermine if semantics contributed todisagreement on Round Three statements.For the SAE section, all items had at leastone participant mark ―disagree,‖ indicatingthat he or she would not include the item asa quality indicator, even if they wereprovided the opportunity to wordsmith thatitem.they were then prompted to explain how theindicator would need to be changed.FindingsObjective one sought to determinewhat constitutes quality SAE according toexperts in the profession. For ease ofcompleting the instrument for round two,items were categorized in the followingareas: records (n 6), supervision (n 8),satisfaction (n 4), SAE characteristics (n 15),instruction(n 9),andrecognition/awards (n 4). Due to thelength of this manuscript, the tablesummarizing the results was not included.Round two resulted in only 2 of the 46quality SAE statements reaching consensus,as defined by 100% of respondents markingeither a ―4‖ (agree) or a ―5‖ (strongly agree)for that particular item. Of those, one (50%)item came from the supervision area and one(50%) item came from the satisfaction area.In addition, 17 of the 46 quality SAEstatements were determined not to be qualityindicators of SAE and removed from thestudy, as defined by less than 75% of therespondents marking either a ―4‖ (agree) orTable 1Agreement Levels for SAE Statements in Round ThreeStatementTeacher has supervision time for SAETopic areaaSupervision% Agree100.0Student has up-to-date records on SAERecords100.0SAEs involve goal-settingSAE Charac.100.0A diversity/variety of SAE types is promotedSAE Charac.100.0Teacher is enthusiastic and informed about SAEInstruction96.6SAE includes skill developmentSAE Charac.96.6Opportunities exists for SAE‘s to be showcasedSAE Charac.96.6Each student maintains a portfolio of their experiences with SAERecords96.6All students have an investment of time, energy and/or moneySAE Charac.96.6Advisory committee is satisfied with SAEsSatisfaction96.6Training plans are used for placement SAEsSAE Charac.93.1Journal of Agricultural Education36Volume 50, Number 3, 2009

Jenkins & KitchelIdentifying Quality Indicators StatementSAE planning is based on agricultural content standardsTopic areaaSAE Charac.SAE is taught as part of the curriculumInstruction% Agree93.193.1Agriculture teacher maintains accurate records of all SAE supervision Supervision93.1Students apply for related awardsRec./Awards89.7SAE program has evidence of growthSAE Charac.89.7A quality records keeping implementation program is in operationRecords89.7School administrators are satisfied with SAEsSatisfaction86.2SAE is viewed as a program versus a projectSatisfaction86.2Parents are involved with their child(ren)‘s SAESupervision82.8All students are engaged in (have a) SAESAE Charac.82.8Recordkeeping time is allocated during classRecords82.4Signed SAE agreements are on fileSAE Charac.79.3SAE is supervised year-roundSupervision79.3SAE involves continuous instructionInstruction79.3By end of 2nd grading period, all students should be engaged in SAEs Instruction72.4Note. 100% agreement (marked 4 or 5) consensus, 75% agreement undecided, 75%agreement reject.aSAE Charac. SAE Characteristic; Rec. /Awards Recognition/Awards.Objective two sought to determine whatconstitutes quality FFA according to expertsin the profession. Two independent codersdeveloped 65 quality SAE statements for theround two questionnaire. For ease ofcompleting the instrument for round two,items were categorized in the followingareas: advisor (n 5), support (n 2), POA(n 3), activities/events (n 19), budget (n 3), instruction (n 9), practice/requirements(n 16), diversity (n 2), andstudent/members (n 6). Due to the lengthof this manuscript, the table summarizingthe results was not included.Round two resulted in 13 of the 65quality FFA statements reaching consensus,as defined by 100% of respondents markingeither a ―4‖ (agree) or a ―5‖ (strongly agree).Of those, three (23%) items came from theJournal of Agricultural Educationadvisor area, three (23%) items came fromthe activities/events area, three (23%) itemscame from the practices/requirements area,one (8%) item came from the support area,one (8%) item came from the budget area,one (8%) came from the diversity area, andone (8%) came from the student/memberarea. In addition, 16 of the 65 quality FFAstatements were determined not to be qualityindicators of FFA and removed from thestudy, as defined by less than 75% of therespondents marking either a ―4‖ (agree) ora ―5‖ (strongly agree). The area wasundecided on the remaining 36 quality FFAstatements, meaning 99.9% to 75% of therespondents marked either a ―4‖ (agree) or a―5‖ (strongly agree). Therefore, thosestatements were included on the round threequestionnaire.37Volume 50, Number 3, 2009

Jenkins & KitchelIdentifying Quality Indicators As illustrated in Table 2, 6 of the 36FFA statements in round three reachedconsensus. Of those, five (83%) items camefrom the instruction area and one (17%)item came from the activities/events area.The remaining 30 FFA statements all had anagreement percentage of 75% or better,meaning 75% or more of the participantsmarked a ―4‖ (agree) or ―5‖ (strongly agree).Therefore, none of the FFA statements wererejected in round three. The participants whodisagreed on the remaining 30 FFAstatements received the statements in roundfour.Table 2Agreement Levels for FFA Statements in Round ThreeStatementThe FFA chapter plans and conducts award and recognition programsTopic areaaAct./Events% Agree100.0Instruction100.0Instruction100.0The local FFA chapter is in good standing with the state and nationalInstructionassociationsThe chapter has an accurate constitution and/or bylaws that is reviewed InstructionregularlyThe local FFA chapter is student ledInstruction100.0Chapter advisor provides assistance to members in completing chapter Advisorand individual applications and reports, but does not complete theapplications and reports for themFFA members are satisfied with the FFA chapterSupport96.7The program of activities includes activities in the following areas:member development, chapter development and communitydevelopment activities/eventsRegularly scheduled FFA chapter business meetings are heldPOA96.7Act./Events96.7The chapter provides community service opportunities for membersFFA activities/events relate to the courses and topics included in theinstructionChapter has student recruitment .7Chapter uses a committee structure to plan and conduct its activitiesInstruction96.7Member dues are collected and submitted to the state association by the Instructionpublished deadlineChapter maintains an active public relations/public awareness program Instruction96.7The chapter is involved in the schoolInstruction96.7Chapter keeps high standards for its members no matter what thesituationThe FFA chapter has the financial resources to support the POAInstruction96.7Budget96.6Chapter budget is communicated to members and administration asappropriateBudget96.6Instruction in personal and leadership development is provided for allFFA membersFFA serves as a connecting activity for SAE and InstructionJournal of Agricultural Education38100.0100.096.796.7Volume 50, Number 3, 2009

Jenkins & KitchelIdentifying Quality Indicators Topic areaaAdvisorStatementExtended contract for FFA advisor% Agree93.3FFA members are involved in the planning and implementation of aPOAchallenging Program of Activities (POA)/ Program of Work (POW)FFA members participate in FFA activities above the chapter levelAct./Events93.3Chapter members attend their state FFA conventionAct./Events93.3Members serve as officers at local, regional/area, state and nationallevelsTeacher provides instruction about FFA in the classroomAct./Events93.3Instruction93.3The FFA chapter assists students to see and build relations with school, Instructioncommunity, adults, and other studentsThe chapter has a diverse representation of membershipDiversity93.3Pride of membership is evident93.3St./Members93.393.3The POA is distributed "widely" (to each member, administration, etc.) POA90.0All students participate in activities/events of the student organizationAct./Events90.0Chapter officers are elected annuallyInstruction90.0Mentoring exists from older to younger membersChapter builds tradition so students feel they belong to a historicallygreat organizationChapter activities include areas of social s83.3All FFA members participate in one or more of the following:Act./Events82.8proficiency awards program, career development events, FFAdegree program, financial activities (fund-raising, etc.), communitydevelopment, activities that promote safety/health, etc.Note. 100% agreement (marked 4 or 5) consensus, 75% agreement undecided, 75%agreement reject.aAct. /Events Activities/Events; St./Members Students/Members.Round four sought to determine ifsemantics contributed to disagreement onroundthreestatements.Onlyparticipants who disagreed with theinclusion of an item from round threeparticipated in round four. Participantswere asked if changing the wording of theitem would change their agreement oninclusion as a quality indicator. If theyagreed that they would include theindicator if a change were made, they werethen prompted to explain how theindicator would need to be changed.Participants indicated two items that wouldbe included if those items wereJournal of Agricultural Educationreworded. The POA item, ―the program ofactivities includes activities in thefollowing areas: member development,chapter development and communitydevelopment activities/ events‖ would beincluded if the wording was changed toread, ―among other activities, thePOA includes activities in the followingareas: member development, chapterdevelopment and community sitem,―regularlyscheduledFFAchapter business meetings are held‖was accepted as written by theparticipant.39Volume 50, Number 3, 2009

Jenkins & KitchelIdentifying Quality Indicators which implies there is a lack of literaturerelated to these areas. Therefore, it isrecommended that these areas be furtherresearched. It can also be concluded that thepanel does not see eye-to-eye on everystatement proposed as a quality indicator ofSAE. The proposed items ―studentsindependently manage their SAE programs,‖―SAE is leading to some type ofrecognition,‖ and ―students apply for relatedawards‖ are supported by the LPS‘s steps tosuccess for SAE. However, the expert paneldid not reach consensus on these statements;therefore, these statements were notincluded as quality indicators of SAE.There are 19 indicators of quality FFA,as defined by the experts in this study. Theindicators that FFA serves as a connectingactivity for SAE and instruction; the chapterhas an accurate constitution and/or bylaws;well-planned chapter business meetings areheld; the chapter maintains accuratefinancial records; the chapter has a capableand trained officer team; chapter receivessupport from administrators, teachers, andadvisory committee, parents, etc.; hostactivities that are designed to meet the needsof a diverse membership; and the chaptermaintains accurate minutes of all meetingsare quality indicators of FFA. Theseconclusions are consistent with therecommended 11 essentials of a successfulFFA chapter provided in the Official FFAManual. These findings imply that theexpert panel is in line with the literature, andit is recommended that these qualityindicators be embraced by the profession.In addition, the expert panel identifiedthe characteristics of the advisor as anindicator of quality for FFA. Thisconclusion is supported by recommendationmade by Phipps and Osborne (1988) that thechapter advisor plays a large role indeveloping a successful FFA chapter. Theconclusion that FFA members shouldreceiveopportunitiestodevelopcommunication skills and be involved inleadership development is consistent withStaller (2001), who stated that the FFAcomponent, compared with the instructionalcomponent, was best suited to teach lifeskills. Furthermore, this conclusion isconsistent with Lockaby and Vaughn‘s(1999) finding that of the three componentsDiscussionThere were some limitations that shouldbe acknowledged. The use of FFA and SAEversusleadershipdevelopmentandexperiential education limited the focus ofthe responses to the tools of SAE and FFArather than the broader concepts behindthem. If the questions focused on qualityindicators of leadership development andexperiential education, the result could havebeen different. In addition, some items werewritten such that two concepts could haveappeared in one item. The researchers had tobalance avoiding such ―double-barreling‖questions and having such an exorbitantamount of items that some respondentswould have potentially refused toparticipate. However, in later rounds, if theexperts were still uncertain whether an itemshould be an indicator, they could haveoffered suggested changes to that item.There are six quality indicators of SAE,as agreed upon by the experts in this study.The experts identified the need for adiversity of SAE types to be promoted andthat agriculture teachers need to havesupervision time for SAE. This conclusion isconsistent with Steele (1997) who noted thatproviding appropriate SAE opportunities forall students is the most important SAEpractice for summer employment ofagriculture teachers. The conclusion is alsoconsistent with Camp, Clarke, and Fallon(2000) who found that an effective SAE issupervised by an adult. In addition, theexpert panel identified the student havingup-to-date records as a quality indicatorwhich is also consistent with Camp et al.The conclusion that SAEs should be assistedby instructor, parents, and employers isconsistent with Phipps and Osborne (1988)and the National Research Council (1988),who stated that the local agribusinesscommunity should be utilized as a SAEresource. These findings imply that theexperts are in line with the literature and it isrecommended that these quality indicatorsbe embraced by the profession.The experts also identified SAEsinvolving goal setting and the student beingsatisfied with the SAE as indicators ofquality SAE. There is no literature tosupport or reject these quality indicators,Journal of Agricultural Education40Volume 50, Number 3, 2009

Jenkins & KitchelIdentifying Quality Indicators developing SAE using a different set oflenses could meet a variety of students‘needs. If this is the case, perhaps theprofession should look at whether it‘s tooprescriptive in its views of FFA. Furtherresearch is the only way to address theseissues.Because this is an exploratory study,there are several opportunities for furtherresearch. For one, these indicators could beexamined by the rest of the profession—agriculture teachers, teacher educators andstaff state—to see if the experts were in linewith the profession. This would take theresearch into much more of a descriptiveand generalizable nature. In addition, asnoted above, some items do not match withagricultural education literature. Is itpossible that we haven‘t studied those areas?Is there literature outside of the profession tosupport or refute these indicators? Finally,the use of the three-circle model to frame themethods and instrumentation could haveimplications as well. As the 10 x 15 newprogram model task force progresses, theprofession may find the traditional threecircle model needs to be modified,expanded, revisioned, or identified as one ofmany possible program models. Taking thespirit of this study in a more broadinterpretation of agricultural educationprogram could result in indicators withbroader or just different perceptions ofprogram quality.of agricultural education, FFA is the best forteaching values and attitudes to students.These findings imply that the expert panel isin line with the literature, and it isrecommended that these quality indicatorsbe embraced by the profession.The experts also agreed that theindicators of agricultural education studentswho wish to participate in FFA are acceptedas members even if there is an inability topay dues, officers and advisors meetperiodically to plan the work of theorganization, the chapter is student led, thechapter is in good standing with state andnational associations, instruction in personaland leadership development is provided forall FFA members, and chapter plans andconducts award and recognition programsare indicators of quality FFA. There is noliterature in agricultural education to supportor reject these quality indicators, whichimplies there is a lack of literature related tothese areas. Therefore, it is recommendedthat these areas be further researched.The expert panel did not see eye-to-eyeon every statement proposed as an FFAqualityindicator.Proposedqualityindicators such as ―regularly scheduled FFAchapter business meetings are held‖ and ―allstudents enrolled in the agriculturaleducation program are members of the FFA‖are supported by the Official FFA Manual.

determine quality indicators for SAE and FFA according to agricultural education experts (agricultural education teacher educators, state instructional staff, and high school teachers) across the United States. In particular, the objectives were to determine: (1) quality indicators of SAE and (2) quality indicators of FFA.