Software Evaluation: Criteria-based Assessment
SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011Software Evaluation: Criteria-based AssessmentMike Jackson, Steve Crouch and Rob BaxterCriteria-based assessment is a quantitative assessment of the software in terms of sustainability,maintainability, and usability. This can inform high-level decisions on specific areas for softwareimprovement.A criteria-based assessment gives a measurement of quality in a number of areas. These areas arederived from ISO/IEC 9126-1 Software engineering — Product quality1 and include usability,sustainability and maintainability.The assessment involves checking whether the software, and the project that develops it, conformsto various characteristics or exhibits various qualities that are expected of sustainable software. Themore characteristics that are satisfied, the more sustainable the software. Please note that not allqualities have equal weight e.g. having an OSI-approved open source licence is of more importancethan avoiding TAB characters in text files.In performing the evaluation, you may want to consider how different user classes affect theimportance of the criteria. For example, for Usability-Understandability, a small set of well-defined,accurate, task-oriented user documentation may be comprehensive for Users but inadequate forDevelopers. Assessments specific to user classes allow the requirements of these specific user classesto be factored in and so, for example, show that a project rates highly for Users but poorly forDevelopers, or vice versa.Scoring can also be affected by the nature of the software itself e.g. for Learnability one couldenvisage an application that has been well-designed, offers context-sensitive help etc. andconsequently is so easy to use that tutorials aren’t needed. Portability can apply to both the softwareand its development infrastructure e.g. the open source software OGSA-DAI2 can be built, compiledand tested on Unix, Windows or Linux (and so is highly portable for Users and User-Developers).However, its Ruby test framework cannot yet run on Windows, so running integration tests wouldinvolve the manual setup of OGSA-DAI servers (so this is far less portable for Developers and,especially, Members).The assessment criteria are grouped as follows.CriterionSub-criterionNotes – to what extent is/does the software UsabilityUnderstandabilityEasily n?BuildabilityStraightforward to build on a supported system?InstallabilityStraightforward to install on a supported system?LearnabilityEasy to learn how to use its functions?IdentityProject/software identity is clear and unique?CopyrightEasy to see who owns the project/software?LicencingAdoption of appropriate iappropriate,well-structureduser1
SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011GovernanceEasy to understand how the project is run and thedevelopment of the software managed?CommunityEvidence of current/future community?AccessibilityEvidence of current/future ability to download?TestabilityEasy to test correctness of source code?PortabilityUsable on multiple platforms?SupportabilityEvidence of current/future developer support?AnalysabilityEasy to understand at the source level?ChangeabilityEasy to modify and contribute changes to developers?EvolvabilityEvidence of current/future development?InteroperabilityInteroperable with other required/related software?The rest of this document covers each category in greater depth, with lists of questions that we useat the Software Sustainability Institute when compiling detailed software evaluation reports.2
SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011UsabilityUnderstandabilityHow straightforward is it to understand:Yes/No, supporting comments if warranted What the software does and its purpose?The intended market and users of thesoftware? The software’s basic functions? The software’s advanced functions?High-level description of what/who the softwareis for is available.High-level description of what the software doesis available.High-level description of how the software worksis available.Design rationale is available – why it does it theway it isDescriptions of intended use cases are available.Case studies of use are available.DocumentationLooking at the user documentation, what is itsYes/No, supporting comments if warranted Quality? Completeness? Accuracy? Appropriateness? Clarity?Provides a high-level overview of the software.Partitioned into sections for users, user-developersand developers (depending on the software).States assumed background and expertise of thereader, for each class of user.Lists resources for further information.Further information is suitable for the level of thereader, for each class of user.Is task-oriented.Consists of clear, step-by-step instructions.Gives examples of what the user can see at eachstep e.g. screen shots or command-line excerpts.For problems and error messages, the symptomsand step-by-step solutions are provided.Does not use terms like “intuitive”, “user friendly”,“easy to use”, “simple” or “obviously”, unless as3
SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011part of quotes from satisfied usersStates command names and syntax, says whatmenus to use, lists parameters and error messagesexactly as they appear or should be typed.Uses teletype-style fonts for commandline inputs and outputs, source code fragments,function names, class names etc.For Java, the package names of classes are statedthe first time a class is mentioned.English language descriptions of commands orerrors are provided but only to complement theabove.Plain-text files (e.g. READMEs) use indentation andunderlining (e.g. and ---) to structure the text.Plain-text files (e.g. READMEs) do not use TABcharacters to indent the text.API documentation e.g. JavaDoc or Doxygen,documents APIs completely e.g. configuration files,property names etc.Is held under version control alongside the code.Is on the project web site.Documentation on the project web site makes itclear what version of the software thedocumentation applies to.BuildabilityHow straightforward is it to:Yes/No, supporting comments if warranted Meet the pre-requisites for building thesoftware on a build platform? Build the software on a build platform?Web site has instructions for building thesoftware.Source distributionsbuilding the software.haveinstructionsforAn automated build (e.g. Make, ANT, customsolution) is used to build the software.Web site lists all third-party dependencies thatare not bundled, along with web addresses,suitable versions, licences and whether these aremandatory or optional.Source distributions list all third-partydependencies that are not bundled, along withweb addresses, suitable versions, licences andwhether these are mandatory or optional.Dependencymanagementisusedtoautomatically download dependencies (e.g. ANT,4
SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011Ivy, Maven or custom solution).All mandatory third-party dependencies arecurrently available.All optional third-partycurrently available.dependenciesareTests are provided to verify the build hassucceeded.InstallabilityHow straightforward is it to:Yes/No, supporting comments if warranted Meet the pre-requisites for the software on atarget platform? Install the software onto a target platform? Configure the software following installationfor use? Verify the installation for use?Note that in some cases build and install may beone and the same.Web site has instructions for installing thesoftware.Binary distributions haveinstalling the software.instructionsforWeb site lists all third-party dependencies thatare not bundled, along with web addresses,suitable versions, licences and whether these aremandatory or optional.Binary distributions list all third-partydependencies that are not bundled, along withweb addresses, suitable versions, licences andwhether these are mandatory or optional.Dependencymanagementisusedtoautomatically download dependencies (e.g. ANT,Ivy, Maven or custom solution).All mandatory third-party dependencies arecurrently available.All optional third-partycurrently available.dependenciesareTests are provided to verify the install hassucceeded.When an archive (e.g. TAR.GZ or ZIP) isunpacked, it creates a single directory with thefiles within. It does not spread its contents allover the current directory.When software is installed, its contents areorganised into sub-directories (e.g. docs fordocumentation, libs for dependent libraries) as5
SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011appropriate.All source and binary distributions contain aREADME.TXT with project name, web site,how/where to get help, version, date, licenceand copyright (or where to find this information),location of entry point into user doc.All GUIs contain a Help menu with commands tosee the project name, web site, how/where toget help, version, date, licence and copyright (orwhere to find this information), location of entrypoint into user doc.All other content distributed as an archivecontains a README.TXT with project name, website, nature, how /where to get help, date.Installers allow user to select where to installsoftware.Uninstallers uninstall every file or warns user ofany files that were not removed and where theseare.LearnabilityYes/No, supporting comments if warrantedHow straightforward is it to learn how toachieve: Basic functional tasks? Advanced functional tasks?A getting started guide is provided outlining abasic example of using the software.Instructions are provided for many basic usecases.Instructions are provided supporting all usecases.Reference guides are provided for all commandline, GUI and configuration options.API documentation is provided for userdevelopers and developers.6
SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011Sustainability and maintainabilityIdentityYes/No, supporting comments if warrantedTo what extent is the identity of theproject/software clear and unique both withinits application domain and generally?Project/software has its own domain name.Project/software has a logo.Project/software has a distinct name within itsapplication area. A search by Google on thename plus keywords from the application areathrows up the project web site in the first pageof matches.Project/software has a distinct nameregardless of its application area. A search byGoogle on the name plus keywords from theapplication area throws up the project website in the first page of matches.Project/software name does not throw upembarrassing “did you mean ” hits on Google.Project/software name does not violate anexisting trade-mark.Project/software name is trade-marked.CopyrightYes/No, supporting comments if warrantedTo what extent is it clear who wrote the softwareand owns its copyright?Web site states copyright.Web site states who developed/develops thesoftware, funders etc.If there are multiple web sites then these all stateexactly the same copyright, licencing andauthorship.Each source code file has a copyright statement.If supported by the language, each source code filehas a copyright statement embedded within aconstant.Each source code file has a licence header.LicencingHas an appropriate licence been adopted?Yes/No, supporting comments if warrantedWeb site states licence.Software (source and binaries) has a licence.Software has an open source licence.7
SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011Software has an Open Software Initiative3 (OSI)recognised licence.GovernanceYes/No, supporting comments if warrantedTo what extent does the project make itsmanagement, or how its software development ismanaged, transparent?Project has defined a governance policy.Governance policy is publicly available.CommunityYes/No, supporting comments if warrantedTo what extent does/will an active user communityexist for this product?Web site has statementusers/developers/members.ofnumberofWeb site has success stories.Web site has quotes from satisfied users.Web site has list of important partners orcollaborators.Web site has list of the project’s publications.Web site has list of third-party publications thatcite the software.Web site has list of software that uses/bundles thissoftware.Users are requested to cite the project ifpublishing papers based on results derived fromthe software.Users are required to cite a boilerplate citation ifpublishing papers based on results derived fromthe software.Users exist who are not members of the project.Developers exist who are not members of theproject.AccessibilityTo what extent is the software accessible?Yes/No, supporting comments if warrantedBinary distributions are available (whether forfree, payment, registration).Binary distributions are freely available.Binary distributions are available without theneed for any registration or authorisation of3http://www.opensource.org/8
SOFTWARE EVALUATION: CRITERIA-BASEDNOVEMBER 2011access by the project.Source distributions are available (whether forfree, payment, registration).Source distributions are freely available.Source distributions are available without theneed for any registration or authorisation ofaccess by the project.Access to source code repository is available(whether for free, payment, registration).Anonymous read-only access to source coderepository.Ability to browse source code repository online.Repository is hosted externally to a singleorganisation/institution in a sustainable thirdparty repository (e.g. SourceForge, GoogleCode,LaunchPad, GitHub) which will live beyond thelifetime of any current funding line.Downloads page shows evidence of regularreleases (e.g. six monthly, bi-weekly, etc.).TestabilityHow straightforward is it to test the software toverify modifications?Yes/No, supporting comments if warrantedProject has unit tests.Project has integration tests.For GUIs, project uses autom
SOFTWARE EVALUATION: CRITERIA-BASED NOVEMBER 2011 1 Software Evaluation: Criteria-based Assessment Mike Jackson, Steve Crouch and Rob Baxter Criteria-based assessment is a quantitative assessment of the software in terms of sustainability, maintainability, and usability.
based a ssessment, which is a software evaluation methodology described by Jackson, Crouch and Baxter in their paper 'Software Evaluation: Criteria-based Assessment'. The second evaluation method that I decided to use was Nielsen's Heuristics. The . criteria-based assessment gives the measure of quality in the areas of usability,
DRAFT MYP OBJECTIVES AND ASSESSMENT CRITERIA 3 Science Assessment Criteria The following assessment criteria have been established by the IB for the sciences subject area of the MYP. All assessment in each year of the MYP must be based on the age appropriate versio
The software requirements for each group are presented in Attachment 3. 2.7 Develop Evaluation Criteria Each software group was evaluated using criteria developed specifically for that group which addresses factors important to risk practitioners. The evaluation criteria for each group are presented in
Table 2. MYP Assessment Criteria Subject groups must address all strands of all four assessment criteria a minimum of at least twice in each year of the MYP. In reality, several formative assignments will utilize the criteria to ensure that summative assessments graded using the criteria are well prepared for. Criteria Subject Group MYP Subjects
Tender Evaluation in Complex Procurement Better Practice Guide UNCONTROLLED IF PRINTED 12 September 2017 2 Table of Contents . evaluation criteria contained in the request documentation. These criteria will also be set out in the TEP. The evaluation criteria are used to assist the evaluation team to objectively assess tenders and
Plans Preparation Manual, Volume 1 January 1, 2017 Design Geometrics and Criteria 2-1 Chapter 2 Design Geometrics and Criteria . 2.0 General . The implementation of design criteria is outlined in the following text. 1. Design Criteria: The design criteria presented in this manual are intended as
History of Nursing Education Unit Summary of Standards and Criteria Analysis and Summary of Strengths and Areas Needing Improvement SECTION TWO: STANDARDS AND CRITERIA I. Mission and Governance.8 NLNAC Criteria 1 NLNAC Criteria 2 NLNAC Criteria 3 NLNAC Criteria 4 II.
hardware IP re-use and consistency accross product families and higher level programming language makes the development job far more convenient when dealing with the STM32 families. HIGH-PERFORMANCE HIGH DEGREE OF INTEGRATION AND RICH CONNECTIVITY STM32H7: highest performance STM32 MCUs with advanced features including DSP and FPU instructions based on Cortex -M7 with 1 to 2 Mbytes of .
