Summary and Description of the Project
What Is OSDDP?
Implemented by the professional writing program at Purdue in the fall of 2004 as a new approach to networked learning, collaborative education, and public writing, OSDDP (the Open Source Development and Documentation Project) has produced many success stories among students and instructors in various business and technical writing classes at Purdue. However, teaching the OSDDP project also presents many challenges to instructors of professional writing, especially new instructors, as it requires a thorough understanding of Open Source products and the concepts and philosophy underlying these products as well as genre knowledge about the deliverables that students are expected to produce throughout the project. This guide introduces new PW instructors to one particular OSDDP project, the OSDDP usability project and provides tips and resources that aid teaching.
What Does the OSDDP Usability Project Entail?
For the OSDDP Usability Project, students are expected to complete two main tasks: 1) conduct usability testing of an open source application (software, content management system etc.) or documentation (tutorial or user manual); 2) write up the test results in the form of a usability report and present it to the open source community by posting it to Purdue's OSDDP site under creative commons.
Why Assign the OSDDP Usability Project?
This project provides students with an opportunity to 1) learn more about how users/customers in real-world situations interact with commercial and non-commercial products (in this case, software applications and manuals); 2) use usability testing as a research method to generate raw data about such interactions; 3) practice public writing or writing that is subject to public reviews and revisions.
Although this project involves some individual elements, the majority of the project is the result of collaboration. This project involves approximately 6-7 weeks of work with each week typically consisting of two 75-minute class meetings or three 50-minute class meetings.
What Can This Teaching Guide Do for Me?
This teaching guide includes a brief summary of the project, suggested group and individual deliverables as well as evaluation criteria, week-by-week suggested activities with annotations, resources for instructors, and sample deliverables produced by business writing students at Purdue.
Suggested Deliverables and Evaluation Criteria
For this project, students are expected to produce 7 group deliverables and 4 individual deliverables.
Description: This one-to-two page single-spaced memo proposes an OSDDP project that fits with the level of expertise of the usability research team in doing primary research as well as their interest in a particular open source product (software application or user manual/tutorial). The proposal should include, but not be limited, to the following elements: description of the product to be tested by the student researchers, rationale underlying their choice of this particular product, qualifications of the students as usability researchers, as well as potential benefits of the project for the developers of the product, target users of the product, and the team of student researchers themselves.
Evaluation:The proposal needs to be both informative and persuasive. It should convince the instructor that what is being proposed is both worthwhile and manageable in a 6 or 7-week time frame.
Example: See page 1 of the pdf document attached in the "media" box.
Team project plan with Gantt chart
Description: The team plan should answer several important questions that the team needs to consider at early stages of the project, such as who is the typical user of the product to be tested, what are some of the characteristics of the "typical" or "average" user, what does the typical user use the product for, what methods will be used to recruit and select users, what problems the student researchers anticipate to occur before, during, and/or after the test sessions, what role(s) each team member plays and so forth. The Gantt chart is a visual way of presenting the timeline by mapping out major tasks to be completed vertically and the expected length of time it takes to complete each task horizontally.
Evaluation: The group plan should demonstrate signs of careful thinking into the whole process that the team will go through from selecting the product to be tested to submitting final draft of the usability report by thoroughly answering questions mentioned above. The Gantt chart should represent the timeline clearly and accurately.
Example: See pages 2-3 of the pdf document attached in the "media" box.
Periodic oral progress report
Description: This is an informal oral report on what the team has achieved in a particular week and what needs to be done in the following week. It's an opportunity for each team to share their work-in-progress with other teams and get feedback and suggestions.
Evaluation: The report should be informative and concise. It should update other teams on both the products/deliverables that the team has produced AND the process the team is going through.
Usability testing materials
Description: The usability testing materials package should include pre-test background survey, observational data sheet to record user reactions to and interactions with the product, and follow-up survey or interview questions.
Evaluation: While testing materials differ significantly depending on what is to be tested, well-developed testing materials packages share several things in common. They should reflect clear and well-articulated testing objectives as well as maintain a good balance between good planning and room for flexibility. They maximize both validity and reliability of the tests.
Example: See pages 14-17 of the pdf document attached in the "media" box.
Description: Each team should select a reasonable number of users as participants in the tests. (For a 3-person team, 6-9 users would be good enough. More than 9 may not be manageable considering the time constraints.) Each team should try to select users from a variety of backgrounds, taking into consideration such factors as gender, race, education, prior knowledge about the product, level of expertise in the product and so forth. In addition, to help enhance the reliability of test results, users should not have a previous connection with any of the team members. Each test session should last approximately 45-60 minutes.
Evaluation: The test sessions will not be evaluated on their own as quality of the tests will be reflected in the report itself, especially in the analysis, findings, and discussion sections of the report.
Description: The report should provide adequate information on the background of the product tested, the methods used (user profiles, researcher roles, testing facility, data collection and analysis methods and procedures) as well as thorough discussion of test results (key findings and recommendations). Encourage your students to follow the CIF (Common Industry Format) guidelines developed by NIST (National Institute of Standards and Testing) and adapt these guidelines for their own purposes. Click here for a sample handout on the structure of a usability report based on the CIF guidelines.
Evaluation: The report should allow the readers to see not only the results of the test sessions but also the process that the usability researchers have gone through. The researchers should be able to identify patterns in the raw data they've collected by using appropriate analytic tools/frameworks and justifying their choice. They should interpret their findings in a fair manner, and the recommendations they make should match and be based on their findings.
Example: See pages 4-17 of the pdf document attached in the "media" box.
Description: Each presentation should run 15-20 minutes. In addition, presenters should be prepared to answer questions from the audience. Each Q-A session lasts about 5-10 minutes. All presentations must include multimedia elements and be accompanied by a handout. Each team member will be expected to take part in the presentation.
Evaluation: Use the following categories to evaluate the oral presentations: content and organization, audience awareness, slide design and use of visuals, and delivery.
Professional profile of skills and interests
Description: The profile should include the following information: name, major and minor, expertise/specialties (tie this to the tasks to be accomplished), products wanting to test, and contact information. The purpose of this assignment is to help students find out whom they are likely to work well with. This information can also be used later by the teams to assign specific roles to each individual member.
Evaluation: This step itself won't be graded but will be used to help form teams.
Description:It details each individual's contributions to the group project by describing what each individual has done in a particular week and the amount of time spent.
Evaluation: This step itself will not be graded, but will be used as one factor in your determination of each individual's participation grade for this project.
Description: This one-to-two page single-spaced reflective piece describes each individual's experience of doing primary research in the form of usability testing, analyzing data and reporting on key findings in a clear and logical manner.
Evaluation: While description of certain aspects of the project is necessary, the bulk of this document should go beyond description and should answer such questions as "what worked? what didn't work and why? what would I do differently next time, knowing what I know now?" and so forth.
Peer collaborative project evaluation document
Description: This document, accessible to the instructor only, describes each individual's contribution to the project in detail. It tells you who deserves more or less credit than others based on such descriptions. This document can be in paragraph or memo format; or you can use a form similar to the one being used by English 420 (Business Writing) instructors at Purdue.
Evaluation: This document itself won't be graded, but will be used as one factor in your determination of each individual student's participation grade for this project.
Week-by-Week Suggested Activities
A 7-week time frame is used here, with each week typically consisting of two 75-minute class meetings or three 50-minute class meetings. Feel free to adapt it to your own class schedule in any way you deem necessary.
Week 1: Introduce Project and Form Usability Teams
- Provide a list of open source applications (Wikipedia, Open Office package, Drupal, Mozilla Firefox, Mozilla Thunderbird etc.) and have students download one of their choice and experiment with it.
- Have students discuss in groups their individual experience using a particular open source application.
- Read the Open Source definition by OSI (Open Source Initiative) and discuss open source as both product and philosophy.
- Discuss the advantages and disadvantages of open source products.
- Introduce the usability project from both a process and a product perspective, go over deliverables to be produced.
- Form usability teams.
Week 2: Select and Propose Product to Be Tested, Set up and Conduct Mock Tests
- Have each team develop learning goals; that is, what they hope to learn as usability researchers and writers in the process of completing the project.
- Select product to be tested and write project proposal.
- Set up mock tests on a product that the students are likely to use for this project or have been using for the course in general (e.g., a software application that students use often in this class such as PowerPoint, an online resource such as the school library electronic database of journal articles, the course website if applicable etc.). As a class, develop testing materials for this particular product, including pre-test background survey and observational data sheet.
- Set up test sessions, decide when and where to conduct the tests, assign roles (who will be the users, and who will be the researchers/observers).
- Conduct mock tests.
- Discuss mock tests from both the user's and the researcher's perspective.
- Develop post-test interview questions and conduct post-test interviews with selected user participants.
Week 3: Analyze Mock Tests and Develop Testing Materials
- Analyze test results. Discuss multiple ways in which data can be analyzed and advantages and disadvantages of each approach.
- Have each team develop their own pre-test survey and observational data sheet.
- Peer review drafts of testing materials submitted by each team.
- Finalize testing materials, using peer and instructor comments.
Week 4: Conduct Usability Tests and Analyze Data
- Conduct usability tests as well as post-test interviews.
- Group conferences with instructor to talk about how to analyze data.
Week 5: Draft Usability Reports and Present Data Visually
- Go over the CIF (Common Industry Format) guidelines developed by NIST (National Institute of Standards and Testing). Discuss how to adapt these guidelines to the class project (Click here for a sample handout on the structure of usability report developed as a result of such a discussion).
- Draft report.
- Discuss the rhetoric of using visuals to present data (charts, graphs, tables, etc..)
- Construct visuals to be used in the report.
Week 6: Peer Review Draft, Conference, and Revise Draft
- Have each group develop their own peer review sheet by compiling a list of questions for their reviewers to consider while commenting on their draft.
- Exchange drafts and conduct peer review sessions.
- Group conferences with instructor to discuss revision.
Week 7: Present, Reflect, and Evaluate
- Have each team present the project to the class and answer questions.
- Have each individual student write a reflective piece on what they’ve learned, encourage them to tie their reflections to the learning goals they developed at the beginning of the project to find out what learning goals have been achieved and what haven’t; what they wish they had done differently towards these learning goals.
- Have team members evaluate anonymously each individual member’s contributions to the project by describing who did exactly what as well as who deserves more or less credit than others.
Resources for Instructors
Open Source and OSDDP at Purdue
Open Source Initiative: Open source definition, products, philosophy, and more.
Drupal: An open source content management platform, which can support a variety of websites ranging from personal weblogs to large community-driven websites.
Moodle: An open source course management system, designed to help educators create effective online learning communities.
GNU: An open source operating system, alternative to UNIX.
Wikipedia: An open source encyclopedia available in 10 different languages.
Open Office Suite: An open source multilingual and multi-platform suite, which is compatible with and can be used as an alternative to the Microsoft Office suite.
Mozilla Firefox and Mozilla Thunderbird: Mozilla Firefox is an open source alternative to IE and other commercial browsers. Mozilla Thunderbird is an open source alternative to Outlook and other commercial inbox organizers.
Open Source Development and Documentation Project at Purdue: OSDDP mission statement, press release, pedagogical rationale, sample projects, teaching tips, discussion forums.
Project Management Tips and Free Software
Project KickStart Website: Project management tips provided by consultants and trainers who helped develop Project KickStart, a project management software package.
All PM (Project Managers) Website: Specifically created for project managers in the workplace, has a large collection of materials ranging from glossary, articles and tips, white papers, to document templates, many of which can be adapted for classroom use. Also has a discussion forum where you can post questions about project management and get answers from project managers all over the country. Membership free.
MindTools' Guide to Gantt Chart: Offers a good conceptual explanation of Gantt chart as a project management tool.
SmartDraw Website: Free trial version of SmartDraw, a software program for creating Gantt chart and other charts, as well as Gantt chart definition/explanation and examples.
Usability Testing and Usability Report: Principles, Techniques, and Examples
Usable Information Technology Website: Primarily column articles, reports, and newsletters on usability by the website's creator, Jakob Nielsen, a former Sun Microsystems engineer and a well published usability researcher with expertise in website usability research, especially the heuristic evaluation method.
Information & Design Website: Tips on the design, analysis, evaluation stages of usability testing provided by Information & Design, a usability consultancy based in Melbourne, Australia. Sample testing materials also included.
CIF Guidelines — Common Industry Format for Usability Test: Guidelines for creating a usability report using the format, recommended sections in a usability report, checklist etc.
Design Technologies Website: A large collection of usability resources, including heuristics, methods, style guides, and links to other resources.
Microsoft Usability Website: Microsoft Usability Publications, including guidelines, techniques, and sample reports.
Dialog Design Usability Website: Standard report format and sample report, good in informal testing situations when testing needs to be done in a fast and easy way.
Gnome Usability: A good example of a formal report, when thoroughness is more important than efficiency.
Sample Project Deliverables
In this section, you will find some deliverables produced for the OSDDP usability project by a group of students enrolled in English 420 (Business Writing), with all the names taken out. You may use these as examples or a discussion-starter in your own classroom if usability report writing is a component in your curriculum.
These deliverables are included in one PDF document attached in the "media" box, and they are separated by page numbers.
- Page 1: Project Proposal
- Page 2-3: Group Project Plan
- Page 4-17: Usability Report
- Page 14-17: Usability Testing Materials (the Appendix section of the report)