Our evaluation process for online applications and services attempts to address some of the common barriers to evaluating privacy practices. Privacy concerns and needs vary widely based on the app and the context where it is used. For example, it makes sense for a student information system to collect a home address; however, it wouldn't make sense for an online calculator to collect a home address. Because our process pairs a transparency evaluation with a qualitative evaluation, we can track what a policy covers alongside the strengths and weaknesses of how data is handled. Lastly, our summary evaluation allows us to highlight the implications of an app's privacy practices alongside the goals and contexts within which the app can be used.
The evaluation process contains four steps:
- Step 1: Overview - Select a product family to evaluate high-level details related to the application or service.
- Step 2: Triage - Answer initial questions related not to the policy text itself, but to the vendor's practices.
- Step 3: Transparency - Answer questions about the text of the policies. Questions include the following details:
- Transparency selection: Do the policies discuss the issues raised in the question?
- Qualitative selection: Does the vendor indicate whether or not it engages in the particular practice described?
- Notes: Is there anything noteworthy, exceptional, or egregious regarding the details of the question that should be noted?
- Policy references: Can text within the various policies be highlighted and associated with a particular question?
- Step 4: Summary - Summarize and describe the various policy details for Safety, Privacy, Security, Compliance, and the complete app.