Article

Step 6: Impact Analysis

You will need to measure the impact your project has or has not made against your original intent. Reflect on the social values and ethical principles you have or have not promoted. Finally you will need to think about how to follow up so your sensing project can continue to create value for your city.

In the end, Crowd Sensing should be about co-creating social impact with the public. You need to show quantitative and qualitative results in the right ways to keep momentum, your stakeholders involved, and the public engaged.

6.1 Questions

Making people aware and getting them involved to the point where they want to participate is difficult so you have to show quantitative and qualitative results in the right way to keep the public engaged. At this point there are also operational, tactical and strategic considerations that relate to accountability issues and follow-up.  For example re-establishing ownership, governance, escalation paths, roles, and responsibilities among other considerations such as: 

  • Can you continue by designing interventions and involving the public?
  • Are you measuring changes, results, and social impact? 
  • Are you being transparent and publishing the data and results with visualisations and engaging stories? 
  • Have you been speaking with the public to generate a “data plus dialogue”?
  • Are you communicating your project, creating awareness and accountability? 
  • How can I reflect on the impact I have made in the context of my project goals as well as in an ethical context taking into account social values? 
  • Is it possible to keep momentum and enable a continuous discussion? 
  • Who will now own this project and how will they be accountable?
  • Have you established a follow-up plan and strategy?

 

6.2 – Project Retrospective

Now is a good time to reflect on your experience, look at the original intent, review your goals and revisit the Decision Canvass you worked on in the beginning of your journey to responsible and ethical crowd sensing.  Project post-mortems and assessing lessons learned is important but think about the unique aspects of crowd sensing and your project in a broader sense.  It’s not just about measuring performance against KPIs, it’s also about how you did or did not achieve your goals in responsible and ethical ways.  Here are some suggestions to help you assess your project’s impact:

  • Revisit your original Decision Canvas with project stakeholders.
  • Think about Lessons Learned. Document and share your experience.
  • Measure the changes you have made with an impact analysis report to quantity the before and after of your project and any interventions you conducted. 
  • Publish the results and tell your story.  It is important to be transparent and share your results. 

No doubt your organisation will have its own way of reflecting on a project. We hope to add some value at this stage by emphasising the ethical aspects specifically related to crowd sensing projects.  Now would be a good time to (re) visit The Responsible Sensing Lab https://www.ams-institute.org/urban-challenges/urban-data-intelligence/responsible-sensing-lab/  and read the cases: Human Scan Car, Transparent Charging Station, and Camera Shutter. Three well documented projects you can refer to. 

 

6.2.1 Reflecting: Social and Ethical Impact

It is difficult to measure success in a crowd sensing project you may or may not have achieved the goals you established. You may have had some serious problems or surprising results but what is most important is understanding how your project impacted the public. Did you change perceptions or behaviour and what does this mean? After evaluating the quantitative results of your project you should consider engaging the public again to get some feedback and gather the qualitative data about social impact.  

 

6.2.2 Evaluating Ethical Performance 

Responsible and ethical crowd sensing as a practice is just now beginning to take shape. If we design and implement our projects keeping social values and ethical principles in mind we are pioneering important work. The impact we create in our cities is much more than solving problems that have to do with how people move in public spaces but how we approach solving these problems with privacy and dignity at the same time.

Now is a good time to hold an Ethical Performance Workshop with your project stakeholders. You can revisit the Applying an Ethical Framework and Measuring Ethical Principles in Step 3 (LINK) or visit some of the data ethics links below to help you think about how to evaluate your ethical performance:

https://gdpr-info.eu/issues/privacy-by-design/
https://tada.city
https://www.ams-institute.org/urban-challenges/urban-data-intelligence/responsible-sensing-lab/
https://citiesfordigitalrights.org
https://en.wikipedia.org/wiki/Value_sensitive_design
https://dataschool.nl/en/deda/app/

For a specific example of a TADA Data Ethics Workshop to evaluate ethical performance, download this report from the Recreational Crowd Study conducted by CITIXL in collaboration with Marineterrien Amsterdam, City of Amsterdam CTO office and TADA.

See the example workshop results:  TADA Recreational Space Study Workshop PDF

Project stakeholders should be evaluating the ethical performance for each of their experiments. Teams are free to define their own values and principles, but most importantly see how they can continually improve on responsible sensing during the project life cycle. In the illustration below, you see a comparison of how teams ranked themselves on a hypothetical research project (Based on the TADA ethical framework). 

Tada cafe

Cafe owners using Computer Vision to measure social distancing on their terrace

Tada business

Business district impact analysis of 3D Sensors in a street during lockdown

Tada museum

Museum managers using WiFi Sniffers for smart ticketing to improve waiting times

Tada health

Health department using MM Waves to detect after hour activities of playgrounds

Tada campus

Campus security using Sound Sensors to classify sounds like guns or breaking glass

Tada facility

Facility managers using Motion Sensors to determine safe capacity of student assembly

 

6.2.3 Follow-up

At this point there are also operational, tactical and strategic considerations that relate to accountability issues and follow-up.  For example re-establishing ownership, governance, escalation paths, roles, and responsibilities among other considerations.

In agile projects there are various approaches to continuous improvement.  Specific to responsible and ethical crowd sensing it is useful to map out your journey, consider what impact you have had, and determine a path forward or a productive conclusion. Here are some considerations:

  • Can you amplify the positive impact you have had with your project?
  • Is it possible to scale, duplicate, and/or distribute your project?
  • If you continue, how can you improve your project responsibly and ethically?
  • If you conclude your project, how can you and the public benefit from lessons learned? 

When we talk about agile development, design thinking, or experimentation in Living Labs it’s important to remember that these are all non-linear approaches to creative collaboration and conclusions are less important than applying lessons learned to improving your project or designing a better solution to test again. Here are some links to help you think about follow up so you can build on your experience and continue to create impact with your project. 

Watch this video externally on: Vimeo

Watch this video externally on: Vimeo

 

Image credits

Header image: Papier Hier - donker blauw

Icon image: Papier Hier - donker blauw

Media

Tada cafe Tada business Tada museum Tada health Tada campus Tada facility OSCM Interview - step 6 - Ido Nap OSCM Toolkit  Interview step 6 Paul Manwaring