In our previous blog post, we explained the initial stages of our pipeline dedicated to pinpointing street lights within point clouds. These stages encompassed data collection, the creation of a training set, semantic segmentation of our point clouds, extraction of street lights from segmented point clouds, and the exploration of ethical considerations.
In this current blog post, we delve into the concluding segments of the pipeline, specifically addressing steps 7 and 8, as visualised in Figure 1. Additionally, we will present some noteworthy results and provide insights into our project's fairness evaluation.
Figure 1: Street light localization pipeline
Step 7: Validation of streetlights
Following the extraction of potential street lights in step 6, a crucial step involves their validation. For this purpose, we've implemented a pole tool that enables a swift review of all images featuring potential street lights, allowing annotators to label them accurately. In addition to options like True Positive and False Positive, annotators can express uncertainty regarding the correct label or suggest adjustments to the street light fit (from which we derive location, height, and skewness). Subsequently, these instances undergo further analysis and modification.
To expedite the labeling process, we collaborated with Spectrum Intelligence, a specialised company in data annotation. People on the autism spectrum, possessing advanced skills in patience, repetitiveness, and precision, carry out these annotation tasks. In partnership with Spectrum Intelligence, we have successfully labeled a total of 150,000 images of potential street lights.
Step: 8 Matching of streetlights
After validating street lights in step 7, the next phase involves matching the identified street lights in point clouds with those in the existing database. This process unfolds through the following steps:
- Removal of hanging street lights on cables (as the project exclusively focuses on standing street lights).
- Elimination of potential duplicates in both point cloud and database street lights.
- Exclusion of False Positives in the point cloud data identified in step 7.
- Matching street lights from point clouds with those in the database when they are closest to each other.
Figure 2 illustrates matches within a meter of each other in Amsterdam Oost. Our technique demonstrates the capability to match street lights across the entire district, highlighting the effectiveness of our matching method.
Figure 2: Matched street lights in Oost within one meter.
Feedback rounds and integration with existing registry
To seamlessly integrate our matching results into the existing registries, we engage in multiple feedback rounds with the public lighting department. The process commences with a discussion of our results in Amsterdam Oost and the exploration of potential methods for transferring matches from this district. Once a mutually agreed-upon approach is established, we move forward with matching each district and subsequently transferring the matched streetlight information to the public lighting department. This collaborative process ensures a thorough and effective integration of our results into the existing infrastructure.
Results
So, how does this all work out? We manage to match 76% of the street lights that are in the current database. This means that for a large part of the street lights, the current information can be made more precise and richer. The remaining 24% of the street lights we cannot match. About half of those are due to a lack of point cloud coverage in the area where the street light is expected. We see this a lot for example in parks, where the cars that gather the data cannot enter. The other half needs additional investigation.
In addition, we retrieve the information of more than 9.000 street lights that are not in the database yet. Some of these street lights might be on private terrain and not the responsibility of the municipality. But there are definitely some new additions to be made to the database!
Fairness Evaluation
In addition to looking at overall results, we are looking into results per neighborhood, to check whether certain groups of people benefit more from our work than others. We do observe that the match percentage varies wildly over the city. Matching percentages are notably low in neighborhoods predominantly consisting of parks (like Westergasfabriek and Vondelpark-West) and in new areas (like Pampusbuurt-West and Cruquiusbuurt). However, beyond these instances, discerning a clear pattern proves challenging. For quite a lot of neighborhoods, a low point cloud coverage doesn't seem to be the reason. Maybe the occuring street light types or street scenes are the cause, but that requires further investigation.
Next, we check how the results relate to the demographics of the neighborhoods. We looked into six sensitive variables (based on relevance and availability in the BBGA): nationality (% no migration background), gender (% women), age (% 65+), marital status (% one-person households), illness (% physical disability) and social-economic status (% low SES). We are looking for correlations between these variables per neighborhood and the match percentage per neighborhood. As the demographic variables are not normally distributed, we are using Spearman's correlation. The correlation coefficients are between –0.11 and 0.14 and we consider this as not alarming. We conduct a similar analysis for point cloud coverage per neighborhood, where once again we do not find very strong associations (between –0.18 and 0.12). Therefore, no additional action needs to be taken.
Impact
At present, the public lighting department is integrating the new data into their asset registries. These updated and expanded registries will enhance their asset management capabilities. Looking ahead, we aspire to implement our pipeline annually, aiming to continually improve asset management practices in the future.
*Header Source: Dock bridge near canals in Amsterdam
Source: amsterdamintelligence.com