�ݺ�ߣ

Editor's Notes

• #3: In this work where we create a Dense semantic map of an urban region. By Dense semantic map we mean an overhead bird’s eye view of a region associated with object labels This image shows a street image (below ) and the associated labelled map on the top. Adding semantic information helps in making an autonomous vehicle help to understand its surroundings. With dense mapping a better boundary estimate can be achieved which helps in navigation for autonomous vehicles in structured environments.
• #4: For this purpose we use street level images captured from cameras mounted on a vehicle. Street level images can be taken in an relatively inexpensive manner, e.g. using cameras in vehicle. They are often generally more detailed and can be captured in high frequency. They have a wealth of semantic information that can be exploited effectively to our purpose. In our experiments the images are captured by a highway maping company called yotta. The have taken images across hundreds of km’s across UK which consists of urban, semi urban and highway images.
• #5: Our frame work of semantic mapping consists of broadly two parts. The semantic mapping framework comprises of two main stages. Once the street level images are acquired, the semantic image segmentation is performed on them individually. These results at local level are aggregated and fed into a global ground plane labelling system which outputs our dense semantic map.
• #6: Our frame work of semantic mapping consists of broadly two parts. The semantic mapping framework comprises of two main stages. Once the street level images are acquired, the semantic image segmentation is performed on them individually. These results at local level are aggregated and fed into a global ground plane labelling system which outputs our dense semantic map.
• #7: Our frame work of semantic mapping consists of broadly two parts. The semantic mapping framework comprises of two main stages. Once the street level images are acquired, the semantic image segmentation is performed on them individually. These results at local level are aggregated and fed into a global ground plane labelling system which outputs our dense semantic map.
• #8: In this section we talk about the image segmentation/labelling problem. The image segmentation problem is to label every image pixels in the image into a particular class label such as car, road , building etc. Labelling is a hard problem, exponential number of possibility (millions of pixels, 10-100’s of labels). We would like to have a system where we give an input street image to an automatic labeller which will give us a semantically labelled image., like in this example telling where is road/car/pavement/building etc.
• #9: We use Conditional Random field (CRF). This has achieved state of the art results in classification of road scene data in recent years. In this framework,. The image is described as a grid graph. All the pixels in the image are the vertices of the graph. each pixel is modelled as a variable which takes a value from the label set which we need to find out.
• #10: We define an cost on the graph, whose components are given by the nodes which is called as Epix, , edges calles as E pairs and the region (E regions). The final labelling is given by x that minimises this energy.
• #11: We now define each of the energy component in details. Epix models the individual pixels cost of taking a label. Generally this is the classifier cost. In our case it is computed via the boosting approach. This is shown in the example, the green node has the cost of taking the label car.2 and road 0.1. similarly all the cost of other nodes are also computed.
• #12: The next cost item is the pairwise cost which models the pixel neighbourhood region. This kind of cost will try to enforce the consistency in label among adjacent pixels. Thus it is Sensitive to edges and preserves boundaries in images. Here in this example you can see the cost of two pixels taking the same label car is zero but non zero otherwise.
• #13: The final cost terms models the behaviour of the group of the pixels defining a region. Here the region is found by unsupervised segmentetiontion technique like manshift, and encourages the entire region to be classified into a same class label. In this example the entire clique will take a cost for an object label.
• #14: Solved in a graph cut approach. This is the alpha expansion inference. In each step, one of the label is expanded (which results in lowering the energy) and the entire process is iterated for a number of times. Here in this example first we show the expansion step for road, then building, then sky. Finally we achieve at the final solution when there is no reduction in energy value.
• #16: Solved in a graph cut approach. This is the alpha expansion inference. In each step, one of the label is expanded (which results in lowering the energy) and the entire process is iterated for a number of times. Here in this example first we show the expansion step for road, then building, then sky. Finally we achieve at the final solution when there is no reduction in energy value.
• #17: Solved in a graph cut approach. This is the alpha expansion inference. In each step, one of the label is expanded (which results in lowering the energy) and the entire process is iterated for a number of times. Here in this example first we show the expansion step for road, then building, then sky. Finally we achieve at the final solution when there is no reduction in energy value.
• #18: Solved in a graph cut approach. This is the alpha expansion inference. In each step, one of the label is expanded (which results in lowering the energy) and the entire process is iterated for a number of times. Here in this example first we show the expansion step for road, then building, then sky. Finally we achieve at the final solution when there is no reduction in energy value.
• #19: In this stage all the semantically segmented street images are aggregated and fed into an labeller which will output the ground plane map.
• #20: A graph is defined over the ground plane , and each pixel of the ground plane is a vertex/node in the graph, with adjacent vertices connected. For each ground plane pixel a voting scheme determines the cost of the label it takes. The pairwise connections are added to get smoothes in the output solution.
• #21: Firstly for aggregation of results to determine the per-pixel classifier cost in the ground plane , we assume a flat world. We define a ground plane patch given the camera viewing direction and project it on to the camera. This gives us a relationship between the image and the ground plane.
• #22: Firstly for aggregation of results to determine the per-pixel classifier cost in the ground plane , we assume a flat world. We define a ground plane patch given the camera viewing direction and project it on to the camera. This gives us a relationship between the image and the ground plane.
• #23: Firstly for aggregation of results to determine the per-pixel classifier cost in the ground plane , we assume a flat world. We define a ground plane patch given the camera viewing direction and project it on to the camera. This gives us a relationship between the image and the ground plane.
• #24: A histogram of label prediction is build for ground plane pixel by aggregating in a voting manner
• #25: Multiple view is added accross time and cameras
• #34: Our dataset consist of the subset of all the images taken by the Yotta dcl. We work on images of 14.8 km road track. The images are from Pembroke city UK. We have labelled the images and made the dataset available. We have 13 object categories. Split the training /testing as 44/42 images. The training images are taken in a small but representative subset of the data.
• #35: First row shows the street images, and the object labelling in the second row. The last row denotes the corresponding ground truth.
• #36: This example shows the semantic map result. In the top-right we can see the track of the vehicle taking images.
• #37: As the semantic labelling of a ground plane map is expensive, we use a back-projection method to evaluate the map. We project the ground plane map results into the image domain and evaluate in the image domain Labelling of ground plane from google maps is expensive and
• #38: This video shows our output. Firs we see the images taken by the car, as the car moves around pembroke city UK. The street level images are semantically segmented in the crf framework. These information are aggregated to build the map. The map is continuously updated as more and more information is added from the street level images. We can see the vehicle turns which is reflected in the map. This is a closeup view of the map, with pavement and cars being shown in the map and the images. Cars in the images captured in the image and shown in the map. This is another close up view where the pavement is flanked by the road and car-park. This is captured in the overhead generated map.
• #39: Finally to conclude, we have presented a method of performing a dense sematic mapping . We showed experiments on large track of data, and we make this dataset available. We hope to make it real time in future with fast binary features and fast maen field based inference for labelling. Thank you.
• #40: Thank you
• #43: Multiple view is added accross time and cameras