nent
This section shows the quantity as well as quality of experimental results of the first stage in our framework, then we discuss about strong points and weak points of this
component.
4.3.1 Initial Results
In preliminary stage, we first trained GAN-based image translation method with our customized dataset. Our dataset contains 14937 daytime images and 14879 nighttime images which are in two day and night domain, respectively. This experiment shows that initial results contain some failure cases, especially wrong located traffic and vehicle lights.
As the mentioned problem of dataset above, the very first results show that the image-to-image translation task is able to translate an image from daytime domain
58
Daytime Images Translated images
Figure 4.4: Results at 280k iterations. These images seem to be wrong located traffic and vehicle lights.
to nighttime domain without paired dataset. In Figure 4.4, we found that the results contain the features of nighttime domain, but there are a various wrong sparkling points like vehicle light and traffic light. This problem is explained in the following section.
4.3.2 With Perceptual Loss Refinement Results
Figure 4.5: Image-to-Image translation results with additional Perceptual loss.
59
Fine Cases. The Figure 4.5 illustrates the results of image-to-image translation task
in all segmentation framework. Almost features from daytime images are relatively converted to nighttime images. Especially, the light of vehicle or traffic looks more suitable and realistic, which is solved by adding the perceptual loss.
Figure 4.6: Image-to-Image translation results. It is too dark even for human vision.
Failure Cases. Although the majority of the results acceptably look convincing, some failure cases still exist such as in Figure 4.6 and 4.7. It is obviously seen that the translated images in the third column are too dark for the human vision. Translated images with the perceptual loss in Figure 4.6 are lack of illuminance, we hardly see any object without totally concentrating on the dark images. A glance at these images only give some bright points.
In contrast, not only for the dark, but also for the bright, there are so many bright images were translated by the trained model in Figure 4.7. For example, the translated images with the perceptual loss show the wrong color of sky in night condition, in this case, brightness instead of darkness sky. Moreover, the color of buildings should be also dark instead of bright. They are tough issues because of the wide range of data distributions. To explain this, we found some mismatches of dataset when grouping them into two separable daytime and nighttime domains.
Quantitative Result of Day2Night Translation Component. As mentioned FID above, we also calculate the differences between translated and real images in Table 4.2. FIDnignt denotes the distance between translated nighttime images and real nighttime images. The similarity with daytime images is true for FI Dpay
60
Figure 4.7: Image-to-Image translation results. The results look too bright in comparison with nighttime distribution.
ID Method FID_Night | FID_Day
1 | UNIT w/o Perceptual Loss 98.39 74.45
2 | UNIT w Perceptual Loss 97.68 81.92
Table 4.2: Quantitative Result of Day2Night Translation Component.
Discussion. In the first stage of the whole framework, image-to-image translation phase, we crawl data from the NEXET dataset and appropriately customize for training the UNIT model. After using default configuration for training, we recognise that the results are too bright and have a various shiny sparkling points demonstrating for wrong vehicle and traffic light. The very first results show that the translation model could perform well in its task. The daytime images are consistently transformed to nighttime images, which is the main purpose of translation method. To address the problem of too brightness and a vast shiny sparkling spots, we modify the loss function via adding the perceptual loss. The perceptual loss (VGG loss) helps to decrease the
large number of brightness points in the translated images via comparing the extracted features through VGG model. We finally succeed in dealing with this problem although there are still some tough failure cases.
To the initial results without the perceptual loss, we found the explanation for some failure cases is dataset problem. When double checking the dataset, we realized that there is a wide range of images which contains a huge amount of sparkling light. Taking Figure 4.8 for instance, these pictures look too bright due to many vehicles and traffic lights. That leads the model to learning the features of vehicle light and generating them randomly in the translated images.
61
Figure 4.8: Examples of shiny sparkling of vehicle light in dataset.
When observing the failure cases, particularly extremely dark images, we also found that there is also a lot of extremely dark images in the dataset. It can be seen in Figure 4.9, these are some images taken from the nighttime domains. It is really hard to gain information from these pictures even by human vision.
The similarity is true for the bright images. We also found there are many bright images in the nighttime domains. In Figure 4.10, we see some examples of twilight images or even daytime images taken from the nighttime set. These images increase noise in the translation task, which lead to generating some images with brightness condition.