An onboard automatic change detection system for disaster monitoring

This paper presents a new approach to disaster monitoring using an automatic change detection system onboard small satellites that features image tiling and fuzzy inference. Unlike other onboard change detection systems for satellites, the proposed system performs change detection on an image tile level rather than on a pixel-by-pixel basis. This image tiling approach allows for more robust change detection performance in the presence of misregistration errors. An important block in the automatic change detection system is the fuzzy inference engine, which generates control signals that trigger different onboard tasks such as image compression, issuing of warning alerts, transmission and rescheduling. The proposed scheme uses not only spectral information as the input data but also cloud cover information to improve the change detection results. Experimental results on accuracy of change detection and flood detection using satellite images are presented.


Introduction
Man-made and natural disasters can cause devastation, resulting in casualties and loss of infrastructure and livestock on an enormous scale. For example, the earthquake and tsunami disaster that struck southeast Asia in December 2004 (Lay et al. 2005) resulted in the death of 160 000 people, and 5 million more people were displaced. The total cost of damage is estimated to be around USD 2 billion . In the following year, hurricane Katrina resulted in the flooding of New Orleans, USA (NOAA 2005, Travis 2005, causing damage amounting to USD 60 billion (Brakenridge et al. 2006). As natural disasters can cause great loss of life and infrastructure it is very important to have in place systems that can effectively predict, detect and monitor such events, facilitating rapid response and rescue efforts.
Earth Observation (EO) satellites flying in low Earth orbit (LEO) have been used as a tool to monitor disasters by providing satellite imagery. These satellites supply valuable data for assessment of the damaged areas (Borrero 2005, Chen et al. 2005, Miura et al. 2006. However, they do not have onboard capabilities to carry out realtime processing and decision-making operations, necessary for rapid responses to disasters, including mapping, prediction and issuing of warning alerts. Traditionally, EO satellites follow a 'store-and-forward' scheme, in which the captured images are stored onboard before being transmitted to the ground station. This mechanism causes delay and is not suitable for rapid response, as the images are delivered for subsequent processing only when the satellite arrives at the nearest ground station.
Because of the lack of a global network of ground stations, the transmission of the images can be delayed as LEO satellites pass several orbits before reaching the required ground station.
Intelligent processing capabilities onboard EO satellites could speed up the response time significantly by extracting information from the images before they are transmitted to the ground. For example, selective image compression could be performed onboard the satellite so that only relevant images are transmitted to the ground. The captured images could also be processed onboard to check for changes in the monitored areas. In fact, the ability to detect temporal changes in images is one of the most important functions in intelligent image processing systems for hazard and disaster monitoring applications. This capability could ensure a rapid response that would greatly benefit the monitoring of disasters such as floods, earthquakes and oil spills.
An automatic change detection system for small satellites is being developed at the Surrey Space Centre, targeted at onboard flood monitoring (Yuhaniz et al. 2007). This paper presents a new approach to change detection based on image tiling and fuzzy inference. Unlike other change detection systems, the proposed system performs change detection on a tile-by-tile rather than on a pixel-by-pixel basis. The paper is organized as follows: §2 discusses background issues and related work; §3 describes the proposed onboard automatic change detection system; in §4 we discuss the experimental results; and in §5 we present our conclusions.

Background
Small EO satellites are a recently developed category of EO satellites weighing less than 500 kg whose main function is to observe the Earth. The small satellite concept has been pioneered by Surrey Satellite Technology Limited (SSTL). Unlike the large EO satellites that take much longer to build, small EO satellites offer a cheaper and faster approach to satellite development. In addition, because of their lower design and manufacturing costs, small satellites offer a viable platform for EO constellations in LEO, which provide more frequent observations (Stephens 2007). For example, the Disaster Monitoring Constellation (DMC) developed by SSTL, which comprises five microsatellites, provides daily imaging of any point on the globe. The standard DMC platform carries 32 m multispectral imagers with a very wide swath of 600 m · 600 km. The DMC satellites have captured images of all disaster areas since 2003, including the 2004 tsunami in southeast Asia. However, image processing onboard small satellites is a very challenging task because of limited computing resources.
There are two main image processing functions onboard an EO satellite: image compression and autonomous image analysis and selection. The purpose of the image analysis and selection function is to analyse, detect and classify the image according to certain characteristics. Automated remotely sensed change detection analysis is a particularly useful feature for disaster monitoring onboard satellites. A general process of change detection takes as inputs two images, a reference image and a sensed image, which are captured at different times. The two images need to be preprocessed so that they are in the same space representation. Preprocessing is an important component that involves the operation of registration. The output of the change detection is a change image that contains change and no-change information. The change image is then analysed to extract changes relevant to the user. The main problems in change detection systems for remote sensing are finding the threshold 6122 S. S. Yuhaniz and T. Vladimirova between change and no-change areas, the sensitivity to registration noise, and cloud cover in the case of optical images. The need for disaster monitoring from space and the emergence of small EO satellite platforms has initiated a number of research projects targeted at advanced onboard image processing. This has enabled the launch of satellites with sophisticated imaging payloads. Onboard image compression and cloud detection are the earliest onboard image processing units that have been developed and launched on small EO satellites, as demonstrated by UoSat-5 (Montpetit et al. 1993). Fire detection based on image processing has also been demonstrated onboard the BIRD satellite (Haller et al. 2002).
Change detection is relatively less developed than the other onboard image processing functions because of the complexity of the change detection process. Furthermore, high accuracy of registration is needed to ensure the accuracy of the change detection analysis because change detection methods are sensitive to misregistration errors. The only known change detection system onboard a satellite is the National Aeronautics and Space Administration (NASA) EO-1 spacecraft, which belongs to the category of large satellites. EO-1 performs change detection as part of its onboard flood detection algorithm (Ip et al. 2006) and cryospheric detection algorithms (Dogget et al. 2006). The algorithms were developed under the Jet Propulsion Laboratory (JPL) Autonomous Sciencecraft Experiment. The project includes the development of onboard algorithms for image classification, compression and change detection. The change detection process is performed by comparing the total number of detected feature pixels with uploaded threshold values to trigger follow-up observations. A different approach to onboard change detection is proposed in Spigai and Oller (2002), where the change detection is used together with image compression and neural network-based classification. When changes are detected in an image tile, neural network classification is performed to identify the features in the changed image tile and the remaining unchanged areas are then compressed. However, this work does not discuss issues related to the implementation of the change detection approach in a satellite platform.
Some researchers have analysed the effect of image misregistration on the accuracy of remotely sensed change detection. Dai and Khorram (1998) reported that change detection accuracy drops dramatically within the first pixel of misregistration. Townshend et al. (1992) performed an evaluation of the effect of misregistration errors on moderate-resolution satellite imagery, concluding that highly accurate registration is needed to achieve a reliable change detection system. This is confirmed by the evaluation of the accuracy and computing performance of several image registration and change detection methods on different computing platforms (Yuhaniz et al. 2005).
A review of the literature has revealed that there is a trend towards increasing computing power and resources onboard satellites, which will enable the implementation of advanced image processing functions, such as change detection, in future remote sensing satellite payloads (Vladimirova et al. 2006).

The change detection processing model
Change detection analysis is not a straightforward task, and is different from other image processing tasks that only process a single image, such as image compression and image encryption. It is a multi-temporal analysis, where two or more co-registered images are compared to establish the changes. The quality of the image registration is very important as the change detection process is sensitive to registration errors. In this section a new change detection processing model is proposed that uses information from a group of pixels rather than a single pixel to overcome this problem.
The traditional change detection process works on a pixel-by-pixel basis. Corresponding pairs of pixels in the two input images are compared, resulting in a value that represents the degree of change. A threshold then determines whether the pixel is considered changed or unchanged. The output of the change detection analysis is a change image that contains changed and unchanged pixels. The threshold value selection is a crucial task in change detection analysis as a wrong threshold would lead to a wrong result. The threshold is always set manually from trials or statistically from the mean of the change image as there are no sharp boundaries between changed and unchanged areas.
In the proposed model both images are automatically split into tiles of a predefined size. Change features are detected by comparing each tile to its matching reference tile, which increases the efficiency of change detection in the presence of misregistration in the images. To overcome the threshold selection problem, a fuzzy inference engine block has been introduced in the system (Yuhaniz et al. 2007). A decision is then made based on the comparison results using an inference engine. In this change detection processing method, the output of the system is a collection of change-detected image tiles.

Image tiling and comparison
Change detection methods can broadly be divided into two types: pre-classification and post-classification (Dai and Khorram 1998). Pre-classification methods identify changes by comparing the reference and sensed images pixel by pixel, and then a threshold is used to separate the changed and not changed pixels. Post-classification methods perform feature classification on the two images, followed by comparison of the classified images pixel by pixel to identify changed features.
Unlike the traditional pixel-to-pixel change detection process, this work proposes a change detection model by tiling the reference and sensed images and detecting changes on a tile-by-tile basis. Let X S be the sensed image from the satellite imager and X R the reference image. The approach is to split the images into a set of N image tiles X Si (i = 1, . . ., N) and X Ri (i = 1, . . ., N) of user-defined size. Each set of image tiles for the sensed and reference images is compared to find their degree of change. The output of this step is a set of image tiles that have a degree of change above the threshold, represented by the set The set of degree of change values, X D , becomes the input to the fuzzy inference engine described in §3.2, which determines whether the image tile is considered changed or not changed. Three strategies to calculate the degree of change set X D , are defined, as explained below.
Option 1: Percentage of changed pixels in each pair of image tiles. X Di is calculated by determining the percentage of the total number of changed pixels over the total number of pixels in each tile:

S. S. Yuhaniz and T. Vladimirova
where n is the total number of pixels in the image tile, and a is the total number of pixels in each image tile that have satisfied the following condition: where x s and x r are the pixel values for images X S and X R , respectively, and T is the threshold that separates changed and not changed pixels.
Option 2: Correlation coefficient for each pair of image tiles. X Di is determined by calculating the correlation coefficient of the pair of image tiles. The correlation coefficient measures the similarity between the pixels in the sensed image tile and its corresponding reference image tile. The correlation coefficient for image tile i is calculated as: Option 3: Difference of means for each pair of image tiles. X Di is determined by calculating the mean for each tile i of images X S and X R :

Fuzzy inference engine
A large and growing body of literature has investigated the use of fuzzy logic in connection with the classical change detection analysis. Most of the existing methods for automatic change detection use crisp models for the selection of the threshold value. In such models, the boundary between changed and unchanged areas is determined without taking into account the uncertainty caused by the overlapping between those areas. Fuzzy logic has been used to solve the change detection threshold problem by mapping the image histogram to its fuzzy membership function (Metternicht 1999), or by mapping the log-ratio image histogram to its fuzzy membership function (Carincotte et al. 2006). The use of fuzzy logic in postclassification comparison change detection techniques has been developed by Deer and Eklund (2002). In their work, fuzzy classifiers are used instead of traditional hard classifiers during the feature classification steps before detecting changes. Inference engines are another application of fuzzy sets and logic. Fuzzy inference is the process of formulating the mapping from a given input to an output using fuzzy logic (Kong and Kosko 1992). The mapping then provides a basis from which decisions can be made, or patterns discerned. It has been used in many applications including computer vision, data classification and automatic control. A fuzzy inference system has three components: fuzzy membership functions, fuzzy operator, and if-then rules. A simple example of using a fuzzy inference engine for image classification can be found in Nedeljkovic (2004).
Although the use of fuzzy logic in change detection analysis is an active research area, there are very few literature sources on using fuzzy inference in a remote sensing change detection system. De Souza et al. (2002) proposed the use of a fuzzy inference system to detect land cover changes. The input to the system is a degree of change for each pair of pixels that represents the absolute value of the differences between two images of the same band. This method uses a pixel-to-pixel comparison scheme to create the change image, which is prone to the classic change detection problems of sensitivity to misregistration errors.
In the current research work, a fuzzy inference processing block is introduced, called the fuzzy inference engine (Yuhaniz et al. 2007). The block diagram of the fuzzy inference engine is shown in figure 1. The fuzzy inference engine receives two input variables and generates one output variable. The two input variables represent: (1) the degree of change value from the change detection processing block and (2) the percentage of cloud cover from the cloud cover processing block. The output variable denotes the control alerts that will be sent to other subsystems for subsequent actions to be triggered.
The advantage of this approach is that there is no direct pixel-to-pixel comparison, which makes it more robust to misregistration noise. Moreover, this approach allows for easy future modification if additional parameters are needed to be included in the inference engine; for example, to obtain current weather information from terrestrial weather databases. The outputs of the inference engine are warning alerts and control signals to other onboard subsystems to perform image compression, schedule the next imaging task or transmit the changed image tile, etc.
The fuzzy inference engine in figure 1 uses the fuzzy model of Mamdani and Assilian (1975), which is the most frequently used fuzzy methodology. This model was originally proposed as an attempt to control a steam engine and boiler combination, and the input and output represented in linguistic form so that they can be readily understood by the user. Thus, the variables for the fuzzy inference engine can be easily designed and adapted from expert decision-making experience. Figure 2 shows the graph of the fuzzy membership function for the input variable X D , or 'degree-of-change'. The range of this variable is divided into three fuzzy sets called 'low-change', 'medium-change' and 'high-change'. Figure 3 shows the membership function for the input variable 'perc-cloud-pixel', which represents the percentage of clouded pixels in each image tile. Similar to the 'degree-of-change' variable, this variable is divided into three fuzzy sets called 'low-cloud', 'medium-cloud' and 'high-cloud'. Figure 4 shows the fuzzy membership function of the output variable 'control-signal', which generates control signals for the subsequent onboard subsystems. A 'no-alert', 'medium-strengthalert' or 'high-strength-signal' value determines the next relevant processing task. Nine rules are created for the fuzzy inference engine, which is realized by the fuzzy associative memory shown in table 1. The rules indicate the type of outputs that are generated when two input variables, 'degree-of-change' and 'perc-cloud-pixel', are applied to the inference engine. For example, when the input variable of 'degree-ofchange' has a 'low-change' value, and the input variable 'perc-cloud-pixel' has a 'highcloud' value, then the output that is generated is 'medium-strength-alert'.

System architecture
This section outlines the system architecture of the proposed automatic change detection system for a small satellite shown in figure 5. The system comprises a database, which is implemented in the form of a hard disk drive, to store reference images and newly generated images.  The rectangular boxes in figure 5 denote the processing blocks of the system: image tiling, image registration, cloud detection, change detection, fuzzy inference engine and database update. The cylindrical block denotes an onboard database to store the results of the processing. The rhomboidal blocks denote the data that flow between the processing blocks: (1) the sensed image, which comes from the onboard imager, (2) the reference image, retrieved from the onboard database, which is compared with the sensed image, and (3) the processed image tile from the change detection processing block, which is used as the input to the fuzzy inference engine block.
The system operation starts when receiving an image from the imager. Global positioning system (GPS) data are used to 'stamp' the images with the geolocation at the time of capture, providing additional information for the subsequent tasks of image coregistration and identification of the reference image in the database. After receiving the sensed image, the image is tiled according to a predetermined size. Next, each of the sensed image tiles is co-registered to the reference image using the ground control points available on both of the images.
Cloud detection is performed with the Automatic Cloud Cover Assessment (ACCA) algorithm (El-Araby et al. 2005), which has been used in the processing of Landsat images. Next, the registered tile images are compared using one of the degree of change calculation options described in §3.1. The output of the change detection and image comparison processing block then becomes the input to the fuzzy inference engine, which generates control alerts of various strength to activate different onboard processing tasks, such as image compression, image scheduling and high priority transmission.

Experimental results
In this section we present experimental results on the accuracy assessment of the change detection system with synthetic and satellite images.

Testing with synthetic images
Three image data sets with synthetic changes were used to test the accuracy performance of the proposed change detection model as specified in different types of noise are tested: misregistration errors and salt-and-pepper noise. Test image set 1 consists of the reference image and a 'sensed' image generated by introducing synthetic changes in the original Pentagon image. Image data sets 2 and 3 contain 'sensed' images that, in addition to the changes in data set 1, include artificially introduced 1-pixel shift misalignment and salt-and-pepper noise, respectively.
One way of measuring the accuracy of feature detection is based on the confusion error matrix in table 3. Omission error (or producer's accuracy) measures the error caused by detecting changed areas as non-changed, which results into a missed change alert. Commission error (or user's accuracy) measures the error caused by detecting unchanged areas as changed, which will generate a false alarm. Overall accuracy measures the accuracy of the change detection method without taking into account the source of error (omission or commission).
The Pentagon images (512 · 512 pixels) are tiled into 1024 tiles of size 16 · 16 pixels each. As proposed and discussed in §3.1, three options are used to calculate the degree of change in the change detection processing block. Option 1 is based on using the percentage of changed pixels, option 2 uses a correlation coefficient and option 3  uses the difference of means. The accuracy is measured by comparing the results from the fuzzy inference engine with ground truth images using the confusion error matrix in table 3. However, as there are three categories of results from the fuzzy inference engine (high-strength-alert, medium-strength, no alert) and two categories of ground truth (changed and unchanged), the results are adjusted accordingly. The results of the accuracy assessment of the proposed change detection model for the image data sets specified in table 2 are presented in tables 4 and 5. Table 5 also contains accuracy results for the three image data sets when the change detection is carried out using the image differencing method for comparison purposes. This method subtracts each pixel value in one image from the corresponding pixel value in the second image to get the change image. Data with non-zero values are change areas and zero or near-zero values are no-change areas in the resultant change image. Table 4 shows that the fuzzy inference engine works well on image data set 1 with all three degree of change calculation options because most of the unchanged image tiles generate a no-alert signal. This is to be expected because this data set is the simplest case of a change detection problem, where there is no misalignment and no noise exists in the test image. The accuracy results on image data set 1 in table 5 show that option 3 is the best method among the tested options, with overall accuracy of 97% and a low omission error of 2.5%.
The results for image data set 2 present different outcomes for the three processing options of the degree of change. It can clearly be seen from table 4 that option 1, which involves pixel-to-pixel comparison, has failed to perform, producing an overall accuracy of 5.86% (table 5). Options 2 and 3 perform better than option 1, and also better than the image differencing method, which shows a very poor performance at the overall accuracy of 2.13%. This is to be expected as this data set simulates misregistration errors. The pixel-by-pixel image differencing method also performs very badly, with 2.13% overall accuracy, for the same reasons.
Image data set 3 gives rise to better results than image data set 2, with all three degree of change methods producing high overall accuracy results of more than 95%, as shown in table 5. This shows that the salt-and-pepper noise does not affect the change detection process as much as the misregistration noise does.

Testing with satellite images
The proposed change detection scheme was also tested on multispectral satellite images for the purpose of flood detection. Three spectral bands were used: Near-Infrared (NIR), Green and Red. Three satellite image data sets 4, 5 and 6 are shown in figures 7, 8 and 9, respectively. The images represent different parts of the coastal   As flood detection is a special case of change detection (finding changes of nonwater to water areas), the flood detection process is performed in the change detection processing block (figure 5). The proposed automatic change detection system, presented in §3, uses tile information instead of single pixel information for decision making. In the flood monitoring process this concept is modified to detect flood. A detailed block diagram for the flood detection processing block is presented in figure 10. The inputs to the flood detection processing block are the reference and the sensed image tiles that have been registered in the image registration processing block. Based on performance evaluation of a number of flood detection methods, it has been found that NIR/Red differencing gives the best results (Yuhaniz et al. 2007). In this method image pixels are determined as water if the ratio of NIR and Red bands is lower than a threshold denoted by T 0 . The value of the threshold T 0 is defined by the histogram of NIR/Red, which is in bimodal distribution. T 0 is located between the two peaks of the histogram, where one peak is for water and the other is for non-water areas (Sheng et al. 2001).
The change detection process in the flood detection block in figure 10 is performed on the ratio image of the NIR and Red bands using the proposed change detection scheme as described in §3, whereby the process generates a degree of change value for each pair of NIR/Red image tiles. The degree of change becomes the input to the fuzzy inference engine, which decides if the tile is considered flooded or not. If T 0 is used to threshold the pixels, some ambiguous pixels would be wrongly classified as water or non-water. However, using fuzzy logic, not only the water and non-water pixels but also ambiguous pixels can be considered in the flood detection process. The pixel values can be either water, ambiguous or non-water. The membership function will define how close the ambiguous pixels are to either water or non-water pixels.
Ground truth images are generated manually by visual inspection for all the three satellite data sets as shown in figures 11(a), 12(a) and 13(a). The white colour on the ground truth images represents the flooded areas and the black colour the areas not flooded. Figures 11(b), 12(b) and 13(b) show the results of the flood detection process, with the black areas denoting the unchanged areas. The flood monitoring accuracy results are presented in table 6. The assessment of the accuracy results is summarized in table 7, which also includes data based on a pixel-by-pixel method for comparison purposes.
A total of 70 flooded image tiles generate high and medium strength alerts for image data set 4 as seen in figure 6. This result shows that the proposed system successfully generates flood alerts for most of the flooded areas, although a few  (table 7). The system achieves an overall accuracy of 93.76%, with a commission error of 16.67% and a small omission error of 4.62%. While the system generates high or medium strength alerts for 50 of the flooded image tiles of test image set 5, it produces a commission error of 40.63% as shown in table 7. Nevertheless, the overall accuracy of the system on this test data set is 92.96%, with an omission error of 5.23%. It can be seen from the results for the image data set 6 that the system successfully detects  most of the flooded and non-flooded areas with an overall accuracy of 92.48%, a lower commission error of 31.58% and an omission error of 5.11%. Table 7 shows that the proposed method performs well, generating better results than the pixel-by-pixel method for image data sets 4 and 5 and achieving comparable results with respect to image data set 6.

Discussion
The test results presented above confirm that the proposed approach based on tile-bytile change detection and fuzzy inference is more robust against registration noise. The system also achieves good accuracy for flood detection. The fuzzy inference engine is shown to be an effective tool and a suitable replacement for pixel-by-pixel change detection methods, which are based on manually detected crisp threshold values. Careful tuning of the fuzzy rules has the potential to improve the change detection accuracy significantly.
A novel solution to flood detection for small satellites is proposed in Yuhaniz et al. (2007), combining GPS reflectometry data and optical images. The ability of GPS reflections to penetrate cloud cover will act as a valuable complement and backup for the flood maps produced from the optical images. The fuzzy inference engine could easily be modified to take input from GPS signals as well as other complementary data sources.
As the tile-by-tile approach works over a larger area, the change metrics are 'averaged' and some information could be lost if the image tile size is too large. However, as the image tile decreases in size, the effects of misregistration would increase. Smaller image tile sizes also increase the processing time. Experimentation with different tile sizes for various UK-DMC multispectral images has shown that the execution time increases dramatically if the tile size is reduced below 50 · 50 pixels. Therefore, the proposed approach to change detection using image tiles is best suited to disaster monitoring applications, such as flooding, where large contiguous areas are subject to change.

Conclusions
An automatic change detection system for disaster monitoring onboard small EO satellites is proposed that is based on a simple change detection scheme with novel features such as tile level processing and the use of a fuzzy inference engine to assist in the decision-making process.
A new change detection model using image tiling is introduced that enables change detection to be effective even when misregistration errors exist in the images. Unlike the traditional pixel-to-pixel change detection process, this approach compares reference and sensed image tiles to generate a degree of change value. Three methods for the calculation of the degree of change value are defined and evaluated. It is verified experimentally that the scheme is more robust to errors caused by misregistration than traditional pixel-by-pixel methods.
The use of cloud cover information in addition to spectral images as input data improves the change detection results. The fuzzy inference engine could be easily modified to allow for addition of new parameters in future system upgrades. For example, GPS reflectometry signals that penetrate clouds and weather information from terrestrial databases could be incorporated to improve the accuracy of the flood monitoring process.
As the proposed change detection scheme is based on a tile-by-tile processing method, the selection of the right size of the image tile is an important issue, which requires further investigation. For the system to be suitable for onboard use, a faster hardware implementation needs to be realized to reduce the lengthy execution time for certain image sizes. Future work will also address aspects related to mitigation of radiation effects and low-power consumption, as well as application of the proposed change detection scheme to other types of satellite imagery, such as hyperspectral and radar images.