In two-dimensional mirrored aperture synthesis (2-D MAS), the rank of the transformation matrix can affect the accuracy of the solved cosine visibilities; therefore, it has an impact on the accuracy of the reconstructed brightness temperature image. In this article, the influence of the rank deficient on the accuracy of the reconstructed brightness temperature image of 2-D MAS is discussed. An analysis of the rank-deficient error is performed by computing its impact on the radiometric accuracy for a reference scene. Two correction methods based on multiple measurements are proposed to correct the rank-deficient error. The simulations and experiments are carried out to demonstrate the effectiveness of the proposed methods.
Pan-sharpening is an effective method to obtain high-resolution multispectral images by fusing panchromatic (PAN) images with fine spatial structure and low-resolution multispectral images with rich spectral information. In this article, a multiscale pan-sharpening method based on dynamic convolutional neural network is proposed. The filters in dynamic convolution are generated dynamically and locally by the filter generation network which is different from the standard convolution and strengthens the adaptivity of the network. The dynamic filters are adaptively changed according to the input images. The proposed multiscale dynamic convolutions extract detail feature of PAN image at different scales. Multiscale network structure is beneficial to obtain effective detail features. The weights obtained by the weight generation network are used to adjust the relationship among the detail features in each scale. The GeoEye-1, QuickBird, and WorldView-3 data are used to evaluate the performance of the proposed method. Compared with the widely used state-of-the-art pan-sharpening approaches, the experimental results demonstrate the superiority of the proposed method in terms of both objective quality indexes and visual performance.
Contextual information has been demonstrated to be helpful for airborne image segmentation. However, most of the previous works focus on the exploitation of spatially contextual information, which is difficult to segment isolated objects, mainly surrounded by uncorrelated objects. To alleviate this issue, we attempt to take advantage of the co-occurrence relations between different classes of objects in the scene. Especially, similar to other works, convolutional features are first extracted to capture the spatially contextual information. Then, a feature decoupling module is designed to encode the class co-occurrence relations into the convolutional features; thus, the most discriminative features can be decoupled. Finally, the segmentation result is inferred from the decoupled features. The whole process is integrated to form an end-to-end network, named class-guided feature decoupling network (CGFDN). Experimental results on two widely used benchmark data sets show that CGFDN obtains competitive results (>90% overall accuracy (OA) on 5-cm-resolution Potsdam and >91% OA on 9-cm-resolution Vaihingen) in comparison with several state-of-the-art models.
Due to the influences of imaging conditions, spectral imagery can be coarse and contain a large number of mixed pixels. These mixed pixels can lead to inaccuracies in the land-cover class (LC) mapping. Super-resolution mapping (SRM) can be used to analyze such mixed pixels and obtain the LC mapping information at the subpixel level. However, traditional SRM methods mostly rely on spatial correlation based on linear distance, which ignores the influences of nonlinear imaging conditions. In addition, spectral unmixing errors affect the accuracy of utilized spectral properties. In order to overcome the influence of linear and nonlinear imaging conditions and utilize more accurate spectral properties, the SRM based on spatial–spectral correlation (SSC) is proposed in this work. Spatial correlation is obtained using the mixed spatial attraction model (MSAM) based on the linear Euclidean distance. Besides, a spectral correlation that utilizes spectral properties based on the nonlinear Kullback–Leibler distance (KLD) is proposed. Spatial and spectral correlations are combined to reduce the influences of linear and nonlinear imaging conditions, which results in an improved mapping result. The utilized spectral properties are extracted directly by spectral imagery, thus avoiding the spectral unmixing errors. Experimental results on the three spectral images show that the proposed SSC yields better mapping results than state-of-the-art methods.
Extensive attention has been widely paid to enhance the spatial resolution of hyperspectral (HS) images with the aid of multispectral (MS) images in remote sensing. However, the ability in the fusion of HS and MS images remains to be improved, particularly in large-scale scenes, due to the limited acquisition of HS images. Alternatively, we super-resolve MS images in the spectral domain by the means of partially overlapped HS images, yielding a novel and promising topic: spectral superresolution (SSR) of MS imagery. This is challenging and less investigated task due to its high ill-posedness in inverse imaging. To this end, we develop a simple but effective method, called joint sparse and low-rank learning (J-SLoL), to spectrally enhance MS images by jointly learning low-rank HS–MS dictionary pairs from overlapped regions. J-SLoL infers and recovers the unknown HS signals over a larger coverage by sparse coding on the learned dictionary pair. Furthermore, we validate the SSR performance on three HS–MS data sets (two for classification and one for unmixing) in terms of reconstruction, classification, and unmixing by comparing with several existing state-of-the-art baselines, showing the effectiveness and superiority of the proposed J-SLoL algorithm. Furthermore, the codes and data sets will be available at https://github.com/danfenghong/IEEE_TGRS_J-SLoL, contributing to the remote sensing (RS) community.
Convolutional neural networks (CNNs) have been widely used for hyperspectral image classification. As a common process, small cubes are first cropped from the hyperspectral image and then fed into CNNs to extract spectral and spatial features. It is well known that different spectral bands and spatial positions in the cubes have different discriminative abilities. If fully explored, this prior information will help improve the learning capacity of CNNs. Along this direction, we propose an attention-aided CNN model for spectral–spatial classification of hyperspectral images. Specifically, a spectral attention subnetwork and a spatial attention subnetwork are proposed for spectral and spatial classifications, respectively. Both of them are based on the traditional CNN model and incorporate attention modules to aid networks that focus on more discriminative channels or positions. In the final classification phase, the spectral classification result and the spatial classification result are combined together via an adaptively weighted summation method. To evaluate the effectiveness of the proposed model, we conduct experiments on three standard hyperspectral data sets. The experimental results show that the proposed model can achieve superior performance compared with several state-of-the-art CNN-related models.
Due to the high dimensionality of hyperspectral images (HSIs), more training samples are needed in general for better classification performance. However, surface materials cannot always provide sufficient training samples in practice. HSI classification with small size training samples is still a challenging problem. Multiview learning is a feasible way to improve the classification accuracy in the case of small training samples by combining information from different views. This article proposes a new spatial window-based multiview intact feature learning method (SWMIFL) for HSI classification. In the proposed SWMIFL, multiple features that reflect different information of the original image are extracted and spatial windows are imposed on training samples to select unlabeled samples. Then, multiview intact feature learning is performed to learn the intact feature of the training and unlabeled samples. Considering that neighboring samples are likely to belong to the same class, labels of spatial neighboring samples are determined by two factors including the labels of training samples that locate in the spatial window and the labels learned from the intact feature. Finally, unlabeled samples that have same labels under these two factors are treated as new training samples. Experimental results demonstrate that the proposed SWMIFL-based classification method outperforms several well-known HSI classification methods on three real-world data sets.
Anomaly detection is a key problem in hyperspectral image (HSI) analysis with important remote sensing applications. Traditional methods for hyperspectral anomaly detection are mostly based on the distinctive statistical features of the HSIs. However, the anomaly-detection performance of these methods has been negatively impacted by two major limitations: 1) failure to consider the spatial pixel correlation and the ground-object correlation and 2) the existence of the mixing pixels caused by both lower spatial resolution and higher spectral resolution, which leads to higher false-alarm rates. In this article, these two problems are largely solved through a novel hyperspectral anomaly-detection method based on image super-resolution (SR) and spatial correlation. The proposed method encompasses two innovative ideas. First, based on the spectral variability in the anomaly targets, an extended linear mixing model can be obtained with more accurate ground-object information. Then, image SR is used to improve the spatial resolution of the HSIs by injecting the ground-object information from the mixing model. This alleviates the effect of mixed pixels on anomaly detection. Second, spatial correlation is exploited jointly with the global Reed-Xiaoli (GRX) method and the ground-object correlation detection for anomaly detection. Experimental results show that the proposed method not only effectively improves the hyperspectral spatial resolution and reduces the false-alarm rate but also increases the detectability with the spatial correlation information. Furthermore, the results for the real HSIs demonstrate that the proposed method achieves higher rates of anomaly detection with lower false-alarm rates.
RGB image spectral super-resolution (SSR) is a challenging task due to its serious ill-posedness, which aims at recovering a hyperspectral image (HSI) from a corresponding RGB image. In this article, we propose a novel hybrid 2-D–3-D deep residual attentional network (HDRAN) with structure tensor constraints, which can take fully advantage of the spatial–spectral context information in the reconstruction progress. Previous works improve the SSR performance only through stacking more layers to catch local spatial correlation neglecting the differences and interdependences among features, especially band features; different from them, our novel method focuses on the context information utilization. First, the proposed HDRAN consists of a 2D-RAN following by a 3D-RAN, where the 2D-RAN mainly focuses on extracting abundant spatial features, whereas the 3D-RAN mainly simulates the interband correlations. Then, we introduce 2-D channel attention and 3-D band attention mechanisms into the 2D-RAN and 3D-RAN, respectively, to adaptively recalibrate channelwise and bandwise feature responses for enhancing context features. Besides, since structure tensor represents structure and spatial information, we apply structure tensor constraint to further reconstruct more accurate high-frequency details during the training process. Experimental results demonstrate that our proposed method achieves the state-of-the-art performance in terms of mean relative absolute error (MRAE) and root mean square error (RMSE) on both the “clean” and “real world” tracks in the NTIRE 2018 Spectral Reconstruction Challenge. As for competitive ranking metric MRAE, our method separately achieves a 16.06% and 2.90% relative reduction on two tracks over the first place. Furthermore, we investigate HDRAN on the other two HSI benchmarks noted as t-
e CAVE and Harvard data sets, also demonstrating better results than state-of-the-art methods.
Determining the optimal number of endmember sources, which is also called “virtual dimensionality” (VD), is a priority for hyperspectral unmixing (HU). Although the VD estimation directly affects the HU results, it is usually solved independently of the HU process. In this article, a saliency-based autonomous endmember detection (SAED) algorithm is proposed to jointly estimate the VD in the process of endmember extraction (EE). In SAED, we first demonstrate that the abundance anomaly (AA) value is an important feature of undetected endmembers since pure pixels have larger AA values than “distractors” (i.e., mixed pixels and pure pixels of detected endmembers). Then, motivated by the fact that endmembers usually gather in certain local regions (superpixels) in the scene, due to spatial correlation, a superpixel prior is introduced in SAED to distinguish endmembers from noise. Specifically, the undetected endmembers are defined as visual stimuli in the AA subspace, the EE is formulated as a salient region detection problem, and the VD is automatically determined when there are no salient objects in the AA subspace. Since the spatial-contextual information of the endmembers is exploited during the saliency analysis, the proposed method is more robust than the spectral-only methods, which was verified using both real and synthetic hyperspectral images.
Endmember extraction (EE) plays a crucial part in the hyperspectral unmixing (HU) process. To obtain satisfactory EE results, the EE can be considered as the multiobjective optimization problem to optimize the volume maximization (VM) and root-mean-square error (RMSE) simultaneously. However, it is often quite challenging to balance the conflict of these objectives. In order to tackle the challenges of multiobjective EE, we present a $({mu + lambda })$ multiobjective differential evolution algorithm ( $({mu +lambda })$ -MODE) based on ranking multiple mutations. In the $({mu + lambda })$ -MODE algorithm, ranking multiple mutations are adopted to create the mutant vectors via the scaling factor pool to enhance the population diversity. Moreover, mutant vectors employ the binary crossover operator to generate the trial vectors through a crossover control parameter pool in $({mu + lambda })$ -MODE to take advantage of the good information of the population. In addition, $({mu + lambda })$ -MODE utilizes the fast nondominated sorting approach to sort the parent and trial vectors, and then selects the elitism offspring as the next population via the $({mu + lambda })$ selection strategy. Eventually, experimental comparative results in three real HSIs reveal that our proposed $({mu + lambda })$ -MODE is superior to other EE methods.
The low spatial resolution associated with imaging spectrometers has caused subpixel target detection to become a special problem in hyperspectral image (HSI) processing that poses considerable challenges. In subpixel target detection, the size of the target is smaller than that of a pixel, making the spatial information of the target almost useless so that a detection algorithm must rely on the spectral information of the image. To address this problem, this article proposes a subpixel target detection algorithm for hyperspectral remote sensing imagery based on background endmember extraction. First, we propose a background endmember extraction algorithm based on robust nonnegative dictionary learning to obtain the background endmember spectrum of the image. Next, we construct a hyperspectral subpixel target detector based on pixel reconstruction (HSPRD) to perform pixel-by-pixel target detection on the image to be tested using the background endmember spectral matrix and the spectra of known ground targets. Finally, the subpixel target detection results are obtained. The experimental results show that, compared with other existing subpixel target detection methods, the algorithm proposed here can provide the optimum target detection results for both synthetic and real-world data sets.
Compressive sensing (CS) has recently been demonstrated as an enabling technology for hyperspectral sensing on remote and autonomous platforms. The power, on-board storage, and computation requirements associated with the high dimensionality of hyperspectral images (HSI) are still limiting factors for many applications. A recent work has exploited the benefits of CS to perform HSI classification directly in the compressively sensed band domain (CSBD). Since the required number of compressively sensed bands (CSBs) needed to achieve full band performance varies with the complexity of an image scene, this article presents a progressive band processing (PBP) approach, called progressive CSB classification (PCSBC), to adaptively determine an appropriate number of CSBs required to achieve full band performance, while also providing immediate feedback from progressions of class classification predictions carried out by PCSBC. By taking advantage of PBP, new progression metrics and stopping criteria are also designed for PCSBC. Four real-world HSIs are used to demonstrate the utility of PCSBC.
The nonnegative matrix factorization (NMF) combining with spatial–spectral contextual information is an important technique for extracting endmembers and abundances of hyperspectral image (HSI). Most methods constrain unmixing by the local spatial position relationship of pixels or search spectral correlation globally by treating pixels as an independent point in HSI. Unfortunately, they ignore the complex distribution of substance and rich contextual information, which makes them effective in limited cases. In this article, we propose a novel unmixing method via two types of self-similarity to constrain sparse NMF. First, we explore the spatial similarity patch structure of data on the whole image to construct the spatial global self-similarity group between pixels. And according to the regional continuity of the feature distribution, the spectral local self-similarity group of pixels is created inside the superpixel. Then based on the sparse expression of the pixel in the subspace, we sparsely encode the pixels in the same spatial group and spectral group respectively. Finally, the abundance of pixels within each group is forced to be similar to constrain the NMF unmixing framework. Experiments on synthetic and real data fully demonstrate the superiority of our method over other existing methods.
Low-rank and sparsity-matrix decomposition (LRaSMD) has received considerable interests lately. One of effective methods for LRaSMD is called go decomposition (GoDec), which finds low-rank and sparse matrices iteratively subject to the predetermined low-rank matrix order $m$ and sparsity cardinality $k$ . This article presents an orthogonal subspace-projection (OSP) version of GoDec to be called OSP-GoDec, which implements GoDec in an iterative process by a sequence of OSPs to find desired low-rank and sparse matrices. In order to resolve the issues of empirically determining $p = m+ j$ and $k$ , the well-known virtual dimensionality (VD) is used to estimate $p$ in conjunction with the Kuybeda et al. developed minimax-singular value decomposition (MX-SVD) in the maximum orthogonal complement algorithm (MOCA) to estimate $k$ . Consequently, LRaSMD can be realized by implementing OSP-GoDec using $p$ and $k$ determined by VD and MX-SVD, respectively. Its application to anomaly detection demonstrates that the proposed OSP-GoDec coupled with VD and MX-SVD performs very effectively and better than the commonly used LRaSMD-based anomaly detectors.
In recent years, with the development of deep learning (DL), the hyperspectral image (HSI) classification methods based on DL have shown superior performance. Although these DL-based methods have great successes, there is still room to improve their ability to explore spatial–spectral information. In this article, we propose a 3-D octave convolution with the spatial–spectral attention network (3DOC-SSAN) to capture discriminative spatial–spectral features for the classification of HSIs. Especially, we first extend the octave convolution model using 3-D convolution, namely, a 3-D octave convolution model (3D-OCM), in which four 3-D octave convolution blocks are combined to capture spatial–spectral features from HSIs. Not only the spatial information can be mined deeply from the high- and low-frequency aspects but also the spectral information can be taken into account by our 3D-OCM. Second, we introduce two attention models from spatial and spectral dimensions to highlight the important spatial areas and specific spectral bands that consist of significant information for the classification tasks. Finally, in order to integrate spatial and spectral information, we design an information complement model to transmit important information between spatial and spectral attention features. Through the information complement model, the beneficial parts of spatial and spectral attention features for the classification tasks can be fully utilized. Comparing with several existing popular classifiers, our proposed method can achieve competitive performance on four benchmark data sets.
Variants of deep networks have been widely used for hyperspectral image (HSI)-classification tasks. Among them, in recent years, recurrent neural networks (RNNs) have attracted considerable attention in the remote sensing community. However, complex geometries cannot be learned easily by the traditional recurrent units [e.g., long short-term memory (LSTM) and gated recurrent unit (GRU)]. In this article, we propose a geometry-aware deep recurrent neural network (Geo-DRNN) for HSI classification. We build this network upon two modules: a U-shaped network (U-Net) and RNNs. We first input the original HSI patches to the U-Net, which can be trained with very few images and obtain a preliminary classification result. We then add RNNs on the top of the U-Net so as to mimic the human brain to refine continuously the output-classification map. However, instead of using the traditional dot product in each gate of the RNNs, we introduce a Net-Gated GRU that increases the nonlinear representation power. Finally, we use a pretrained ResNet as a regularizer to improve further the ability of the proposed network to describe complex geometries. To this end, we construct a geometry-aware ResNet loss, which leverages the pretrained ResNet’s knowledge about the different structures in the real world. Our experimental results on real HSIs and road topology images demonstrate that our approach outperforms the state-of-the-art classification methods and can learn complex geometries.
In this article, we propose an end-to-end adaptive spectral–spatial multiscale network to extract multiscale contextual information for hyperspectral image (HSI) classification, which contains spectral feature extraction (FE) and spatial FE subnetworks. In spectral FE aspect, different from previous methods where features are obtained in a single scale, which limits the accuracy improvement, we propose two schemes based on band grouping strategy, and the long short-time memory (LSTM) model is used for perceiving spectral multiscale information. In spatial subnetwork, on the foundation of existing multiscale architecture, the spatial contextual features which are usually ignored by previous literature are successfully obtained under the aid of convolutional LSTM (ConvLSTM) model. Besides, a new spatial grouping strategy is proposed for convenience of ConvLSTM to extract the more discriminative features. Then, a novel adaptive feature combining way is proposed considering the different importance of spectral and spatial parts. Experiments on three public data sets in HSI community demonstrate that our methods achieve competitive results compared with other state-of-the-art methods.
This article proposes a denoiser for hyperspectral (HS) images that consider, not only spatial features, but also spectral features. The method starts by projecting the noisy (observed) HS data onto a lower dimensional subspace and then learns a Gaussian mixture model (GMM) from 3-D patches or blocks extracted from the projected data cube. Afterward, the minimum mean squared error (MMSE) estimates of the blocks are obtained in closed form and returned to their original positions. Experiments show that the proposed algorithm is able to outperform other state-of-the-art methods under Gaussian and Poissonian noise and to reconstruct high-quality images in the presence of stripes.
Due to the limitations of hyperspectral imaging systems, hyperspectral imagery (HSI) often suffers from poor spatial resolution, thus hampering many applications of the imagery. Hyperspectral super resolution refers to fusing HSI and MSI to generate an image with both high spatial and high spectral resolutions. Recently, several new methods have been proposed to solve this fusion problem, and most of these methods assume that the prior information of the point spread function (PSF) and spectral response function (SRF) are known. However, in practice, this information is often limited or unavailable. In this work, an unsupervised deep learning-based fusion method—HyCoNet—that can solve the problems in HSI–MSI fusion without the prior PSF and SRF information is proposed. HyCoNet consists of three coupled autoencoder nets in which the HSI and MSI are unmixed into endmembers and abundances based on the linear unmixing model. Two special convolutional layers are designed to act as a bridge that coordinates with the three autoencoder nets, and the PSF and SRF parameters are learned adaptively in the two convolution layers during the training process. Furthermore, driven by the joint loss function, the proposed method is straightforward and easily implemented in an end-to-end training manner. The experiments performed in the study demonstrate that the proposed method performs well and produces robust results for different data sets and arbitrary PSFs and SRFs.
