This article proposes a novel synthetic aperture radar visual-inertial odometry (SAR-VIO) consisting of an SAR and an inertial measurement unit (IMU), which aims to enable the observation platform to complete successfully a continuous observation mission in the context of low-cost demand and lack of enough navigation information. First, we establish the observation models of the SAR in a continuous observation process based on the SAR frequency-domain imaging algorithm and the SAR time-domain imaging algorithm, respectively. With the preintegrated IMU data, we then propose a method for estimating the geographic locations of the matched targets in the SAR images and verify the condition and correctness of the method. The optimization of the track and the locations of the targets is achieved by bundle adjustment according to the minimum reprojection error criterion, and a sparse point-cloud map can be obtained. Finally, these methods and models are organized into a complete SAR-VIO framework, and the feasibility of the framework is verified through experiments.
Holographic synthetic aperture radar (HoloSAR) tomography is an attractive imaging mode that can retrieve the 3-D scattering information of the observed scene over 360° azimuth angle variation. To improve the resolution and reduce the sidelobes in elevation, the HoloSAR imaging mode requires many passes in elevation, thus decreasing its feasibility. In this article, an imaging method based on iterative adaptive approach (IAA) and generalized likelihood ratio test (GLRT) is proposed for the HoloSAR with limited elevation passes to achieve super-resolution reconstruction in elevation. For the elevation reconstruction in each range-azimuth cell, the proposed method first adopts the nonparametric IAA to retrieve the elevation profile with improved resolution and suppressed sidelobes. Then, to obtain sparse elevation estimates, the GLRT is used as a model order selection tool to automatically recognize the most likely number of scatterers and obtain the reflectivities of the detected scatterers inside one range-azimuth cell. The proposed method is a super-resolving method. It does not require averaging in range and azimuth, thus it can maintain the range-azimuth resolution. In addition, the proposed method is a user parameter-free method, so it does not need the fine-tuning of any hyperparameters. The super-resolution power and the estimation accuracy of the proposed method are evaluated using the simulated data, and the validity and feasibility of the proposed method are verified by the HoloSAR real data processing results.
The water body detection in high-resolution synthetic aperture radar (SAR) images is a challenging task due to the changing interference caused by multiple imaging conditions and complex land backgrounds. Inspired by the excellent adaptability of deep neural networks (DNNs) and the structured modeling capabilities of probabilistic graphical models, the cascaded fully-convolutional network (CFCN) is proposed to improve the performance of water body detection in high-resolution SAR images. First, for the resolution loss caused by convolutions with large stride in traditional convolutional neural network (CNN), the fully-convolutional upsampling pyramid networks (UPNs) are proposed to suppress this loss and realize pixel-wise water body detection. Then considering blurred water boundary, the fully-convolutional conditional random fields (FC-CRFs) are introduced to UPNs, which reduce computational complexity and lead to the automatic learning of Gaussian kernels in CRFs and the higher boundary accuracy. Furthermore, to eliminate the inefficient training caused by imbalanced categorical distribution in the training data set, a novel variable focal loss (VFL) function is proposed, which replaces the constant weighting factor of focal loss with the frequency-dependent factor. The proposed methods can not only improve the pixel accuracy and boundary accuracy but also perform well in detection robustness and speed. Results of GaoFen-3 SAR images are presented to validate the proposed approaches.
Modern spaceborne synthetic aperture radar (SAR) features a capacity of multiple imaging modes. It comes with Earth-observation data archives consisting of SAR images acquired in various modes with different resolutions and coverage. In this context, in addition to using single-mode images for SAR interferometry (InSAR), exploiting images acquired in different imaging modes for InSAR can provide extra interferograms and, thus, favors the retrieval of interferometric information. The interferometric processing of multimode image pairs requires special considerations due to significant variations in the Doppler spectra. Conventionally, the InSAR technique only uses the spectral band common in both master and slave images, and the remaining band is discarded before interferogram formation. Therefore, conventional processing cannot make full use of the observed data, and the interferogram quality is limited by the common band spectra. In this article, by exploiting the conventionally discarded spectrum, we present a new interferometric phase retrieval method for multimode InSAR data to improve interferogram quality. To this end, first, we propose a linear model to characterize the interferometric phase of a multimode image pair based on image spectral relation. Second, we adopt a sparse recovery method to inverse the linear model for the retrieval of the interferometric phase. Finally, we present real-data experiments on TerraSAR-X staring spotlight to sliding spotlight interferometry and Sentinel-1 strip map to Terrain Observation by Progressive Scan (TOPS) interferometry to test the proposed method. The experiment results show that the proposed method can provide interferograms with reduced phase noise and defocusing effect for multimode InSAR.
Low-earth-orbit (LEO) synthetic aperture radar (SAR) can achieve advanced remote sensing applications benefiting from the large beam coverage and long duration time of interested area provided by a geosynchronous (GEO) SAR illuminator. In addition, the receiving LEO SAR system is also cost-effective because the transmitting module can be omitted. In this article, an imaging method for GEO-LEO bistatic SAR (BiSAR) is proposed. First, the propagation delay characteristics of GEO-LEO BiSAR are studied. It is found that the traditional “stop-and-go” propagation delay assumption is not appropriate due to the long transmitting path and high speed of the LEO SAR receiver. Then, an improved propagation delay model and the corresponding range model for GEO-LEO BiSAR are established to lay the foundation of accurate imaging. After analyzing the sampling characteristics of GEO-LEO BiSAR, it is found that only 12.5% sampling data can be acquired in the azimuth direction. To handle the serious sub-Nyquist sampling problem and achieve good focusing results, an imaging method combined with multireceiving technique and compressed sensing is proposed. The multireceiving observation model is first obtained based on the inverse process of a nonlinear chirp-scaling imaging method, which can handle 2-D space-variant echo. Following that, the imaging problem of GEO-LEO BiSAR is converted to an $L_{1}$ regularization problem. Finally, an effective recovery method named complex approximate message passing (CAMP) is applied to obtain the final nonambiguous image. Simulation results show that the proposed method can suppress eight times Doppler ambiguity and obtain the well-focused image with three receiving channels. With the proposed method, the number of required receiving channels can be greatly reduced.
A new method based on the Looking-Around-and-Into (LAaI) mode is proposed for the task of change detection in large-scope Polarimetric Synthetic Aperture Radar (PolSAR) image. Specifically, the LAaI mode consists of two processes named Look-Around and Look-Into, which are accomplished by attention proposal network (APN) and recurrent convolutional neural network (CNN) (Recurrent CNN), respectively. The former provides certain subregions efficiently, and the latter detects changes in subregions accurately. In Look-Around, difference image (DI) of whole PolSAR images is calculated first to get global information; then, APN is established to locate the position of interested subregions intentionally by paying special attention to; next interested subregions that contain changed area in high probability are picked out as candidate-regions. Moreover, candidate-regions are sorted in importance descending order so that highly interested regions have priority to be detected. In Look-Into, candidate-regions of different scales are selected at first; then, Recurrent CNN is constructed and employed to deal with multiscale PolSAR subimages so that clearer and finer change detection results are generated. The process is repeated until all candidate-regions are detected. As a whole, the proposed algorithm based on the LAaI mode looks around whole images first to find out the possible position of changes (candidate-regions generation in Look-Around) and then reveal the exact shape of changes in different scales (multiscale change detection in Look-Into). The effect of APN and Recurrent CNN is verified in experiments, and it shows that the proposed method performs well in the task of change detection in the large-scope PolSAR image.
Ship target detection using large-scale synthetic aperture radar (SAR) images has important application in military and civilian fields. However, ship targets are difficult to distinguish from the surrounding background and many false alarms can occur due to the influence of land area. False alarms always occur with ship target detection because most of the area in large-scale SAR images are treated as background and clutter, and the ship targets are considered unevenly distributing small targets. To address these issues, a ship detection method in large-scale SAR images via CenterNet is proposed in this article. As an anchor-free method, CenterNet defines the target as a point, and the center point of the target is located through key point estimation, which can effectively avoid the missing detection of small targets. At the same time, the spatial shuffle-group enhance (SSE) attention module is introduced into CenterNet. Through SSE, the stronger semantic features are extracted while suppressing some noise to reduce false positives caused by inshore and inland interferences. The experiments on the public SAR-ship-data set show that the proposed method can detect all targets without missed detection with dense-docking targets. For the ship targets in large-scale SAR images from Sentinel 1, the proposed method can also detect targets near the shore and in the sea with high accuracy, which outperforms the methods like faster R-convolutional neural network (CNN), single-shot multibox detector (SSD), you only look once (YOLO), feature pyramid network (FPN), and their variations.
The lack of in situ data has always posed challenges to remote-sensing-data product validation. Herein, the products of sea-ice concentration (SIC) data derived using the arctic radiation and turbulence interaction study (ARTIST) sea ice (ASI) algorithm were evaluated by comparing them with SICs from a high-resolution sea-ice aerial image obtained during the 27th China Antarctic expedition in January 2011. Results suggest that data obtained from the advanced microwave scanning radiometer for the earth-observing system (AMSR-E) underestimate SICs by 19%. We performed step-by-step comparisons among the aerial image, moderate-resolution-imaging spectroradiometer (MODIS), and AMSR-E SIC. These types of comparisons have not been made in previous validation studies. First, SICs acquired from MODIS-Terra imagery were acquired using a tie-point method and corrected by SICs derived from the aerial photography. Second, SICs of MODIS-Aqua images were trained based on the consistency of SIC results between MODIS-Terra and MODIS-Aqua over the selected region on the same day. Finally, the MODIS-Aqua SICs were employed to validate synchronous AMSR-E swath SIC products. The results show that the AMSR-E products underestimate SICs by 8.5% in the marginal ice zone in comparison with MODIS SICs. According to our further analysis between sea-ice types and AMSR-E biases, the higher the proportion of first-year ice, the smaller the AMSR-E SIC bias. In other words, results suggest that the higher the thin ice proportion, the more the AMSR-E underestimates the SIC.
In this article, we first present a generalized methodology for deriving sensor brightness temperature sensor data records (SDR) from antenna temperature data records (TDR) applicable for Advanced Microwave Sounding Unit-A (AMSU-A) instruments. It includes corrections for antenna sidelobe contributions, antenna emission, and radiation perturbation due to the difference of Earth radiance in the main beam and that in the sidelobes that lie outside the main beam but within the Earth disk. For practical purposes, we simplify the methodology by neglecting the components other than the antenna sidelobe contributions to establish a consistent formulation with the legacy AMSU-A antenna pattern correction (APC) formula. The simplified formulation is then applied to the final AMSU-A instrument onboard the Metop-C satellite that was launched in November 2018, in order to compute APC coefficients for deriving SDR from TDR data. Furthermore, the performance of the calculated correction coefficients is validated by calculating the differences between the daily averaged AMSU-A (TDR and SDR) observations against radiative transfer model (O–B) simulations under clear sky conditions, and over open oceans. The validation results show that the derived temperature corrections are channel and scan position dependent, and can add 0.2–1.6 K to the antenna temperatures. In addition, the derived SDR O-B results exhibit a reduced and more symmetric scan angle-dependent bias when compared with corresponding TDR antenna temperatures.
Compressive spectral imaging (SI) (CSI) acquires few random projections of an SI reducing acquisition, storage, and, in some cases, processing costs. Then, this acquisition framework has been widely used in various tasks, such as target detection, video processing, and fusion. Particularly, compressive spectral image fusion (CSIF) aims at obtaining a high spatial–spectral resolution SI from two sets of compressed measurements: one from a hyperspectral image with a high-spectral low-spatial resolution, and one from a multispectral image with high-spatial low-spectral resolution. Most of the literature approaches include prior information, such as global low rank, smoothness, and sparsity, to solve the resulting ill-posed CSIF inverse problem. More recently, the high self-similarities exhibited by SIs have been successfully used to improve the performance of CSI inverse problems, including a nonlocal low-rank (NLLR) prior. However, to the best of our knowledge, this NLLR prior has not been implemented in the solution of the CSIF inverse problem. Therefore, this article formulates an approach that jointly includes the global low rank, the smoothness, and the NLLR priors to solve the CSIF inverse problem. The global low-rank prior is introduced with the linear mixture model that describes the SI as a linear combination of a set of few end-members to specific abundances. In this article, either the end-members are accurately estimated from the compressed measurements or initialized from a fast reconstruction of the hyperspectral image. Also, it assumes that the abundances preserve the smoothness and NLLR priors of the SI so that the fused image is obtained from the end-members and abundances that result when minimizing a cost function including the sum of two data fidelity terms and two regularizations: the smoothness and the NLLR. Simulations over three data sets show that the proposed approach increases the CSIF performance compared with literature -
pproaches.
The trade-off between feature representation power and spatial localization accuracy is crucial for the dense classification/semantic segmentation of remote sensing images (RSIs). High-level features extracted from the late layers of a neural network are rich in semantic information, yet have blurred spatial details; low-level features extracted from the early layers of a network contain more pixel-level information but are isolated and noisy. It is therefore difficult to bridge the gap between high- and low-level features due to their difference in terms of physical information content and spatial distribution. In this article, we contribute to solve this problem by enhancing the feature representation in two ways. On the one hand, a patch attention module (PAM) is proposed to enhance the embedding of context information based on a patchwise calculation of local attention. On the other hand, an attention embedding module (AEM) is proposed to enrich the semantic information of low-level features by embedding local focus from high-level features. Both proposed modules are lightweight and can be applied to process the extracted features of convolutional neural networks (CNNs). Experiments show that, by integrating the proposed modules into a baseline fully convolutional network (FCN), the resulting local attention network (LANet) greatly improves the performance over the baseline and outperforms other attention-based methods on two RSI data sets.
Developed here is an algorithm for determining the infrared (IR) cloud-top phase for advanced Himawari imager (AHI) measurements from the Japanese geostationary satellite Himawari-8. The tests and decision tree used in the AHI algorithm are different from those in the Moderate Resolution Imaging Spectroradiometer (MODIS) Level-2 cloud product algorithm. Verification of AHI cloud-top phase results with the Cloud–Aerosol Lidar with orthogonal polarization product over a four-month period from March to June of 2017 over the North Pacific gives hit rates of 80.20% (66.33%) and 86.51% (80.61%) for liquid-water and randomly oriented-ice cloud tops, respectively, if clear-sky pixels are excluded (included) from the statistics. Also made are intercomparisons between AHI and MODIS IR cloud-top phase products over the North Pacific in June 2017. AHI liquid-water-phase determinations agree with MODIS liquid-water-phase determinations at an agreement rate of 83.68%, showing a dependence on MODIS zenith angles. The agreement rate of ice phase classifications between AHI and MODIS is 93.54%. The MODIS IR product contains some unreasonable ice-phase pixels over oceans, as well as uncertain-phase pixels over land, and has limitations for daytime liquid-water-phase identifications over the Indo-China Peninsula. Limitations of the AHI cloud-top phase algorithm are mainly caused by difficulties in identifying liquid-water-phase clouds over sun-glint regions and during twilight.
In the last five years, deep learning has been introduced to tackle the hyperspectral image (HSI) classification and demonstrated good performance. In particular, the convolutional neural network (CNN)-based methods for HSI classification have made great progress. However, due to the high dimensionality of HSI and equal treatment of all bands, the performance of these methods is hampered by learning features from useless bands for classification. Moreover, for patchwise-based CNN models, equal treatment of spatial information from the pixel-centered neighborhood also hinders the performance of these methods. In this article, we propose an end-to-end residual spectral–spatial attention network (RSSAN) for HSI classification. The RSSAN takes raw 3-D cubes as input data without additional feature engineering. First, a spectral attention module is designed for spectral band selection from raw input data by emphasizing useful bands for classification and suppressing useless bands. Then, a spatial attention module is designed for the adaptive selection of spatial information by emphasizing pixels from the same class as the center pixel or those are useful for classification in the pixel-centered neighborhood and suppressing those from a different class or useless. Second, two attention modules are also used in the following CNN for adaptive feature refinement in spectral–spatial feature learning. Third, a sequential spectral–spatial attention module is embedded into a residual block to avoid overfitting and accelerate the training of the proposed model. Experimental studies demonstrate that the RSSAN achieved superior classification accuracy compared with the state of the art on three HSI data sets: Indian Pines (IN), University of Pavia (UP), and Kennedy Space Center (KSC).
Hyperspectral (HS) pansharpening, as a special case of the superresolution (SR) problem, is to obtain a high-resolution (HR) image from the fusion of an HR panchromatic (PAN) image and a low-resolution (LR) HS image. Though HS pansharpening based on deep learning has gained rapid development in recent years, it is still a challenging task because of the following requirements: 1) a unique model with the goal of fusing two images with different dimensions should enhance spatial resolution while preserving spectral information; 2) all the parameters should be adaptively trained without manual adjustment; and 3) a model with good generalization should overcome the sensitivity to different sensor data in reasonable computational complexity. To meet such requirements, we propose a unique HS pansharpening framework based on a 3-D generative adversarial network (HPGAN) in this article. The HPGAN induces the 3-D spectral–spatial generator network to reconstruct the HR HS image from the newly constructed 3-D PAN cube and the LR HS image. It searches for an optimal HR HS image by successive adversarial learning to fool the introduced PAN discriminator network. The loss function is specifically designed to comprehensively consider global constraint, spectral constraint, and spatial constraint. Besides, the proposed 3-D training in the high-frequency domain reduces the sensitivity to different sensor data and extends the generalization of HPGAN. Experimental results on data sets captured by different sensors illustrate that the proposed method can successfully enhance spatial resolution and preserve spectral information.
As one of the most important algorithms in target detection, constrained energy minimization (CEM) has been widely used and developed in recent years. However, it is easy to verify that the target detection result of CEM varies with the data origin, which is apparently unreasonable since the distribution of the target of interest is objective and, therefore, unrelated to the selection of data origin. The clever eye (CE) algorithm tries to solve this problem by adding the data origin as a new variable from the perspective of the filter output energy. However, due to the nonconvexity of the objective function, CE can only obtain locally optimal solutions by using the gradient ascent method. In this article, we find a striking conclusion that there exists an analytical solution for CE that corresponds to the solution of a linear equation and further prove that all the solutions of the linear equation are globally optimal.
Conventional low-rank (LR)-based hyperspectral image (HSI) denoising models generally convert high-dimensional data into 2-D matrices or just treat this type of data as 3-D tensors. However, these pure LR or tensor low-rank (TLR)-based methods lack flexibility for considering different correlation information from different HSI directions, which leads to the loss of comprehensive structure information and inherent spatial–spectral relationship. To overcome these shortcomings, we propose a novel multidirectional LR modeling and spatial–spectral total variation (MLR-SSTV) model for removing HSI mixed noise. By incorporating the weighted nuclear norm, we obtain the weighted sum of weighted nuclear norm minimization (WSWNNM) and the weighted sum of weighted tensor nuclear norm minimization (WSWTNNM) to estimate the more accurate LR tensor, especially, to remove the dead-line noise better. Gaussian noise is further denoised and the local spatial–spectral smoothness is preserved effectively by SSTV regularization. We develop an efficient algorithm for solving the derived optimization based on the alternating direction method of multipliers (ADMM). Extensive experiments on both synthetic data and real data demonstrate the superior performance of the proposed MLR-SSTV model for HSI mixed noise removal.
Class-wise adversarial adaptation networks are investigated for the classification of hyperspectral remote sensing images in this article. By adversarial learning between the feature extractor and the multiple domain discriminators, domain-invariant features are generated. Moreover, a probability-prediction-based maximum mean discrepancy (MMD) method is introduced to the adversarial adaptation network to achieve a superior feature-alignment performance. The class-wise adversarial adaptation in conjunction with the class-wise probability MMD is denoted as the class-wise distribution adaptation (CDA) network. The proposed CDA does not require labeled information in the target domain and can achieve an unsupervised classification of the target image. The experimental results using the Hyperion and Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) hyperspectral data demonstrated its efficiency.
Recently, convolution neural network (CNN)-based hyperspectral image (HSI) classification has enjoyed high popularity due to its appealing performance. However, using 2-D or 3-D convolution in a standalone mode may be suboptimal in real applications. On the one hand, the 2-D convolution overlooks the spectral information in extracting feature maps. On the other hand, the 3-D convolution suffers from heavy computation in practice and seems to perform poorly in scenarios having analogous textures along with consecutive spectral bands. To solve these problems, we propose a mixed CNN with covariance pooling for HSI classification. Specifically, our network architecture starts with spectral–spatial 3-D convolutions that followed by a spatial 2-D convolution. Through this mixture operation, we fuse the feature maps generated by 3-D convolutions along the spectral bands for providing complementary information and reducing the dimension of channels. In addition, the covariance pooling technique is adopted to fully extract the second-order information from spectral–spatial feature maps. Motivated by the channel-wise attention mechanism, we further propose two principal component analysis (PCA)-involved strategies, channel-wise shift and channel-wise weighting, to highlight the importance of different spectral bands and recalibrate channel-wise feature response, which can effectively improve the classification accuracy and stability, especially in the case of limited sample size. To verify the effectiveness of the proposed model, we conduct classification experiments on three well-known HSI data sets, Indian Pines, University of Pavia, and Salinas Scene. The experimental results show that our proposal, although with less parameters, achieves better accuracy than other state-of-the-art methods.
Blind hyperspectral unmixing is the process of expressing the measured spectrum of a pixel as a combination of a set of spectral signatures called endmembers and simultaneously determining their fractional abundances in the pixel. Most unmixing methods are strictly spectral and do not exploit the spatial structure of hyperspectral images (HSIs). In this article, we present a new spectral–spatial linear mixture model and an associated estimation method based on a convolutional neural network autoencoder unmixing (CNNAEU). The CNNAEU technique exploits the spatial and the spectral structure of HSIs both for endmember and abundance map estimation. As it works directly with patches of HSIs and does not use any pooling or upsampling layers, the spatial structure is preserved throughout and abundance maps are obtained as feature maps of a hidden convolutional layer. We compared the CNNAEU method to four conventional and three deep learning state-of-the-art unmixing methods using four real HSIs. Experimental results show that the proposed CNNAEU technique performs particularly well and consistently when it comes to endmembers’ extraction and outperforms all the comparison methods.
The fusion of hyperspectral (HS) and multispectral (MS) images designed to obtain high-resolution HS (HRHS) images is a very challenging work. A series of solutions has been proposed in recent years. However, the similarity in the structure of the HS image has not been fully used. In this article, we present a novel HS and MS image-fusion method based on nonlocal low-rank tensor approximation and sparse representation. Specifically, the HS image and the MS image are considered the spatially and spectrally degraded versions of the HRHS image, respectively. Then, the nonlocal low-rank constraint term is adopted in order to form the nonlocal similarity and the spatial–spectral correlation. Meanwhile, we add the sparse constraint term to describe the sparsity of abundance. Thus, the proposed fusion model is established and its optimization is solved by alternative direction method of multipliers (ADMM). The experimental results on three synthetic data sets and one real data set show the advantages of the proposed method over several state-of-the-art competitors.
