Cloud Computing in Remote Sensing PDF by Lizhe Wang, Jining Yan and Yan Ma

By

Cloud Computing in Remote Sensing
By Lizhe Wang, Jining Yan and Yan Ma

Cloud Computing in Remote Sensing

Contents

Preface xi
1 Remote Sensing and Cloud Computing 1
1.1 Remote Sensing . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Remote sensing de_nition . . . . . . . . . . . . . . . . . 1
1.1.2 Remote sensing big data . . . . . . . . . . . . . . . . . 2
1.1.3 Applications of remote sensing big data . . . . . . . . 3
1.1.4 Challenges of remote sensing big data . . . . . . . . . 5
1.1.4.1 Data integration challenges . . . . . . . . . . 5
1.1.4.2 Data processing challenges . . . . . . . . . . 5
1.2 Cloud Computing . . . . . . . . . . . . . . . . . . . . . . . . 6
1.2.1 Cloud service models . . . . . . . . . . . . . . . . . . . 6
1.2.2 Cloud deployment models . . . . . . . . . . . . . . . . . 7
1.2.3 Security in the Cloud . . . . . . . . . . . . . . . . . . . 7
1.2.4 Open-source Cloud frameworks . . . . . . . . . . . . . 8
1.2.4.1 OpenStack . . . . . . . . . . . . . . . . . . . 8
1.2.4.2 Apache CloudStack . . . . . . . . . . . . . . 10
1.2.4.3 OpenNebula . . . . . . . . . . . . . . . . . . 10
1.2.5 Big data in the Cloud . . . . . . . . . . . . . . . . . . 12
1.2.5.1 Big data management in the Cloud . . . . . 12
1.2.5.2 Big data analytics in the Cloud . . . . . . . 12
1.3 Cloud Computing in Remote Sensing . . . . . . . . . . . . . 14

2 Remote Sensing Data Integration in a Cloud Computing
Environment 17
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Background on Architectures for Remote Sensing Data Integration  18
2.2.1 Distributed integration of remote sensing data . . . . 18
2.2.2 OODT: a data integration framework . . . . . . . . . 19
2.3 Distributed Integration of Multi-Source Remote Sensing Data 20
2.3.1 The ISO 19115-basedc metadata transformation . . . . 20
2.3.2 Distributed multi-source remote sensing data integration 22
2.4 Experiment and Analysis . . . . . . . . . . . . . . . . . . . . 24
2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 

3 Remote Sensing Data Organization and Management in a
Cloud Computing Environment 29
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 Preliminaries and Related Techniques . . . . . . . . . . . . . . 31
3.2.1 Spatial organization of remote sensing data . . . . . . . 31
3.2.2 MapReduce and Hadoop . . . . . . . . . . . . . . . . . 32
3.2.3 HBase . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2.4 Elasticsearch . . . . . . . . . . . . . . . . . . . . . . . 33
3.3 LSI Organization Model of Multi-Source Remote Sensing Data 35
3.4 Remote Sensing Big Data Management in a Parallel File System 38
3.4.1 Full-text index of multi-source remote sensing metadata 38
3.4.2 Distributed data retrieval . . . . . . . . . . . . . . . . 40
3.5 Remote Sensing Big Data Management in the Hadoop Ecosystem . . . 42
3.5.1 Data organization and storage component . . . . . . . 42
3.5.2 Data index and search component . . . . . . . . . . . 43
3.6 Metadata Retrieval Experiments in a Parallel File System . . 45
3.6.1 LSI model-based metadata retrieval experiments in a parallel _le system . . . . . 45
3.6.2 Comparative experiments and analysis . . . . . . . . . 48
3.6.2.1 Comparative experiments . . . . . . . . . . . 48
3.6.2.2 Results analysis . . . . . . . . . . . . . . . . 49
3.7 Metadata Retrieval Experiments in the Hadoop Ecosystem . . 51
3.7.1 Time comparisons of storing metadata in HBase . . . 52
3.7.2 Time comparisons of loading metadata from HBase to
Elasticsearch . . . . . . . . . . . . . . . . . . . . . . . 52
3.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

4 High Performance Remote Sensing Data Processing in a
Cloud Computing Environment 55
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.2 High Performance Computing for RS Big Data: State of the Art 58
4.2.1 Cluster computing for RS data processing . . . . . . . 58
4.2.2 Cloud computing for RS data processing . . . . . . . . 59
4.2.2.1 Programming models for big data . . . . . . 60
4.2.2.2 Resource management and provisioning . . . 60
4.3 Requirements and Challenges: RSCloud for RS Big Data . . . 61
4.4 pipsCloud: High Performance Remote Sensing Clouds . . . . 62
4.4.1 The system architecture of pipsCloud . . . . . . . . . 63
4.4.2 RS data management and sharing . . . . . . . . . . . 65
4.4.2.1 HPGFS: distributed RS data storage with
application-aware data layouts and copies . . . 67
4.4.2.2 RS metadata management with NoSQL
database . . . . . . . . . . . . . . . . . . . . 68
4.4.2.3 RS data index with Hilbert R+tree . . . . . 69
4.4.2.4 RS data subscription and distribution . . . . . 71
4.4.3 VE-RS: RS-speci_c HPC environment as a service . . 72
4.4.3.1 On-demand HPC cluster platforms with
bare-metal provisioning . . . . . . . . . . . . 73
4.4.3.2 Skeletal programming for RS big data processing
. . . . . . . . . . . . . . . . . . . . . . . . 76
4.4.4 VS-RS: Cloud-enabled RS data processing system . . . 77
4.4.4.1 Dynamic workow processing for RS applications
in the Cloud . . . . . . . . . . . . . . . 78
4.5 Experiments and Discussion . . . . . . . . . . . . . . . . . . 82
4.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5 Programming Technologies for High Performance Remote
Sensing Data Processing in a Cloud Computing Environment 89
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.3 Problem De_nition . . . . . . . . . . . . . . . . . . . . . . . 92
5.3.1 Massive RS data . . . . . . . . . . . . . . . . . . . . . 92
5.3.2 Parallel programmability . . . . . . . . . . . . . . . . 93
5.3.3 Data processing speed . . . . . . . . . . . . . . . . . . 94
5.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 94
5.4.1 Generic algorithm skeletons for remote sensing applications
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.4.1.1 Categories of remote sensing algorithms . . . 98
5.4.1.2 Generic RS farm-pipeline skeleton . . . . . . 98
5.4.1.3 Generic RS image-wrapper skeleton . . . . . 102
5.4.1.4 Generic feature abstract skeleton . . . . . . . 105
5.4.2 Distributed RS data templates . . . . . . . . . . . . . 108
5.4.2.1 RSData templates . . . . . . . . . . . . . . . 108
5.4.2.2 Dist RSData templates . . . . . . . . . . . . . 111
5.5 Experiments and Discussion . . . . . . . . . . . . . . . . . . 115
5.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
6 Construction and Management of Remote Sensing Produc-
tion Infrastructures across Multiple Satellite Data Centers 121
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 123
6.3 Infrastructures Overview . . . . . . . . . . . . . . . . . . . . 124
6.3.1 Target environment . . . . . . . . . . . . . . . . . . . 124
6.3.2 MDCPS infrastructures overview . . . . . . . . . . . . 125
6.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 128
6.4.1 Data management . . . . . . . . . . . . . . . . . . . . 128
6.4.1.1 Spatial metadata management for
co-processing . . . . . . . . . . . . . . . . . . 130
6.4.1.2 Distributed _le management . . . . . . . . . . 131
6.4.2 Workow management . . . . . . . . . . . . . . . . . . 133
6.4.2.1 Workow construction . . . . . . . . . . . . . 136
6.4.2.2 Task scheduling . . . . . . . . . . . . . . . . . 137
6.4.2.3 Workow fault-tolerance . . . . . . . . . . . . 141
6.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
6.5.1 Related experiments on dynamic data management . . 142
6.5.2 Related experiments on workow management . . . . 146
6.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.6.1 System architecture . . . . . . . . . . . . . . . . . . . . 147
6.6.2 System feasibility . . . . . . . . . . . . . . . . . . . . . 148
6.6.3 System scalability . . . . . . . . . . . . . . . . . . . . 148
6.7 Conclusions and Future Work . . . . . . . . . . . . . . . . . 148
7 Remote Sensing Product Production in an OpenStack-Based
Cloud Computing Environment 151
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.2 Background and Related Work . . . . . . . . . . . . . . . . . 153
7.2.1 Remote sensing products . . . . . . . . . . . . . . . . 153
7.2.1.1 Fine processing products . . . . . . . . . . . 154
7.2.1.2 Inversion index products . . . . . . . . . . . 154
7.2.1.3 Thematic products . . . . . . . . . . . . . . . 154
7.2.2 Remote sensing production system . . . . . . . . . . . 155
7.3 Cloud-Based Remote Sensing Production System . . . . . . 156
7.3.1 Program framework . . . . . . . . . . . . . . . . . . . 156
7.3.2 System architecture . . . . . . . . . . . . . . . . . . . . 157
7.3.3 Knowledge base and inference rules . . . . . . . . . . . 159
7.3.3.1 The upper and lower hierarchical relationship
database . . . . . . . . . . . . . . . . . . . . 159
7.3.3.2 Input/output database of every kind of remote
sensing product . . . . . . . . . . . . . . . . 160
7.3.3.3 Inference rules for production demand data
selection . . . . . . . . . . . . . . . . . . . . . 161
7.3.3.4 Inference rules for workow organization . . . 161
7.3.4 Business logic . . . . . . . . . . . . . . . . . . . . . . . 162
7.3.5 Active service patterns . . . . . . . . . . . . . . . . . . 165
7.4 Experiment and Case Study . . . . . . . . . . . . . . . . . . . 167
7.4.1 Global scale remote sensing production . . . . . . . . . 167
7.4.2 Regional scale mosaic production . . . . . . . . . . . . 168
7.4.3 Local scale change detection . . . . . . . . . . . . . . . 170
7.4.3.1 Remote sensing data cube . . . . . . . . . . . . 171
7.4.3.2 Local scale time-series production . . . . . . . 171
7.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
8 Knowledge Discovery and Information Analysis from Remote
Sensing Big Data 175
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
8.2 Preliminaries and Related Work . . . . . . . . . . . . . . . . 176
8.2.1 Knowledge discovery categories . . . . . . . . . . . . . 176
8.2.2 Knowledge discovery methods . . . . . . . . . . . . . . 178
8.2.3 Related work . . . . . . . . . . . . . . . . . . . . . . . 179
8.3 Architecture Overview . . . . . . . . . . . . . . . . . . . . . 180
8.3.1 Target data and environment . . . . . . . . . . . . . . 180
8.3.2 FRSDC architecture overview . . . . . . . . . . . . . . . 181
8.4 Design and Implementation . . . . . . . . . . . . . . . . . . . 182
8.4.1 Feature data cube . . . . . . . . . . . . . . . . . . . . 182
8.4.1.1 Spatial feature object in FRSDC . . . . . . . 182
8.4.1.2 Data management . . . . . . . . . . . . . . . 182
8.4.2 Distributed executed engine . . . . . . . . . . . . . . . 184
8.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
8.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
9 Automatic Construction of Cloud Computing Infrastructures
in Remote Sensing 191
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
9.2 De_nition of the Remote Sensing Oriented Cloud Computing
Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . 192
9.2.1 Generally used cloud computing infrastructure . . . . 193
9.2.2 Remote sensing theme oriented cloud computing infrastructure . . . . 193
9.3 Design and Implementation of Remote Sensing Oriented Cloud
Computing Infrastructure . . . . . . . . . . . . . . . . . . . . 195
9.3.1 System architecture design . . . . . . . . . . . . . . . 195
9.3.2 System workow design . . . . . . . . . . . . . . . . . 196
9.3.3 System module design . . . . . . . . . . . . . . . . . . 198
9.4 Key Technologies of Remote Sensing Oriented Cloud Infrastructure
Automatic Construction . . . . . . . . . . . . . . . . . . 200
9.4.1 Automatic deployment based on OpenStack and Salt- Stack . . . . . 200
9.4.2 Resource monitoring based on Ganglia . . . . . . . . . 203
9.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
10 Security Management in a Remote-Sensing-Oriented Cloud
Computing Environment 207
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
10.2 User Behavior Authentication Scheme . . . . . . . . . . . . . 209
10.2.1 User behavior authentication set . . . . . . . . . . . . 209
10.2.2 User behavior authentication process . . . . . . . . . . 210
10.3 The Method for User Behavior Trust Level Prediction . . . . 213
10.3.1 Bayesian network model for user behavior trust prediction  . . 213
10.3.2 The calculation method of user behavior prediction . . 214
10.3.2.1 Prior probability calculation of user behavior attribute level . . . 214
10.3.2.2 Conditional probability of behavioral authentication set . . . 215
10.3.2.3 Method of calculating behavioral trust level . 216
10.3.3 User behavior trust level prediction example and analysis 216
10.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

11 A Cloud-Based Remote Sensing Information Service System
Design and Implementation 221
11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
11.2 Remote Sensing Information Service Mode Design . . . . . . 223
11.2.1 Overall process of remote sensing information service mode . . . 223
11.2.2 Service mode design of RSDaaS . . . . . . . . . . . . . 224
11.2.3 Service mode design of RSDPaaS . . . . . . . . . . . . 225
11.2.4 Service mode design of RSPPaaS . . . . . . . . . . . . 226
11.2.5 Service mode design of RSCPaaS . . . . . . . . . . . . 228
11.3 Architecture Design . . . . . . . . . . . . . . . . . . . . . . . 229
11.4 Functional Module Design . . . . . . . . . . . . . . . . . . . 233
11.4.1 Function module design of RSDaaS . . . . . . . . . . . 233
11.4.2 Function module design of RSDPaaS . . . . . . . . . . 234
11.4.3 Function module design of RSPPaaS . . . . . . . . . . 235
11.4.4 Function module design of RSCPaaS . . . . . . . . . . . 237
11.5 Prototype System Design and Implementation . . . . . . . . 238
11.5.1 RSDaaS subsystem . . . . . . . . . . . . . . . . . . . . 240
11.5.2 RSDPaaS subsystem . . . . . . . . . . . . . . . . . . . 242
11.5.3 RSPPaaS subsystem . . . . . . . . . . . . . . . . . . . 243
11.5.4 RSCPaaS subsystem . . . . . . . . . . . . . . . . . . . 244
11.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Bibliography 247
Index 279

This book is US$10
To get free sample pages OR Buy this book


Share this Book!

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.