Geo-spatial Big Data Mining Techniques - Ijcaonline

International Journal of Computer Applications (0975 – 8887)Volume 135 – No.11, February 2016(Bação, 2006) also introduced the issues of Geospatial DataMining (GDM) which fitted it into the broader setting of GIScand provide the framework for Geo-Spatial Data Mining(GSDM). GSDM included three generations of GIS datamodels are existing, viz. (i) the CAD data model, (ii) the Georelational data model, and (iii) the object–relation data model.The last one abstracts geographical entities as object ofclasses with attributes, behavior and the associated rules andrelationships between objects. Some of the benefits of theGeo-database model are that they are close to humanunderstanding and are expressions of real world objects. Theyalso have a better capacity of expansibility, and therelationships between spatial data are fully expressed. All ofspatial data together with attribute data can be stored andcentrally managed in a DBMS. There was a provision to editgeographic data simultaneously by many users. Yin and Su(2006) implemented a Model for Geospatial Database asNational Fundamental Geographic Information which usedthe DBMS Oracle9i to store geospatial objects, and the Geodatabase model was applied to describe and organize thegeospatial entities and their relations.The spatial features of a particular location can be stored inthe geographic databases, in which each one of those featuresis usually located or stored in a different relation. The processof data preparation, storing and analyzing is a time consumingtask which is an issue of concern in the spatial data miningsystems. One of the obvious solutions is to automate thisprocess. Bogorny et al. (2007) presented a package named―Weka-GDPM ‖, which was an extension of Weka to supportthe automation of the spatial data preparation, storing andanalyzing for mining from geographic data. The fig. 4 showsthe different geographic data storage methods under thegeographical database management systems (GDBMS)following the Open Geo-spatial Consortium (OGC)specifications. It also contains a knowledge repository forstoring well known geographic associations extracted fromgeographic database schemas, Geo-Ontologies, and thoseprovided by the users. It can seen that the first part of thediagram contains different data mining algorithms for the taskof extracting the needed knowledge out of the database. Thesecond part of this diagram is a new one which takes care ofthe spatial data preparation and it is located in the center tosolve the problem that exists between the GDBMS and thedata mining tools. The JDBC is used to reach to the data. Thedata is retrieved, preprocessed, and transformed into a singletable format according to the user specifications.Fig. 4: Integrated spatial data mining framework(Bogorny et al. 2007)4. GEO-SPATIAL DATA AS BIG DATAAccording to some authors there have been big-bangexplosion of data in recent years. The data acquisitions at therates of terabytes per day are quite common. The spatial dataon mobile phone users, rail and air travelers, marketing,consumers, goods production and wide range of dailyactivities are producing large volume of data which are ofinterest for Data Mining. The remote sensing data are noexception to these. A large number of meteorological, landand ocean observations sensors on board satellites arecontinuously down pouring data from space. The analyses ofsuch large data present their own challenges, even thoughwith highly powerful processors and high speed data accesspossible presently.One of the characteristics of this volume of data generated istheir formats. The data may be structured, semi-structured andun-structured formats, which further complicate their analysis.As a historical study of the term ―Big Data‖, it is noticed thatit used first time in by Mashey(1997). Diebold (2000)presented the first academic paper with the words ―Big Data‖in the title in 2000, and Doug Laney was the first one tointroduce the 3 Vs characterizing the Big Data. The Big Datadata are sets whose size (volume), complexity (variability),and rate of growth (velocity) make them complex to becollected, managed, processed or analyzed by currenttechnologies and tools (Bhosale and Gadekar, 2014). The BigData analysis require ingenious steps for data analysis,characterized by its three main components: variety, velocityand volume” (Sagiroglu & Sinanc, 2013).On the top of this framework there are:1.The Metadata retrieval module: for retrieving allrelevant information from a database,2.The Dependence Elimination module: forverifying all associations between the target featuretype and all relevant feature types,3.The Spatial Join module: for computing ips, and4.The Transformation module: for transposing aswell as discretizing the Spatial Join module outputinto the single table representation, understandableby data mining algorithmsSagiroglu and Sinanc (2013) compared two of the existing bigdata analysis platforms, viz. Hadoop framework and HighPerformance Computing (HPC) clusters (HPCC).The hadoopis a known open source Java based framework, whichincludes, firstly, a system of several main parts starting withthe distributed file system, and the secondly the system thatmanages the workflow and the distribution of the processes inthe system. Apache Hadoop is an open source implementationof Google’s MapReduce framework; it comes with itsdistributed file own system (HDFS), which was derived fromGoogle File System (GFS). The HDFS has many benefitssuch as low cost of storing the data, data redundancy byreplication, huge storage abilities, balanced utilization ofstorage, high fault-tolerance, high throughput rate, andscalability. These two main parts of Apache hadoop are theHDFS and MapReduce are shown in Fig. 5. The Hadoopframework also provides many other sub-projects, viz. HBase,Pig, Hive, Sqoop, Avro, Oozie, Chukwa, Flume, andZooKeeper, each with its specific focus.30

International Journal of Computer Applications (0975 – 8887)Volume 135 – No.11, February 2016Fig. 5: Comparison of Hadoop and HPCC frameworks(Sagoroglu and Sinanc, 2013)The HPCC Systems is also an open source, distributed, dataintensive computing platform. It provides big data workflowmanagement services. The HPCC data model is defined by theuser. The three main HPCC components are: HPCC DataRefinery (Thor), HPCC Data Delivery Engine (Roxie), andEnterprise Control Language (ECL). Here, in this paper, theHadoop framework is dealt with in more details; especially incontext of geo-spatial data analysis.5. HADOOP FRAMEWORK FORSPATIAL DATA ANALYSISThe main concept of the framework is segregated in two parts,viz. the Hadoop Distributed File System (HDFS) for storingdata and the MapReduce programming model to process thedata (fig. 6). It is a distributed data processing framework. Itwas first developed by Google and later it was adopted byApache Foundation. It is capable of managing large volumedata efficiently, using a large number of commodity hardwareas Nodes forming a hadoop cluster. The MapReduce model ofdata analysis is composed of map and reduce functions. Theconcepts of the framework are based on divide and conquerrule, in which, the big volume data are divided into smallclusters and processed parallel as SIMD or MIMD techniques.The map function reads data and converts it into key-valuepairs. Later the reduce function, after shuffling the key-valuepairs are converted into key-multiple values pairs. A job isdivided into the following five tasks: (1) Iteration over theinput data, (2) Computation of key/value pairs from each ofinput data, (3) Grouping of all intermediate values by key, (4)Iteration over the resulting groups, and (5) Reduction of eachgroup.Economides et al. (2013) presented a new Spatial DatabaseDesign or Spatial Database Management System (SDBMS)named MIGIS which is a Hadoop based framework forhandling efficiently complex spatial queries with highperformance. This Hadoop framework is further extended byYSmart and RESQUE. A typical database needs an index inorder to achieve fast query processing. The R-Tree basedindices group nearby entities and represent them using theirminimum bounding rectangle. The R-Tree can be built inthree-phases, in the first phase; each of close input is dividedaccording to the size properties. In the second phase, hadoopproduces lower level R-Trees processes out of the divideddata. And in the last phase, the lower level R-Trees arecombined to form complete R-Tree index of the data.Cary et al. (2009) showed how MapReduce model solvedfollowing two typical and representative spatial dataprocessing problems fast and efficiently: The large scale construction of R-Trees, a popularindexing mechanism for spatial search queryprocessing, and Processing digital aerial imagery and computationand storing image quality characteristics asmetadata.The key contribution of this work was presentation of atechnique for large scale building R-Trees based on theMapReduce model, and also to show how MapReduce can beapplied to big parallel processing of raster data. They alsoevaluated the algorithms performance using differentmatrices.The algorithm had following three phases(1) Computation of partitioning function (f): It is afunction calculated quantity using the inputs for thisphase which are the data set and a positive number R,which represents the number of partitions.(2) R-Tree construction: The partitioning function (f)calculated in the first phase is used by Mappers to dividethe data set into R partitions. Also build R-Tree indicesin each of the input partitions.(3) R-Tree consolidation: This phase combines the Rindividual R-Trees, built in the second phase, under asingle root node to form the final R-Tree index of thedata set.For supporting spatial queries on Hadoop, many newtechniques have come to the surface. But, most of themrequire internal modifications of frameworks. Lee et al. (2014)have come up with a spatial index especially for big data,based on a hierarchical spatial data structure stored indistributed file storage systems. This spatial index has severaladvantages; such as: it can be implemented without changingthe internal implementation of the existing storage systems,simple and efficient filtering, and supports to updates ofspatial objects.Fig. 6: Architecture of Hadoop Distributed File System(HDFS) (Source: http://hadoop.apache.org)Hadoop can be used for many applications involving largevolume of Geospatial as well as Spatio-temporal dataanalysis, Biomedical Imagery analysis, and even forsimulation of various physical, chemical and computationallyintensive application biological processes. However, in thispaper, we emphasize the spatial usage of hadoop framework.For this spatial indexing, a hierarchical spatial data structure,called geo-hash is used. It is a geo-coding system of latitudesand longitudes. The longer the geo-hash code the smaller thebounding. There are two categories of the existing spatialquery processing techniques, viz: In the first, small representatives of spatial objects areused like the k-Nearest Neighbour (kNN) selectionqueries.31

International Journal of Computer Applications (0975 – 8887)Volume 135 – No.11, February 2016 In the second, the whole data set is used for the queryingpurpose. This type of query called low selectivity query.meters and less) images. The Very High Resolution (VHR)images allow recognition of structures in the images.The other basis queries for mainly spatial data are:More examples applications are: Containing: This query returns all spatial objectscontaining the given search geometry. ContainedIn: This query returns all spatial objectscontained by the given search geometry Intersects: This query returns all spatial objects thatintersecting a given search geometry. WithinDistance: This query (or also called range query)returns all spatial objects within a given distance from agiven search geometry.Above tasks require feature extraction and selection, indexing,machine learning, pattern matching etc. Most of these tasksoften deal with segments and objects instead of pixels.Computation parameter of similarity between image patches(Housdorff distance) is computationally expensive. The largescaling of these algorithms (to global applications) requiresefficient and novel algorithmic solutions, and also requireexascale computing infrastructure to support large scale toglobal spatiotemporal applications. In such situations, thedistributed and parallel computing techniques, such as Apachehadoop framework, CUDA computing architecture and MPIbased architecture etc., come handy.The Spatiotemporal data is one type of the spatial data thatalso can have a very big size and is a problem while analyzingthe data. Also, the very large volume of the spatiotemporaldata generated by different social media networks is veryimportant and useful in various domains like the commercialdomain or more importantly in the disaster mapping ornational security applications. Also much of the huge amountof data that Google generates is spatiotemporal data. Toprocess such type of large volumes of data, there is a need forresearch in developing efficient management techniques andanalytical infrastructure for such massive to big spatial data.One of the main requirements of any data mining algorithm isthat it should take into account the spatial and temporalautocorrelations existing, if any. The explicit modeling ofspatial dependencies increases computational complexity. TheData mining primitives that explicitly model spatialdependencies are (Vatsavai et al., 2012): Spatial Autoregressive (SAR) model in which for theprediction, often spatial dependencies are modeled usingregression technique, Markov Random Field (MRF) model in which thespatial dependencies in classification are often modeledas the extension of a priori probabilities in a Bayesianclassification framework, and Gaussian Processes Learning and Mixture Models formodeling spatial heterogeneity in classification of largegeographic regions. Used also in change detectionstudies.There are many examples of applications which deal with biggeospatial data. For example, the application on BiomassMonitoring requires high temporal resolution satellite remotesensing imagery. The MODIS instrument on board NASA’sTerra satellite is providing a opportunity for continuousmonitoring of biomass over large geographic regions. Sincethe data at global scale is difficult to handle due to its largevolume and formats complexity, the data from the MODISsensor is organized into tiles of 10o 10olatitudes andlongitudes (4800 x 4800 MODIS pixels). Also, thecomputational complexity of change detection algorithms isvery high due to varying atmospheric contributions, varyingsensor parameters, sensors look angles and scan angles, inaddition, season dependent solar illumination levels.The second example, in the above category, can be searchingfor complex patterns in geospatial data. Most of the patternrecognition as well as the machine learning algorithms areper-pixel. Such methods work well for thematic classificationof moderate to high to very high resolution (pixel size of 5 Recognizing complex spatial patterns in an urban areasto map informal and formal settlements,Recognizing critical infrastructure establishments (suchas nuclear, thermal, and chemical plants, airports,shopping and sports complexes), andImage based content search and retrieval on applicationsof classification and clustering techniques.There are many data mining packages are in vogue and quitepopular among the data miners. To name some most popularpackages are Open Office, WEKA, KNIMIE, andRapidMiner. The Rapid Miner has a clear and user friendlyinterface that makes it easy to learn. It enables to design thedata mining work flow visually and investigate results whileexecution. Though, the above packages are highly rich infunctionalities and visualization tools, but they havelimitations in terms of size of data being handled andefficiency of large scale data processing.6. RADOOPPrekopcsak etal.(2011) added Hadoop extension toRapidMiner, calling the extended version as Radoop (fig. 7).The extension was designed to achieve a close integration ofhadoop and rapid miner and provide hadoop functionalitiescommonly used in memory-based RapidMiner processes.Also it keeps the convenient RapidMiner features likemetadata transformations and breakpoints.The user interface is one important part of any platform orframework, and it should be a user friendly and make theinteraction of the user with the platform easier and faster. Inthe case of the Hadoop revolution many distributed analyticsystems have been developed that are strong and faulttolerance and support many other features. But these systemsare usually very hard to use and interact with them.Radoop creates a process by adding the RadoopNest metaoperator containing general settings for the cluster (such asthe IP address of the Hadoop master node), and all otherRadoop operators are used inside this meta-operator.Prekopcsak et al. (2011) describe data handling and theseveral possibilities of uploading the data to a cluster forprocessing as well as the data preprocessing and modelingoperators of Radoop.For the data handling in RapidMiner, the data tables areobjects of ExampleSet and normally stored in memory. InRadoop, the data tables are stored in Apache Hive and useHadoopExampleSet object to describe the data stored. TheHadoopExampleSet not only stores several pointers and32

International Journal of Computer Applications (0975 – 8887)Volume 135 – No.11, February 2016settings, but all data is stored in Hive in the distributed filesystem resulting in no significant memory consumptionduring Radoop based data processing. Also, the Reader andWriter operators are implemented to enable transfer of largedata files from right inside RapidMiner. These operators workwith parsing every row in the dataset which has an overhead.So the big CSV (Comma Separated Variable) formatted filescan be uploaded to HDFS and also loaded to Hive using thecommand line interface. The Store and Retrieve are extremelypowerful operators to write and read back intermediateresults.3.Complex spatial queries, such as spatial cross-matchingor overlay (large scale spatial join) and nearest neighborqueries,4.Integrated spatial and feature queries, for example,feature aggregation queries in a selected spatial regions,and5.Global spatial pattern queries, include queries onfinding high density regions, or queries to find directionalpatterns of spatial objects.Traditional Methods for Spatial Queries have the followinglimitations:1.Managing and querying spatial data on a massive scale,2.Reducing the I/O bottleneck by partitioning of data onmultiple parallel SDBMSs disks,3.Optimzing for computationally intensive operations suchas geometric computations,4.Locking of effective spatial partitioning mechanism(s)to balance data loads and task loads across databasepartitions, and5.High data loading overheads.7.1 ResqueFig. 7: Radoop Architecture (Prekopcsak et al. 2011)Radoop has a built in excellent powerful mechanism for thedata transformations using views in Hive Query Language(HiveQL). It was expensive data writing in the results table tothe distributed file system only when needed. Radoopsupports many data transformations, such as selectingattributes, generating new attributes, filtering examples,sorting, renaming, t

no unique set of data mining algorithms that can be used in all application domains. But we can apply different types of the data mining algorithms as an integrated architecture or hybrid models to data sets to increase the robustness of the mining system. GeoMiner, a spatial data mining system prototype was developed on the top of the DBMiner .