Hadoop: The Definitive Guide - Grut Computing
on editi datEd ph &U4t s e dReviHadoop: The Definitive GuideUsing Hadoop 2 exclusively, author Tom White presents new chapterson YARN and several Hadoop-related projects such as Parquet, Flume,Crunch, and Spark. You’ll learn about recent changes to Hadoop, andexplore new case studies on Hadoop’s role in healthcare systems andgenomics data processing. Learn fundamental components such as MapReduce, HDFS,and YARN Explore MapReduce in depth, including steps for developingapplications with it Set up and maintain a Hadoop cluster running HDFS andMapReduce on YARN Learn two data formats: Avro for data serialization and Parquetfor nested data Use data ingestion tools such as Flume (for streaming data) andSqoop (for bulk data transfer) Understand how high-level data processing tools like Pig, Hive,Crunch, and Spark work with Hadoop Learn the HBase distributed database and the ZooKeeperdistributed configuration serviceyou have the “Now opportunityto learnabout Hadoop from amaster—not only of thetechnology, but also of common sense andplain talk.”—Doug CuttingClouderaTom White, an engineer at Cloudera and member of the Apache SoftwareFoundation, has been an Apache Hadoop committer since 2007. He has writtennumerous articles for oreilly.com, java.net, and IBM’s developerWorks, and speaksregularly about Hadoop at industry conferences.US 49.99Twitter: @oreillymediafacebook.com/oreillyWhitePROGR AMMING L ANGUAGES/HADOOPFOURTH EDITIONHadoop:The Definitive GuideGet ready to unlock the power of your data. With the fourth edition ofthis comprehensive guide, you’ll learn how to build and maintain reliable,scalable, distributed systems with Apache Hadoop. This book is ideal forprogrammers looking to analyze datasets of any size, and for administratorswho want to set up and run Hadoop clusters.HadoopThe Definitive GuideSTORAGE AND ANALYSIS AT INTERNET SCALECAN 57.99ISBN: 978-1-491-90163-2Tom White
on editi datEd ph &U4t s e dReviHadoop: The Definitive GuideUsing Hadoop 2 exclusively, author Tom White presents new chapterson YARN and several Hadoop-related projects such as Parquet, Flume,Crunch, and Spark. You’ll learn about recent changes to Hadoop, andexplore new case studies on Hadoop’s role in healthcare systems andgenomics data processing. Learn fundamental components such as MapReduce, HDFS,and YARN Explore MapReduce in depth, including steps for developingapplications with it Set up and maintain a Hadoop cluster running HDFS andMapReduce on YARN Learn two data formats: Avro for data serialization and Parquetfor nested data Use data ingestion tools such as Flume (for streaming data) andSqoop (for bulk data transfer) Understand how high-level data processing tools like Pig, Hive,Crunch, and Spark work with Hadoop Learn the HBase distributed database and the ZooKeeperdistributed configuration serviceyou have the “Now opportunityto learnabout Hadoop from amaster—not only of thetechnology, but also of common sense andplain talk.”—Doug CuttingClouderaTom White, an engineer at Cloudera and member of the Apache SoftwareFoundation, has been an Apache Hadoop committer since 2007. He has writtennumerous articles for oreilly.com, java.net, and IBM’s developerWorks, and speaksregularly about Hadoop at industry conferences.US 49.99Twitter: @oreillymediafacebook.com/oreillyWhitePROGR AMMING L ANGUAGES/HADOOPFOURTH EDITIONHadoop:The Definitive GuideGet ready to unlock the power of your data. With the fourth edition ofthis comprehensive guide, you’ll learn how to build and maintain reliable,scalable, distributed systems with Apache Hadoop. This book is ideal forprogrammers looking to analyze datasets of any size, and for administratorswho want to set up and run Hadoop clusters.HadoopThe Definitive GuideSTORAGE AND ANALYSIS AT INTERNET SCALECAN 57.99ISBN: 978-1-491-90163-2Tom White
FOURTH EDITIONHadoop: The Definitive GuideTom White
Hadoop: The Definitive Guide, Fourth Editionby Tom WhiteCopyright 2015 Tom White. All rights reserved.Printed in the United States of America.Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions arealso available for most titles (http://safaribooksonline.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.Editors: Mike Loukides and Meghan BlanchetteProduction Editor: Matthew HackerCopyeditor: Jasmine KwitynProofreader: Rachel HeadJune 2009:First EditionOctober 2010:Second EditionMay 2012:Third EditionApril 2015:Fourth EditionIndexer: Lucie HaskinsCover Designer: Ellie VolckhausenInterior Designer: David FutatoIllustrator: Rebecca DemarestRevision History for the Fourth Edition:2015-03-19:First release2015-04-17:Second releaseSee http://oreilly.com/catalog/errata.csp?isbn 9781491901632 for release details.The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Hadoop: The Definitive Guide, the coverimage of an African elephant, and related trade dress are trademarks of O’Reilly Media, Inc.Many of the designations used by manufacturers and sellers to distinguish their products are claimed astrademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademarkclaim, the designations have been printed in caps or initial caps.While the publisher and the author have used good faith efforts to ensure that the information and instruc‐tions contained in this work are accurate, the publisher and the author disclaim all responsibility for errorsor omissions, including without limitation responsibility for damages resulting from the use of or relianceon this work. Use of the information and instructions contained in this work is at your own risk. If any codesamples or other technology this work contains or describes is subject to open source licenses or the intel‐lectual property rights of others, it is your responsibility to ensure that your use thereof complies with suchlicenses and/or rights.ISBN: 978-1-491-90163-2[LSI]
For Eliane, Emilia, and Lottie
Table of ContentsForeword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiPreface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xixPart I.Hadoop Fundamentals1. Meet Hadoop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Data!Data Storage and AnalysisQuerying All Your DataBeyond BatchComparison with Other SystemsRelational Database Management SystemsGrid ComputingVolunteer ComputingA Brief History of Apache HadoopWhat’s in This Book?356688101112152. MapReduce. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19A Weather DatasetData FormatAnalyzing the Data with Unix ToolsAnalyzing the Data with HadoopMap and ReduceJava MapReduceScaling OutData FlowCombiner FunctionsRunning a Distributed MapReduce JobHadoop Streaming1919212222243030343737v
RubyPython37403. The Hadoop Distributed Filesystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43The Design of HDFSHDFS ConceptsBlocksNamenodes and DatanodesBlock CachingHDFS FederationHDFS High AvailabilityThe Command-Line InterfaceBasic Filesystem OperationsHadoop FilesystemsInterfacesThe Java InterfaceReading Data from a Hadoop URLReading Data Using the FileSystem APIWriting DataDirectoriesQuerying the FilesystemDeleting DataData FlowAnatomy of a File ReadAnatomy of a File WriteCoherency ModelParallel Copying with distcpKeeping an HDFS Cluster 7476774. YARN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79Anatomy of a YARN Application RunResource RequestsApplication LifespanBuilding YARN ApplicationsYARN Compared to MapReduce 1Scheduling in YARNScheduler OptionsCapacity Scheduler ConfigurationFair Scheduler ConfigurationDelay SchedulingDominant Resource FairnessFurther Readingvi Table of Contents808182828385868890949596
5. Hadoop I/O. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97Data IntegrityData Integrity in decsCompression and Input SplitsUsing Compression in MapReduceSerializationThe Writable InterfaceWritable ClassesImplementing a Custom WritableSerialization FrameworksFile-Based Data StructuresSequenceFileMapFileOther File Formats and Column-Oriented FormatsPart MapReduce6. Developing a MapReduce Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141The Configuration APICombining ResourcesVariable ExpansionSetting Up the Development EnvironmentManaging ConfigurationGenericOptionsParser, Tool, and ToolRunnerWriting a Unit Test with MRUnitMapperReducerRunning Locally on Test DataRunning a Job in a Local Job RunnerTesting the DriverRunning on a ClusterPackaging a JobLaunching a JobThe MapReduce Web UIRetrieving the ResultsDebugging a JobHadoop 65167168172Table of Contents vii
Remote DebuggingTuning a JobProfiling TasksMapReduce WorkflowsDecomposing a Problem into MapReduce JobsJobControlApache Oozie1741751751771771781797. How MapReduce Works. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185Anatomy of a MapReduce Job RunJob SubmissionJob InitializationTask AssignmentTask ExecutionProgress and Status UpdatesJob CompletionFailuresTask FailureApplication Master FailureNode Manager FailureResource Manager FailureShuffle and SortThe Map SideThe Reduce SideConfiguration TuningTask ExecutionThe Task Execution EnvironmentSpeculative ExecutionOutput 971982012032032042068. MapReduce Types and Formats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209MapReduce TypesThe Default MapReduce JobInput FormatsInput Splits and RecordsText InputBinary InputMultiple InputsDatabase Input (and Output)Output FormatsText OutputBinary Outputviii Table of Contents209214220220232236237238238239239
Multiple OutputsLazy OutputDatabase Output2402452459. MapReduce Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247CountersBuilt-in CountersUser-Defined Java CountersUser-Defined Streaming CountersSortingPreparationPartial SortTotal SortSecondary SortJoinsMap-Side JoinsReduce-Side JoinsSide Data DistributionUsing the Job ConfigurationDistributed CacheMapReduce Library ClassesPart 79Hadoop Operations10. Setting Up a Hadoop Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283Cluster SpecificationCluster SizingNetwork TopologyCluster Setup and InstallationInstalling JavaCreating Unix User AccountsInstalling HadoopConfiguring SSHConfiguring HadoopFormatting the HDFS FilesystemStarting and Stopping the DaemonsCreating User DirectoriesHadoop ConfigurationConfiguration ManagementEnvironment SettingsImportant Hadoop Daemon 93294296Table of Contents ix
Hadoop Daemon Addresses and PortsOther Hadoop PropertiesSecurityKerberos and HadoopDelegation TokensOther Security EnhancementsBenchmarking a Hadoop ClusterHadoop BenchmarksUser Jobs30430730930931231331431431611. Administering Hadoop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317HDFSPersistent Data StructuresSafe ModeAudit LoggingToolsMonitoringLoggingMetrics and JMXMaintenanceRoutine Administration ProceduresCommissioning and Decommissioning NodesUpgradesPart IV.317317322324325330330331332332334337Related Projects12. Avro. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345Avro Data Types and SchemasIn-Memory Serialization and DeserializationThe Specific APIAvro DatafilesInteroperabilityPython APIAvro ToolsSchema ResolutionSort OrderAvro MapReduceSorting Using Avro MapReduceAvro in Other Languagesx Table of Contents346349351352354354355355358359363365
13. Parquet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367Data ModelNested EncodingParquet File FormatParquet ConfigurationWriting and Reading Parquet FilesAvro, Protocol Buffers, and ThriftParquet MapReduce36837037037237337537714. Flume. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381Installing FlumeAn ExampleTransactions and ReliabilityBatchingThe HDFS SinkPartitioning and InterceptorsFile FormatsFan OutDelivery GuaranteesReplicating and Multiplexing SelectorsDistribution: Agent TiersDelivery GuaranteesSink GroupsIntegrating Flume with ApplicationsComponent CatalogFurther 9940015. Sqoop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401Getting SqoopSqoop ConnectorsA Sample ImportText and Binary File FormatsGenerated CodeAdditional Serialization SystemsImports: A Deeper LookControlling the ImportImports and ConsistencyIncremental ImportsDirect-Mode ImportsWorking with Imported DataImported Data and HiveImporting Large able of Contents xi
Performing an ExportExports: A Deeper LookExports and TransactionalityExports and SequenceFilesFurther Reading41741942042142216. Pig. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423Installing and Running PigExecution TypesRunning Pig ProgramsGruntPig Latin EditorsAn ExampleGenerating ExamplesComparison with DatabasesPig ctionsMacrosUser-Defined FunctionsA Filter UDFAn Eval UDFA Load UDFData Processing OperatorsLoading and Storing DataFiltering DataGrouping and Joining DataSorting DataCombining and Splitting DataPig in PracticeParallelismAnonymous RelationsParameter SubstitutionFurther 17. Hive. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471Installing HiveThe Hive Shellxii Table of Contents472473
An ExampleRunning HiveConfiguring HiveHive ServicesThe MetastoreComparison with Traditional DatabasesSchema on Read Versus Schema on WriteUpdates, Transactions, and IndexesSQL-on-Hadoop AlternativesHiveQLData TypesOperators and FunctionsTablesManaged Tables and External TablesPartitions and BucketsStorage FormatsImporting DataAltering TablesDropping TablesQuerying DataSorting and AggregatingMapReduce ScriptsJoinsSubqueriesViewsUser-Defined FunctionsWriting a UDFWriting a UDAFFurther 9149650050250250350350350550850951051151351818. Crunch. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519An ExampleThe Core Crunch APIPrimitive OperationsTypesSources and TargetsFunctionsMaterializationPipeline ExecutionRunning a PipelineStopping a PipelineInspecting a Crunch Plan520523523528531533535538538539540Table of Contents xiii
Iterative AlgorithmsCheckpointing a PipelineCrunch LibrariesFurther Reading54354554554819. Spark. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549Installing SparkAn ExampleSpark Applications, Jobs, Stages, and TasksA Scala Standalone ApplicationA Java ExampleA Python ExampleResilient Distributed DatasetsCreationTransformations and ActionsPersistenceSerializationShared VariablesBroadcast VariablesAccumulatorsAnatomy of a Spark Job RunJob SubmissionDAG ConstructionTask SchedulingTask ExecutionExecutors and Cluster ManagersSpark on YARNFurther 6556556656957057057157420. HBase. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575HBasicsBackdropConceptsWhirlwind Tour of the Data ModelImplementationInstallationTest DriveClientsJavaMapReduceREST and ThriftBuilding an Online Query Applicationxiv Table of Contents575576576576578581582584584587589589
Schema DesignLoading DataOnline QueriesHBase Versus RDBMSSuccessful ServiceHBasePraxisHDFSUIMetricsCountersFurther Reading59059159459759859960060060160160160121. ZooKeeper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603Installing and Running ZooKeeperAn ExampleGroup Membership in ZooKeeperCreating the GroupJoining a GroupListing Members in a GroupDeleting a GroupThe ZooKeeper ServiceData atesBuilding Applications with ZooKeeperA Configuration ServiceThe Resilient ZooKeeper ApplicationA Lock ServiceMore Distributed Data Structures and ProtocolsZooKeeper in ProductionResilience and PerformanceConfigurationFurther 27627630634636637637639640Table of Contents xv
Part V.Case Studies22. Composable Data at Cerner. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643From CPUs to Semantic IntegrationEnter Apache CrunchBuilding a Complete PictureIntegrating Healthcare DataComposability over FrameworksMoving Forward64364464464765065123. Biological Data Science: Saving Lives with Software. . . . . . . . . . . . . . . . . . . . . . . . . . . . 653The Structure of DNAThe Genetic Code: Turning DNA Letters into ProteinsThinking of DNA as Source CodeThe Human Genome Project and Reference GenomesSequencing and Aligning DNAADAM, A Scalable Genome Analysis PlatformLiterate programming with the Avro interface description language (IDL)Column-oriented access with ParquetA simple example: k-mer counting using Spark and ADAMFrom Personalized Ads to Personalized MedicineJoin In65565665765966066166266366566766824. Cascading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669Fields, Tuples, and PipesOperationsTaps, Schemes, and FlowsCascading in PracticeFlexibilityHadoop and Cascading at ShareThisSummary670673675676679680684A. Installing Apache Hadoop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685B. Cloudera’s Distribution Including Apache Hadoop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 691C. Preparing the NCDC Weather Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 693D. The Old and New Java MapReduce APIs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 697Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 701xvi Table of Contents
ForewordHadoop got its start in Nutch. A few of us were attempting to build an open source websearch engine and having trouble managing computations running on even a handfulof computers.
Hadoop The Definitive Guide STORAGE AND ANALYSIS AT INTERNET SCALE n d. PROGRAMMING LANGUAGES/HADOOP Hadoop: The Definitive Guide ISBN: 978-1-491-90163-2 US 49.99 CAN 57.99
1: hadoop 2 2 Apache Hadoop? 2 Apache Hadoop : 2: 2 2 Examples 3 Linux 3 Hadoop ubuntu 5 Hadoop: 5: 6 SSH: 6 hadoop sudoer: 8 IPv6: 8 Hadoop: 8 Hadoop HDFS 9 2: MapReduce 13 13 13 Examples 13 ( Java Python) 13 3: Hadoop 17 Examples 17 hoods hadoop 17 hadoop fs -mkdir: 17: 17: 17 hadoop fs -put: 17: 17
May 02, 2018 · D. Program Evaluation ͟The organization has provided a description of the framework for how each program will be evaluated. The framework should include all the elements below: ͟The evaluation methods are cost-effective for the organization ͟Quantitative and qualitative data is being collected (at Basics tier, data collection must have begun)
Silat is a combative art of self-defense and survival rooted from Matay archipelago. It was traced at thé early of Langkasuka Kingdom (2nd century CE) till thé reign of Melaka (Malaysia) Sultanate era (13th century). Silat has now evolved to become part of social culture and tradition with thé appearance of a fine physical and spiritual .
On an exceptional basis, Member States may request UNESCO to provide thé candidates with access to thé platform so they can complète thé form by themselves. Thèse requests must be addressed to esd rize unesco. or by 15 A ril 2021 UNESCO will provide thé nomineewith accessto thé platform via their émail address.
̶The leading indicator of employee engagement is based on the quality of the relationship between employee and supervisor Empower your managers! ̶Help them understand the impact on the organization ̶Share important changes, plan options, tasks, and deadlines ̶Provide key messages and talking points ̶Prepare them to answer employee questions
Dr. Sunita Bharatwal** Dr. Pawan Garga*** Abstract Customer satisfaction is derived from thè functionalities and values, a product or Service can provide. The current study aims to segregate thè dimensions of ordine Service quality and gather insights on its impact on web shopping. The trends of purchases have
2006: Doug Cutting implements Hadoop 0.1. after reading above papers 2008: Yahoo! Uses Hadoop as it solves their search engine scalability issues 2010: Facebook, LinkedIn, eBay use Hadoop 2012: Hadoop 1.0 released 2013: Hadoop 2.2 („aka Hadoop 2.0") released 2017: Hadoop 3.0 released HADOOP TIMELINE Daimler TSS Data Warehouse / DHBW 12
Baldrige is based on a set of beliefs and behaviors. These core values and concepts are the foundation for integrating key performance and operational requirements within a results-oriented framework that creates a basis for action, feedback, and ongoing success: Systems perspective Visionary leadership Customer-focused excellence
