Red Hat Ceph Storage 4 Architecture Guide
Red Hat Ceph Storage 4Architecture GuideGuide on Red Hat Ceph Storage ArchitectureLast Updated: 2021-01-11
Red Hat Ceph Storage 4 Architecture GuideGuide on Red Hat Ceph Storage Architecture
Legal NoticeCopyright 2021 Red Hat, Inc.The text of and illustrations in this document are licensed by Red Hat under a Creative CommonsAttribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA isavailable athttp://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you mustprovide the URL for the original version.Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift,Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United Statesand other countries.Linux is the registered trademark of Linus Torvalds in the United States and other countries.Java is a registered trademark of Oracle and/or its affiliates.XFS is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United Statesand/or other countries.MySQL is a registered trademark of MySQL AB in the United States, the European Union andother countries.Node.js is an official trademark of Joyent. Red Hat is not formally related to or endorsed by theofficial Joyent Node.js open source or commercial project.The OpenStack Word Mark and OpenStack logo are either registered trademarks/service marksor trademarks/service marks of the OpenStack Foundation, in the United States and othercountries and are used with the OpenStack Foundation's permission. We are not affiliated with,endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.All other trademarks are the property of their respective owners.AbstractThis document provides architecture information for Ceph Storage Clusters and its clients.
Table of ContentsTable of Contents. . . . . . . . . . . 1. .THECHAPTER. . . . .CEPH. . . . . . ARCHITECTURE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3. . . . . . . . . . . . .CHAPTER. . . . . . . . . . 2. . THE. . . . . CORE. . . . . . .CEPH. . . . . . COMPONENTS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5. . . . . . . . . . . . .2.1. PREREQUISITES52.2. CEPH POOLS52.3. CEPH AUTHENTICATION62.4. CEPH PLACEMENT GROUPS72.5. CEPH CRUSH RULESET2.6. CEPH INPUT/OUTPUT OPERATIONS2.7. CEPH REPLICATION2.8. CEPH ERASURE CODING2.9. CEPH OBJECTSTORE2.10. CEPH BLUESTORE99101113132.11. CEPH SELF MANAGEMENT OPERATIONS2.12. CEPH HEARTBEAT2.13. CEPH PEERING1415152.14. CEPH REBALANCING AND RECOVERY2.15. CEPH DATA INTEGRITY15162.16. CEPH HIGH AVAILABILITY2.17. CLUSTERING THE CEPH MONITOR1616. . . . . . . . . . . 3.CHAPTER. . THE. . . . . CEPH. . . . . . .CLIENT. . . . . . . .COMPONENTS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18.3.1. PREREQUISITES3.2. CEPH CLIENT NATIVE PROTOCOL18183.3. CEPH CLIENT OBJECT WATCH AND NOTIFY3.4. CEPH CLIENT MANDATORY EXCLUSIVE LOCKS3.5. CEPH CLIENT OBJECT MAP1819203.6. CEPH CLIENT DATA STRIPPING21. . . . . . . . . . . 4.CHAPTER. . .CEPH. . . . . .ON-DISK. . . . . . . . . .ENCRYPTION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25. . . . . . . . . . . 5.CHAPTER. . CEPH. . . . . . .ON-WIRE. . . . . . . . . . ENCRYPTION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26.1
Red Hat Ceph Storage 4 Architecture Guide2
CHAPTER 1. THE CEPH ARCHITECTURECHAPTER 1. THE CEPH ARCHITECTURERed Hat Ceph Storage cluster is a distributed data object store designed to provide excellentperformance, reliability and scalability. Distributed object stores are the future of storage, because theyaccommodate unstructured data, and because clients can use modern object interfaces and legacyinterfaces simultaneously.For example:APIs in many languages (C/C , Java, Python)RESTful interfaces (S3/Swift)Block device interfaceFilesystem interfaceThe power of Red Hat Ceph Storage cluster can transform your organization’s IT infrastructure and yourability to manage vast amounts of data, especially for cloud computing platforms like RHEL OSP.Red Hat Ceph Storage cluster delivers extraordinary scalability–thousands of clients accessingpetabytes to exabytes of data and beyond.At the heart of every Ceph deployment is the Red Hat Ceph Storage cluster. It consists of three types ofdaemons:Ceph OSD Daemon: Ceph OSDs store data on behalf of Ceph clients. Additionally, Ceph OSDsutilize the CPU, memory and networking of Ceph nodes to perform data replication, erasurecoding, rebalancing, recovery, monitoring and reporting functions.Ceph Monitor: A Ceph Monitor maintains a master copy of the Red Hat Ceph Storage clustermap with the current state of the Red Hat Ceph Storage cluster. Monitors require highconsistency, and use Paxos to ensure agreement about the state of the Red Hat Ceph Storagecluster.Ceph Manager: The Ceph Manager maintains detailed information about placement groups,process metadata and host metadata in lieu of the Ceph Monitor— significantly improvingperformance at scale. The Ceph Manager handles execution of many of the read-only Ceph CLIqueries, such as placement group statistics. The Ceph Manager also provides the RESTfulmonitoring APIs.Ceph client interfaces read data from and write data to the Red Hat Ceph Storage cluster. Clients needthe following data to communicate with the Red Hat Ceph Storage cluster:The Ceph configuration file, or the cluster name (usually ceph) and the monitor addressThe pool nameThe user name and the path to the secret key.3
Red Hat Ceph Storage 4 Architecture GuideCeph clients maintain object IDs and the pool names where they store the objects. However, they do notneed to maintain an object-to-OSD index or communicate with a centralized object index to look upobject locations. To store and retrieve data, Ceph clients access a Ceph Monitor and retrieve the latestcopy of the Red Hat Ceph Storage cluster map. Then, Ceph clients provide an object name and poolname to librados, which computes an object’s placement group and the primary OSD for storing andretrieving data using the CRUSH (Controlled Replication Under Scalable Hashing) algorithm. The Cephclient connects to the primary OSD where it may perform read and write operations. There is nointermediary server, broker or bus between the client and the OSD.When an OSD stores data, it receives data from a Ceph client— whether the client is a Ceph Block Device,a Ceph Object Gateway, a Ceph Filesystem or another interface— and it stores the data as an object.NOTEAn object ID is unique across the entire cluster, not just an OSD’s storage media.Ceph OSDs store all data as objects in a flat namespace. There are no hierarchies of directories. Anobject has a cluster-wide unique identifier, binary data, and metadata consisting of a set of name/valuepairs.Ceph clients define the semantics for the client’s data format. For example, the Ceph block devicemaps a block device image to a series of objects stored across the cluster.NOTEObjects consisting of a unique ID, data, and name/value paired metadata can representboth structured and unstructured data, as well as legacy and leading edge data storageinterfaces.4
CHAPTER 2. THE CORE CEPH COMPONENTSCHAPTER 2. THE CORE CEPH COMPONENTSA Red Hat Ceph Storage cluster can have a large number of Ceph nodes for limitless scalability, highavailability and performance. Each node leverages non-proprietary hardware and intelligent Cephdaemons that communicate with each other to:Write and read dataCompress dataEnsure durability by replicating or erasure coding dataMonitor and report on cluster health— also called 'heartbeating'Redistribute data dynamically— also called 'backfilling'Ensure data integrity; and,Recover from failures.To the Ceph client interface that reads and writes data, a Red Hat Ceph Storage cluster looks like asimple pool where it stores data. However, librados and the storage cluster perform many complexoperations in a manner that is completely transparent to the client interface. Ceph clients and CephOSDs both use the CRUSH (Controlled Replication Under Scalable Hashing) algorithm. The followingsections provide details on how CRUSH enables Ceph to perform these operations seamlessly.2.1. PREREQUISITESA basic understanding of distributed storage systems.2.2. CEPH POOLSThe Ceph storage cluster stores data objects in logical partitions called 'Pools.' Ceph administrators cancreate pools for particular types of data, such as for block devices, object gateways, or simply just toseparate one group of users from another.From the perspective of a Ceph client, the storage cluster is very simple. When a Ceph client reads orwrites data using an I/O context, it always connects to a storage pool in the Ceph storage cluster. Theclient specifies the pool name, a user and a secret key, so the pool appears to act as a logical partitionwith access controls to its data objects.In actual fact, a Ceph pool is not only a logical partition for storing object data. A pool plays a critical rolein how the Ceph storage cluster distributes and stores data. However, these complex operations arecompletely transparent to the Ceph client.Ceph pools define:Pool Type: In early versions of Ceph, a pool simply maintained multiple deep copies of an object.Today, Ceph can maintain multiple copies of an object, or it can use erasure coding to ensuredurability. The data durability method is pool-wide, and does not change after creating the pool.The pool type defines the data durability method when creating the pool. Pool types arecompletely transparent to the client.Placement Groups: In an exabyte scale storage cluster, a Ceph pool might store millions of dataobjects or more. Ceph must handle many types of operations, including data durability viareplicas or erasure code chunks, data integrity by scrubbing or CRC checks, replication,5
Red Hat Ceph Storage 4 Architecture Guiderebalancing and recovery. Consequently, managing data on a per-object basis presents ascalability and performance bottleneck. Ceph addresses this bottleneck by sharding a pool intoplacement groups. The CRUSH algorithm computes the placement group for storing an objectand computes the Acting Set of OSDs for the placement group. CRUSH puts each object into aplacement group. Then, CRUSH stores each placement group in a set of OSDs. Systemadministrators set the placement group count when creating or modifying a pool.CRUSH Ruleset: CRUSH plays another important role: CRUSH can detect failure domains andperformance domains. CRUSH can identify OSDs by storage media type and organize OSDshierarchically into nodes, racks, and rows. CRUSH enables Ceph OSDs to store object copiesacross failure domains. For example, copies of an object may get stored in different serverrooms, aisles, racks and nodes. If a large part of a cluster fails, such as a rack, the cluster can stilloperate in a degraded state until the cluster recovers.Additionally, CRUSH enables clients to write data to particular types of hardware, such as SSDs, harddrives with SSD journals, or hard drives with journals on the same drive as the data. The CRUSH rulesetdetermines failure domains and performance domains for the pool. Administrators set the CRUSHruleset when creating a pool.NOTEAn administrator CANNOT change a pool’s ruleset after creating the pool.Durability: In exabyte scale storage clusters, hardware failure is an expectation and not anexception. When using data objects to represent larger-grained storage interfaces such as ablock device, losing one or more data objects for that larger-grained interface can compromisethe integrity of the larger-grained storage entity— potentially rendering it useless. So data loss isintolerable. Ceph provides high data durability in two ways:Replica pools will store multiple deep copies of an object using the CRUSH failure domain tophysically separate one data object copy from another. That is, copies get distributed toseparate physical hardware. This increases durability during hardware failures.Erasure coded pools store each object as K M chunks, where K represents data chunks andM represents coding chunks. The sum represents the number of OSDs used to store theobject and the the M value represents the number of OSDs that can fail and still restoredata should the M number of OSDs fail.From the client perspective, Ceph is elegant and simple. The client simply reads from and writes topools. However, pools play an important role in data durability, performance and high availability.2.3. CEPH AUTHENTICATIONTo identify users and protect against man-in-the-middle attacks, Ceph provides its cephxauthentication system, which authenticates users and daemons.NOTEThe cephx protocol does not address data encryption for data transported over thenetwork or data stored in OSDs.Cephx uses shared secret keys for authentication, meaning both the client and the monitor cluster havea copy of the client’s secret key. The authentication protocol enables both parties to prove to eachother that they have a copy of the key without actually revealing it. This provides mutual authentication,6
CHAPTER 2. THE CORE CEPH COMPONENTSwhich means the cluster is sure the user possesses the secret key, and the user is sure that the clusterhas a copy of the secret key.CephxThe cephx authentication protocol operates in a manner similar to Kerberos.A user/actor invokes a Ceph client to contact a monitor. Unlike Kerberos, each monitor can authenticateusers and distribute keys, so there is no single point of failure or bottleneck when using cephx. Themonitor returns an authentication data structure similar to a Kerberos ticket that contains a session keyfor use in obtaining Ceph services. This session key is itself encrypted with the user’s permanent secretkey, so that only the user can request services from the Ceph monitors. The client then uses the sessionkey to request its desired services from the monitor, and the monitor provides the client with a ticketthat will authenticate the client to the OSDs that actually handle data. Ceph monitors and OSDs share asecret, so the client can use the ticket provided by the monitor with any OSD or metadata server in thecluster. Like Kerberos, cephx tickets expire, so an attacker cannot use an expired ticket or session keyobtained surreptitiously. This form of authentication will prevent attackers with access to thecommunications medium from either creating bogus messages under another user’s identity or alteringanother user’s legitimate messages, as long as the user’s secret key is not divulged before it expires.To use cephx, an administrator must set up users first. In the following diagram, the client.admin userinvokes ceph auth get-or-create-key from the command line to generate a username and secret key.Ceph’s auth subsystem generates the username and key, stores a copy with the monitor(s) andtransmits the user’s secret back to the client.admin user. This means that the client and the monitorshare a secret key.NOTEThe client.admin user must provide the user ID and secret key to the user in a securemanner.2.4. CEPH PLACEMENT GROUPSStoring millions of objects in a cluster and managing them individually is resource intensive. So Cephuses placement groups (PGs) to make managing a huge number of objects more efficient.A PG is a subset of a pool that serves to contain a collection of objects. Ceph shards a pool into a seriesof PGs. Then, the CRUSH algorithm takes the cluster map and the status of the cluster into account anddistributes the PGs evenly and pseudo-randomly to OSDs in the cluster.Here is how it works.When a system administrator creates a pool, CRUSH creates a user-defined number of PGs for the7
Red Hat Ceph Storage 4 Architecture GuideWhen a system administrator creates a pool, CRUSH creates a user-defined number of PGs for thepool. Generally, the number of PGs should be a reasonably fine-grained subset of the data. For example,100 PGs per OSD per pool would mean that each PG contains approximately 1% of the pool’s data.The number of PGs has a performance impact when Ceph needs to move a PG from one OSD toanother OSD. If the pool has too few PGs, Ceph will move a large percentage of the data simultaneouslyand the network load will adversely impact the cluster’s performance. If the pool has too many PGs,Ceph will use too much CPU and RAM when moving tiny percentages of the data and thereby adverselyimpact the cluster’s performance. For details on calculating the number of PGs to achieve optimalperformance, see PG Count.Ceph ensures against data loss by storing replicas of an object or by storing erasure code chunks of anobject. Since Ceph stores objects or erasure code chunks of an object within PGs, Ceph replicates eachPG in a set of OSDs called the "Acting Set" for each copy of an object or each erasure code chunk of anobject. A system administrator can determine the number of PGs in a pool and the number of replicas orerasure code chunks. However, the CRUSH algorithm calculates which OSDs are in the acting set for aparticular PG.The CRUSH algorithm and PGs make Ceph dynamic. Changes in the cluster map or the cluster statemay result in Ceph moving PGs from one OSD to another automatically.Here are a few examples:Expanding the Cluster: When adding a new host and its OSDs to the cluster, the cluster mapchanges. Since CRUSH evenly and pseudo-randomly distributes PGs to OSDs throughout thecluster, adding a new host and its OSDs means that CRUSH will reassign some of the pool’splacement groups to those new OSDs. That means that system administrators do not have torebalance the cluster manually. Also, it means that the new OSDs contain approximately thesame amount of data as the other OSDs. This also means that new OSDs do not contain newlywritten OSDs, preventing "hot spots" in the cluster.An OSD Fails: When a OSD fails, the state of the cluster changes. Ceph temporarily loses oneof the replicas or erasure code chunks, and needs to make another copy. If the primary OSD inthe acting set fails, the next OSD in the acting set becomes the primary and CRUSH calculates anew OSD to store the additional copy or erasure code chunk.By managing millions of objects within the context of hundreds to thousands of PGs, the Ceph storagecluster can grow, shrink and recover from failure efficiently.For Ceph clients, the CRUSH algorithm via librados makes the process of reading and writing objectsvery simple. A Ceph client simply writes an object to a pool or reads an object from a pool. The primaryOSD in the acting set can write replicas of the object or erasure code chunks of the object to thesecondary OSDs in the acting set on behalf of the Ceph client.If the cluster map or cluster state changes, the CRUSH computation for which OSDs store the PG willchange too. For example, a Ceph client may write object foo to the pool bar. CRUSH will assign theobject to PG 1.a, and store it on OSD 5, which makes replicas on OSD 10 and OSD 15 respectively. IfOSD 5 fails, the cluster state changes. When the Ceph client reads object foo from pool bar, the clientvia librados will automatically retrieve it from OSD 10 as the new primary OSD dynamically.The
Ceph Manager: The Ceph Manager maintains detailed information about placement groups, process metadata and host metadata in lieu of the Ceph Monitor— s ignificantly improving performance at scale. The Ceph Manager handles execution of many of the read-only Ceph CLI queries, such as placement group statistics.
A Red Hat Ceph Storage cluster is the foundation for all Ceph deployments. After deploying a Red Hat Ceph Storage cluster, there are administrative operations for keeping a Red Hat Ceph Storage cluster healthy and performing optimally. The Red Hat Ceph Storage Administration Guide helps storage administrators to perform such tasks as:
Ceph Storage Cluster. Ceph's monitoring and self-repair features minimize administration overhead. You can configure a Ceph Storage Cluster on non-identical hardware from different manufacturers. Ceph Storage for Oracle Linux Release 3.0 is based on the Ceph Community Luminous release (v12.2.5).
Ceph and MySQL represent highly complementary technologies, providing: Strong synergies . MySQL, OpenStack, and Ceph are often chosen to work together. Ceph is the leading open source software-defined storage solution. MySQL is the leading open source rela-tional database management system (RDBMS). 1 Moreover, Ceph is the number-one block .
The Red Hat Ceph Storage environment makes use of industry standard servers that form Ceph nodes for scalability, fault-tolerance, and performance. Data protection methods play a vital role in deciding the total cost of ownership (TCO) of a solution. Ceph allows the user to set different data protection methods on different storage pools.
Ceph features, then packages changes into predictable, stable, enterprise-quality releases. Red Hat Ceph Storage 3.2 is based on the Ceph community ‘Luminous’ version 12.2.1, to which Red Hat was a leading code contributor. As a self-healing, self-managing, unified storage platform with no single point of
Red Hat Enterprise Linux 6 Security Guide A Guide to Securing Red Hat Enterprise Linux Mirek Jahoda Red Hat Customer Content Services mjahoda@redhat.com Robert Krátký Red Hat Customer Content Services Martin Prpič Red Hat Customer Content Services Tomáš Čapek Red Hat Customer Content Services Stephen Wadeley Red Hat Customer Content Services Yoana Ruseva Red Hat Customer Content Services .
As 20 melhores certificações e cursos do Red Hat Linux Red Hat Certified System Administrator (RHCSA) Engenheiro Certificado Red Hat (RHCE) Red Hat Certified Enterprise Application Developer Red Hat Certified Architect (RHCA) Engenheiro certificado pela Red Hat no Red Hat OpenStack. Administração do Red Hat Enterprise Linux (EL) Desenvolvedor de microsserviços corporativos com .
ASTM C 76 specification for reinforced sewer and storm drain pipes. (Only on special requests) inTernaTional & local approVals ISO 9001 registered firm. M.P.W Approval QualiTy assurance Concrete pipe factory has an independent quality control department with a well equipped laboratory, aided by experienced staff to give the necessary backup for the quality assurance program .
