Ceph Overview

Overview

• What is CEPH?

• Ceph Architecture

• Ceph Componets

• Ceph Storage System

• Ceph Client

• Ceph Deployment

• Use Case

What is CEPH?

• Massively scalable storage system

• Reliable Autonomic Distributed Object Store (RADOS)

• Object, block, and file system storage in a single unified storage cluster

• Object-based Storage

Ceph Architecture

 

 

Ceph Componets

OSDs

Stores data, handles data replication, recovery, backfilling, rebalancing, provide information to Mons

Monitors

Maintains maps of the cluster state,monitor map, OSD map, PG map, CRUSH map

MDSs

Stores metadata on behalf of the Ceph Filesystem ( POSIX file system )

 

 

OSD Servers(Object Storage Device)

• Intelligent Storage Servers

• Serve stored objects to clients

• OSD is primary for some objects

  • Responsible for replication, re-balancing, recovery

• OSD is secondary for some objects

  • Under control of the primary

  • Capable of becoming primary

MON Servers

• Must be an odd number of monitors.

• Maintain the cluster map

  • MON Map

  • OSD Map

  • MDS Map

  • PG Map

  • CRUSH Map

MDS Servers (Metadate Servers)

• The Ceph Metadata Server daemon (MDS)

  • Provides the POSIX information needed by file systems that enables

    Ceph FS to interact with the Ceph Object Store

  • It remembers where data lives within a tree

    Clients accessing CephFS data first make a request to an MDS, which provides what they need to get files from the right OSDs

• If you aren’t running CephFS, MDS daemons do not need to be deployed

Calamari

Calamari is WebUI to monitor a Ceph cluster

 

Placement Groups (PGs)

• The cluster is split into sections

• Each section is called a “Placement Group” (PG).

• A PG contains a collection of objects

• A PG is replicated across a set of devices

• An object’s PG is determined by CRUSH

  • Hash the object name

  • Against the number of PGs configured

• The PG location is determined by CRUSH

  • According to the cluster state

  • According to the desired protection

  • According to the desired placement strategie

Pools

Pools are logical partitions
Pools Provide the following attributes:

• Ownership/access
• For each protection type respectively
• Number of placement groups
• CRUSH placement rule

PGs within a pool are dynamically mapped to OSDs
Two types of pools:

• Replicated (historical default)

• Erasure Coded (EC Pools)

Native Protocal (LIBRADOS)

Pool Operations
Snapshots and Copy-on-write Cloning
Read/Write Objects – Create or Remove
Create/Set/Get/Remove XATTRs (Key/Value Pairs)
Compound operations and dual-ack semantics
Object Classes

Ceph Storage Cluster

The foundation for all Ceph deployments

• Ceph Monitor

  maintains a master copy of the cluster map

• Ceph OSD Daemon

  stores data as objects on a storage node

Ceph Block Device

Ubiquity of block device interfaces makes a virtual block device an ideal candidate to interact

 

• kernel modules, KVMs such as Qemu, integrate with Ceph block devices

• Snapshots – create snapshots of the images to retain a history of an image’s state

• RBD Mirroring – asynchronously mirrored between two Ceph clusters

• Librbd

Ceph Rados Gateway

Object storage interface emulating both Amazon S3 and Openstack Swift.

 

• Accessible through a ReST-ful HTTP interface

• ReST APIs for Amazon S3 and OpenStack Swift protocols

• Supports Regions, Zones, Users, ACLs, Quotas etc.. similar to S3/Swift

• RGW nodes spanning multiple geographical locations (Federated Gateways)

Ceph Filesystem

POSIX-compliant filesystem that uses a Ceph Storage Cluster to store its data

 

• Ceph Filesystem requires at least one Ceph Metadata Server

• Client – Mount CephFS as ceph-fuse, mount.ceph

• Client has network connectivity and the proper authentication keyring.

Ceph Client

Block Device(RBD)

block devices with snapshotting and cloning kernel objects (KO) and a QEMU hypervisor that uses librbd

Object Storage(RGW)

RESTful APIs with interfaces that are compatible with Amazon S3 and OpenStack Swift

Filesystem(CephFS)

POSIX compliant filesystem usable with mount, filesytem in user space (FUSE).

Ceph Deployment

• Puppet

  maintained through the OpenStack community

• chef

  Installation via Chef cookbooks is possible

  Chef cookbooks initially developed by Inktank

• ansible

  Installation via Ansible is possible

• juju

  developed by Clint Byrum

• crowbar

  Ceph barclamps for Crowbar exist and are actively maintained(upstream repository)