Black Blocks

Kubernetes, meet OpenStack Cinder

Adam Litke -
Principle Software Engineer - Red Hat
FOSDEM 2018 - 04 February 2018

Why Persistent Storage?

  • Containers are ephemeral
  • Start with state (ie. data analytics)
  • Share data between pods
  • Keep results after termination (ie. database)

Storage in kubernetes

  • Operator pattern
  • Self service
  • Modular and extensible

Kubernetes Volume Types

  • Cloud Provider: based on the hosting cloud
    • AWS, Azure, GCE, and Cinder
  • Software Defined Storage:
    • Ceph, Gluster, ScaleIO, StorageOS
  • Raw: Direct references to existing storage
    • ISCSI, FibreChannel, NFS
  • More via FlexVolume and CSI

Storage Classes

  • Provide a way to classify available storage
    • QoS: Fast SSDs Vs. Slow HDDs
    • Volume Type: Cloud Vs. On-Prem
    • Any arbitrary policy
  • Reclaim Policy: How to handle discarded volumes
  • Provisioner: How are volumes created / deleted
  • Parameters: Describe volume properties or behavior

Volume Objects

  • Storage is independent of Pod lifecycle
  • Persistent Volumes (PVs)
    • Global object representing a unit of storage
  • Persistent Volume Claims (PVCs)
    • Namespaced object used to request storage
  • A request is granted by binding a PVC to a PV

Dynamic provisioning

  • Self-service volume creation and deletion
  • Configured per Storage Class
  • Dynamic provisioner
    • Watches PVCs in a Storage Class
    • Provisions a volume of the requested size
    • New PV is bound to the PVC
    • On PVC deletion PV handled per reclaim policy

Trends and Requests

  • Hybrid storage environments
  • File-mode and Block-mode
  • Snapshots, cloning, resize
  • QoS and metrics
  • Vendor-optimized drivers


  • Mature and stable API
  • Create, delete, attach, detach, snapshot, clone
  • Migration, backup, QoS, consistency groups, ...
  • 70+ vendor drivers and CI infrastructure
  • Active community of 500+ contributors

Block Storage as a Service

  1. Deploy cinder
  2. Create storage class
  3. Deploy cinder provisioner

Block Storage as a Service

Provision Volume

  1. User creates PVC
  2. Provisioner creates cinder volume
  3. Provisioner connects cinder volume
  4. Provisioner creates PV object
  5. Kubelet binds PVC to PV

User: create PVC

                            kind: PersistentVolumeClaim
                            apiVersion: v1
                              name: test-claim
                              storageClassName: standalone-cinder
                                - ReadWriteOnce
                                  storage: 1Gi

Provisioner: create volume

                            options := volumes_v2.CreateOpts{
                                Name:             name,
                                Size:             sizeGB,
                                VolumeType:       volType,
                                AvailabilityZone: availability,
                                SourceVolID:      sourceVolID,
                            vol, err := volumes_v2.Create(vs, &options).Extract()

Provisioner: Get connection

                            var conn volumeConnection
                            err := volumeactions.InitializeConnection(vs, volumeID, &opt).ExtractInto(&conn)

                            struct {                                       
                                VolumeID: "ff7b41e1-09ab-4502-b672-e490378183e2",
                                Name: "cinder-dynamic-pvc-3f025a35-ffe4-11e7-a6aa-525400e1c724",
                                AuthMethod: "CHAP",
                                AuthUsername: "jhd9NXVedRvR5M5r8ybC",
                                AuthPassword: "3H7eXNWDzPzqPBdS",                                
                                TargetPortal: "",
                                TargetIqn: "",
                                TargetLun: 0

Provisioner: create PV

                                kind: PersistentVolume
                                    cinderVolumeId: ff7b41e1-09ab-4502-b672-e490378183e2
                                  name: pvc-3efe2d9c-ffe4-11e7-9d5b-525400e1c724
                                  - ReadWriteOnce
                                    storage: 1Gi
                                    kind: PersistentVolumeClaim
                                    name: test-claim
                                    namespace: default
                                    chapAuthSession: true
                                    iscsiInterface: default
                                    lun: 0
                                      name: pvc-3efe2d9c-ffe4-11e7-9d5b-525400e1c724-secret
                                  persistentVolumeReclaimPolicy: Delete
                                  storageClassName: standalone-cinder
                                  phase: Bound

Consume Volume

  1. User creates Pod
  2. Kubelet attaches and mounts volume
  3. Kubelet starts pod
  4. User deletes Pod
  5. Kubelet unmounts and detaches volume

Note that cinder is not involved

User: Create Pod

                                kind: Pod
                                  name: demo
                                  - name: demo
                                    image: alpine:latest
                                    - name: data
                                      mountPath: /storage
                                  - name: data
                                      claimName: test-claim                                

Kubelet attaches volume

                            $ sudo iscsiadm -m session
                            tcp: [3],1\
                                volume-c1aa5a52-8aef-4e2f-8dfa-5edea788568a (non-flash)

                            $ mount | grep kubelet/pods
                            /dev/sda on /var/lib/kubelet/pods/0673fee0-ffc4-11e7-9d5b-525400e1c724/ \
                                volumes/ \
                                type ext4 (rw,relatime,seclabel,stripe=16,data=ordered)

Delete Volume

  1. User deletes PVC
  2. Provisioner deletes cinder volume
  3. Kubelet removes PV

Live demo!


  • Dynamic provisioning
  • Vendor abstraction
  • Efficient storage operations
  • Snapshots and cloning


  • Cinder deployment and reconfiguration
  • Should vendors focus on cinder or native kubernetes? Both?
  • CSI: The next generation kubernetes storage API

Future Work

  • Fibre Channel support
  • Provision existing cinder volumes
  • Clone existing cinder volumes

Join Us!