Black Blocks

Kubernetes, meet OpenStack Cinder

Adam Litke - alitke@redhat.com
Principle Software Engineer - Red Hat
FOSDEM 2018 - 04 February 2018

Why Persistent Storage?

  • Containers are ephemeral
  • Start with state (ie. data analytics)
  • Share data between pods
  • Keep results after termination (ie. database)

Storage in kubernetes

  • Operator pattern
  • Self service
  • Modular and extensible

Kubernetes Volume Types

  • Cloud Provider: based on the hosting cloud
    • AWS, Azure, GCE, and Cinder
  • Software Defined Storage:
    • Ceph, Gluster, ScaleIO, StorageOS
  • Raw: Direct references to existing storage
    • ISCSI, FibreChannel, NFS
  • More via FlexVolume and CSI

Storage Classes

  • Provide a way to classify available storage
    • QoS: Fast SSDs Vs. Slow HDDs
    • Volume Type: Cloud Vs. On-Prem
    • Any arbitrary policy
  • Reclaim Policy: How to handle discarded volumes
  • Provisioner: How are volumes created / deleted
  • Parameters: Describe volume properties or behavior

Volume Objects

  • Storage is independent of Pod lifecycle
  • Persistent Volumes (PVs)
    • Global object representing a unit of storage
  • Persistent Volume Claims (PVCs)
    • Namespaced object used to request storage
  • A request is granted by binding a PVC to a PV

Dynamic provisioning

  • Self-service volume creation and deletion
  • Configured per Storage Class
  • Dynamic provisioner
    • Watches PVCs in a Storage Class
    • Provisions a volume of the requested size
    • New PV is bound to the PVC
    • On PVC deletion PV handled per reclaim policy

Trends and Requests

  • Hybrid storage environments
  • File-mode and Block-mode
  • Snapshots, cloning, resize
  • QoS and metrics
  • Vendor-optimized drivers

Cinder

  • Mature and stable API
  • Create, delete, attach, detach, snapshot, clone
  • Migration, backup, QoS, consistency groups, ...
  • 70+ vendor drivers and CI infrastructure
  • Active community of 500+ contributors

Block Storage as a Service

  1. Deploy cinder
  2. Create storage class
  3. Deploy cinder provisioner

Block Storage as a Service

Provision Volume

  1. User creates PVC
  2. Provisioner creates cinder volume
  3. Provisioner connects cinder volume
  4. Provisioner creates PV object
  5. Kubelet binds PVC to PV

User: create PVC


                            kind: PersistentVolumeClaim
                            apiVersion: v1
                            metadata:
                              name: test-claim
                            spec:
                              storageClassName: standalone-cinder
                              accessModes:
                                - ReadWriteOnce
                              resources:
                                requests:
                                  storage: 1Gi

Provisioner: create volume


                            options := volumes_v2.CreateOpts{
                                Name:             name,
                                Size:             sizeGB,
                                VolumeType:       volType,
                                AvailabilityZone: availability,
                                SourceVolID:      sourceVolID,
                            }
                            vol, err := volumes_v2.Create(vs, &options).Extract()

Provisioner: Get connection


                            var conn volumeConnection
                            err := volumeactions.InitializeConnection(vs, volumeID, &opt).ExtractInto(&conn)

                            struct {                                       
                                VolumeID: "ff7b41e1-09ab-4502-b672-e490378183e2",
                                Name: "cinder-dynamic-pvc-3f025a35-ffe4-11e7-a6aa-525400e1c724",
                                AuthMethod: "CHAP",
                                AuthUsername: "jhd9NXVedRvR5M5r8ybC",
                                AuthPassword: "3H7eXNWDzPzqPBdS",                                
                                TargetPortal: "192.168.122.221:3260",
                                TargetIqn: "iqn.2010-10.org.openstack:volume-ff7b41e1-09ab-4502-b672-e490378183e2",
                                TargetLun: 0
                            }

Provisioner: create PV


                                kind: PersistentVolume
                                metadata:
                                  annotations:
                                    cinderVolumeId: ff7b41e1-09ab-4502-b672-e490378183e2
                                    pv.kubernetes.io/provisioned-by: openstack.org/standalone-cinder
                                  name: pvc-3efe2d9c-ffe4-11e7-9d5b-525400e1c724
                                spec:
                                  accessModes:
                                  - ReadWriteOnce
                                  capacity:
                                    storage: 1Gi
                                  claimRef:
                                    kind: PersistentVolumeClaim
                                    name: test-claim
                                    namespace: default
                                  iscsi:
                                    chapAuthSession: true
                                    iqn: iqn.2010-10.org.openstack:volume-ff7b41e1-09ab-4502-b672-e490378183e2
                                    iscsiInterface: default
                                    lun: 0
                                    secretRef:
                                      name: pvc-3efe2d9c-ffe4-11e7-9d5b-525400e1c724-secret
                                    targetPortal: 192.168.122.221:3260
                                  persistentVolumeReclaimPolicy: Delete
                                  storageClassName: standalone-cinder
                                status:
                                  phase: Bound

Consume Volume

  1. User creates Pod
  2. Kubelet attaches and mounts volume
  3. Kubelet starts pod
  4. User deletes Pod
  5. Kubelet unmounts and detaches volume

Note that cinder is not involved

User: Create Pod


                                kind: Pod
                                metadata:
                                  name: demo
                                spec:
                                  containers:
                                  - name: demo
                                    image: alpine:latest
                                    volumeMounts:
                                    - name: data
                                      mountPath: /storage
                                  volumes:
                                  - name: data
                                    persistentVolumeClaim:
                                      claimName: test-claim

Kubelet attaches volume


                            $ sudo iscsiadm -m session
                            tcp: [3] 192.168.122.221:3260,1 iqn.2010-10.org.openstack:\
                                volume-c1aa5a52-8aef-4e2f-8dfa-5edea788568a (non-flash)

                            $ mount | grep kubelet/pods
                            /dev/sda on /var/lib/kubelet/pods/0673fee0-ffc4-11e7-9d5b-525400e1c724/ \
                                volumes/kubernetes.io~iscsi/pvc-1c8961d3-ffc3-11e7-9d5b-525400e1c724 \
                                type ext4 (rw,relatime,seclabel,stripe=16,data=ordered)

Delete Volume

  1. User deletes PVC
  2. Provisioner deletes cinder volume
  3. Kubelet removes PV

Live demo!

Advantages

  • Dynamic provisioning
  • Vendor abstraction
  • Efficient storage operations
  • Snapshots and cloning

TBD

  • Cinder deployment and reconfiguration
  • Should vendors focus on cinder or native kubernetes? Both?
  • CSI: The next generation kubernetes storage API

Future Work

  • Fibre Channel support
  • Provision existing cinder volumes
  • Clone existing cinder volumes

Join Us!