Datapath operations

• A VM or host accessing volume contents
• These are the most common and most important
• Lots of IO
• Long running
• Narrow in scope

Example: VM volume access

Example: Host volume access

Metadata operations

• Adding / removing / rearranging storage objects
• Changing storage domain metadata
• Minimal IO
• Short running
• Can have broad scope

Example: create volume

Example: delete image

Preventing conflicts

• Requirement: data integrity
• Goal: maximize concurrency
• Interaction between storage objects is complex
• Orchestration required across several domains
  User actions
  Hosts
  Local threads

Same VM on multiple hosts

Conflicting metadata updates

Run VM during snapshot

Shared storage locking

• Implemented using Sanlock
• Lockspace is on shared storage
• Leases grant hosts exclusive access to storage resources
  • Storage domain lease: needed for metadata changes
  • Volume lease: protects volume contents

More about sanlock

• Host IDs
  • Every host has a unique ID
  • Uniqueness is enforced by SANlock
  • IDs must be periodically renewed
  • Failure to renew will surrender all resource leases
• Resource leases
  • Represent an arbitrary resource (storage or otherwise)
• Misbehaving hosts will be fenced (rebooted)

Handling interruptions

• Some steps in a task are never completed
• Happen naturally or due to bugs
  • Power or network outage
  • Hardware failure
  • Software failure
• Must be carefully mitigated to keep storage coherent
• Approaches
  • Storage task manager with rollback capability
  • Storage transactions with garbage collection

Interrupted volume creation

Interrupted volume copy

Storage transactions

• Storage commands must be a single transaction
• A transaction is opened with a marker operation
• Subsequent steps accumulate "garbage" on storage
• A transaction is committed by converting the start marker

Example

Garbage collection

• Runs periodically on an arbitrary host
• Identifies candidates by finding markers
• Acquires necessary locks for the candidate
• Verifies the candidate should be collected
• Cleans garbage associated with the marker
• Removes the marker

Identify candidate

Acquire locks

Clean

Remove marker

Identify candidate

Acquire locks

Abort

Example: Create volume

Acquire domain lease

Acquire image lock

Create volatile image directory

Create volatile metadata file

Create lease file

Create volume data file

Commit metadata file

Commit image directory

Release image lock

Release domain lease

Example: Remove volume

Existing volume

Acquire domain lease

Acquire image lock

Make image volatile

Invoke the garbage collector

Release image lock

Release domain lease

Example: Clone volume

Existing volume

Create another volume

Acquire source image lock

Acquire target image lock

Acquire source volume lease

Acquire target volume lease

Mark target volume illegal

Copy data

Progress event

Mark target volume legal

Release target volume lease

Release source volume lease

Release target image lock

Release source image lock

Completion event