Skip to content

Volumes (create/delete/snapshot/extend..)

Understanding what a volume is

OpenStack provides two classes of block storage, ephemeral storage and persistent volumes. Ephemeral storage exists only for the life of an instance, it will persist across reboots of the guest operating system but when the instance is deleted so is the associated storage. The size of the Ephemeral storage is defined in the flavor of the virtual machine and is fixed for all virtual machines of that particular flavor. The service level for ephemeral storage is dependent on the underlying hardware.

Volumes are persistent virtualized block devices independent of any particular instance. Volumes can be attached to a single instance at a time (i.e. not like a distributed filesystem), but they may be detached or reattached to a different instance while retaining all data, much like a USB drive. Volumes are stored on redundant storage and are therefore more resilient against hardware failures.

You can define additional volumes to be attached to a virtual machine in addition to the system disk. This is useful if you need more space than is provided on the initial flavor and can also allow some interesting flexiblity for snapshotting or moving volumes around between virtual machines.

Volume type

Describes the type of storage that is being requested. The following types are currently available:

Info

  • the CEPH_1_perf1 type is allocated for projects by default.
  • You can mix a VM with perf1 and a volume with perf2 or vice versa. Applications writing to the VM disk will be rate limited to 500 iops while the ones writing to the volume will be rate limited to 1000 iops.
Storage Type Description Datacenter Location Maximum Performance
CEPH_1_perf1 The 'CEPH_1_perf1' volume type provides a reliable store which is tolerant to disk/server/rack failure without user impact or data loss. This is the default. D3 cross racks 200MB/s and 500 IO operations (both, read and write)
CEPH_1_perf2 The 'CEPH_1_perf2' volume type provides a reliable store which is tolerant to disk/server/rack failure without user impact or data loss. D3 cross racks 400MB/s and 1'000 IO operations (both, read and write)

Warning

the CEPH_1_perf2 type is only available upon request, please contact our support to enable it for your project.

Volume creation and mount

Below are the 5 steps to achieve creating a volume and attaching it to an instance. Each steps is described below.

  1. Choose a volume type. We offer multiple types of volumes with different caracteristics, check the table below for more details.
  2. Define a disk volume of the size required with the volume name defined (in the example below the volume is named testvol and 100 GB big).
  3. Check that the volume is available for mounting using openstack volume list.
  4. Attach that volume to the instance that you would like it to be available on.
  5. Log in to the instance and format/mount the volume as required.

In details:

Creating a 100 GB volume

taylor@laptop:~$ openstack volume create --description "Volume for documentation" --size 100 testvol
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2021-02-25T12:43:22.000000           |
| description         | Volume for documentation             |
| encrypted           | False                                |
| id                  | bb17fd17-9c52-4a62-9e85-42fff55f80d6 |
| multiattach         | False                                |
| name                | testvol                              |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 100                                  |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | CEPH_1                               |
| updated_at          | None                                 |
| user_id             | b1580497f51e4d10b9110c60c154562c     |
+---------------------+--------------------------------------+

The size is measured in GB.

The status reported will become available when the volume has been created. This status can be checked using $ openstack volume show <VOLUME-(ID|NAME)>.

Attaching a volume

Once the volume has been created, it can be attached to the virtual machine.

$ openstack server add volume <INSTANCE-(ID-NAME)> <VOLUME-(ID-NAME)>

In this example:

$ openstack server add volume infomaniak-vm-1 testvol

The arguments are:

infomaniak-vm-1 is the virtual machine where you would like to attach the image. This is as previously created using openstack server create.

testvol is the volume name or id you can retrieve from openstack volume list as created above

Info

The optional device argument to specify the device name should not be used. It does not work properly, and will be removed in the near future.

Log into the VM to check and format the disk:

debian@infomaniak-vm-1:~$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   20G  0 disk
└─sda1   8:1    0   20G  0 part /
sdb      8:16   0  100G  0 disk

Create a file system on it and mount it. Using a volume label is useful in the future to avoid needing to find the exact device name as you attach different volumes. Using a label, such as testvol in the example below, which is the same as the name in OpenStack also makes tracking easier when volumes are not mounted. The filesystem type can be one supported by the Linux distribution but XFS and Ext4 are the most widely used.

Installing parted on Ubuntu and Debian flavored distros:

$ sudo apt-get install parted

Installing parted on CentOS and RHEL:

$ sudo yum install parted

$ sudo parted -s /dev/sdb mklabel msdos mkpart primary ext4 0% 100%
$ sudo mkfs -t ext4 -L testvol /dev/sdb1
$ sudo mount -L testvol /mnt
$ df -h

If you'd like non-root users to be able to access the mounted volume similar to the way /tmp is accessible , you'll need to set the sticky bit on the mount point.

$ sudo chmod +t /mnt

The filesystem should be added to /etc/fstab if the desired behaviour is to mount automatically on restart. Using the LABEL parameter here will ensure the correct volume is chosen if multiple volumes are attached.

LABEL=testvol /mnt ext4 noatime,nodiratime,user_xattr,nofail 0 0

Info

The use of nofail is recommended to skip (and not block on) mounting the volume if it is unavailable, e.g. in case of network issues. Remove this option from the fstab line if you want your VM to block the boot process if the volume is unavailable.

Info

The use of nobarrier for the mount options is not recommended as volumes are accessed via a cache, so ignoring the correct ordering of journal commits may leave you with a corrupted file system in case of a hardware problem.

Removing disk volume

The steps to properly remove disk volumes are:

  1. Umount the volume on the virtual machine umount /mnt
  2. Detach it from the virtual machine
  3. Delete the volume (optional depends on your needs)

Info

Detaching a volume while it is being mounted is possible, but the next access on the VM to that volume will return I/O errors.

Then use the openstack command line interface to detach the volume from the virtual machine:

$ openstack server remove volume infomaniak-vm-1 testvol

where infomaniak-vm-1 is the virtual machine and the second parameter, testvol, is the volume as shown in openstack volume list.

Check that the volume is in state available again.

If that's the case, the volume is now ready to either be attached to another virtual machine or, if it is not needed any longer, to be completely removed (note that this step cannot be reverted!):

$ openstack volume delete testvol

Your volume will now go into state deleting and completely disappear from the openstack volume list output

Volume snapshots and backups

The picture below show the difference between a snapshot and a backup.

Volume snapshot vs backup

Volume snapshots

A user can also create snapshots of a volume, that then later can be used to create other volumes or to rollback to a previous state. Volume snapshots are pointers in the read/write history of a volume. The creation of a snapshot takes a few seconds and it can be done while the volume is in-use.

Once the snapshot is done, you can use it to create other volumes based on this snapshot. Creation time for these volumes may depend on the type of volume you are creating as it may entitle some data transfer.

$ openstack volume snapshot create --volume testvol testvolsnap
taylor@laptop:~$ openstack volume snapshot create --volume testvol testvolsnap
+-------------+--------------------------------------+
| Field       | Value                                |
+-------------+--------------------------------------+
| created_at  | 2021-02-25T13:46:24.628894           |
| description | None                                 |
| id          | 0ec94901-93b5-43c1-86bd-735572574776 |
| name        | testvolsnap                          |
| properties  |                                      |
| size        | 100                                  |
| status      | creating                             |
| updated_at  | None                                 |
| volume_id   | 71bd47d8-b51e-4f12-9c51-0445fc36c9e4 |
+-------------+--------------------------------------+

Info

If the volume is in-use, you may need to specify --force

You can list the volume snapshots with the following command.

taylor@laptop:~$ openstack volume snapshot list
+--------------------------------------+-------------+-------------+-----------+------+
| ID                                   | Name        | Description | Status    | Size |
+--------------------------------------+-------------+-------------+-----------+------+
| 0ec94901-93b5-43c1-86bd-735572574776 | testvolsnap | None        | available |  100 |
+--------------------------------------+-------------+-------------+-----------+------+

Once the volume snapshot is in available state, you may create other volumes based on that snapshot. You don't need to specify the size of the volume, it will use the size of the snapshot.

taylor@laptop:~$ openstack volume create --snapshot testvolsnap  --description "Volume from a snapshot" testvolfromsnapshot
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2021-02-25T13:49:52.000000           |
| description         | Volume from a snapshot               |
| encrypted           | False                                |
| id                  | 7d4e4db5-4041-4451-8add-e01dbdb64c0f |
| multiattach         | False                                |
| name                | testvolfromsnapshot                  |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 100                                  |
| snapshot_id         | 0ec94901-93b5-43c1-86bd-735572574776 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | CEPH_1_perf1                         |
| updated_at          | None                                 |
| user_id             | b1580497f51e4d10b9110c60c154562c     |
+---------------------+--------------------------------------+

Check the volume is available:

taylor@laptop:~$ openstack volume list
+--------------------------------------+---------------------+-----------+------+------------------------------------------+
| ID                                   | Name                | Status    | Size | Attached to                              |
+--------------------------------------+---------------------+-----------+------+------------------------------------------+
| 7d4e4db5-4041-4451-8add-e01dbdb64c0f | testvolfromsnapshot | available |  100 |                                          |
| 71bd47d8-b51e-4f12-9c51-0445fc36c9e4 | testvol             | in-use    |  100 | Attached to infomaniak-vm-1 on /dev/sdb  |
+--------------------------------------+---------------------+-----------+------+------------------------------------------+

Listing/Deleting snapshots

Firstly, list the snapshots

taylor@laptop:~$ openstack volume snapshot list
+--------------------------------------+-------------+-------------+-----------+------+
| ID                                   | Name        | Description | Status    | Size |
+--------------------------------------+-------------+-------------+-----------+------+
| 0ec94901-93b5-43c1-86bd-735572574776 | testvolsnap | None        | available |  100 |
+--------------------------------------+-------------+-------------+-----------+------+

Then to delete a snapshot issue the following command:

$ openstack volume snapshot delete testvolsnap

Volume Backup

A user can also create a backup of a volume. This volume can only be used later to create or replace other volumes. The backup will be stored on a cold storage which is cheaper than Ceph performant storage used for snapshots. Backups can be done while the volume is in-use without disruption, note that they take a significant amount of time (~4 hours for 500 GB).

To create a volume backup:

use --force if the volume is in-use

$ openstack volume backup create --force <VOLUME-(ID|NAME)>

To list backups:

$ openstack volume backup list
+--------------------------------------+------+-------------+-----------+------+
| ID                                   | Name | Description | Status    | Size |
+--------------------------------------+------+-------------+-----------+------+
| 8e146735-d9cc-4a96-9c83-47e71e263963 | None | None        | available |  500 |
+--------------------------------------+------+-------------+-----------+------+

Identifying volumes

In cases where multiple volumes are attached to the same instance, it is desirable to establish a mapping between volume IDs (in OpenStack) and the block devices inside the guest. Volumes appear in /dev/disk/by-id as entries of the form scsi-0QEMU_QEMU_HARDDISK_ID where ID is a subset of the OpenStack volume ID. For instance, a volume like

taylor@laptop:~$ openstack volume list
+--------------------------------------+---------------------+-----------+------+------------------------------------------+
| ID                                   | Name                | Status    | Size | Attached to                              |
+--------------------------------------+---------------------+-----------+------+------------------------------------------+
| 71bd47d8-b51e-4f12-9c51-0445fc36c9e4 | testvol             | in-use    |  100 | Attached to infomaniak-vm-1 on /dev/sdb  |
+--------------------------------------+---------------------+-----------+------+------------------------------------------+

testvol would appear in the VM instance as follow

debian@infomaniak-vm-1:~$ ls /dev/disk/by-id
scsi-0QEMU_QEMU_HARDDISK_71bd47d8-b51e-4f12-9  scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0  scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0-part1

scsi-0QEMU_QEMU_HARDDISK_71bd47d8-b51e-4f12-9 : 71bd47d8-b51e-4f12-9 is a fragment of the volume ID 71bd47d8-b51e-4f12-9c51-0445fc36c9e4

Extending volumes

A volume can be extended while maintaining the existing contents, assuming the file system supports resizing (Ext4/XFS for example).

The steps are as follows:

  1. Extend the volume
  2. Extend the filesystem

Note that you might need to reboot your VM in case the new size isn't taken into account.

$ openstack volume set --size 150 testvol

Then on the guest:

# mount /dev/sdb /mnt/
# resize2fs /dev/sdb
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/vdc is mounted on /mnt/clone; on-line resizing required
old_desc_blocks = 2, new_desc_blocks = 4
The filesystem on /dev/vdc is now 6553600 blocks long.

For XFS, a similar procedure can be followed using the xfs_growfs command.

Info

Volumes can be extended, but not shrunked. There is no support for shrinking existing volumes. The procedure given above has been tested with Ext4 and XFS filesystems only.

Common Error

Failed to set volume size: Volume is in in-use state, it must be available before size can be extended

The volume you're trying to resize is in-use which means most probably attached to a VM. You have 2 options:

  • Method 1. You can force the resize with the cinder client as the option isn't yet available for openstack client

Info

Cinder client requires Volume_ID and not Volume_Names

sudo apt install python3-cinderclient
taylor@laptop:~$ cinder --os-volume-api-version 3.42 extend 71bd47d8-b51e-4f12-9c51-0445fc36c9e4 150

You might need to stop/start your VM in order to take in account the new size (donc use reboot, it wouldn't have the same effect).

taylor@laptop:~$ openstack server stop infomaniak-vm-1
taylor@laptop:~$ openstack server start infomaniak-vm-1
  • Method 2 : Detach the volume from the instance before resizing it

debian@infomaniak-vm-1:~$ sudo umount /mnt

Info

Detaching a volume while it is being mounted is possible, but the next access on the VM to that volume will return I/O errors.

Then use the openstack command line interface to detach the volume from the virtual machine:

$ openstack server remove volume infomaniak-vm-1 testvol where infomaniak-vm-1 is the virtual machine and the second parameter, testvol, is the volume as shown in openstack volume list.

You can now resize your volume with the command above, when done, attach the volume again to your instance $ openstack server add volume infomaniak-vm-1 testvol

Retyping volumes

The type of a volume can be changed in order to change the characteristics of an already existing volume (e.g. one that has already data stored). One use case for retyping volumes is to change their IO capabilities, e.g. a change from type CEPH_1_perf1 to type CEPH_1_perf2. Another use case is when you want to change where the volume is located, e.g. a change from type CEPH_1 to type CEPH_2 (2 independant clusters)

Info

Volumes need to be detached (available) in order to be retyped.

From our example retyping volume from type CEPH_1_perf1 to type CEPH_1_perf2 can be done as follow:

$ openstack volume set --type CEPH_1_perf2 --retype-policy on-demand testvol

The parameter retype-policy is required when the retype entails a migration, i.e. a change of the area in which the volume is hosted.

Info

Retyping will silently fail if the volume is still attached or if the retyping is not possible.

Common Error Messages

From the openstack volume command, the following errors can be received

Error Cause
VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed Gigabytes quota.ERROR: openstack Request Entity Too Large (HTTP 413) Your total space for the volume service exceeds the allowance for the project. If you require further quota, please contact the helpdesk. Please note that additional quota is only granted for shared projects. Additional personal project quota requests will not be accepted.
VolumeLimitExceeded: Maximum number of volumes allowed (N) exceeded You have exceeded the number of volumes. Please contact the helpdesk to request to raise this limit for a project. Additional personal project quota requests will not be accepted.
BadRequest: Failed to delete volume with name or ID: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) If you are deleting a volume and has associated snapshots, you need to delete them first, before deleting the volume.
Back to top