Volumes (create/delete/snapshot/extend..)
Understanding what a volume is
OpenStack provides two classes of block storage, ephemeral storage and persistent volumes. Ephemeral storage exists only for the life of an instance, it will persist across reboots of the guest operating system but when the instance is deleted so is the associated storage. The size of the Ephemeral storage is defined in the flavor of the virtual machine and is fixed for all virtual machines of that particular flavor. The service level for ephemeral storage is dependent on the underlying hardware.
Volumes are persistent virtualized block devices independent of any particular instance. Volumes can be attached to a single instance at a time (i.e. not like a distributed filesystem), but they may be detached or reattached to a different instance while retaining all data, much like a USB drive. Volumes are stored on redundant storage and are therefore more resilient against hardware failures.
You can define additional volumes to be attached to a virtual machine in addition to the system disk. This is useful if you need more space than is provided on the initial flavor and can also allow some interesting flexiblity for snapshotting or moving volumes around between virtual machines.
Volume type
Describes the type of storage that is being requested. The following types are currently available:
Info
- the CEPH_1_perf1 type is allocated for projects by default.
- You can mix a VM with
perf1and a volume withperf2or vice versa. Applications writing to the VM disk will be rate limited to 500 iops while the ones writing to the volume will be rate limited to 1000 iops.
| Storage Type | Description | Datacenter | Location | Maximum Performance |
|---|---|---|---|---|
| CEPH_1_perf1 | The 'CEPH_1_perf1' volume type provides a reliable store which is tolerant to disk/server/rack failure without user impact or data loss. This is the default. | D3 | cross racks | 200MB/s and 500 IO operations (both, read and write) |
| CEPH_1_perf2 | The 'CEPH_1_perf2' volume type provides a reliable store which is tolerant to disk/server/rack failure without user impact or data loss. | D3 | cross racks | 400MB/s and 1'000 IO operations (both, read and write) |
Warning
the CEPH_1_perf2 type is only available upon request, please contact our support to enable it for your project.
Volume creation and mount
Below are the 5 steps to achieve creating a volume and attaching it to an instance. Each steps is described below.
- Choose a volume type. We offer multiple types of volumes with different caracteristics, check the table below for more details.
- Define a disk volume of the size required with the volume name defined (in the example below the volume is named testvol and 100 GB big).
- Check that the volume is available for mounting using openstack volume list.
- Attach that volume to the instance that you would like it to be available on.
- Log in to the instance and format/mount the volume as required.
In details:
Creating a 100 GB volume
$ openstack volume create --description "Volume for documentation" --size 100 testvol
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-02-25T12:43:22.000000 |
| description | Volume for documentation |
| encrypted | False |
| id | bb17fd17-9c52-4a62-9e85-42fff55f80d6 |
| multiattach | False |
| name | testvol |
| properties | |
| replication_status | None |
| size | 100 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | CEPH_1 |
| updated_at | None |
| user_id | b1580497f51e4d10b9110c60c154562c |
+---------------------+--------------------------------------+
The size is measured in GB.
The status reported will become available when the volume has been created. This status can be checked using $ openstack volume show <VOLUME-(ID|NAME)>.
Attaching a volume
Once the volume has been created, it can be attached to the virtual machine.
openstack server add volume <INSTANCE-(ID-NAME)> <VOLUME-(ID-NAME)>
In this example:
openstack server add volume infomaniak-vm-1 testvol
The arguments are:
infomaniak-vm-1 is the virtual machine where you would like to attach the image. This is as previously created using openstack server create.
testvol is the volume name or id you can retrieve from openstack volume list as created above
Info
The optional device argument to specify the device name should not be used. It does not work properly, and will be removed in the near future.
Log into the VM to check and format the disk:
debian@infomaniak-vm-1:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20G 0 disk
└─sda1 8:1 0 20G 0 part /
sdb 8:16 0 100G 0 disk
Create a file system on it and mount it. Using a volume label is useful in the future to avoid needing to find the exact device name as you attach different volumes. Using a label, such as testvol in the example below, which is the same as the name in OpenStack also makes tracking easier when volumes are not mounted. The filesystem type can be one supported by the Linux distribution but XFS and Ext4 are the most widely used.
Installing parted on Ubuntu and Debian flavored distros:
sudo apt-get install parted
Installing parted on CentOS and RHEL:
sudo yum install parted
sudo parted -s /dev/sdb mklabel msdos mkpart primary ext4 0% 100%
sudo mkfs -t ext4 -L testvol /dev/sdb1
sudo mount -L testvol /mnt
df -h
If you'd like non-root users to be able to access the mounted volume similar to the way /tmp is accessible , you'll need to set the sticky bit on the mount point.
sudo chmod +t /mnt
The filesystem should be added to /etc/fstab if the desired behaviour is to mount automatically on restart. Using the LABEL parameter here will ensure the correct volume is chosen if multiple volumes are attached.
LABEL=testvol /mnt ext4 noatime,nodiratime,user_xattr,nofail 0 0
Info
The use of nofail is recommended to skip (and not block on) mounting the volume if it is unavailable, e.g. in case of network issues. Remove this option from the fstab line if you want your VM to block the boot process if the volume is unavailable.
Info
The use of nobarrier for the mount options is not recommended as volumes are accessed via a cache, so ignoring the correct ordering of journal commits may leave you with a corrupted file system in case of a hardware problem.
Removing disk volume
The steps to properly remove disk volumes are:
- Umount the volume on the virtual machine
umount /mnt - Detach it from the virtual machine
- Delete the volume (optional depends on your needs)
Info
Detaching a volume while it is being mounted is possible, but the next access on the VM to that volume will return I/O errors.
Then use the openstack command line interface to detach the volume from the virtual machine:
openstack server remove volume infomaniak-vm-1 testvol
where infomaniak-vm-1 is the virtual machine and the second parameter, testvol, is the volume as shown in openstack volume list.
Check that the volume is in state available again.
If that's the case, the volume is now ready to either be attached to another virtual machine or, if it is not needed any longer, to be completely removed (note that this step cannot be reverted!):
openstack volume delete testvol
Your volume will now go into state deleting and completely disappear from the openstack volume list output
Volume snapshots and backups
The picture below show the difference between a snapshot and a backup.
Volume snapshots
A user can also create snapshots of a volume, that then later can be used to create other volumes or to rollback to a previous state. Volume snapshots are pointers in the read/write history of a volume. The creation of a snapshot takes a few seconds and it can be done while the volume is in-use.
Once the snapshot is done, you can use it to create other volumes based on this snapshot. Creation time for these volumes may depend on the type of volume you are creating as it may entitle some data transfer.
$ openstack volume snapshot create --volume testvol testvolsnap
+-------------+--------------------------------------+
| Field | Value |
+-------------+--------------------------------------+
| created_at | 2021-02-25T13:46:24.628894 |
| description | None |
| id | 0ec94901-93b5-43c1-86bd-735572574776 |
| name | testvolsnap |
| properties | |
| size | 100 |
| status | creating |
| updated_at | None |
| volume_id | 71bd47d8-b51e-4f12-9c51-0445fc36c9e4 |
+-------------+--------------------------------------+
Info
If the volume is in-use, you may need to specify --force
You can list the volume snapshots with the following command.
$ openstack volume snapshot list
+--------------------------------------+-------------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+-------------+-------------+-----------+------+
| 0ec94901-93b5-43c1-86bd-735572574776 | testvolsnap | None | available | 100 |
+--------------------------------------+-------------+-------------+-----------+------+
Once the volume snapshot is in available state, you may create other volumes based on that snapshot. You don't need to specify the size of the volume, it will use the size of the snapshot.
$ openstack volume create --snapshot testvolsnap --description "Volume from a snapshot" testvolfromsnapshot
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2021-02-25T13:49:52.000000 |
| description | Volume from a snapshot |
| encrypted | False |
| id | 7d4e4db5-4041-4451-8add-e01dbdb64c0f |
| multiattach | False |
| name | testvolfromsnapshot |
| properties | |
| replication_status | None |
| size | 100 |
| snapshot_id | 0ec94901-93b5-43c1-86bd-735572574776 |
| source_volid | None |
| status | creating |
| type | CEPH_1_perf1 |
| updated_at | None |
| user_id | b1580497f51e4d10b9110c60c154562c |
+---------------------+--------------------------------------+
Check the volume is available:
$ openstack volume list
+--------------------------------------+---------------------+-----------+------+------------------------------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+---------------------+-----------+------+------------------------------------------+
| 7d4e4db5-4041-4451-8add-e01dbdb64c0f | testvolfromsnapshot | available | 100 | |
| 71bd47d8-b51e-4f12-9c51-0445fc36c9e4 | testvol | in-use | 100 | Attached to infomaniak-vm-1 on /dev/sdb |
+--------------------------------------+---------------------+-----------+------+------------------------------------------+
Listing/Deleting snapshots
Firstly, list the snapshots
$ openstack volume snapshot list
+--------------------------------------+-------------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+-------------+-------------+-----------+------+
| 0ec94901-93b5-43c1-86bd-735572574776 | testvolsnap | None | available | 100 |
+--------------------------------------+-------------+-------------+-----------+------+
Then to delete a snapshot issue the following command:
openstack volume snapshot delete testvolsnap
Cannot delete snapshot
When you try to delete a snapshot, it can fail due to volume dependencies (Happens when volume has been created from snapshot).
First you need to iterate over all your volumes, to ensure that no one is linked to the snapshot that we want to delete.
for i in $(openstack volume list -c ID -f value); do echo "volume: $i"; openstack volume show $i -c snapshot_id -f value; done
Then, if you find volumes that are linked to the snapshot we wanna delete, you have to delete volume first.
openstack volume delete volume_id
Finally, we can delete our snapshot
openstack volume snapshot delete testvolsnap
Volume Backup
A user can also create a backup of a volume. This volume can only be used later to create or replace other volumes. The backup will be stored on a cold storage which is cheaper than Ceph performant storage used for snapshots. Backups can be done while the volume is in-use without disruption, note that they take a significant amount of time (~4 hours for 500 GB).
To create a volume backup:
use --force if the volume is in-use
openstack volume backup create --force <VOLUME-(ID|NAME)>
To list backups:
$ openstack volume backup list
+--------------------------------------+------+-------------+-----------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+------+-------------+-----------+------+
| 8e146735-d9cc-4a96-9c83-47e71e263963 | None | None | available | 500 |
+--------------------------------------+------+-------------+-----------+------+
Identifying volumes
In cases where multiple volumes are attached to the same instance, it is desirable to establish a mapping between volume IDs (in OpenStack) and the block devices inside the guest. Volumes appear in /dev/disk/by-id as entries of the form scsi-0QEMU_QEMU_HARDDISK_ID where ID is a subset of the OpenStack volume ID. For instance, a volume like
$ openstack volume list
+--------------------------------------+---------------------+-----------+------+------------------------------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+---------------------+-----------+------+------------------------------------------+
| 71bd47d8-b51e-4f12-9c51-0445fc36c9e4 | testvol | in-use | 100 | Attached to infomaniak-vm-1 on /dev/sdb |
+--------------------------------------+---------------------+-----------+------+------------------------------------------+
testvol would appear in the VM instance as follow
$ ls /dev/disk/by-id
scsi-0QEMU_QEMU_HARDDISK_71bd47d8-b51e-4f12-9 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0-part1
scsi-0QEMU_QEMU_HARDDISK_71bd47d8-b51e-4f12-9 : 71bd47d8-b51e-4f12-9 is a fragment of the volume ID 71bd47d8-b51e-4f12-9c51-0445fc36c9e4
Extending volumes
A volume can be extended while maintaining the existing contents, assuming the file system supports resizing (Ext4/XFS for example).
The steps are as follows:
- Extend the volume
- Extend the filesystem
Note that you might need to reboot your VM in case the new size isn't taken into account.
openstack volume set --size 150 testvol
Then on the guest:
# mount /dev/sdb /mnt/
# resize2fs /dev/sdb
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/vdc is mounted on /mnt/clone; on-line resizing required
old_desc_blocks = 2, new_desc_blocks = 4
The filesystem on /dev/vdc is now 6553600 blocks long.
For XFS, a similar procedure can be followed using the xfs_growfs command.
Info
Volumes can be extended, but not shrunked. There is no support for shrinking existing volumes. The procedure given above has been tested with Ext4 and XFS filesystems only.
Common Error
Failed to set volume size: Volume is in in-use state, it must be available before size can be extended
The volume you're trying to resize is in-use which means most probably attached to a VM. You have 2 options:
- Method 1. You can force the resize with the
cinderclient as the option isn't yet available foropenstackclient
Info
Cinder client requires Volume_ID and not Volume_Names
sudo apt install python3-cinderclient
cinder --os-volume-api-version 3.42 extend 71bd47d8-b51e-4f12-9c51-0445fc36c9e4 150
You might need to stop/start your VM in order to take in account the new size (donc use reboot, it wouldn't have the same effect).
openstack server stop infomaniak-vm-1
openstack server start infomaniak-vm-1
- Method 2 : Detach the volume from the instance before resizing it
sudo umount /mnt
Info
Detaching a volume while it is being mounted is possible, but the next access on the VM to that volume will return I/O errors.
Then use the openstack command line interface to detach the volume from the virtual machine:
openstack server remove volume infomaniak-vm-1 testvol
testvol, is the volume as shown in openstack volume list.
You can now resize your volume with the command above, when done, attach the volume again to your instance
openstack server add volume infomaniak-vm-1 testvol
Retyping volumes
The type of a volume can be changed in order to change the characteristics of an already existing volume (e.g. one that has already data stored). One use case for retyping volumes is to change their IO capabilities, e.g. a change from type CEPH_1_perf1 to type CEPH_1_perf2. Another use case is when you want to change where the volume is located, e.g. a change from type CEPH_1 to type CEPH_2 (2 independant clusters)
Info
Volumes need to be detached (available) in order to be retyped.
From our example retyping volume from type CEPH_1_perf1 to type CEPH_1_perf2 can be done as follow:
openstack volume set --type CEPH_1_perf2 --retype-policy on-demand testvol
The parameter retype-policy is required when the retype entails a migration, i.e. a change of the area in which the volume is hosted.
Info
Retyping will silently fail if the volume is still attached or if the retyping is not possible.
Common Error Messages
| Error | Cause |
|---|---|
| VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed Gigabytes quota.ERROR: openstack Request Entity Too Large (HTTP 413) | Your total space for the volume service exceeds the allowance for the project. If you require further quota, please contact the helpdesk. Please note that additional quota is only granted for shared projects. Additional personal project quota requests will not be accepted. |
| VolumeLimitExceeded: Maximum number of volumes allowed (N) exceeded | You have exceeded the number of volumes. Please contact the helpdesk to request to raise this limit for a project. Additional personal project quota requests will not be accepted. |
| BadRequest: Failed to delete volume with name or ID: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) | If you are deleting a volume and has associated snapshots, you need to delete them first, before deleting the volume. |
Volume stuck in status reserved |
Various situations lead to a volume being in a reserved state. It usually means it can only be attached to a very specific instance. To list the attachments and their status issue the command cinder attachment-list. Identify the attachment(s) ID corresponding to your volume then delete them using the command cinder attachment-delete <attachment ID> |