Console access and event log
Instance consoles can be accessed through both the command line tools and the dashboard. In addition, OpenStack maintains a list of actions performed on a host which can be used to diagnose problems.
Horizon
Access to the graphical console can be made via Horizon. Select the node on the initial page you wish to connect to and then the console tab. A screen as follows should appear
The text console can be accessed using the Log tab. This shows the text output including history.
VNC or RDP via a URL
The URL for the console can be retrieved via the unified CLI.
$ openstack console url show my-instance-name
+-------+---------------------------------------------------------------------------------------------------+
| Field | Value |
+-------+---------------------------------------------------------------------------------------------------+
| type | novnc |
| url | https://ver1-api.cloud.infomaniak.ch:443/vnc_auto.html?token=ea8f9a2c-8396-442b-86f7-675e2cab026b |
+-------+---------------------------------------------------------------------------------------------------+
A browser can then be pointed directly at the URL to retrieve the console. This can be used if you need to embed the console access into a web interface.
Replacing init
In certain situations it can be necessary to intercept the boot process and get a shell prompt on the instance to execute administrative tasks. An example would be to correct a problem in fstab which prevents the normal boot process to succeed.
To interrupt the normal boot process on a CC7 instance:
- Get console access as described above
- Reset the instance (e.g. by clicking on the 'Send CtrlAltDel' button in the top right corner of the console)
- When the kernel list appears, use the up/down arrows to select the kernel you'd like to boot into and press 'e' (to edit the kernel command line)
- Scroll down to find the kernel command line (it contains the kernel name and 'ro') and append 'rd.break enforcing=0'
- Press Ctrl-x to boot the system with these parameters
- Once you have a root shell, remount the file system with 'mount -oremount,rw /sysroot'
- Make the file system your root file system with 'chroot /sysroot'
You can now do the tasks for which you booted into single user mode. For instance, run 'passwd' to change the password and 'touch /.autorelabel' to relabel files after the password change.
Once done:
- Exit the chroot environment with 'exit'
- Remount the filesystem as readonly with 'mount -oremount,ro /sysroot'
- Reboot the system with 'exit'
Info
'init=/bin/bash' will replace init with bash and give you a shell with a command prompt will boot into single user mode (and eventually ask you for the root password)
Kernel panic after reboot
There have been cases where instances get rebooted and end up in kernel panic. In such cases logs like the following appear in the console:
Procide a KernelPanic example
The most common reason for this is an incomplete kernel update (e.g. the yum process to install a new kernel failed to create a proper initrd initial ram-disk) A solution would be to rollback to a previous kernel following these steps:
- Get console access as described above.
- Hard reboot the instance. At this point the console should show that the instance disconnected.
- Reload the page in your browser to reconnect, and enter the GRUB menu
- be quick or you might miss it! You may need multiple retries...
- Using the arrows, select the previous kernel, and boot into it.
- After this, the instance should boot successfully into the kernel selected in the previous step.
Once you login, it is strongly advised to remove and reinstall the kernel that caused problems.
Info
There is a timer in the kernel selection menu, so your actions have to be quick enough.
Once you hit either the up or down arrow the timer will stop.
The instance might not respond to the 'Send CtrlAltDel' button, this is why it is suggested to hard reboot instead.
Command line tools
Console log
Unlike the dashboard access, the command line tools can access the console history in text. This is particularly useful for Linux instances where crash dump output can be checked.
$ openstack console log show my-instance-name
Info
The console log is reset when a instance is hard rebooted.
Event log
Each instance has an event log associated with it. This gives the times and actions performed on the instance. A message may also be provided to give further information. The event log for the instance can be obtained as follows
$ openstack server event list my-instance-name
Example:
taylor@laptop:~$ openstack server event list infomaniak-instance-2
+------------------------------------------+--------------------------------------+----------+----------------------------+
| Request ID | Server ID | Action | Start Time |
+------------------------------------------+--------------------------------------+----------+----------------------------+
| req-69493241-e8ab-423b-8b8c-1e8a82f8b109 | 0c2b0e8f-0a59-4d81-9545-fd90dc7fee73 | resize | 2021-02-25T09:42:27.000000 |
| req-db5e2c1f-0b47-4fd2-90e3-b622d8aa26a7 | 0c2b0e8f-0a59-4d81-9545-fd90dc7fee73 | unrescue | 2021-02-25T09:32:08.000000 |
| req-8006aed2-a180-4981-bd77-bd2e46f5a91a | 0c2b0e8f-0a59-4d81-9545-fd90dc7fee73 | rescue | 2021-02-25T09:17:21.000000 |
| req-b696554b-021b-4da8-a28b-18e62167e528 | 0c2b0e8f-0a59-4d81-9545-fd90dc7fee73 | create | 2021-02-25T08:57:27.000000 |
+------------------------------------------+--------------------------------------+----------+----------------------------+
This can be useful to identify when actions were performed on the instance.
OpenStack API
$ openstack server event show my-instance-name req-7bf4a18c-8cc3-4741-9924-e7cbh230c5e2
+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | start |
| events | [{u'finish_time': u'2018-06-11T12:29:18.000000', u'start_time': u'2018-06-11T12:29:12.000000', u'traceback': None, u'event': u'compute_start_instance', u'result': u'Success'}] |
| instance_uuid | 5f8837b9-abb2-4992-bfd8-140220nd6816 |
| message | None |
| project_id | 29d90dc7-2558-41h9-a947-836c0d1df2fb |
| request_id | req-7bf4a18c-8cc3-4731-9924-e7cb2230c5e2 |
| start_time | 2020-06-11T12:29:11.000000 |
| user_id | my-user-name |
+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+