One of the great features to manage Azure Linux VMs is the serial console, which gives direct access to the serial port (ttyS0) on the Linux VM. You don’t need SSH or opening ports on your NSG to have console access to your server. The feature works by default on all Linux distributions (SUSE may be an exception if you are using an older version). There are a couple of simple requirements that must be met: a password-based user is required and the boot diagnostics settings must be enabled. We wrote how to manage Azure boot diagnostics in a previous article, which you can access here.) Every time that you need to enter the recovery mode, you will be grateful for this Azure feature. The Red Hat recovery tools and recovery mode could be used for a simple mistake like a wrong entry on the /etc/fstab, a file system configured in the /etc/fstab, a hardware module, firewall settings, GRUB (grand unified boot loader), SSH configuration, and so forth. In this article, we are going to cover the process of recovering the Red Hat Linux system using all three methods available. If you are studying for Red Hat certification, that is a must to know before scheduling your exam. We will also see how Microsoft Azure features can help a Linux administrator reset the root password without previous knowledge of the password.
Red Hat booting process and recovery tools available
The GRUB is comprised of two stages. The first stage comes from the MBR partition that is loaded into the memory by the BIOS. The second stage is when we see the GRUB menu, and we can select which kernel/operation system we want to use, and it reads the kernel into memory and transfer the control to the kernel to complete the bootup process.
However, we can interact with the second stage and use three available methods to perform configuration to repair the system:
- Break: It doesn’t require the root password, and the / is mounted in read-only mode
- Emergency: It does require root password. Only root is mounted in read-only mode, only a few services are loaded, the network is not activated, and no other file systems are attempted to be mounted.
- Rescue: It does require root password, all file systems are mounted, and some essential services are started.
It is a best practice to fix /etc/fstab and other configuration issues on Linux using either rescue (first option) or emergency as your plan B.
Recovering the root password
Let’s assume the worst-case scenario — you don’t know the root password, and you don’t have any other available user (in Azure, we have some ways around it, and we will get there in a minute).
When you don’t have the root password, the maintenance and rescue modes are not available because they require you to enter the root password on the very first step.
We are left with the rd.break mode, which can change the root password. Using the serial console future in the Azure Portal, wait for the GRUB menu to show up and type e.
Using rd.break, we are interrupting the boot process by not allowing the systemd to take over and continue the normal booting process of the operating system.
We can use the arrow to navigate, and we need to go to the line that starts with linux16, which is the command line that instructs the kernel to take over during the boot process. We are going to add rd.break at the end of that line (you can use the right arrow to get to the end of the line or ctrl-e). After adding the new string, press ctrl-x to start the boot process.
Although some of the steps I’m going to use here are not required, I like to use them to remember the actual actions without reading the product documentation every time. First, I want to see what we have mounted after we get access to the console by running mount -l (Item 1).
Then, we will look for the /sysroot from the list to remember that it is on read-only mode, and we need to change that to be able to change the root’s password.
The following commands will be used to mount the /sysroot as read-write. Our next step is to chroot jail, meaning our reality will be inside of the /sysroot folder.
mount -o remount,rw /sysroot chroot /sysroot
We have access to the root of our operating system. We are going to run whoami to make sure that we are root, and then we will change the password, instruct SELinux to relabel all files with the correct contexts.
whoami passwd touch /.autorelabel exit exit
Using emergency mode
We need to add the string emergency at the end of the linux16 line, as depicted in the image below.
After entering the root’s credentials, we can see that only the root was mounted.
Using rescue mode
The rescue mode follows the same logic from the previous attempts. The only difference is to add a rescue string at the end of the linux16 line.
The difference when using rescue is the number of mounted file system partitions.
Azure and creating new user capabilities
We went through the process to use rd.break to change the root password. As long as you have an operating system, we can take advantage of Microsoft Azure to add a new user and use that user to reset the root password — and with no downtime involved!
In the Linux VM blade, we have the Reset Password option, and it allows the creation of a brand-new user. In the example below, we are adding the user batman to Red Hat Linux.
Now that we have a user on the desired VM, we can authenticate using the new credentials and use sudo passwd root and reset the root’s password.
Featured image: Shutterstock