Rescue Mode¶
There are times when a server becomes unreachable over SSH due to broken networking, a bad install, misconfiguration, a kernel upgrade, bad firewall rules, etc. When you can’t log into your server at all (you don't have the root password or the server won’t boot up) you can use our Rescue Mode, which loads a vanilla Alpine Linux image into your server's RAM.
If you have your server's root password and your SSH key, you should use our S.O.S (Serial over SSH) service.
Requirements¶
Layer 3 Networking - In order for us to be able to load the Rescue OS onto your server, it must be reachable at an IP address. Therefore, your server must be in a Layer 3 or Hybrid networking mode. If your server is currently in a pure Layer 2 configuration, you will have to convert it to a Layer 3 or Hybrid networking mode in order to use Rescue Mode.
SSH Keys - In order to access the server after it boots into the Rescue OS, you will need to have a valid SSH key. The Rescue OS uses the SSH public keys that are associated with your server when you initially provisioned it.
Booting into Rescue Mode¶
To enter Rescue Mode, you can find Rescue Mode from your server's main Overview page. Click on Server Actions, and select Rescue OS from the dropdown.
This will reboot the server, load the Rescue OS Alpine Linux image and then boot into Alpine Linux.
Once the server has booted, you can SSH as root using authorized SSH keys.
If you had previously used SSH to connect to the server, you may get a warning that the host key has changed, and to remove the host key from your known hosts file on the local machine before you can connect.
ssh root@SERVER_IP_HERE
>
,------. ,--. ,--.
| .---' ,---. ,--.,--.`--',--,--, `--',--. ,--.
| `--, | .-. || || |,--.| \,--. \ `' /
| `---.' '-' |' '' '| || || || | / /. \
`------' `-| | `----' `--'`--''--'`--''--' '--'
`--'
===============================================
Rescue environment based on Alpine Linux $(cat /etc/alpine-release)
Use "apk" package manager for additional utilities.
See docs at http://wiki.alpinelinux.org
localhost:~#
Mounting the Original Root Partition¶
do you need to already know the root partition find it with df -h in Ubuntu (ha, yep you do need to know) or can you just guess?
Next, you'll need to mount the original OS's root partition, the examples on this page mount it to /mnt.
mount -t ext4 /dev/sda3 /mnt
If you don't already know where the original OS's root partition is, you can try to guess it. Many times it is dev/sda3. For some of our servers it may be in dev/mdxxx where xxx is the number of the md device.
Once you have successfully mounted a partition, cd to it.
cd /mnt
Run ls to see the original OS's filesystem.
localhost:/mnt# ls
>
bin libx32 modules.dep run
boot lost+found modules.dep.bin sbin
dev media modules.devname snap
etc mnt modules.order srv
home modules.alias modules.softdep sys
initrd modules.alias.bin modules.symbols tmp
kernel modules.builtin modules.symbols.bin usr
lib modules.builtin.alias.bin opt var
lib32 modules.builtin.bin proc vdso
lib64 modules.builtin.modinfo root
If you have successfully accessed the root partition, you can repair it.
Common Use Cases and Repairs¶
Resetting the Root Password¶
In cases when you can normally SSH into your server and can’t use SOS because you have forgotten or haven’t saved the root password, you can reset it through Rescue Mode. There are two ways to do this.
Option 1
After mounting the root filesystem to /mnt, we can use chroot to load the mounted filesystem and directly use the sudo passwd command to modify the existing root password.
chroot /mnt /bin/bash
sudo passwd
Option 2
You can also simply modify the shadow file containing the code for the root password, which will result in the root account being password-less.
vi /mnt/etc/shadow
On the first line, there is a "root:$X$SK5xfLB1ZW:0:0...".
In order to delete the password, delete everything between the first and second colon.
Thus "root:$X$SK5xfLB1ZW:0:0..." gets converted to: "root::0:0..."_
Keep in mind this will remove the root password completely, and after a reboot you will have to use our SOS Console to access the machine using password authentication, and set a new password.
Recovering Files¶
If the server cannot boot into the original OS anymore and you need to recover any important files, you can use scp from Rescue Mode to move them somewhere else.
So if you have a demo folder located in a /demo folder in your home directory, you can use scp to save them to your local machine.
scp -r root@Server_IP:/mnt/home/demo /User/Downloads
Resize or Add Partitions¶
By default, the boot drive will have a root partition that utilizes all the drive space. Some servers have additional unformatted drives but if you need to create multiple partitions in the same boot drive, you can go into Rescue Mode to shrink the root partition and create new ones.
The process for doing this can differ depending on the filesystem that you use but if using an OS with ext4 such as Ubuntu, you can use resize2fs and gdisk to resize your filesystem/partition and create additional ones.
It's worth noting that the default cloud-init configuration file in our images has the growpart and resizefs modules enabled which will resize your root partition back to it's max drive size if you only shrink the partition but not create additional ones. If you would like to avoid this behavior, you can edit the cloud-init configuration file found at /etc/cloud/cloud.cfg to remove the -growpart and -resizefs modules. Once you save the file, cloud-init should not attempt to resize the partition every time the OS boots.
Additional Packages for Alpine Linux¶
The Alpine Linux image used for Rescue Mode may be missing packages that are required for special operations. For example, to run fsck you need to install the e2fsprogs package.
apk add e2fsprogs
After successful installation, you can run fsck on sda1 for example.
fsck.ext4 -f -y /dev/sda1
You may need other packages, like zfs if you are formatting a partition with a filesystem other than ext4. You can find all the available packages to install in the Alpine Linux package directory.
Returning to Your Original Operating System¶
In order to get back to the Original OS, we can simply reboot the server.
From Alpine Linux:
To reboot Alpine Linux, use the reboot command.
reboot
From the console:
To reboot your server from its Overview page, click on Server Actions, and select Reboot from the dropdown.
Once the server has finished rebooting from either process, it will boot back into your original operating system.