how to fix broken fstab that causes / read-only on remote VM?

Question

I broke my fstab by a multi line paste of commands on an AWS ec2.

by some sort of miracle, the os still boots. output of mount:

ec2-user@host:~ $ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=1001720k,nr_inodes=250430,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
/dev/xvda1 on / type xfs (ro,relatime,attr2,inode64,noquota)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14220)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=203952k,mode=700,uid=1000,gid=1000)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=203952k,mode=700)

stuff I've tried:

ec2-user@host:~ $ sudo mount -o remount,rw /dev/xda1 /
Cannot create a temporary file: /var/tmp/scl5pGkRA
ec2-user@host:~ $ sudo remount / rw
Cannot create a temporary file: /var/tmp/sclUmQEPx
ec2-user@host:~ $ sudo mount -o rw,remount /
Cannot create a temporary file: /var/tmp/sclwlQGn3
ec2-user@host:~ $ mount -o rw,remount /
mount: only root can use "--options" option
ec2-user@host:~ $ su root
Password:
su: Authentication failure
ec2-user@host:/tmp $ pkexec mount -o rw,remount /
==== AUTHENTICATING FOR org.freedesktop.policykit.exec ===
Authentication is needed to run `/usr/bin/mount' as the super user
Authenticating as: EC2 Default User (ec2-user)
Password:
polkit-agent-helper-1: pam_authenticate failed: Authentication failure
==== AUTHENTICATION FAILED ===
Error executing command as another user: Not authorized

as you can see, there is no root password, so impossible to su in. impossible to sudo su since sudo wants to create temp files. now what ?

Answer 1

Ok so I didn't find a soft solution.

I had to go to the management console, create a new virtual machine in the same network, turn off the old VM.

go to "volume" and detach the ECB disk. re-attach it to the new VM, it will plug on /dev/sdf (renamed to xvdf for some reason).

And then the UUID is the same so mount with sudo mount --rw -t xfs -o nouuid /dev/xvdf1 somedirectory

fixup the fstab with sudoedit somedirectory/etc/fstab

umount, logout, go back to the management console, shut down the new VM. back to "volumes" detach again, and re-attach to the old VM using /dev/xvda.

restart the old VM. fixed.
this takes one hour :'( it's sad