It was also actually pretty fun!
read this over on Linux community a few days ago
Damn lol wish I would have come out sooner lmao Seems a tad more complicated though, I don’t remember having to compile anything at least
I’ve never gotten pivot_root to work well / as documented. Major props for being able to pull this off. Hopefully you didn’t have much time where a connection-drop would require a reboot. SSH in the new root on an alternate port before the pivot? idk.
Really? It seemed to go pretty smooth for me, SSH dropped once I think when I reloaded the services (expected), but after that I was able to connect back fine on the same port no issue
Hopefully you don’t lose power or something bad happens or poor their goes your data
If that happens, they just fall back to the first option of waiting for their provider to set up the IPMI.
I have never gotten it to work. It always implodes
Can someone ELI5? Are you freefloating an operating system on your RAM?
I had a server I rented from a provider in a data center and I wanted to image (dd) the drive for archival before I decommissioned it from my infrastructure.
Normally, you can’t really do that with the OS running and you would have to shut down and insert a live USB or something and temporarily boot from that. The server being a faraway rental the only option was to open a ticket so that they could enable an out-of-band management option like KVM or IPMI. Which would allow you to control a machine as if you had a physical monitor, keyboard and mouse in front of you. With that you can attach flash drives, shutdown, restart, see the POST/BIOS/UEFI screens etc remotely .
But, I didn’t want to wait 6-8 hours for them to enable that so instead I put together a process that would “boot” me into another distro “installed” into a RAM disk (kinda like how live CD/USB works) from the currently installed and running OS without rebooting
From there I could unmount the boot disk and do what ever I wanted from there, I could have even wiped the disk entirely and installed a entirely different distro if I wanted
How do you do that? What minimal distro did you use? Did you make it yourself? How reliable is pivot-root? How many tries did it take you to do that successfully?
RAM is still memory, so you can put anything in there, like an OS.
With pivot_root, you can change where the root is. So you pivot into your OS in RAM, which becomes the root of the system.
At that point, you can do pretty much anything you would normally do with your OS, like unmount a HDD.
I’ve never personally done it, but that’s a simplified explanation of what OP did.
Once I figured out how to netboot the os into memory that’s how I run all my nodes :)
I’ve been killing myself in my free time to get a NanoPi R6S to boot from an SD card; wish I had this expertise.
Not sure about your hardware, but try to look up if it requires a special kernel, like the RPi 5 (which only runs on raspbian because they ship that kernel)
Heh, this expertise is built on hundreds of “Why tf won’t X boot from Y!!” and solid research skills lol you’ll get there
What happens if the SSH session closes before dd finishes? Sounds pretty badass but I don’t think I would trust this approach in prod lol
You could use screen or tmux for a persistent terminal session.
True, but I was more thinking about the issue of reconnecting in general when you just nuked sshd.
As long as you only copy off the disk, you can just reboot and the whole system in RAM vanishes and the normal system boots again for the second try.
Well the 1 SSH session bit was for dramatic meme-effect lol, you can actually connect back without issue (at least it did for me) so worst case if you weren’t working in tmux you’d just have to start dd again
Worst worst case, you’d just end up back where you were probably heading before anyways, KVM/IPMI
Sure, I suppose as long as sshd is up and running in the ramdisk environment (which you mentioned in another comment, along with all other services) you could always reconnect. Very neat and clever!
Impressive, sounds like magic, tbh! You knou any tutorials?
Not really, I pulled it together from a bunch of random posts lol
Maybe I should write one, but in essence you:
-
Stop all non-OS essential services
-
Create a filesystem in a chunk of RAM
-
Pull essential OS files from the installed OS into it recreating needed directories (Though you could probably just use a tiny pre-built distro but meh)
-
Pivot root into it
-
Reload services (when they restart they’ll be restarted in the context of where you pivot rooted, prior they’re still running under the context of the installed OS)
-
Unmount the boot drive
-
Then do what ever you need to do
-
???
-
Profit
I work in this field and I think this is 🔥 af
How exactly so you pivot root? Simply chroot or something more involved?
chroot isn’t enough, you need “real” pivot_root
chroot just changes the shells root point, pivot_root actually changes the root mount point and enables this trick to work
Holy fuck
This is god-tier technique, kudos
FYI you can use kexec and a prepared initrd to do something similar with only one command.
Imagining this process, despite having never done 50% of the steps on the list, makes my brain imagine this:
-
Yeah, I’d love to see a write up on this to follow.Sounds like useful practice in the lab if nothing else.
i experimented with this some time ago, see my post here: https://lemmy.ml/post/18706002/12772832
i had experimented with kexec and takeover.sh to install a distro that was not available by my provider.
it resulted in some scripts i now (triggered by this thread) have published (in a nonready state):
http://github.com/tobinq/goaround
the scripts may be in bad shape but i successfully changed one preinstalled ubuntu to a devuan with what is in these scripts. however i didn’t work on them for month now and am not sure about that last state… so its experimental only.
Give me more of this and less of the politics. This is what I come to Lemmy for.
Weird
No, you
subscribe to the linux comms and not to the politics comms
Check out dattobd sometime. It’s a tool that allows you to make a snapshot of a block device for backups, while the OS is running. Useful if your filesystem doesn’t support snapshots.
I also vote for a writeup. This sounds awesome!
@cm0002
I need that!Is there a tutorial for that? I have an Old outdated Synology nas that I wish to replace the system with an Open Media Vault.
I did this once to install a different distro on a free oracle VPS lol