The quest to save a lost apt full-upgrade
apt full-upgradeThe quest to save a lost apt full-upgrade
During one of my numerous bullseye to bookworm migration, one that should have been painless and straightforward,
the chaotic-evil ancient daemon of computers decided to test my nerves.
As usual, I have cast byobu-enable once in my shell, which means I have the passive advantage of always being in a
tmux session. That normally prevents many accident to happen, like giving you an easy saving roll in case of network
failure during a complex invocation like apt upgrade.
So, well prepared for my quest, and having a bit of time in front of me (which is the last and most useful advantage you
can have in those situation), I ssh'd to my doomed server and began facing the trials.
sed the sources.list files, now apt update, then apt full-upgrade and 💥... wooops, missed that trap1 ... and I
wake up deep into oblivion...
No problem, I think, I'll just cast another ssh spell, get back my running session, and continue answering apt
whether or not I want to replace my modified config scroll with the new one shipped at the magic academy.
That's precisely where I started to realize why this routine quest has a slight chance of turning to nightmare.
ssh -v would correctly ask my gatekeeper for the challenge, which means the communication runs fine with the dungeon,
but then the invocation stopped at the following lines:
debug1: Sending environment.
debug1: channel 0: setting env LANG = "fr_FR.UTF-8"
with nothing more appearing, and most importantly, no interactive shell spawning.
No problem, I'll just use a longer, but simpler invocation to test how things are going: ssh server.of.doom ls. Lo and
behold, this gives me the scrolls of my home! That's good news, the server is not dead, and can still take commands. But
no matter what spell I try to cast that way, my sweet tmux won't come back, and neither will a new shell.
Time to open the old magic manuals and find some clever tricks.
bash --norc is the first one taking me forward. With that, you can invoke the bash spell without it reading all
its nice personalization scrolls, like ~/.bashrc. It's just a nice and simple bash, but of course, it'll have no
$PS1, thus the prompt will be empty, and the behavior appears just like the previous one with the stuck ssh. You
absolutely need to cast some simple spell like ls in it to see if it works! I knew that already, so didn't loose any
time here, but for fellow wizards still learning their basics, that tip might help.
As my final goal is to complete the infamous apt full-upgrade command, better directly cast a privileged shell:
ssh server.of.doom sudo -S su -c \'bash --norc\'.
Once I reach back the root shell level of my dungeon, things get pretty much easier: I can roll my pseption to
check for monsters, and obviously find a zombie I'm afraid to stumble upon: byobu. As ssh is not a necromancy
spell, it obviously can't deal with zombies, but still this zombie is retaining my beloved apt in chains, and I need
to free it if I want to bring its cast to completion.
To break the chains, I need a tool. Fortunately for me, I learned a long time ago an almighty spell that can lend me all
the knowledge and wisdom I want. This quickly brings me to the discovery of that powerful new tool: reptyr.
With that tool, I realize the approach won't be to fight the undead, but to steal what I need without them realizing it.
That new reptyr spell is luckily very easy to cast, but I still have a final twist and turn: I'm in a silence zone
that prevents me from apt installing anything new, for obvious reasons. Good thing I've also learned about the arcanes
of -static sticks and scp to very quickly recover from being silenced.
Now that I'm all stuffed and know my way around that uncanny dungeon, I quickly cut the chains and free my primary objective. My terminal is unfortunately not in a very good shape, probably lost a few HP around some mysterious glyphs, but it's still walking and I can rely on it to complete the spell.
Comes the time of the final boss fight: systemctl reboot, but it seem that I've recovered my apt spell so well that
I managed to trigger fear out of it, and there is basically no fight at all.
A last dpkg --configure -a spell to ensure the exit doors are opened, but that wasn't even necessary.
Until next time, farewell o/
normal procedure should be apt-get update && apt-get upgrade && apt-get full-upgrade, but in my case, I wonder if it would have changed anything. Still, that's the only step I missed when preparing the quest, and won't forget it next time, just in case.