ls: Segmentation fault (core dumped), or: howto repair a broken glibc


terminal linux

When trying to install a newer glibc, specifically 2.34, I got into this problem:

I added the “locally installed” lib directory to the system paths this way

chmod 0755 /root/ #yes yes that's bad. just to see if it _can_ work...

echo '/root/glibc-2.34/lib' > /etc/ld.so.conf.d/new_libc.conf

And reloaded

ldconfig

Then… Here is the history of this adventure copy pasted with only minial editing, and adding some of my thoughts along the way.

root@simone:~# mv

Segmentation fault (core dumped)

root@simone:~# ls

Segmentation fault (core dumped)

root@simone:~# LD_LIBRARY_PATH=/usr/lib ls

Segmentation fault (core dumped)

root@simone:~# rm

Segmentation fault (core dumped)

root@simone:~# cd

root@simone:~# cd /etc

root@simone:/etc# ping

Segmentation fault (core dumped)

root@simone:/etc# mv

Segmentation fault (core dumped)

root@simone:/etc# apt

Segmentation fault (core dumped)

root@simone:/etc# ll

Segmentation fault (core dumped)

root@simone:/etc# ln

Segmentation fault (core dumped)

root@simone:/etc# ls

Segmentation fault (core dumped)

root@simone:/etc# chmod

Segmentation fault (core dumped)

root@simone:/etc# peek

 #no error

It seems bash builtins are working…

root@simone:/etc# fopen

root@simone:/etc# fopen ld.so.c

ld.so.cache   ld.so.conf    ld.so.conf.d/

Oh, and tab-completion still works!?

root@simone:/etc# fopen ld.so.conf.d/

fakeroot-x86_64-linux-gnu.conf  new_libc.conf                   

libc.conf                       x86_64-linux-gnu.conf           

root@simone:/etc# fopen ld.so.conf.d/new_libc.conf 

root@simone:/etc# fopen ld.so.conf.d/libc.conf 

root@simone:/etc# cat ld.so.conf.d/libc.conf 

Segmentation fault (core dumped)

Well, still not…

root@simone:/etc# python3

Segmentation fault (core dumped)

root@simone:/etc# cat ld.so.cache 

Segmentation fault (core dumped)

Okay, the library loading is broken. This means I only have the executables currently loaded in RAM, nothing else. At least the (root) shell itself, bash, is part of that.

This means the OS is irrevocably (?) broken and needs a reinstall. Nevertheless, feel free to continue reading…

Also in there is bash path completion, which can do some things (like replace ls for crude tasks) …

root@simone:/etc# echo

root@simone:/etc# echo > /etc/ld.so.conf.d/new_libc.conf

 #no error

Oh, at least I can wipe files… Best would be to remove new_libc.conf actually, but now the problem is wiped away. That is, only after the next ldconfig, but it can’t run!

root@simone:/etc# echo > /etc/ld.so.cache

root@simone:/etc# rm

Segmentation fault (core dumped)

Well, still no. But now I just need to ldconfig again, to rebuild the cache correctly…

Some research:

https://unix.stackexchange.com/questions/272606/locally-installing-glibc-2-23-causes-all-programs-to-segfault

https://askubuntu.com/questions/1353235/how-to-recover-from-a-corrupted-glibc-on-ubuntu-20-04-server

Did some research, stumbled on the ld.so thing: /lib/ld.so can be used to bypass the normal system libraries loader and config (for the given executable in argv). On this system, it happens to be at /lib64/ld-linux-x86-64.so.2 (exactly like here).

Switching to another terminal (the one where glibc-2.34 was built, as it’s one of the few precious still-in-RAM things left) …

root@simone:~/src/glibc-2.34-build# /lib64/ld-linux-x86-64.so.2 --library-path /lib /usr/bin/ldconfig

/usr/bin/ldconfig: error while loading shared libraries: /usr/bin/ldconfig: cannot open shared object file: No such file or directory

root@simone:~/src/glibc-2.34-build# /lib64/ld-linux-x86-64.so.2 --library-path /lib --help

--help: error while loading shared libraries: --help: cannot open shared object file

root@simone:~/src/glibc-2.34-build# /lib64/ld-linux-x86-64.so.2 --library-path /lib

/lib64/ld-linux-x86-64.so.2: missing program name
Try '/lib64/ld-linux-x86-64.so.2 --help' for more information.

root@simone:~/src/glibc-2.34-build# /lib64/ld-linux-x86-64.so.2 --help

Usage: ld.so [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
You have invoked `ld.so', the helper program for shared library executables.
This program usually lives in the file `/lib/ld.so', and special directives
in executable files using ELF shared libraries tell the system's program
loader to load the helper program from this file.  This helper program loads
the shared libraries needed by the program executable, prepares the program
to run, and runs it.  You may invoke this helper program directly from the
command line to load and run an ELF executable file; this is like executing
that file itself, but always uses this helper program from the file you
specified, instead of the helper program file specified in the executable
file you run.  This is mostly of use for maintainers to test new versions
of this helper program; chances are you did not intend to run this program.
    --list                list all dependencies and how they are resolved
    --verify              verify that given object really is a dynamically linked
object we can handle
    --inhibit-cache       Do not use /etc/ld.so.cache
    --library-path PATH   use given PATH instead of content of the environment
variable LD_LIBRARY_PATH
    --inhibit-rpath LIST  ignore RUNPATH and RPATH information in object names
in LIST
    --audit LIST          use objects named in LIST as auditors
    --preload LIST        preload objects named in LIST

root@simone:~/src/glibc-2.34-build# /lib64/ld-linux-x86-64.so.2 --inhibit-cache /usr/ls

/usr/ls: error while loading shared libraries: /usr/ls: cannot open shared object file: No such file or directory

root@simone:~/src/glibc-2.34-build# /lib64/ld-linux-x86-64.so.2 --inhibit-cache /usr/bin/ls

abi-versions.h      elf       libc-modules.stmp  linkobj        rtld-offsets.h.d       tcb-offsets.h

[…]

dummy.o.dt      libc-modules.h   link-defines.h.d   rtld-offsets.h      sysvipc

Tadaa, found an incantation that works!!

root@simone:~/src/glibc-2.34-build# /lib64/ld-linux-x86-64.so.2 --inhibit-cache /usr/bin/bash

root@simone:~/src/glibc-2.34-build# echo $SHELL

/bin/bash

root@simone:~/src/glibc-2.34-build# echo $SHLVL

2

root@simone:~/src/glibc-2.34-build# rm

Segmentation fault (core dumped)

Well, yeah the libraries are borked on the system level, a dirty env wasn’t the problem…

root@simone:~/src/glibc-2.34-build# exit

root@simone:~/src/glibc-2.34-build# /lib64/ld-linux-x86-64.so.2 --inhibit-cache /usr/bin/rm /etc/ld.so.conf.d/new_libc.conf

root@simone:~/src/glibc-2.34-build# echo $?

0

Aha, as expected? It worked !!

root@simone:~/src/glibc-2.34-build#

root@simone:~/src/glibc-2.34-build#

root@simone:~/src/glibc-2.34-build# /lib64/ld-linux-x86-64.so.2 --inhibit-cache /usr/bin/rm /etc/ld.so.cache

root@simone:~/src/glibc-2.34-build#

root@simone:~/src/glibc-2.34-build# /lib64/ld-linux-x86-64.so.2 --inhibit-cache /usr/sbin/ldconfig

Finishing without an error: it should have rebuilt the library cache back correctly (without /etc/ld.so.conf.d/new_libc.conf)

root@simone:~/src/glibc-2.34-build# /lib64/ld-linux-x86-64.so.2 --inhibit-cache /usr/bin/bash
root@simone:~/src/glibc-2.34-build# test

root@simone:~/src/glibc-2.34-build# ls

abi-versions.h            elf                    libc-modules.stmp  linkobj                 rtld-offsets.h.d         tcb-offsets.h

[…]

Aha! yes it works again!

root@simone:~/src/glibc-2.34-build# exit

root@simone:~/src/glibc-2.34-build#