You were running a program installed in your Linux/Unix system and suddenly that program get stop and it log error in its logs memory segmentation fault error or you might executing a command in Linux / Unix server say you are working on LVM extension or VG scanning but you are unable to do so as you are getting segmentation fault error in output or in syslog log in Linux/Unix system as segfault example of error reporting in syslog log:
Dec 01 11:47:27 localhost kernel: lvs[15909]: segfault at 0000000000000000 rip 0000000000000000 rsp 00007fff8c65c048 error 14
Dec 01 11:49:39 localhost kernel: vgdisplay[16103]: segfault at0000000000000000 rip 0000000000000000 rsp 00007fffb6830258 error 14
Dec 01 11:49:39 localhost kernel: vgscan[16104]: segfault at 0000000000000000 rip 0000000000000000 rsp 00007fff064dcf18 error 14
Dec 01 11:49:40 localhost kernel: pvscan[16105]: segfault at 0000000000000000 rip 0000000000000000 rsp 00007fff685f4d08 error 14
Dec 01 11:49:40 localhost kernel: lvs[16106]: segfault at 0000000000000000 rip 0000000000000000 rsp 00007ffff15b3fd8 error 14
Dec 01 11:49:40 localhost kernel: pvs[16107]: segfault at 0000000000000000 rip 0000000000000000 rsp 00007fff80a0a438 error 14
Dec 01 11:49:40 localhost kernel: vgs[16108]: segfault at 0000000000000000 rip 0000000000000000 rsp 00007fffe3289cc8 error 14
So what is a memory Segmentation and its fault ?
Generally Segmentation is a division of system primary memory into segments or section. It is a unique distinct location for program or Operating system itself in memory space hence each program or Operating system process has to access it’s own distinct segmentation address in memory.
A Segmentation fault is a type of error which occur when a program or command try to access memory location which is not allowed, by doing this program or command violate the memory access restriction hence this error occur. Example: A program or command foo tries to read or write outside the memory that is allocated for it, or to write memory that can only be read by doing this signal SIGSEGV will be generated which means Signal Segment Violation or Segmentation fault.
Image: Kernel Panic trigger in Linux system due to Segmentation Fault
What are the Causes of Segmentation fault ?
The following are some typical causes of a segmentation fault:
- Attempting to access a nonexistent memory address (outside process’s address space)
- Attempting to access memory the program does not have rights to (such as kernel structures in process context)
- Attempting to write read-only memory (such as code segment)
Buggy programs example coded in C or C ++ language can also trigger such errors that result in invalid memory access:
- Dereferencing a null pointer, which usually points to an address that’s not part of the process’s address space
- Dereferencing or assigning to an uninitialized pointer (wild pointer, which points to a random memory address)
- Dereferencing or assigning to a freed pointer (dangling pointer, which points to memory that has been freed/deallocated/deleted)
- A buffer overflow
- A stack overflow
- Attempting to execute a program that does not compile correctly. (Some compilers will output an executable file despite the presence of compile-time errors.)
How this error can be addressed?
- Always try to download and install the program from authenticate source or repository of Linux / Unix.
- Keep track or check the bug report of software and Operating System distributor site if you find bug(s) consider to update or patch the installed software or Operating System itself if update version or patches are released by them. Mostly distributor sites provide bug reporting option for unknown bug, BugZilla is good example of Red Hat Bug tracking and reporting site.
- Consider to enable core dump for program like Apache core dump or for Operating System as when segment fault occur it trigger core dump of memory which capture memory current state during error. Refer: How to enable core dump for RHEL Linux.
- If you are programmer be careful with memory allocations and deletions while developing the program and track down errors as they occur.
- Sometime it is not possible where the program get crashed bug could be there as when program is first accessed in memory a faulty allocation memory in program could trigger crash hence consider to use debugger program for debugging the bug in program. In Linux/Unix command or process which is triggering segment fault error using strace with that command or process id could prove useful in finding root cause. An example below of strace command output of Firefox browser process id (PID) in this example below you can see how strace is tracing Firefox process system call in Linux system.
[root@cj-pers-laptop ~]# pidof firefox
30096 30039 29930 29904 29901
[root@cj-pers-laptop ~]#
[root@cj-pers-laptop ~]# strace -p 29901
strace: Process 29901 attached
restart_syscall(<... resuming interrupted poll ...>) = 1
recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"\16\0kAB\0\240\3\0\0>\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 64
futex(0x7f52a55aa018, FUTEX_WAKE_PRIVATE, 1) = 1
recvmsg(4, 0x7ffec5044be0, 0) = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(4, 0x7ffec5044aa0, 0) = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(4, 0x7ffec5044bd0, 0) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=11, events=POLLIN}, {fd=40, events=POLLIN}, {fd=49, events=POLLIN}, {fd=66, events=POLLIN}], 7, 0) = 1 ([{fd=40, revents=POLLIN}])
read(40, "\372", 1) = 1
recvmsg(4, 0x7ffec5044bd0, 0) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=11, events=POLLIN}, {fd=40, events=POLLIN}, {fd=49, events=POLLIN}, {fd=66, events=POLLIN}], 7, 0) = 0 (Timeout)
recvmsg(4, 0x7ffec5044bd0, 0) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7f52a55aa018, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f52a55b10a0, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=11, events=POLLIN}, {fd=40, events=POLLIN}, {fd=49, events=POLLIN}, {fd=66, events=POLLIN}], 7, 0) = 0 (Timeout)
futex(0x7f52a55aa018, FUTEX_WAIT_PRIVATE, 2, NULL) = 0
futex(0x7f52a55aa018, FUTEX_WAKE_PRIVATE, 1) = 0
recvmsg(4, {msg_name(0)=NULL, msg_iov(1)=[{"\16\0nAT\311\240\3\0\0>\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 64
futex(0x7f52a5600018, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7f52a5600018, FUTEX_WAKE_PRIVATE, 1) = 0
recvmsg(4, 0x7ffec5044bd0, 0) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=11, events=POLLIN}, {fd=40, events=POLLIN}, {fd=49, events=POLLIN}, {fd=66, events=POLLIN}], 7, 0) = 0 (Timeout)
recvmsg(4, 0x7ffec5044aa0, 0) = -1 EAGAIN (Resource temporarily unavailable)
recvmsg(4, 0x7ffec5044bd0, 0) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=11, events=POLLIN}, {fd=40, events=POLLIN}, {fd=49, events=POLLIN}, {fd=66, events=POLLIN}], 7, 0) = 1 ([{fd=40, revents=POLLIN}])
read(40, "\372", 1) = 1
recvmsg(4, 0x7ffec5044bd0, 0) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=11, events=POLLIN}, {fd=40, events=POLLIN}, {fd=49, events=POLLIN}, {fd=66, events=POLLIN}], 7, 0) = 0 (Timeout)
recvmsg(4, 0x7ffec5044bd0, 0) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=11, events=POLLIN}, {fd=40, events=POLLIN}, {fd=49, events=POLLIN}, {fd=66, events=POLLIN}], 7, 0) = 0 (Timeout)
recvmsg(4, 0x7ffec5044bd0, 0) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=11, events=POLLIN}, {fd=40, events=POLLIN}, {fd=49, events=POLLIN}, {fd=66, events=POLLIN}], 7, 0) = 0 (Timeout)
recvmsg(4, 0x7ffec5044bd0, 0) = -1 EAGAIN (Resource temporarily unavailable)
... Output Truncate
- Sometime faulty hardware memory or driver could cause the problem hence consider to replace the faulty memory or update the driver available on hardware vendor site.
Reference Sources:
https://en.wikipedia.org/wiki/Segmentation_fault
https://www.cyberciti.biz/tips/segmentation-fault-on-linux-unix.html
https://smallbusiness.chron.com/segmentation-fault-linux-27699.html
http://www.gnu.org/software/libc/manual/html_node/Program-Error-Signals.html
Charanjit is currently working as a Cloud Architect at Mphasis, with 17 years of experience in IT infrastructure projects, implementation, and support. While his main role is as a DevOps engineer, he holds a Cloud Architect position and has strong skills in cloud technologies and automation. His expertise includes Terraform, Ansible, AWS, Azure DevOps, Azure Cloud, VMware, and Linux systems.
Charanjit is passionate about automating tasks and improving processes. He uses tools like Terraform and Azure DevOps to build and manage cloud infrastructure and streamline deployment. He also enjoys using Shell scripts and Ansible playbooks to make systems run more efficiently.
In his free time, Charanjit enjoys learning about new technologies and sharing his knowledge through his blog. When he’s not working, he likes listening to music, having a cup of coffee, and relaxing in nature.
You can connect with Charanjit on Twitter, Facebook, LinkedIn, or email him at charanjit.cheema@cjcheema.com.
Charanjit is passionate about automating tasks and improving processes. He uses tools like Terraform and Azure DevOps to build and manage cloud infrastructure and streamline deployment. He also enjoys using Shell scripts and Ansible playbooks to make systems run more efficiently.
In his free time, Charanjit enjoys learning about new technologies and sharing his knowledge through his blog. When he’s not working, he likes listening to music, having a cup of coffee, and relaxing in nature.
You can connect with Charanjit on Twitter, Facebook, LinkedIn, or email him at charanjit.cheema@cjcheema.com.
Latest posts by Charanjit Cheema (see all)