{"id":4068,"date":"2014-12-10T00:48:51","date_gmt":"2014-12-09T16:48:51","guid":{"rendered":"http:\/\/rmohan.com\/?p=4068"},"modified":"2014-12-10T00:49:40","modified_gmt":"2014-12-09T16:49:40","slug":"kernel-crash-reportcrash-dump-analysis","status":"publish","type":"post","link":"https:\/\/mohan.sg\/?p=4068","title":{"rendered":"Kernel Crash Report\/Crash Dump Analysis"},"content":{"rendered":"

Kernel Crash Report\/Crash Dump Analysis<\/h3>\n
<\/div>\n
\n
\n
In my previous post, we have configured how to capture kernel dump for reference click on the link\u00a0kernel crash dump<\/a><\/div>\n
<\/div>\n
\n
Here in this article,we master the basic usage of crash utility to open the dumped memory core and process the information\u00a0contained therein and to intercept the output.<\/div>\n

\"Fundamentals-Complete-Memory-Dump-Analysis-Logo\"<\/a><\/p>\n<\/div>\n

<\/div>\n
<\/div>\n
<\/div>\n
\n
find out the dumped kernel location and start analyzing the code.<\/div>\n
#crash \/usr\/lib\/debug\/lib\/modules\/2.6.32-279.el6.i686\/vmlinux \/var\/crash\/127.0.0.1-2013-07-18-09\\:40\\:28\/vmcore<\/div>\n
<\/div>\n
\n
KERNEL: \/usr\/lib\/debug\/lib\/modules\/2.6.32-279.el6.i686\/vmlinux<\/div>\n
DUMPFILE: \/var\/crash\/127.0.0.1-2013-07-18-09:40:28\/vmcore \u00a0[PARTIAL DUMP]<\/div>\n
CPUS: 1<\/div>\n
DATE: Thu Jul 18 09:40:21 2013<\/div>\n
UPTIME: 00:37:21<\/div>\n
LOAD AVERAGE: 934.79, 206.96, 67.74<\/div>\n
TASKS: 5494<\/div>\n
NODENAME: <hostname><\/div>\n
RELEASE: 2.6.32-279.el6.i686<\/div>\n
VERSION: #1 SMP Fri Jun 22 10:59:55 UTC 2012<\/div>\n
MACHINE: i686 \u00a0(2933 Mhz)<\/div>\n
MEMORY: 895.6 MB<\/div>\n
PANIC: “Oops: 0002 [#1] SMP ” (check log for details)<\/div>\n
PID: 6847<\/div>\n
COMMAND: “bash”<\/div>\n
TASK: ea142aa0 \u00a0[THREAD_INFO: db8ae000]<\/div>\n
CPU: 0<\/div>\n
STATE: TASK_RUNNING (PANIC)<\/div>\n<\/div>\n
<\/div>\n
Explanation of code is as below :-<\/div>\n
<\/div>\n
\n
KERNEL:<\/b> specifies the kernel running at the time of the crash.<\/div>\n
DUMPFILE:<\/b> is the name of the dumped memory core.<\/div>\n
CPUS:<\/b> is the number of CPUs on your machine.<\/div>\n
DATE:<\/b> specifies the time of the crash.<\/div>\n
TASKS:<\/b> indicates the number of tasks in the memory at the time of the crash. Task is a set of program instructions loaded into memory.<\/div>\n
NODENAME:<\/b> is the name of the crashed host.<\/div>\n
RELEASE:<\/b> and VERSION:<\/b> specify the kernel release and version.<\/div>\n
MACHINE:<\/b> specifies the architecture of the CPU.<\/div>\n
MEMORY:<\/b> is the size of the physical memory on the crashed machine.<\/div>\n
PANIC:<\/b> specifies what kind of crash occurred on the machine.<\/div>\n<\/div>\n
<\/div>\n
Panic refers to the use of magic keys(SysRq), which we\u00a0deliberately trigger for a crash.<\/div>\n
<\/div>\n
\n
SysRq (System Request)<\/b> refers to Magic Keys<\/b>, which allow you to send instructions directly to the kernel. They can be invoked using a keyboard sequence or by echoing letter commands to \/proc\/sysrq-trigger<\/b>, provided the functionality is enabled. We have discussed this in the Kdump part.<\/div>\n<\/div>\n
<\/div>\n
I attacked system by Denial of service(DoS) using the system to consume all its resources by forking.\u00a0forkbomb<\/a><\/div>\n
<\/div>\n
if you have looked load average of the crashed kernel its too high(934.79, 206.96, 67.74<\/b>) and the process responsible was PID:<\/b>6847<\/b><\/div>\n
\u00a0<\/b><\/div>\n
PANIC: “Oops: 0002 [#1] SMP ” has the value below.
\n<\/b><\/div>\n
<\/div>\n
\n
\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 value<\/b><\/div>\n
—————————————————————–<\/div>\n
Bit 0 1<\/b><\/div>\n
——————————————————————<\/b><\/div>\n
0<\/b> No page found Invalid access<\/div>\n
1<\/b> Read or Execute Write<\/div>\n
2<\/b> Kernel mode User mode<\/div>\n
3<\/b> Not instruction fetch Instruction fetch<\/div>\n
<\/div>\n
from our PANIC analysis it is clear that “we have a page not found during a write operation in Kernel mode; the fault was not an Instruction Fetch.<\/b>“<\/div>\n<\/div>\n
\u00a0<\/b><\/div>\n
we have used “\/proc\/sysrq-trigger” from the command line to dump our kernel in previous post, but if your system is unresponsive then you would be unable to trigger. In such cases we enable SysRq feature<\/b> so that we could use magic keys to collect the dump of the crashed kernel.<\/div>\n
<\/div>\n
\n
#echo “1” > \/proc\/sys\/kernel\/sysrq<\/div>\n
or add an entry to \/etc\/sysctl.conf \u00a0<\/b><\/div>\n
<\/div>\n
#vim\u00a0\/etc\/sysctl.conf \u00a0<\/b><\/div>\n
kernel.sysrq = 1<\/div>\n<\/div>\n
<\/div>\n
Once configured, you will be able to use magic keys [ alt + PrintScreenSysRq + <options> ]<\/div>\n
<\/div>\n
Options<\/b> as are below.<\/div>\n
<\/div>\n
\n
‘b’ \u00a0– Will immediately reboot the system without syncing or unmounting your disks.<\/div>\n
‘c’ \u00a0– Will perform a system crash by a NULL pointer\u00a0deference\u00a0 A crash dump will be taken if configured.<\/div>\n
‘d’ \u00a0– Shows all locks that are held.<\/div>\n
‘e’ \u00a0– Send a SIGTERM to all processes, except for init.<\/div>\n
‘f’\u00a0 – Will call oom_kill to kill a memory hog process.<\/div>\n
‘g’\u00a0 – Used by kgdb (kernel debugger)<\/div>\n
‘h’ \u00a0– Will display help<\/div>\n
‘i’ \u00a0– Send a SIGKILL to all processes, except for init.<\/div>\n
‘j’ \u00a0– Forcibly “Just thaw it” – filesystems frozen by the FIFREEZE ioctl.<\/div>\n
‘k’ \u00a0– Secure Access Key (SAK) Kills all programs on the current virtual console.<\/div>\n
‘l’ \u00a0 – Shows a stack backtrace for all active CPUs.<\/div>\n
‘m’ – Will dump current memory info to your console.<\/div>\n
‘n’ \u00a0– Used to make RT tasks nice-able<\/div>\n
‘o’ \u00a0– Will shut your system off (if configured and supported).<\/div>\n
‘p’ \u00a0– Will dump the current registers and flags to your console.<\/div>\n
‘q’ \u00a0– Will dump per CPU lists of all armed hrtimers (but NOT regular timer_list timers) and detailed information about all clockevent devices.<\/div>\n
‘r’ \u00a0– Turns off keyboard raw mode and sets it to XLATE.<\/div>\n
‘s’ \u00a0– Will attempt to sync all mounted filesystems.<\/div>\n
‘t’ \u00a0– Will dump a list of current tasks and their information to your console.<\/div>\n
‘u’ – Will attempt to remount all mounted filesystems read-only.<\/div>\n
‘v’\u00a0– Forcefully restores framebuffer console<\/div>\n
‘v’\u00a0– Causes ETM buffer dump [ARM-specific]<\/div>\n
‘w’\u00a0– Dumps tasks that are in uninterruptable (blocked) state.<\/div>\n
‘x’\u00a0– Used by xmon interface on ppc\/powerpc platforms. Show global PMU Registers on sparc64.<\/div>\n
‘y’\u00a0– Show global CPU Registers [SPARC-64 specific]<\/div>\n
‘z’\u00a0– Dump the ftrace buffer<\/div>\n
<\/div>\n
[ Alt + SysRq + c ] – Crash collected by rebooting the system.<\/div>\n
\"kernel-crash\"<\/a><\/div>\n
<\/div>\n
<\/div>\n
\n
We have almost\u00a0analyzed upto certain extent, however there are few of the commands which can help us in understanding more.<\/div>\n
Let us look few more basic commands which can be helpful.<\/div>\n
<\/div>\n
crash> help<\/div>\n
<\/div>\n
* \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0files \u00a0 \u00a0 \u00a0 \u00a0 \u00a0mach \u00a0 \u00a0 \u00a0 repeat \u00a0 \u00a0 \u00a0timer<\/div>\n
alias \u00a0 \u00a0 \u00a0 \u00a0 foreach \u00a0 \u00a0 mod \u00a0 \u00a0 \u00a0 \u00a0runq \u00a0 \u00a0 \u00a0 \u00a0 \u00a0tree<\/div>\n
ascii \u00a0 \u00a0 \u00a0 \u00a0 fuser \u00a0 \u00a0 \u00a0 \u00a0 mount \u00a0 \u00a0 search \u00a0 \u00a0 \u00a0 union<\/div>\n
bt \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 gdb \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 net \u00a0 \u00a0 \u00a0 \u00a0 set \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 vm<\/div>\n
btop \u00a0 \u00a0 \u00a0 \u00a0 \u00a0help \u00a0 \u00a0 \u00a0 \u00a0 \u00a0p \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0sig \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 vtop<\/div>\n
dev \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 ipcs \u00a0 \u00a0 \u00a0 \u00a0 \u00a0ps \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 struct \u00a0 \u00a0 \u00a0 \u00a0 waitq<\/div>\n
dis \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0irq \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0pte \u00a0 \u00a0 \u00a0 \u00a0 swap \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 whatis<\/div>\n
eval \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 kmem \u00a0 \u00a0 \u00a0ptob \u00a0 \u00a0 \u00a0 \u00a0sym \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0wr<\/div>\n
exit \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 list \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 ptov \u00a0 \u00a0 \u00a0 \u00a0sys \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0q<\/div>\n
extend \u00a0 \u00a0 \u00a0 log \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 rd \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 task<\/div>\n
<\/div>\n
bt – backtrace –\u00a0Display a kernel stack backtrace :=<\/b><\/div>\n
The sequence of numbered lines, starting with the hash sign (#) is the call trace. It’s a list of kernel functions executed just prior to the crash. This gives us a good indication<\/p>\n
of what happened before the system went down.<\/div>\n<\/div>\n
\n
\u00a0<\/b><\/div>\n
crash> bt<\/b><\/div>\n<\/div>\n
\n
PID: 6847 \u00a0 TASK: ea142aa0 \u00a0CPU: 0 \u00a0 COMMAND: “bash”<\/div>\n
\u00a0#0 [db8af618] crash_kexec at c049b75c<\/div>\n
\u00a0#1 [db8af66c] oops_end at c083fe92<\/div>\n
.<\/div>\n
.<\/div>\n
(output omitted …)<\/div>\n
<\/div>\n<\/div>\n
foreach – display command data for multiple tasks in the system :=<\/b><\/div>\n
This command allows for a an examination of various kernel data associated\u00a0with any, or all, tasks in the system, without having to set the context<\/p>\n
\u00a0to each targeted task.<\/div>\n<\/div>\n
\n
<\/div>\n
crash>foreach bt<\/b><\/div>\n
.<\/b><\/div>\n
.<\/b><\/div>\n
(output omitted..)<\/div>\n
<\/div>\n
log – dump system message buffer :=<\/b><\/div>\n
\u00a0<\/b><\/div>\n
\n
The log command dumps \u00a0the kernel log buffer contents \u00a0inchronological order . This \u00a0is \u00a0similar to \u00a0what \u00a0you would see when you type dmesg<\/b> on a running machine. This is useful when you want to look at the panic or oops message. An oops is triggered by some exception. It is a dump of the CPU register’s state and kernel stack at that instant . From the panic message, we can find hints as to how the panic was triggered (e. g. the function or process \u00a0or pid or command or address that triggered the panic), the register’s information, kernel module list, whether the kernel is<\/p>\n
tainted with proprietary kernel modules loaded, and so \u00a0on..<\/div>\n<\/div>\n
\n
<\/div>\n<\/div>\n
\n
crash>log<\/b><\/div>\n<\/div>\n
(output omitted..)<\/div>\n
.<\/div>\n
.<\/div>\n
SysRq : Trigger a crash<\/div>\n
\n
\n
BUG: unable to handle kernel NULL pointer dereference at (null)<\/div>\n
IP: [<c06a0d8f>] sysrq_handle_crash+0xf\/0x20<\/div>\n
*pdpt = 00000000116c3001 *pde = 0000000000000000<\/div>\n
Oops: 0002 [#1] SMP<\/div>\n<\/div>\n<\/div>\n
\n
.<\/div>\n<\/div>\n
\n
.<\/div>\n<\/div>\n
\n
\n
Pid: 6847, comm: bash Not tainted<\/b> 2.6.32-279.el6.i686 #1 innotek GmbH\u00a0<\/b><\/div>\n
EIP: 0060:[<c06a0d8f>] EFLAGS: 00010096 CPU: 0<\/div>\n
EIP is at sysrq_handle_crash+0xf\/0x20<\/div>\n
.<\/div>\n<\/div>\n<\/div>\n
\n
.<\/div>\n<\/div>\n
\n
(output omitted..)<\/div>\n<\/div>\n
\n
<\/div>\n<\/div>\n
\n
we don’t observe any tainted flags on the kernel, each flag has its own meaning.<\/div>\n<\/div>\n
\n
<\/div>\n<\/div>\n
\n
\n
P \u2014 Proprietary module has been loaded.<\/div>\n
F \u00a0\u2014 Module has been forcibly loaded.<\/div>\n
S \u2014 SMP with a CPU not \u00a0designed for SMP .<\/div>\n
R \u2014 User forced a module unload.<\/div>\n
M \u2014 System experienced a machine check exception.<\/div>\n
B \u2014 System has \u00a0hit \u00a0bad_page.<\/div>\n
U \u2014 Users pace- defined naughtiness .<\/div>\n
A \u2014 \u00a0ACPI \u00a0table over ridden.<\/div>\n
W \u2014 Taint on warning.<\/div>\n
<\/div>\n
ps – Display process status :=<\/b><\/div>\n
Display process status information This command displays process status for selected, or all, processes in the system. If no arguments are entered, the process data is displayed for all processes.\u00a0The active task is marked with “>”<\/div>\n
\n
<\/div>\n
crash>ps<\/div>\n<\/div>\n
\n
<\/div>\n
\u00a0 \u00a06846 \u00a0 5711 \u00a0 0 \u00a0d169d550 \u00a0UN \u00a0 0.1 \u00a0 \u00a05992 \u00a0 1112 \u00a0bash<\/div>\n
> \u00a06847 \u00a0 \u00a0 \u00a01 \u00a0 0 \u00a0ea142aa0 \u00a0RU \u00a0 0.1 \u00a0 \u00a05992 \u00a0 1320 \u00a0bash<\/div>\n
\u00a0 \u00a06848 \u00a0 \u00a0 \u00a01 \u00a0 0 \u00a0d169d000 \u00a0UN \u00a0 0.1 \u00a0 \u00a05992 \u00a0 1368 \u00a0bash<\/div>\n
<\/div>\n
when you observe from the log, there are number of bash instances. Hence total number of bash instances are as below<\/div>\n
<\/div>\n
\n
crash> ps | fgrep bash | wc -l<\/div>\n
5363<\/div>\n
crash><\/div>\n
<\/div>\n
\u00a0vm – virtual memory :=<\/b><\/div>\n
This command displays basic virtual memory information of a context, consisting of a pointer to its mm_struct and page dirctory, its RSS and total virtual memory size; and a list of pointers to each vm_area_struct<\/b>, its starting and ending address, vm_flags value<\/b>, and file path name.<\/div>\n
\u00a0<\/b><\/div>\n
\n
crash> vm<\/div>\n
PID: 6847 \u00a0 TASK: ea142aa0 \u00a0CPU: 0 \u00a0 COMMAND: “bash”<\/div>\n
\u00a0 \u00a0MM \u00a0 \u00a0 \u00a0 PGD \u00a0 \u00a0 \u00a0RSS \u00a0 \u00a0TOTAL_VM<\/div>\n
e247f740 \u00a0d16c2000 \u00a01320k \u00a0 \u00a05992k<\/div>\n
\u00a0 VMA \u00a0 \u00a0 \u00a0 START \u00a0 \u00a0 \u00a0END \u00a0 \u00a0FLAGS \u00a0FILE<\/div>\n
d16b66f8 \u00a0 \u00a070d000 \u00a0 \u00a070e000 4040075<\/div>\n
d16b6694 \u00a0 \u00a0760000 \u00a0 \u00a077e000 8000875 \u00a0\/lib\/ld-2.12.so<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
.<\/b><\/div>\n
.<\/b><\/div>\n
(output omitted..)
\n<\/b><\/div>\n
\u00a0<\/b><\/div>\n
files – open files :=<\/b><\/div>\n
This command displays information about open files of a context. It prints the context’s current root directory and current working directory, and then for each open file descriptor it prints a pointer to its file struct, a pointer to its dentry struct, a pointer to the inode, the file type, and the pathname.<\/div>\n
<\/div>\n
\n
crash> files 6427<\/div>\n
PID: 6427 \u00a0 TASK: d4150aa0 \u00a0CPU: 0 \u00a0 COMMAND: “bash”<\/div>\n
ROOT: \/ \u00a0 \u00a0CWD: \/var\/crash<\/div>\n
\u00a0FD \u00a0 \u00a0FILE \u00a0 \u00a0 DENTRY \u00a0 \u00a0INODE \u00a0 \u00a0TYPE \u00a0PATH<\/div>\n
\u00a0 0 \u00a0f6698240 \u00a0f3609df8 \u00a0f3610e48 \u00a0FIFO<\/div>\n
\u00a0 1 \u00a0d40791c0 \u00a0f377c8d0 \u00a0f377d1a8 \u00a0FIFO<\/div>\n<\/div>\n
.<\/div>\n
.<\/div>\n
(output omitted..)<\/div>\n
<\/div>\n
runq – run queue :=<\/b><\/div>\n
\n
This command displays the tasks on the run queues<\/p>\n
of each cpu.<\/div>\n<\/div>\n
<\/div>\n<\/div>\n
\n
\n
crash> runq<\/div>\n
CPU 0 RUNQUEUE: c6408680<\/div>\n
\u00a0 CURRENT: PID: 6847 \u00a0 TASK: ea142aa0 \u00a0COMMAND: “bash”<\/div>\n
\u00a0 RT PRIO_ARRAY: c6408778<\/div>\n
\u00a0 \u00a0 \u00a0[no tasks queued]<\/div>\n
\u00a0 CFS RB_ROOT: c64086dc<\/div>\n
\u00a0 \u00a0 \u00a0[120] PID: 7990 \u00a0 TASK: c9bb1000 \u00a0COMMAND: “bash”<\/div>\n
\u00a0 \u00a0 \u00a0[120] PID: 8616 \u00a0 TASK: c13f8aa0 \u00a0COMMAND: “bash”<\/div>\n
\u00a0 \u00a0 \u00a0[120] PID: 7714 \u00a0 TASK: f2485550 \u00a0COMMAND: “bash”<\/div>\n<\/div>\n<\/div>\n
\n
.<\/div>\n
.<\/div>\n
(output omitted..)<\/div>\n<\/div>\n
<\/div>\n
\n
timer – timer queue data :=<\/b><\/div>\n<\/div>\n
Displays the timer queue entries in chronological order, listing the target function names, the current value of jiffies, and the expiration time of each entry.<\/div>\n
<\/div>\n
TVEC_BASES[0]: c0be66a0<\/div>\n
\n
\u00a0JIFFIES<\/div>\n
\u00a01941918<\/div>\n
\u00a0EXPIRES \u00a0TIMER_LIST \u00a0FUNCTION<\/div>\n
\u00a01941920 \u00a0 d4aa1b74 \u00a0 c0466210 \u00a0<process_timeout><\/div>\n
\u00a01941920 \u00a0 d847fb74 \u00a0 c0466210 \u00a0<process_timeout><\/div>\n<\/div>\n
\n
.<\/div>\n
.<\/div>\n
(output omitted..)<\/div>\n<\/div>\n
<\/div>\n
\n
net – network command :=<\/b><\/div>\n<\/div>\n
Display various network related data<\/div>\n
<\/div>\n
\n
crash> net<\/div>\n
NET_DEVICE \u00a0NAME \u00a0 IP ADDRESS(ES)<\/div>\n
\u00a0f70e6820 \u00a0 lo \u00a0 \u00a0 127.0.0.1<\/div>\n
\u00a0f4f32020 \u00a0 eth0 \u00a0 <ipaddress-2><\/div>\n
\u00a0c171e020 \u00a0 eth1 \u00a0 <ipaddress-1><\/div>\n
crash><\/div>\n<\/div>\n
<\/div>\n
we have found some of the basics of kernel dump analysis which might be helpful in knowing what went behind the kernel to crash the system. As a best practice we need to analyze the dump and take necessary actions to avoid the re-occurrences.<\/div>\n
<\/div>\n
“there are lot of administrators who don’t care rebooting server, but need server online”.<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Kernel Crash Report\/Crash Dump Analysis In my previous post, we have configured how to capture kernel dump for reference click on the link kernel crash dump Here in this article,we master the basic usage of crash utility to open the dumped memory core and process the information contained therein and to intercept the output. <\/p>\n

[…]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,4,50],"tags":[],"_links":{"self":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/4068"}],"collection":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4068"}],"version-history":[{"count":2,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/4068\/revisions"}],"predecessor-version":[{"id":4072,"href":"https:\/\/mohan.sg\/index.php?rest_route=\/wp\/v2\/posts\/4068\/revisions\/4072"}],"wp:attachment":[{"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4068"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4068"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mohan.sg\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4068"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}