74419980 kB<\/span> leaving almost no free memory:<\/p>\negrep \"zzz|Slab|MemAvailable\" \/u00\/oracle\/GI\/gridbase\/oracle.ahf\/data\/repository\/suptools\/host\/oswbb\/grid\/archive\/oswmeminfo\/host.vt.ch_meminfo_22.06.29.1000.dat<\/code><\/pre>\nzzz ***Wed Jun 29 10:24:15 CEST 2022\nMemAvailable: 50734336 kB\nSlab: 2928720 kB<\/span>\nzzz ***Wed Jun 29 10:24:45 CEST 2022\nMemAvailable: 50754784 kB\nSlab: 2928428 kB\nzzz ***Wed Jun 29 10:25:15 CEST 2022\nMemAvailable: 50748708 kB\nSlab: 2928400 kB\nzzz ***Wed Jun 29 10:25:45 CEST 2022\nMemAvailable: 4890828 kB\nSlab: 47753392 kB\nzzz ***Wed Jun 29 10:26:15 CEST 2022\nMemAvailable: 853208 kB\nSlab: 52054036 kB\nzzz ***Wed Jun 29 10:26:45 CEST 2022\nMemAvailable: 767924 kB\nSlab: 52249192 kB\nzzz ***Wed Jun 29 10:27:15 CEST 2022\nMemAvailable: 750048 kB\nSlab: 52442408 kB\nzzz ***Wed Jun 29 10:27:45 CEST 2022\nMemAvailable: 673256 kB\nSlab: 52612572 kB\nzzz ***Wed Jun 29 10:28:15 CEST 2022\nMemAvailable: 692748 kB\nSlab: 53122860 kB\nzzz ***Wed Jun 29 10:28:45 CEST 2022\nMemAvailable: 595100 kB\nSlab: 53553064 kB\nzzz ***Wed Jun 29 10:29:16 CEST 2022\nMemAvailable: 568432 kB\nSlab: 53969576 kB\nzzz ***Wed Jun 29 10:29:46 CEST 2022\nMemAvailable: 572700 kB\nSlab: 54801556 kB\nzzz ***Wed Jun 29 10:30:16 CEST 2022\nMemAvailable: 615584 kB\nSlab: 55338836 kB\nzzz ***Wed Jun 29 10:30:46 CEST 2022\nMemAvailable: 599560 kB\nSlab: 55715568 kB\nzzz ***Wed Jun 29 10:31:16 CEST 2022\nMemAvailable: 555600 kB\nSlab: 55861032 kB\nzzz ***Wed Jun 29 10:31:47 CEST 2022\nMemAvailable: 556200 kB\nSlab: 56051892 kB\nzzz ***Wed Jun 29 10:32:17 CEST 2022\nMemAvailable: 491812 kB\nSlab: 56165600 kB\nzzz ***Wed Jun 29 10:32:48 CEST 2022\nMemAvailable: 468500 kB\nSlab: 56284980 kB\nzzz ***Wed Jun 29 10:33:18 CEST 2022\nMemAvailable: 2326488 kB\nSlab: 74419980 kB<\/span><\/code><\/pre>\nACFS buffer cache<\/h1>\n
slabtop provides the breakdown of the slab allocations:<\/p>\n
slabtop<\/code><\/pre>\nActive \/ Total Objects (% used) : 20455612 \/ 20599104 (99.3%)\nActive \/ Total Slabs (% used) : 2798200 \/ 2798200 (100.0%)\nActive \/ Total Caches (% used) : 127 \/ 184 (69.0%)\nActive \/ Total Size (% used) : 76278221.22K \/ 76313158.69K (100.0%)\nMinimum \/ Average \/ Maximum Object : 0.01K \/ 3.70K \/ 64.01K\n\nOBJS ACTIVE USE OBJ SIZE SLABS OBJ\/SLAB CACHE SIZE NAME\n9176426 9176426 100% 0.28K 164002 56 2624032K acfs_ofsbuf_cache\n9176260 9176260 100% 8.00K 2608768 4 83480576K acfs_buffer_cache_4k<\/span>\n420352 420352 100% 0.01K 821 512 3284K kmalloc-8\n378250 378250 100% 0.02K 2225 170 8900K avtab_node\n298880 294703 98% 0.03K 2335 128 9340K kmalloc-32<\/code><\/pre>\nThe whole memory was consumed by ACFS buffer cache. At this point, it wasn’t clear what caused these extreme memory allocations. So, I started observing the slab growth in real time after restarting the VM:<\/p>\n
while :\ndo\ndate\ngrep Slab \/proc\/meminfo\necho \nsleep 1\ndone<\/code><\/pre>\n\nMon Jul 11 14:36:08 CEST 2022\nSlab: 421856 kB\n\nMon Jul 11 14:36:10 CEST 2022\nSlab: 421888 kB\n\nMon Jul 11 14:36:11 CEST 2022\nSlab: 2733208 kB\n\nMon Jul 11 14:36:12 CEST 2022\nSlab: 7733784 kB\n\nMon Jul 11 14:36:13 CEST 2022\nSlab: 13214688 kB\n\nMon Jul 11 14:36:14 CEST 2022\nSlab: 18859680 kB\n\nMon Jul 11 14:36:15 CEST 2022\nSlab: 24406444 kB\n\nMon Jul 11 14:36:16 CEST 2022\nSlab: 29065932 kB\n\nMon Jul 11 14:36:17 CEST 2022\nSlab: 33006508 kB\n\nMon Jul 11 14:36:18 CEST 2022\nSlab: 37035908 kB\n\nMon Jul 11 14:36:19 CEST 2022\nSlab: 40773812 kB\n\nMon Jul 11 14:36:21 CEST 2022\nSlab: 43771604 kB\n\nMon Jul 11 14:36:22 CEST 2022\nSlab: 46700020 kB\n<\/code><\/pre>\nWhile slab was increasing, I traced kernel memory allocations for a very short time:<\/p>\n
\/usr\/share\/bcc\/tools\/trace -T 't:kmem:kmem_cache_alloc' > trace 2> \/dev\/null<\/code><\/pre>\nA simple trace postprocessing revealed that the most allocations were done by the kernel thread acfsnimap<\/span>:<\/p>\ngrep 14:36:16 trace | awk '{print $4}' | sort | uniq -c | sort -k 1 -n -r<\/code><\/pre>\n\n 50962 acfsnimap<\/span>\n 3872 acfssnap_tmp\n 1577 UsmRdRb-02\n 1288 kworker\/4:1H\n 988 UsmRdRb-00\n 503 UsmRdRb-05\n 450 UsmRdRb-04\n 229 UsmRdRb-01\n 207 UsmRdRb-03\n 202 UsmRdRb-06\n 166 grep\n 128 bash\n 40 crscdpd.bin\n 32 asm_imr0_+asm1\n 26 kauditd\n 18 orarootagent.bi\n 17 auditd\n 12 osysmond.bin<\/code><\/pre>\nBecause of the substring “sn” in the name, I made the assumption that the kernel thread acfsnimap<\/span> is involved in ACFS snapshot handling. Consequently, I inspected the acfs log files.<\/p>\nCorrupt snapshot<\/h1>\n
The last entries in the \/u00\/oracle\/GI\/gridbase\/crsdata\/host\/acfs\/event.log.0 when the issue appeared for the first time were indicated that some snapshots were deleted:<\/p>\n
\n2022-06-29 10:25:32.648 (5064080.797) : acfsutil.bin[1958274] Start deleting snapshot from ACFS file system F8: snapshot name:CDB_2022-06-16_15_39_37_PDB dev_path:\/dev\/asm\/pdb_1-417 mountPoint:\/u00\/oracle\/oradata01\/CDB\/PDB\n2022-06-29 10:25:32.709 (5064080.858) : acfsutil.bin[1958274] Successfully deleted<\/span> snapshot from ACFS file system F8: snapshot name:CDB_2022-06-16_15_39_37_PDB dev_path:\/dev\/asm\/pdb_1-417 mountPoint:\/u00\/oracle\/oradata01\/CDB\/PDB\n2022-06-29 10:26:03.097 (5064111.246) : acfsutil.bin[1958752] Start deleting snapshot from ACFS file system F1: snapshot name:CDB_2022-06-16_15_39_37 dev_path:\/dev\/asm\/cdb_1-417 mountPoint:\/u00\/oracle\/oradata01\/CDB\n2022-06-29 10:26:05.264 (5064113.413) : acfsutil.bin[1958752] Successfully deleted snapshot from ACFS file system F1: snapshot name:CDB_2022-06-16_15_39_37 dev_path:\/dev\/asm\/cdb_1-417 mountPoint:\/u00\/oracle\/oradata01\/CDB\n<\/code><\/pre>\nAccording to the event.log, the snapshot were successfully deleted a long time ago. But the “acfsutil snap info” command was still showing “delete in progress” for one of the deleted snapshots:<\/p>\n
acfsutil snap info \/u00\/oracle\/oradata01\/CDB\/PDB\nsnapshot name: CDB_2022-06-16_15_39_37_PDB (delete in progress<\/span>)\nsnapshot creation time: Thu Jun 16 15:40:01 2022\n\n number of snapshots: 0 (active)\n 1 (delete in progress)\n kilosnap state: Enabled\n storage interest tracking: Enabled\n snapshot space usage: 130131570688 ( 121.19 GB )<\/code><\/pre>\nIn other words, even though the “afsutil snap delete” completed without errors, the snapshot hasn’t been deleted yet. Whenever the file system had been mounted, the pending delete bloated the slab until it consumed the whole memory on the server.<\/p>\n
Another error message in acfs.log also indicated that something’s wrong with the snapshot:<\/p>\n
F2 4303519.747 2022-07-11 14:36:10.644 acfssnap_tmp[80720] OfsSnapStorageThread: FAILED CREATE loaded sfeta for snapId 0x1<\/code><\/pre>\nThis problem isn’t related to the database load – the database wasn’t even running. It’s sufficient to mount the file system to reproduce the issue:<\/p>\n
\nsudo \/bin\/mount -t acfs \/dev\/asm\/cdb_1-417 \/u00\/oracle\/oradata01\/CDB\nsudo \/bin\/mount -t acfs \/dev\/asm\/pdb_1-417 \/u00\/oracle\/oradata01\/CDB\/PDB\n<\/code><\/pre>\nThe allocated memory isn’t returned to the OS until a VM restart – it remains allocated even after unmounting the file system with the corrupt snapshot.<\/p>\n
Recovery<\/h1>\n
I tried to limit the memory for the ACFS resource, but that hasn’t worked:<\/p>\n
\/u00\/oracle\/GI\/product\/21.3.0\/bin\/crsctl modify resource ora.data.pdb_1.acfs -attr WORKLOAD_MEMORY_MAX=1024 -unsupported<\/code><\/pre>\nThe first thing to do after hitting this problem is to identify the culprit. The relevant information is in the acfs and event log files in the \/u00\/oracle\/GI\/gridbase\/crsdata\/host\/acfs directory. Then, you have to prevent the memory leak by disabling the file system and the database with the corrupt snapshot, for example:<\/p>\n
\/u00\/oracle\/GI\/product\/21.3.0\/bin\/crsctl modify resource ora.data.pdb_1.acfs -attr AUTO_START=never -unsupported\n\n\/u00\/oracle\/GI\/product\/21.3.0\/bin\/crsctl modify resource ora.pdb.db -attr AUTO_START=never -unsupported<\/code><\/pre>\nFinally, you have to delete the file system, create a new one and restore the database.<\/p>\n
Unfortunately, I’m still unaware of a way to get rid of a snapshot with a pending delete without dropping the whole file system. The snapshot directory \/u00\/oracle\/oradata01\/CDB\/PDB\/.ACFS\/snaps was already empty.<\/p>\n","protected":false},"excerpt":{"rendered":"
The case study of troubleshooting a serious kernel memory leak issue caused by a failed ACFS snapshot deletion. Continue Reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[53,5,55],"tags":[],"class_list":["post-4292","post","type-post","status-publish","format-standard","hentry","category-ebpf","category-oracle","category-slab"],"yoast_head":"\nKernel Memory Leak due to ACFS Snapshots - All-round Database Topics<\/title>\n<meta name=\"description\" content=\"The case study of troubleshooting a serious kernel memory leak issue caused by a failed ACFS snapshot deletion.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Kernel Memory Leak due to ACFS Snapshots - All-round Database Topics\" \/>\n<meta property=\"og:description\" content=\"The case study of troubleshooting a serious kernel memory leak issue caused by a failed ACFS snapshot deletion.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/\" \/>\n<meta property=\"og:site_name\" content=\"All-round Database Topics\" \/>\n<meta property=\"article:published_time\" content=\"2022-07-12T14:43:39+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-07-12T14:43:42+00:00\" \/>\n<meta name=\"author\" content=\"Nenad Noveljic\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@NenadNoveljic\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Nenad Noveljic\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/acfs-slab\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/acfs-slab\\\/\"},\"author\":{\"name\":\"Nenad Noveljic\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\"},\"headline\":\"Kernel Memory Leak due to ACFS Snapshots\",\"datePublished\":\"2022-07-12T14:43:39+00:00\",\"dateModified\":\"2022-07-12T14:43:42+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/acfs-slab\\\/\"},\"wordCount\":473,\"commentCount\":0,\"articleSection\":[\"eBPF\",\"Oracle\",\"slab\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/acfs-slab\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/acfs-slab\\\/\",\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/acfs-slab\\\/\",\"name\":\"Kernel Memory Leak due to ACFS Snapshots - All-round Database Topics\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#website\"},\"datePublished\":\"2022-07-12T14:43:39+00:00\",\"dateModified\":\"2022-07-12T14:43:42+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\"},\"description\":\"The case study of troubleshooting a serious kernel memory leak issue caused by a failed ACFS snapshot deletion.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/acfs-slab\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/acfs-slab\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/acfs-slab\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Kernel Memory Leak due to ACFS Snapshots\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/\",\"name\":\"All-round Database Topics\",\"description\":\"Nenad Noveljic\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/#\\\/schema\\\/person\\\/51458d9dd86dbbdd19f5add451d44efa\",\"name\":\"Nenad Noveljic\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g\",\"caption\":\"Nenad Noveljic\"},\"sameAs\":[\"nenad-noveljic-9b746a6\",\"https:\\\/\\\/x.com\\\/NenadNoveljic\"],\"url\":\"https:\\\/\\\/nenadnoveljic.com\\\/blog\\\/author\\\/nenad\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Kernel Memory Leak due to ACFS Snapshots - All-round Database Topics","description":"The case study of troubleshooting a serious kernel memory leak issue caused by a failed ACFS snapshot deletion.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/","og_locale":"en_US","og_type":"article","og_title":"Kernel Memory Leak due to ACFS Snapshots - All-round Database Topics","og_description":"The case study of troubleshooting a serious kernel memory leak issue caused by a failed ACFS snapshot deletion.","og_url":"https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/","og_site_name":"All-round Database Topics","article_published_time":"2022-07-12T14:43:39+00:00","article_modified_time":"2022-07-12T14:43:42+00:00","author":"Nenad Noveljic","twitter_card":"summary_large_image","twitter_creator":"@NenadNoveljic","twitter_misc":{"Written by":"Nenad Noveljic","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/#article","isPartOf":{"@id":"https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/"},"author":{"name":"Nenad Noveljic","@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa"},"headline":"Kernel Memory Leak due to ACFS Snapshots","datePublished":"2022-07-12T14:43:39+00:00","dateModified":"2022-07-12T14:43:42+00:00","mainEntityOfPage":{"@id":"https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/"},"wordCount":473,"commentCount":0,"articleSection":["eBPF","Oracle","slab"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/","url":"https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/","name":"Kernel Memory Leak due to ACFS Snapshots - All-round Database Topics","isPartOf":{"@id":"https:\/\/nenadnoveljic.com\/blog\/#website"},"datePublished":"2022-07-12T14:43:39+00:00","dateModified":"2022-07-12T14:43:42+00:00","author":{"@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa"},"description":"The case study of troubleshooting a serious kernel memory leak issue caused by a failed ACFS snapshot deletion.","breadcrumb":{"@id":"https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/nenadnoveljic.com\/blog\/acfs-slab\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/nenadnoveljic.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Kernel Memory Leak due to ACFS Snapshots"}]},{"@type":"WebSite","@id":"https:\/\/nenadnoveljic.com\/blog\/#website","url":"https:\/\/nenadnoveljic.com\/blog\/","name":"All-round Database Topics","description":"Nenad Noveljic","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/nenadnoveljic.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/nenadnoveljic.com\/blog\/#\/schema\/person\/51458d9dd86dbbdd19f5add451d44efa","name":"Nenad Noveljic","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a97b796613ea48ec8a7b79c8ffe1c685dcffc920c68121f6238d5caab5070670?s=96&d=mm&r=g","caption":"Nenad Noveljic"},"sameAs":["nenad-noveljic-9b746a6","https:\/\/x.com\/NenadNoveljic"],"url":"https:\/\/nenadnoveljic.com\/blog\/author\/nenad\/"}]}},"_links":{"self":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/4292","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/comments?post=4292"}],"version-history":[{"count":1,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/4292\/revisions"}],"predecessor-version":[{"id":4295,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/posts\/4292\/revisions\/4295"}],"wp:attachment":[{"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/media?parent=4292"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/categories?post=4292"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nenadnoveljic.com\/blog\/wp-json\/wp\/v2\/tags?post=4292"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}