From a22be2cd8ec1e697749dc4c5e33d76200fa7e8db Mon Sep 17 00:00:00 2001 From: Ben Hutchings Date: Thu, 30 Nov 2017 00:29:18 +0000 Subject: [PATCH] mmap: Add an exception to the stack gap for Hotspot JVM compatibility The Hotspot JVM can easily exhaust the default stack, and has a SIGSEGV handler to cope with this by switching to a new stack segment. However, on i386 it creates a single writable and executable page just under the stack limit as a workaround for a bug in Exec Shield. That together with the enlarged stack gap causes the SIGSEGV handler to be triggered when the stack pointer is further away from the stack limit, and it doesn't recognise this as being a stack overflow. This specifically affects programs that use JNI. Hotspot doesn't normally run Java code on the initial thread. Reduce the effective stack guard gap on x86 if the previous vma is a single page allocated as MAP_FIXED. References: https://bugs.debian.org/865303 Signed-off-by: Ben Hutchings Gbp-Pq: Topic bugfix/x86 Gbp-Pq: Name mmap-add-an-exception-to-the-stack-gap-for-hotspot-jvm.patch --- mm/mmap.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index d77f01f5380..6c7cd690205 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2343,6 +2343,16 @@ int expand_downwards(struct vm_area_struct *vma, /* Check that both stack segments have the same anon_vma? */ if (prev && !(prev->vm_flags & VM_GROWSDOWN) && (prev->vm_flags & (VM_WRITE|VM_READ|VM_EXEC))) { + /* + * bwh: Reduce the stack guard gap if this looks like + * Hotspot JVM craziness - see Debian bug #865303 + */ + if (IS_ENABLED(CONFIG_X86) && (prev->vm_flags & VM_FIXED) && + prev->vm_end - prev->vm_start == PAGE_SIZE) { + if (address - prev->vm_end < + min(stack_guard_gap, 4UL << PAGE_SHIFT)) + return -ENOMEM; + } else if (address - prev->vm_end < stack_guard_gap) return -ENOMEM; } -- 2.30.2