The problem is that sbrk(2) checks each new request against the
machine's total RAM + swap space.  It should check each new request
against total RAM + swap space - memory that has already been allocated
to the current process.

Steps to reproduce:
1. Compile & run the program below on a machine with 32 GiB memory & 24
GiB swap
--- eatmem_bad.c
#include <stdio.h>
#include <unistd.h>

intptr_t bigMem = (intptr_t) 128LL*1024*1024*1024;

int main(int argc,char *argv[]) {
     intptr_t i;
     unsigned long long mem;

     if (argc != 1)
         { (void) fprintf(stderr,"Usage: %s\n", argv[0]); return 2; }

     mem=0;
     for (i = bigMem; i >= 16; i /= 2)
         if (sbrk(i) != (void *) -1)
             mem |= i;

     (void) printf("Memory=%llu MiB\n",mem / (1024*1024));

     return 0;
     }
---

Expected (and correct) result:
Memory=56718 MiB

Actual result:
Memory=65535 MiB