Swap space size in C

Discussion in 'C Programming' started by Otto Meijer, Dec 12, 2003.

  1. Otto Meijer

    Otto Meijer Guest

    Hi everyone,

    for one of my projects I need to figure out the size of the
    swap file(s) of a certain system. The problem is I need to do this
    on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
    OpenVMS and Windows.

    This seems much more difficult than I thought origionaly.
    Some platforms don't seem to have any API to query this information.

    Any hints how to tack this would be very welcome

    Otto
    Otto Meijer, Dec 12, 2003
    #1
    1. Advertising

  2. Otto Meijer

    Grumble Guest

    Otto Meijer wrote:
    > for one of my projects I need to figure out the size of the
    > swap file(s) of a certain system. The problem is I need to do this
    > on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
    > OpenVMS and Windows.
    >
    > This seems much more difficult than I thought origionaly.
    > Some platforms don't seem to have any API to query this information.
    >
    > Any hints how to tack this would be very welcome


    AFAICT, your best bet is the Portable Operating System Interface for
    UNIX, aka POSIX.
    Grumble, Dec 12, 2003
    #2
    1. Advertising

  3. Otto Meijer

    Dan Pop Guest

    Re: Swap space size in C

    In <> (Otto Meijer) writes:

    >for one of my projects I need to figure out the size of the
    >swap file(s) of a certain system. The problem is I need to do this
    >on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
    >OpenVMS and Windows.
    >
    >This seems much more difficult than I thought origionaly.
    >Some platforms don't seem to have any API to query this information.
    >
    >Any hints how to tack this would be very welcome


    On a platform by platform basis. Post the question to newsgroups
    dedicated to programming on each of the platforms of interest to you.
    If you get the desired information, write a function for each platform.

    Dan
    --
    Dan Pop
    DESY Zeuthen, RZ group
    Email:
    Dan Pop, Dec 12, 2003
    #3
  4. Otto Meijer

    Dan Pop Guest

    Re: Swap space size in C

    In <brc3oe$a89$> Grumble <> writes:

    >Otto Meijer wrote:
    >> for one of my projects I need to figure out the size of the
    >> swap file(s) of a certain system. The problem is I need to do this
    >> on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
    >> OpenVMS and Windows.
    >>
    >> This seems much more difficult than I thought origionaly.
    >> Some platforms don't seem to have any API to query this information.
    >>
    >> Any hints how to tack this would be very welcome

    >
    >AFAICT, your best bet is the Portable Operating System Interface for
    >UNIX, aka POSIX.


    Which, AFAICT, blissfully ignores the issue.

    Dan
    --
    Dan Pop
    DESY Zeuthen, RZ group
    Email:
    Dan Pop, Dec 12, 2003
    #4
  5. Otto Meijer

    CBFalconer Guest

    Otto Meijer wrote:
    >
    > for one of my projects I need to figure out the size of the
    > swap file(s) of a certain system. The problem is I need to do
    > this on a host of platforms like: HP_UX, Solaris, Linux, AIX,
    > OSF, OpenVMS and Windows.
    >
    > This seems much more difficult than I thought origionaly. Some
    > platforms don't seem to have any API to query this information.
    >
    > Any hints how to tack this would be very welcome


    Yes. Ask on newsgroups that deal with HP_UX, Solaris, Linux, AIX,
    OSF, OpenVMS, and Windows. This group deals with ISO standard
    portable C only. If it isn't in the C standard, it doesn't exist
    (here).

    --
    Chuck F () ()
    Available for consulting/temporary embedded and systems.
    <http://cbfalconer.home.att.net> USE worldnet address!
    CBFalconer, Dec 12, 2003
    #5
  6. Otto Meijer

    Grumble Guest

    Re: Swap space size in C

    Dan Pop wrote:

    > Grumble wrote:
    >
    >> AFAICT, your best bet is the Portable Operating System Interface
    >> for UNIX, aka POSIX.

    >
    > Which, AFAICT, blissfully ignores the issue.


    Then, it seems, I could not tell far enough ;-)
    Grumble, Dec 12, 2003
    #6
  7. (Otto Meijer) writes:
    > Hi everyone,
    >
    > for one of my projects I need to figure out the size of the
    > swap file(s) of a certain system. The problem is I need to do this
    > on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
    > OpenVMS and Windows.
    >
    > This seems much more difficult than I thought origionaly.
    > Some platforms don't seem to have any API to query this information.
    >
    > Any hints how to tack this would be very welcome
    >
    > Otto


    As others have mentioned, this is extremely platform-specific. It may
    not even be possible on some platforms. Some platforms may not have a
    concept of a "swap file" (it could be a swap partition, for example);
    others could have a "swap file" whose size can vary over time, or that
    doesn't have a well-defined size at all.

    You might want to reconsider whether knowing the "size" of the "swap
    file" is even useful. What are you going to do with the information?
    (That's a rhetorical question; any answer is probably off-topic here.)

    --
    Keith Thompson (The_Other_Keith) <http://www.ghoti.net/~kst>
    San Diego Supercomputer Center <*> <http://www.sdsc.edu/~kst>
    Schroedinger does Shakespeare: "To be *and* not to be"
    (Note new e-mail address)
    Keith Thompson, Dec 12, 2003
    #7
  8. Otto Meijer wrote:


    > for one of my projects I need to figure out the size of the
    > swap file(s) of a certain system. The problem is I need to do this
    > on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
    > OpenVMS and Windows.


    I have done it with an exponentially decreasing malloc() loop,

    for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;

    There is no guarantee that it gives meaningful results
    on any system, and may crash the system.

    I did once crash a unix system with a malloc() loop, but
    not with this one.

    The i-8 is a guess of the malloc() overhead, which may or may
    not be related to any real malloc() overhead.

    -- glen
    glen herrmannsfeldt, Dec 13, 2003
    #8
  9. Otto Meijer

    Dan Pop Guest

    Re: Swap space size in C

    In <tOBCb.102332$_M.539115@attbi_s54> glen herrmannsfeldt <> writes:

    >Otto Meijer wrote:
    >
    >
    >> for one of my projects I need to figure out the size of the
    >> swap file(s) of a certain system. The problem is I need to do this
    >> on a host of platforms like: HP_UX, Solaris, Linux, AIX, OSF,
    >> OpenVMS and Windows.

    >
    >I have done it with an exponentially decreasing malloc() loop,
    >
    >for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;
    >
    >There is no guarantee that it gives meaningful results
    >on any system, and may crash the system.


    In particular, it's useless on systems with lazy swap space allocation.

    Dan
    --
    Dan Pop
    DESY Zeuthen, RZ group
    Email:
    Dan Pop, Dec 15, 2003
    #9
  10. Re: Swap space size in C

    Dan Pop wrote:

    (snip on determining swap space)

    >>for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;


    >>There is no guarantee that it gives meaningful results
    >>on any system, and may crash the system.


    > In particular, it's useless on systems with lazy swap space allocation.


    It might still tell you the maximum for one task...

    My favorite lazy swap story was someone trying to determine how fast the
    memory system was, something like:

    int i;
    char *a,*b;
    a=malloc(SIZE);
    b=malloc(SIZE);
    for(i=0;i<1000000;i++) memcpy(a,b,SIZE);

    assuming that SIZE was enough bigger than the cache, so that it would
    have to go out to memory.

    It turned out that in a lazy swap allocation, all the allocated pages
    referenced the same physical page, which was smaller than the cache,
    so that it only tested the cache.

    -- glen
    glen herrmannsfeldt, Dec 16, 2003
    #10
  11. Otto Meijer

    Otto Meijer Guest

    Re: Swap space size in C

    glen herrmannsfeldt <> wrote in message news:<C0sDb.401963$ao4.1321249@attbi_s51>...
    > Dan Pop wrote:
    >
    > (snip on determining swap space)
    >
    > >>for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;

    >
    > >>There is no guarantee that it gives meaningful results
    > >>on any system, and may crash the system.

    >
    > > In particular, it's useless on systems with lazy swap space allocation.

    >
    > It might still tell you the maximum for one task...
    >
    > My favorite lazy swap story was someone trying to determine how fast the
    > memory system was, something like:
    >
    > int i;
    > char *a,*b;
    > a=malloc(SIZE);
    > b=malloc(SIZE);
    > for(i=0;i<1000000;i++) memcpy(a,b,SIZE);
    >
    > assuming that SIZE was enough bigger than the cache, so that it would
    > have to go out to memory.
    >
    > It turned out that in a lazy swap allocation, all the allocated pages
    > referenced the same physical page, which was smaller than the cache,
    > so that it only tested the cache.
    >
    > -- glen



    Okay everyone,

    thanks for some of the pointers, I'm not there yet but getting closer.

    Let me first tell you why I wanted this information in the first place.
    Our application uses very large files (100G +) and these have to be
    procesed in the fastest possible way. We do this my placing a lot of
    data in memory and then run our tricks on it. However there are a number
    of restrictions that stops us of allocation more and more memory.
    We think the following:

    1) Address Space of the CPU. (32 bit only allows for 4G range).
    2) Process limits.
    3) Size of the available swap/page file.

    When a malloc fails I would like to print an apropriate error message
    why it fails. So we can give the user a meaningfull sugestion how to
    improve the job.

    Now 1 and 2 are fairly easy to check. Bit 3 is as you already know
    quite tricky. So just allocating some memory is not a good option
    because we are talking about big systems here and allocates can take
    quite some time. Also at the time I need the info there is none left
    anyway.

    Anway thanks for the info sofar.
    Otto
    Otto Meijer, Dec 16, 2003
    #11
  12. Otto Meijer

    Dan Pop Guest

    Re: Swap space size in C

    In <C0sDb.401963$ao4.1321249@attbi_s51> glen herrmannsfeldt <> writes:

    >Dan Pop wrote:
    >
    >(snip on determining swap space)
    >
    >>>for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;

    >
    >>>There is no guarantee that it gives meaningful results
    >>>on any system, and may crash the system.

    >
    >> In particular, it's useless on systems with lazy swap space allocation.

    >
    >It might still tell you the maximum for one task...
    >
    >My favorite lazy swap story was someone trying to determine how fast the
    >memory system was, something like:
    >
    >int i;
    >char *a,*b;
    >a=malloc(SIZE);
    >b=malloc(SIZE);
    >for(i=0;i<1000000;i++) memcpy(a,b,SIZE);
    >
    >assuming that SIZE was enough bigger than the cache, so that it would
    >have to go out to memory.
    >
    >It turned out that in a lazy swap allocation, all the allocated pages
    >referenced the same physical page, which was smaller than the cache,
    >so that it only tested the cache.


    Nonsense. Such a system would be irremediably broken and perfectly
    useless. Once you write a value to a memory location, that memory
    location must be mapped into your program's address space. If this is
    not possible (it can happen in a system with lazy swap space allocation,
    due to lack of physical swap space), the behaviour is system specific.
    Typical examples are sending a signal to the program or killing *other*
    programs, chosen more or less randomly, to obtain the missing swap space.

    Although such behaviour may sound crazy, the redeeming merits of lazy
    swap space allocation make it the preferred choice of many people.

    Back when using a system with only 128 MB of swap space (eager swap
    allocation would have rendered it next to useless), I knew that it was
    time to kill netscape when I was starting to lose xterms ;-)

    Dan
    --
    Dan Pop
    DESY Zeuthen, RZ group
    Email:
    Dan Pop, Dec 16, 2003
    #12
  13. Otto Meijer

    CBFalconer Guest

    Re: Swap space size in C

    Dan Pop wrote:
    > glen herrmannsfeldt <> writes:
    > >Dan Pop wrote:
    > >
    > >(snip on determining swap space)
    > >
    > >>>for(i=0;j=0x40000000;j>8;j>>=1) if(malloc(i-8)) i += j;

    > >
    > >>>There is no guarantee that it gives meaningful results
    > >>>on any system, and may crash the system.

    > >
    > >> In particular, it's useless on systems with lazy swap space allocation.

    > >
    > >It might still tell you the maximum for one task...
    > >
    > >My favorite lazy swap story was someone trying to determine how fast the
    > >memory system was, something like:
    > >
    > >int i;
    > >char *a,*b;
    > >a=malloc(SIZE);
    > >b=malloc(SIZE);
    > >for(i=0;i<1000000;i++) memcpy(a,b,SIZE);
    > >
    > >assuming that SIZE was enough bigger than the cache, so that it would
    > >have to go out to memory.
    > >
    > >It turned out that in a lazy swap allocation, all the allocated pages
    > >referenced the same physical page, which was smaller than the cache,
    > >so that it only tested the cache.

    >
    > Nonsense. Such a system would be irremediably broken and perfectly
    > useless. Once you write a value to a memory location, that memory
    > location must be mapped into your program's address space. If this is
    > not possible (it can happen in a system with lazy swap space allocation,
    > due to lack of physical swap space), the behaviour is system specific.
    > Typical examples are sending a signal to the program or killing *other*
    > programs, chosen more or less randomly, to obtain the missing swap space.
    >
    > Although such behaviour may sound crazy, the redeeming merits of lazy
    > swap space allocation make it the preferred choice of many people.


    That system may have built some hidden assumptions in, such as
    freshly allocated space contains zeroes, copying zeroes to lazy
    allocations does not constitute dirtying, etc. This could result
    in some blinding speeds. :)

    --
    Chuck F () ()
    Available for consulting/temporary embedded and systems.
    <http://cbfalconer.home.att.net> USE worldnet address!
    CBFalconer, Dec 16, 2003
    #13
  14. Re: Swap space size in C

    (Otto Meijer) writes:
    > glen herrmannsfeldt <> wrote in message
    > news:<C0sDb.401963$ao4.1321249@attbi_s51>...

    [...]
    > Let me first tell you why I wanted this information in the first place.
    > Our application uses very large files (100G +) and these have to be
    > procesed in the fastest possible way. We do this my placing a lot of
    > data in memory and then run our tricks on it. However there are a number
    > of restrictions that stops us of allocation more and more memory.
    > We think the following:
    >
    > 1) Address Space of the CPU. (32 bit only allows for 4G range).
    > 2) Process limits.
    > 3) Size of the available swap/page file.
    >
    > When a malloc fails I would like to print an apropriate error message
    > why it fails. So we can give the user a meaningfull sugestion how to
    > improve the job.


    Just a thought: as long as you can't read the whole thing into memory
    at once, you'll probably get the best performance by reading only as
    much as will fit into real memory, avoiding the swap file. It
    probably depends on your access patterns and half a dozen other things
    that are off-topic here.

    <OT>Something like mmap() might be a possibility, apart from the
    address space limitation (though it might be worth migrating to a
    64-bit CPU).</OT>

    --
    Keith Thompson (The_Other_Keith) <http://www.ghoti.net/~kst>
    San Diego Supercomputer Center <*> <http://www.sdsc.edu/~kst>
    Schroedinger does Shakespeare: "To be *and* not to be"
    (Note new e-mail address)
    Keith Thompson, Dec 16, 2003
    #14
  15. Re: Swap space size in C

    Dan Pop wrote:

    > In <C0sDb.401963$ao4.1321249@attbi_s51> glen herrmannsfeldt writes:


    (snip)

    >>My favorite lazy swap story was someone trying to determine how fast the
    >>memory system was, something like:


    >>int i;
    >>char *a,*b;
    >>a=malloc(SIZE);
    >>b=malloc(SIZE);
    >>for(i=0;i<1000000;i++) memcpy(a,b,SIZE);


    >>assuming that SIZE was enough bigger than the cache, so that it would
    >>have to go out to memory.


    >>It turned out that in a lazy swap allocation, all the allocated pages
    >>referenced the same physical page, which was smaller than the cache,
    >>so that it only tested the cache.


    > Nonsense. Such a system would be irremediably broken and perfectly
    > useless. Once you write a value to a memory location, that memory
    > location must be mapped into your program's address space. If this is
    > not possible (it can happen in a system with lazy swap space allocation,
    > due to lack of physical swap space), the behaviour is system specific.
    > Typical examples are sending a signal to the program or killing *other*
    > programs, chosen more or less randomly, to obtain the missing swap space.


    > Although such behaviour may sound crazy, the redeeming merits of lazy
    > swap space allocation make it the preferred choice of many people.


    > Back when using a system with only 128 MB of swap space (eager swap
    > allocation would have rendered it next to useless), I knew that it was
    > time to kill netscape when I was starting to lose xterms ;-)


    Note that the above program does not write anything to b.

    I may not have remembered the actual example right, though.
    This one would have to actually allocate a, but it doesn't
    ever need to allocate memory for b.

    I don't know if they actually overcommit, or just don't bother
    actually clearing the memory (usually required for security),
    and setting up the tables to describe it.

    -- glen
    glen herrmannsfeldt, Dec 17, 2003
    #15
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Shuo Xiang

    Stack space, global space, heap space

    Shuo Xiang, Jul 9, 2003, in forum: C Programming
    Replies:
    10
    Views:
    2,894
    Bryan Bullard
    Jul 11, 2003
  2. Christian Seberino
    Replies:
    21
    Views:
    1,655
    Stephen Horne
    Oct 27, 2003
  3. Ian Bicking
    Replies:
    2
    Views:
    1,012
    Steve Lamb
    Oct 23, 2003
  4. Ian Bicking
    Replies:
    2
    Views:
    723
    Michael Hudson
    Oct 24, 2003
  5. Niels Dekker (no reply address)

    What swap is called when using std::swap?

    Niels Dekker (no reply address), Jul 19, 2006, in forum: C++
    Replies:
    4
    Views:
    989
    Niels Dekker (no reply address)
    Jul 20, 2006
Loading...

Share This Page