Need to Allocate more space than size_t

[ramasubramanian.rahul]

hi folks
i have a peculiar problem. i have to allocate more than size_t
consequtive bytes on a system . after i do a malloc .. i am unable to
do a realloc because it takes size_t as a new size and not as an
incremental size..
can you tell me which library/system call to use..
Plz consider that all the physical hardware req for such an operation
is available... i know this sounds crazy... but just help me out here
....
Thanks in Advance
Rahul

 

[sam_cit]

i have a peculiar problem. i have to allocate more than size_t
consequtive bytes on a system . after i do a malloc .. i am unable to

consequtive bytes and dynamic allocation ???

 

[mark_bluemel]

hi folks
i have a peculiar problem. i have to allocate more than size_t
consequtive bytes on a system . after i do a malloc .. i am unable to
do a realloc because it takes size_t as a new size and not as an
incremental size..
can you tell me which library/system call to use..

This is probably wandering into "Off-topic for comp.lang.c" - there's
no way in the standard language.

You'd have to delve into the O/S to do this, if it is possible at all.
On un*x-like systems the "brk()" or "sbrk()" calls are probably
relevant, or the memory-mapping calls (mmap()) but...

Are you actually able to deal with more memory than size_t can denote?
You will be limited, one way or another by issues such as
addressability.. If you are working on a 32-bit O/S, then you cannot
(AFAIK) have more than 4Gb of address space for a process, for example.

Plz consider that all the physical hardware req for such an operation
is available...

But if the O/S can't access more than size_t can denote, that's no help
at all..

i know this sounds crazy... > but just help me out here

You probably need to give us more information, but we'll probably point
you at a newsgroup relating to your platform...

 

[Richard Bos]

i have a peculiar problem. i have to allocate more than size_t
consequtive bytes on a system . after i do a malloc .. i am unable to
do a realloc because it takes size_t as a new size and not as an
incremental size..
can you tell me which library/system call to use..

There isn't any. If you need more memory than can be specified in a
size_t, ISO C gives you no way to do so. Ideally, this shouldn't even be
possible; your implementation should make size_t large enough to cater
for every possible memory block size.

Richard

 

[santosh]

hi folks
i have a peculiar problem. i have to allocate more than size_t
consequtive bytes on a system . after i do a malloc .. i am unable to
do a realloc because it takes size_t as a new size and not as an
incremental size..
can you tell me which library/system call to use..
Plz consider that all the physical hardware req for such an operation
is available... i know this sounds crazy... but just help me out here
...

This is not possible using strictly standard C. The type size_t is
specifically meant to store the sizes of objects in a C program and it
should be possible to contain within a size_t object, the size of the
largest possible object guaranteed under standard C.

Generally, one would expect size_t to be equivalent to the largest
unsigned type, under most implementations, though you cannot assume
this. For example, on my implementation here, size_t is a typedef for
unsigned int, even though unsigned long long int, a much larger
unsigned type, is also available.

Anyway, returning to your question, the short answer is that you'll
have to use implementation specific extensions or, more likely, OS
specific system calls and their corresponding types. Asking in a group
for your system might get more helpful responses.

 

[santosh]

consequtive bytes and dynamic allocation ???

Why not? As far as a conforming C program is concerned, the component
bytes of a single dynamically allocated object are consecutive, though
that may not be the case as far as the layout of the object on physical
memory is concerned.

 

[sam_cit]

bytes of a single dynamically allocated object are consecutive, though
that may not be the case as far as the layout of the object on physical
memory is concerned.

I have never seen a requirement to allocate memory dynamically and
to expect that to be in consecutive locations... may be the OP should
explain the actual reason as to why he needs this, if he can...

 

[santosh]

I have never seen a requirement to allocate memory dynamically and
to expect that to be in consecutive locations... may be the OP should
explain the actual reason as to why he needs this, if he can...

The OP seems to be talking about resizing, (enlarging), the allocation
for a single object with a call to realloc(). The bytes of a single
dynamically allocated object in C are consecutive to the program.

If you're talking about the layout of two or more dynamic objects then
yes, they need not be consecutive.

 

[Stephen Sprunk]

hi folks
i have a peculiar problem. i have to allocate more than size_t
consequtive bytes on a system . after i do a malloc .. i am unable to
do a realloc because it takes size_t as a new size and not as an
incremental size..
can you tell me which library/system call to use..

size_t is able to hold the size of the largest object you can allocate
with malloc() or realloc() by definition. That means that, within the
realm of Standard C, there is no way to allocate an object larger than
SIZE_MAX.

Your OS may have mechanisms to allocate larger objects; you'll need to
ask about those in an OS-specific newsgroup.

Plz consider that all the physical hardware req for such an operation
is available... i know this sounds crazy... but just help me out here

Note that any sane implementation will define size_t such that it can
handle anything the system is capable of. It is unlikely that your OS
does, in fact, have something better, because that's what the C
implementors would have used if it were available.

For example, on x86 with PAE, your machine may have 36 bits of physical
memory, but it's impossible to access more than 32 bits of address space
from a single program, so size_t will be 32 bits. Ditto for an 32-bit
program running inside a 64-bit OS; the OS can hand out a different 4GB
address space to each program, but no single program can use more than
4GB on its own. A 64-bit program on the same OS would have a 64-bit
size_t.

S

 

[mark_bluemel]

hi folks
i have a peculiar problem. i have to allocate more than size_t
consequtive bytes on a system . after i do a malloc .. i am unable to
do a realloc because it takes size_t as a new size and not as an
incremental size..

As I and others have pointed out, you can't do this in Standard C and
it's very likely that you can't do it using non-standard techniques
either.

Why not describe what you are trying to achieve with this huge memory
allocation and let us suggest alternative ways of meeting your goal?

 

[Tor Rustad]

hi folks
i have a peculiar problem. i have to allocate more than size_t
consequtive bytes on a system . after i do a malloc .. i am unable to
do a realloc because it takes size_t as a new size and not as an
incremental size..
can you tell me which library/system call to use..
Plz consider that all the physical hardware req for such an operation
is available... i know this sounds crazy... but just help me out here

I suggest getting a C compiler that support your hardware. So, if you
have a 64-bit system, then you need a compiler which support for e.g.
LP64 model. You cannot expect a ILP32 compiler to give you some magic
way to address an object with greater size than 32-bit.

 

[Frederick Gotham]

(e-mail address removed):

hi folks
i have a peculiar problem. i have to allocate more than size_t
consequtive bytes on a system . after i do a malloc .. i am unable to
do a realloc because it takes size_t as a new size and not as an
incremental size..
can you tell me which library/system call to use..
Plz consider that all the physical hardware req for such an operation
is available... i know this sounds crazy... but just help me out here
...

Either:

(1) Your particular implementation intends such large chunks of memory to
be allocated via implementation-specific means (e.g. System API function).
(2) Your particular implementation intends for you to be able to allocate
such large chunks of memory using Standard C, but has neglected to make
"size_t" big enough.

Solutions:

(1) Unfortunately, you'll have to resort to implementation-specific means.
(2) You'll have to contact your compiler manufacturer and ask for a fix, or
perhaps try fix it yourself (e.g. altering the size_t typedef)


That's as much as I can tell you without going Off-Topic here.

 

[Tor Rustad]

Solutions:

(1) Unfortunately, you'll have to resort to implementation-specific means.

The problem is that OP's implementation doesn't support his HW
archtecture.

(2) You'll have to contact your compiler manufacturer and ask for a fix, or
perhaps try fix it yourself (e.g. altering the size_t typedef)

How is changing size_t typedef gonna help?

 

[Frederick Gotham]

Tor Rustad:

The problem is that OP's implementation doesn't support his HW
archtecture.

If you're supposed to be able to allocate a terrabyte of memory without
resorting to implementation-specific means, then "size_t" should be able to
hold a large enough value.

If this is indeed intended, then the implementation is flawed in that
size_t isn't big enough.

If this isn't intended, then you'll have to use implementation-specific
means to achieve your goal.

How is changing size_t typedef gonna help?

Perhaps it's defined(Do you "declare" or do you "define" a typedef?) as
follows in his/her system headers:

typedef long unsigned size_t;

, and perhaps he/she can change it to something platform-specific like:

typedef __int128 unsigned size_t;

Of course, this might mess up his/her entire system.

 

[Random832]

Perhaps it's defined(Do you "declare" or do you "define" a typedef?) as
follows in his/her system headers:

typedef long unsigned size_t;

, and perhaps he/she can change it to something platform-specific like:

typedef __int128 unsigned size_t;

Of course, this might mess up his/her entire system.

You misspelled "will".

He'd have to, at the _very_ least, modify and recompile all his libraries.

 

[Old Wolf]

hi folks
i have a peculiar problem. i have to allocate more than size_t
consequtive bytes on a system . after i do a malloc .. i am unable to
do a realloc because it takes size_t as a new size and not as an
incremental size..
can you tell me which library/system call to use..

You could use calloc( ), it can allocate more memory than
SIZE_MAX. Then copy all the data from your previous allocation,
and free it.

 

[Ian Collins]

You could use calloc( ), it can allocate more memory than
SIZE_MAX. Then copy all the data from your previous allocation,
and free it.

Doing the copy could be a bit of a challenge if you have more memory
than the size of a pointer.

 

[Keith Thompson]

Either:

(1) Your particular implementation intends such large chunks of memory to
be allocated via implementation-specific means (e.g. System API function).
(2) Your particular implementation intends for you to be able to allocate
such large chunks of memory using Standard C, but has neglected to make
"size_t" big enough.

Solutions:

(1) Unfortunately, you'll have to resort to implementation-specific means.
(2) You'll have to contact your compiler manufacturer and ask for a fix, or
perhaps try fix it yourself (e.g. altering the size_t typedef)


That's as much as I can tell you without going Off-Topic here.

No, altering the size_t typedef in your implementation's <stddef.h>
file (assuming it's a file) will not help. It's not absolutely
guaranteed to break the implementation (it's barely conceivable that
everything would magically work correctly with the modified typedef),
but I'd be astonished if it actually worked.

Don't try modify your implementation's system header files unless
you're absolutely sure you know what you're doing. And if you *think*
you're absolutely sure you know what you're doing, you're probably
mistaken.

 

[Keith Thompson]

You could use calloc( ), it can allocate more memory than
SIZE_MAX. Then copy all the data from your previous allocation,
and free it.

Not necessarily. calloc() takes two arguments, and the number of
bytes allocated is their product, but it's likely that an attempt to
allocate more than SIZE_MAX bytes will fail. Also, calloc()
zero-initializes the allocated space; with the amount of memory we're
talking about, that could take a significant amount of time.

If you really need to deal with more than SIZE_MAX bytes, your best
bet is probably to use a file (or to use a system with a larger
SIZE_MAX).

 

[Randy Howard]

(e-mail address removed) wrote
(in article

hi folks
i have a peculiar problem. i have to allocate more than size_t
consequtive bytes on a system .

Then you are using a compiler not properly matched to the
system, otherwise size_t would be plenty big.

can you tell me which library/system call to use..

The existing ones work fine, provided your implementation is
suited to the host architecture.

Plz consider that all the physical hardware req for such an operation
is available... i know this sounds crazy... but just help me out here

Using a 32-bit compiler on a 64-bit system by chance?