place stl container object on shared memory

D

dbtouch

Hi, C++ programmers

I am trying to place an STL container object on shared memory and I
got a Segmentation Fault. Please give me some advice. Please notice
that if I allocate memory locally then I can place pStr on the
address
I sepcify. My platform is SunOS 5.10.


dbtouch


#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <vector>
#include <string>


#define CHECK_RESULT(X, Y) \
if (X==-1) \
{ \
perror(Y); \
exit(EXIT_FAILURE); \
}


typedef std::vector<int> VecInt;
typedef std::string String;


int main() {
// create shared memory
int result=shmget(IPC_PRIVATE, 1024*1024, 0644 | IPC_CREAT);
if (result==-1) {
perror("shmget");
exit(EXIT_FAILURE);
}


int shmid=result;


// attach shared memory
char *ptr=NULL;
ptr=(char*)shmat(shmid, NULL, 0644);
if ((int)ptr==-1) {
perror("shmat");
exit(EXIT_FAILURE);
}


#if 0
VecInt* pVec=new(ptr)VecInt(10);
delete pVec;
#endif


// char* qtr=new char[1024*1024];
// String* pStr=new(ptr)String(); good
String* pStr=new(ptr)String(); // segment fault!!!
if (pStr) {
delete pStr;
printf("test is good\n");
}
result=shmdt(ptr);
CHECK_RESULT(result, "shmdt")


result=shmctl(shmid, IPC_RMID, NULL);
CHECK_RESULT(result, "shctl")


exit(EXIT_SUCCESS);

}
 
R

REH

Hi, C++ programmers

I am trying to place an STL container object on shared memory and I
got a Segmentation Fault. Please give me some advice. Please notice
that if I allocate memory locally then I can place pStr on the
address
I sepcify. My platform is SunOS 5.10.

dbtouch

OT, but you are mostly likely having a memory alignment issue. Beyond
that, I think it's a very bad idea to so such a thing. You are going
to run into issues that the C++ standard library is not design to
handle such as cache coherency and thread-safety.

REH
 
D

dbtouch

Hi, Jeff

Thanks for the advice. I modify my code where I "delete" pStr to pStr-
~String() and my test program is good. I compiled your sample code in
SunOS 5.10 and got this error:

"shm.h", line 62: Error: Formal argument 1 of type char* in call to
shmdt(char*) is being passed void*.
"shm.h", line 85: Where: While instantiating "static
dbtouch::shared_memory<420, 1048576>::detach(void*)".
"shm.h", line 85: Where: Instantiated from non-template code.

It is
static void detach(void_pointer memory) {
check_posix(shmdt(memory), "shmdt");
}
which has problem. Please take a look.

Thanks,

YE LIU
dbtouch said:
I am trying to place an STL container object on shared memory
  String* pStr=new(ptr)String(); // segment fault!!!

No, that's not where the segv is occurring.
  if (pStr) {
    delete pStr;

This is.  Delete doesn't just call the destructor, it also tries to free
the memory.  That's not what you want here.  You just want to call the
destructor.

   pStr->~string();
    printf("test is good\n");
  }

You seem to be new to C++, and have carried a lot of C with you.  I'm
posting a modern C++ version of your code here, in the hope that it will
help you find "the C++ way."  Good luck.

/* POSIX */
extern "C" {
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>

}

/* C++ Standard */
#include <cerrno>
#include <cstdlib>
#include <cstring>
#include <iostream>
#include <stdexcept>
#include <string>

/* Dbtouch library code. */

namespace dbtouch {

     typedef void* void_pointer;

     template<std::size_t Size>
     void check_posix(long i, char const (&command_name)[Size]) {
         if (i == -1L) {
             throw std::runtime_error(
                     std::string( command_name ) +
                     std::strerror(errno) );
         }
     }

     template<int Mode, std::size_t Size>
     class shared_memory
     {
         typedef void const* void_const_pointer;
         typedef shmid_ds* shmid_ds_pointer;

         static int create() {
             int const shmid =
                 shmget(IPC_PRIVATE , Size , Mode | IPC_CREAT);
             check_posix(shmid, "shmget");
             return shmid;
         }

         static void_pointer attach(int shmid) {
             void_pointer const memory =
                     shmat(shmid, void_const_pointer( ), Mode);
             check_posix(reinterpret_cast<long>(memory), "shmat");
             return memory;
         }

         static void detach(void_pointer memory) {
             check_posix(shmdt(memory), "shmdt");
         }

         static void destroy(int shmid) {
             check_posix(
                     shmctl(shmid, IPC_RMID, shmid_ds_pointer( ))
                     , "shmctl");
         }

         int m_shmid;
         void_pointer m_memory;

       public:

         shared_memory( )
           : m_shmid( create() )
           , m_memory( attach(m_shmid) ) { }

         ~shared_memory() {
             detach(m_memory);
             destroy(m_shmid);
         }

         void_pointer get() {
             return m_memory;
         }
     };

}

/* Test of dbtouch library. */

/* Configuration. */
namespace {
     typedef std::string object;
     int const mode = 0644;
     std::size_t const size = 1024 * 1024;

}

void unchecked_main() {
     dbtouch::shared_memory<mode, size> buffer;
     if (object* const p = new(buffer.get()) object) {
         p->~object();
         std::cout << "test is good\n";
     }

}

int main() try {
     unchecked_main();
     return EXIT_SUCCESS;} catch (std::exception const& x) {

     std::cerr << "std::exception: " << x.what() << '\n';
     return EXIT_FAILURE;



}- Hide quoted text -

- Show quoted text -
 
D

dbtouch

Hi, Jeff

After static_cast, the program is good. Thanks for providing a very
helpful example.

dbtouch

[Top-posting elided.]

 > I compiled your sample code in SunOS 5.10 and got this error:
 >
 > "shm.h", line 62: Error: Formal argument 1 of type char* in call to
 > shmdt(char*) is being passed void*.

I didn't realize Solaris still did that.  Once upon a time, before there
was a standard void type, it was common to use char instead.  When void
was introduced to C, newly written void* could just be passed directly
to old functions expecting char*, because C allows implicit conversion
of void* to any other pointer type.

In C++, void* is not implicitly convertible to char*, so you have to
cast manually:

     static void detach(void_pointer memory) {
         check_posix(shmdt(static_cast<char*>(memory)), "shmdt");
     }
 
C

Chris M. Thomasson

I V said:
If I understand it correctly, this dynamically allocates a std::string
in the shared memory. Is that likely to be useful for anything? In many
cases, std::string will itself dynamically allocate memory in which to
store the actual string data. In that case, won't you end up with, in the
shared memory, a pointer to some non-shared memory that contains the
actual string data? Any attempt to access this data from another process
wouldn't work.

Bingo!
 
D

dbtouch

Hi, IV

I understand the point you are making: the example is just a test to
place an object the shared memory. I understand that I have to provide
an allocator using shared memory if I want the content in the
container to be in shared memory.

dbtouch
 
C

Chris M. Thomasson

Jeff Schwab said:
That's very insightful, but the OP only used std::string as a sample type
to establish that placement new could be used with shared memory. Hence my
typedef to "object". He did not try to use the shared memory for
anything. Whether that's useful is another discussion.


Not bingo. It depends, among other things, on the size of the string, and
whether the string implements the small string optimization. In the OP's
code, the string is empty. This may be living on the edge, but nothing is
actually wrong (yet).
;^)


If the OP actually were to use the shared memory for IPC, and really
wanted to store a std::string in it, other logic would be necessary. It's
not just the string's data buffer (which presumably would have to be
allocated by a custom allocator); it's also implementation-dependent
whether two threads of execution can access the string concurrently at
all. Some locking may be in order, and of course, caching issues come
into play.

synchronization would definitely be in order. However, can you be 100% sure
that the `std::string' uses the custom allocator for 100% of its internal
allocations?
 
C

Chris M. Thomasson

Chris M. Thomasson said:
[...]
[...]
If the OP actually were to use the shared memory for IPC, and really
wanted to store a std::string in it, other logic would be necessary. It's
not just the string's data buffer (which presumably would have to be
allocated by a custom allocator); it's also implementation-dependent
whether two threads of execution can access the string concurrently at
all. Some locking may be in order, and of course, caching issues come
into play.

synchronization would definitely be in order. However, can you be 100%
sure that the `std::string' uses the custom allocator for 100% of its
internal allocations?

I am very weary of using STL containers in shared memory even with custom
allocators. For one, you cannot rely on returning pointers from the custom
allocator; you need to think in terms of offsets off the base memory. Unless
you can guarantee that the STL container will use the custom allocator for
100% of all its internal allocations, and that each process maps the memory
at the same address, well, IMVHO, you playing with a ticking time bomb.
 
W

Walt

Jeff Schwab raised a very good point, that you shouldn't delete an
object created with a placement new. I think maybe deleting the
vector corrupted the heap, which may have caused the run-time error in
the String constructor.

As I V pointed out STL containers dynamically allocate additional
memory. Using the default STL allocator, this additional memory will
not be in the shared memory region.

So you could use (generally more primitive) containers that don't
dynamically allocate memory. Like arrays or boost intrusive
containers. This, http://www.geocities.com/wkaras/gen_cpp/avl_tree.html
, could be used as a rough substitute for the map and set templates.

If you are willing to take on a big challenge, you could write (or
maybe find?) an STL-compatible allocator that uses shared memory. To
write such an allocator, this, http://www.geocities.com/wkaras/heapmm/heapmm.html
, might be helpful.
 
C

Chris M. Thomasson

Walt said:
Jeff Schwab raised a very good point, that you shouldn't delete an
object created with a placement new. I think maybe deleting the
vector corrupted the heap, which may have caused the run-time error in
the String constructor.
As I V pointed out STL containers dynamically allocate additional
memory. Using the default STL allocator, this additional memory will
not be in the shared memory region.
So you could use (generally more primitive) containers that don't
dynamically allocate memory. Like arrays or boost intrusive
containers. This, http://www.geocities.com/wkaras/gen_cpp/avl_tree.html
, could be used as a rough substitute for the map and set templates.
If you are willing to take on a big challenge, you could write (or
maybe find?) an STL-compatible allocator that uses shared memory.

I don't think you can use a custom allocator wrt shared memory if you cannot
guarantee that all processes involved map the memory to the same address.
Also, even if you can make this guarantee, can you be 100% sure that the STL
container will use the custom allocator for 100% of all its internal
allocations?

To
write such an allocator, this,
http://www.geocities.com/wkaras/heapmm/heapmm.html
, might be helpful.
 
K

Kai-Uwe Bux

Jeff said:
That's a great question, and I'm not sure what the answer is. I do
expect 100% of the string's elements (i.e. characters) to live in memory
obtained from the allocator, but I'm not aware of any prohibition on the
string using other free store for other purposes, e.g. for a
shared_ptr-style reference count.

Yes, it is a good question. And the standard has an answer [23.1/8]:

Copy constructors for all container types defined in this clause copy an
allocator argument from their respective first parameters. All other
constructors for these container types take an Allocator& argument
(20.1.5), an allocator whose value type is the same as the container?s
value type. A copy of this argument is used for any memory allocation
performed, by these constructors and by all member functions, during
the lifetime of each container object. ...

Note the phrase "for any memory allocation".


Best

Kai-Uwe Bux
 
K

Kai-Uwe Bux

Jeff said:
Kai-Uwe Bux said:
Jeff said:
Chris M. Thomasson wrote:

can you be 100%
sure that the `std::string' uses the custom allocator for 100% of its
internal allocations?
That's a great question, and I'm not sure what the answer is. I do
expect 100% of the string's elements (i.e. characters) to live in memory
obtained from the allocator, but I'm not aware of any prohibition on the
string using other free store for other purposes, e.g. for a
shared_ptr-style reference count.

Yes, it is a good question. And the standard has an answer [23.1/8]:

Let me just take a moment to thank the members of WG21, and of the
various ISO member bodies, for their work and foresight. While no
language is perfect, I am continually astounded by how much the authors
of C++ have gotten right. Even when I don't like the answer to a
particular question, the very existence of an unambiguous answer within
the standard is a testament to the amount and quality of thought
represented by that document.

Agreed. It's somewhat amazing.
Thank you for the quick and satisfying answer. (So yes, Chris, I'm 100%
sure.)

Ah, rats: formally, std::string is not a container, so [23.1/8] does not
apply (sorry for that). Instead, we have [21.1/1]:

For a char-like type charT, the class template basic_string describes
objects that can store a sequence consisting of a varying number of
arbitrary char-like objects (clause 21). The first element of the sequence
is at position zero. Such a sequence is also called a ?string? if the
given char-like type is clear from context. In the rest of this clause,
charT denotes such a given char-like type. Storage for the string is
allocated and freed as necessary by the member functions of class
basic_string, via the Allocator class passed as template parameter. ...

And, unfortunately, the word "string" in the relevant sentence does not
unambiguously refer to the std::string data structure (i.e., basic_string)
could also refer only to the managed sequence of characters.

Fortunately, we also have [21.3.1/1]:

In all basic_string constructors, a copy of the Allocator argument is used
for any memory allocation performed by the constructor or member functions
during the lifetime of the object.

Here, again, we have the crucial phrase "for any memory allocation".


Best

Kai-Uwe Bux
 
C

Chris M. Thomasson

Kai-Uwe Bux said:
Jeff said:
Kai-Uwe Bux said:
Jeff Schwab wrote:

Chris M. Thomasson wrote:

can you be 100%
sure that the `std::string' uses the custom allocator for 100% of its
internal allocations?
That's a great question, and I'm not sure what the answer is. I do
expect 100% of the string's elements (i.e. characters) to live in
memory
obtained from the allocator, but I'm not aware of any prohibition on
the
string using other free store for other purposes, e.g. for a
shared_ptr-style reference count.

Yes, it is a good question. And the standard has an answer [23.1/8]:

Let me just take a moment to thank the members of WG21, and of the
various ISO member bodies, for their work and foresight. While no
language is perfect, I am continually astounded by how much the authors
of C++ have gotten right. Even when I don't like the answer to a
particular question, the very existence of an unambiguous answer within
the standard is a testament to the amount and quality of thought
represented by that document.

Agreed. It's somewhat amazing.
Thank you for the quick and satisfying answer. (So yes, Chris, I'm 100%
sure.)

Ah, rats: formally, std::string is not a container, so [23.1/8] does not
apply (sorry for that). Instead, we have [21.1/1]:

For a char-like type charT, the class template basic_string describes
objects that can store a sequence consisting of a varying number of
arbitrary char-like objects (clause 21). The first element of the
sequence
is at position zero. Such a sequence is also called a ?string? if the
given char-like type is clear from context. In the rest of this clause,
charT denotes such a given char-like type. Storage for the string is
allocated and freed as necessary by the member functions of class
basic_string, via the Allocator class passed as template parameter. ...

And, unfortunately, the word "string" in the relevant sentence does not
unambiguously refer to the std::string data structure (i.e.,
basic_string)
could also refer only to the managed sequence of characters.

Fortunately, we also have [21.3.1/1]:

In all basic_string constructors, a copy of the Allocator argument is
used
for any memory allocation performed by the constructor or member
functions
during the lifetime of the object.

Here, again, we have the crucial phrase "for any memory allocation".

So, just to clarify:

I can be guaranteed that any internal private allocations by a STL container
will use the custom allocator.


Right?
 
K

Kai-Uwe Bux

Chris said:
"Kai-Uwe Bux" <[email protected]> wrote in message
[snip: 23.1/8 about containers]
Fortunately, we also have [21.3.1/1]:

In all basic_string constructors, a copy of the Allocator argument is
used
for any memory allocation performed by the constructor or member
functions
during the lifetime of the object.

Here, again, we have the crucial phrase "for any memory allocation".

So, just to clarify:

I can be guaranteed that any internal private allocations by a STL
container will use the custom allocator.


Right?

Yes, I think that is the only reasonable way to interpret the standard. It
goes for containers [23.1/8] and for strings [21.3.1/1].


Best

Kai-Uwe Bux
 
C

Chris M. Thomasson

Kai-Uwe Bux said:
Chris said:
"Kai-Uwe Bux" <[email protected]> wrote in message
[snip: 23.1/8 about containers]
Fortunately, we also have [21.3.1/1]:

In all basic_string constructors, a copy of the Allocator argument is
used
for any memory allocation performed by the constructor or member
functions
during the lifetime of the object.

Here, again, we have the crucial phrase "for any memory allocation".

So, just to clarify:

I can be guaranteed that any internal private allocations by a STL
container will use the custom allocator.


Right?

Yes, I think that is the only reasonable way to interpret the standard. It
goes for containers [23.1/8] and for strings [21.3.1/1].

Thank you Sir!
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,769
Messages
2,569,580
Members
45,054
Latest member
TrimKetoBoost

Latest Threads

Top