Dynamic C to C Data Transfer

[Mattan]

[...] There are useful tutorials and sometimes
those that just add to the confusion. I've got my fingers crossed for
some good ones to come my way.

For an introduction to sockets that is brief, to the point and
certain to elevate your confusion to a higher level see

http://cs.baylor.edu/~donahoo/practical/CSockets/

and get the book. (Only approx 100 pages) And no, I am in
no way affiliated with the book or its authors.

Happy Holidays,
Mattan

I will now go into hiding in my flame proof OT-shelter...

 

[Richard Heathfield]

Tom said:

For now I am stuck in MicroSoftville. I realize this is disgusting for
many of the guru's here.

On the contrary, we're generally not interested in your platform. It doesn't
matter, where C is concerned.

<snip>

 

[Keith Thompson]

Whatever naming convention you choose the gist of the solution is the
atomic rename() function. You need to copy your file with a temporary
name to the common directory and then rename it to e.g.
'00000001.tom'.

Assuming that rename() happens to be atomic on your system. The C
standard says nothing about this.

It also says nothing about sockets. To the OP: sockets may well be
the ideal solution for your problem, but this is not the place to ask
about them; you should post to a newsgroup that's specific to your
operating system.

 

[Kenny McCormack]

Tom said:



On the contrary, we're generally not interested in your platform. It doesn't
matter, where C is concerned.

To the OP: You might like to google some of RH's posting history, to see
that he *has* engaged in MS-bashing in the past.

 

[Malcolm]

To the OP: You might like to google some of RH's posting history, to see
that he *has* engaged in MS-bashing in the past.

It's only fun.
We all know that Microsoft are the leading personal software company in the
world. They are so good that sometimes it seems they are the only company in
the market. However their OS is not the only one in existence, nor is it
perfect.

If you go to Richard's site you will see instructions for using Visual C++.
He knows that practically everyone needs a copy of this compiler.

 

[Richard]

It's only fun.
We all know that Microsoft are the leading personal software company in the
world. They are so good that sometimes it seems they are the only company in
the market. However their OS is not the only one in existence, nor is it
perfect.

If you go to Richard's site you will see instructions for using Visual C++.
He knows that practically everyone needs a copy of this compiler.

There is perfectly good documentation provided with Visual C++ and by MS
themselves : better to use that as a newbie than to use potentially out
of date and biased instructions from every tom, dick and harry out
there IMO. The information Richard has on his web page is rudimentary to
say the least.

 

[Bill Reid]

For now I am stuck in MicroSoftville. I realize this is disgusting for
many of the guru's here.

Actually, it is an advantage, despite what some "gurus" might
imply. That's because just about everything you need to write the
app is available for FREE and is DOCUMENTED and ACTUALLY
WORKS from the MSDN site. Contrast that with all "community
effort" crap out there that is "still untested" and "looking for volunteers
to document it" (how do you "document" something that doesn't
work, or what's the point?).

Substantial portions of the functionality in my current codebase I
just downloaded right from the site, seems to work fine...

Also, heterogenuous machine architectures make "RPC" somewhat
more problematic to implement, which is why "DCOM" is "PC only"...

For now I consider RPC, sockets, file writing hack, and INI techniques
to all be possibilities. I'll try to learn enough about each to make
the best choice.

The funny thing is, I implemented the EXACT same thing you
appear to be working on (a distributed automatic stock trading
system) using "DCOM" several years ago...in my case though,
I didn't have a choice in the matter, since it was just a demo
app as an example of a stock trading API that had added
"DCOM" functionality as part of its latest version...

I wonder how many other techniques will surface? In a
networking newsgroup it was suggested I consider XML within the .Net
framework and using SOAP. <- Another new acronym in the puzzle for me
that I have not even googled yet (but I will!)

Heh. As I said, Microsoft has gone through several versions
of "RPC", the latest of which was folding "DCOM" into the ".Net
framework" and expanding it to include the "standard" XML and
SOAP protocols.

How many different ways are there to do this? Actually, not that
many, in the most general sense. "CORBA", "DCOM", and "SOAP"
all do pretty much the same thing, which you might suspect because
they all include an "O" ("object") in the acronym...but NONE of this
is required if you don't REALLY need asynchonous "real-time" execution
AND locking capability (which is basically required for "program trading",
at least how I implemented the demo app).

I think the XML is way
out of my league.

Also is overkill, as near as I can tell. You talked about a "40 byte
file"), and stock trade orders don't even require that amount of data.
You just need a few (maybe just one) remote calls with the necessary
arguments (pointer to a 8-character array containing the stock
symbol, enum for type of trade, etc.).

That person also seems willing to point me towards
some useful socket tutorials.

And "sockets" just sounds like massive underkill to me. It's like
asking, "how do I drive to Chicago by car?", and somebody gives
you instructions for machining an engine block...of course, you
didn't help matters any by showing up here with the equivalent of
"I have four wheels, how do I make them go roundy-round?"...

 

[Keith Thompson]

We all know that Microsoft are the leading personal software company in the
world. They are so good that sometimes it seems they are the only company in
the market. However their OS is not the only one in existence, nor is it
perfect.

If you go to Richard's site you will see instructions for using Visual C++.
He knows that practically everyone needs a copy of this compiler.

No, practically everyone does not need a copy of that particular
compiler. I get along just fine without it.

The point, of course, is that it doesn't matter.

 

[Richard Heathfield]

Malcolm said:

It's only fun.

No, it's not only fun. Microsoft have done a lot of harm with their
proprietary, lock-in approach to software, and periodic excoriations are
useful "lest we forget". Of course, if they changed their ways, that would
be different, but how likely is that?

Nevertheless, my point stands. We are *not* interested in the OP's platform.
C is C, no matter where you are.

If you go to Richard's site you will see instructions for using Visual
C++. He knows that practically everyone needs a copy of this compiler.

No, I just don't like seeing people floundering around at Step Zero.

 

[Richard Heathfield]

Richard said:

There is perfectly good documentation provided with Visual C++ and by MS
themselves :

Fabulous. Unfortunately, many people find the instructions for creating a
console program in C incomprehensible, or can't find those instructions at
all.

better to use that as a newbie than to use potentially out
of date and biased instructions from every tom, dick and harry out
there IMO.

I have seen too many newbies struggle with the official docs not to have
sympathy for their plight.

The information Richard has on his web page is rudimentary to
say the least.

Deliberately so. Instead of wading through a million pages of C# this,
dotnet that, and managed C++ the other, you can find a clear, simple,
*short* guide that will get you up and running with your "hello world"
console program in pretty short order. I stand by my decision to publish
that information.

If it's inaccurate, by all means let me know, and I'll fix it.

Alternatively, if you can find a Microsoft page that isn't going to go away
in the next few years and which gives substantially the same information in
about the same space and simplicity, by all means post its URL here and
I'll link to that instead. Or as well. If it's good enough.

 

[CBFalconer]

Richard said:
.... snip ...

Fabulous. Unfortunately, many people find the instructions for
creating a console program in C incomprehensible, or can't find
those instructions at all.


I have seen too many newbies struggle with the official docs not
to have sympathy for their plight.

Don't limit your comment to newbies. I can recall bouncing off the
wall trying to tame that (alleged) system, and I don't consider
myself a newbie.

 

[Roland Pibinger]

Assuming that rename() happens to be atomic on your system. The C
standard says nothing about this.

"Synopsis
#include <stdio.h>
int rename(const char *old, const char *new);

Description
The rename function causes the file whose name is the string pointed
to by old to be henceforth known by the name given by the string
pointed to by new. The file named old is no longer accessible by that
name..."


Best regards,
Roland Pibinger

 

[Chris Dollin]

"Synopsis
#include <stdio.h>
int rename(const char *old, const char *new);

Description
The rename function causes the file whose name is the string pointed
to by old to be henceforth known by the name given by the string
pointed to by new. The file named old is no longer accessible by that
name..."

That doesn't say it's atomic, although the Rationale does -- but
that's non-normative, right?

 

[Tom]

That doesn't say it's atomic, although the Rationale does -- but
that's non-normative, right?

Can you please explain what "atomic" is precisely? I understand its
meaning by context as you guys use it in here ... but a pure
definition would assure I do not guess wrongly.

Thanks!

 

[Stephen Sprunk]

Can you please explain what "atomic" is precisely? I understand its
meaning by context as you guys use it in here ... but a pure
definition would assure I do not guess wrongly.

"Atomic" comes from the Greek "atomos", which means indivisible. If you
perform an atomic operation, no outside observer (such as a signal
handler, another thread/process, etc) should be able to see the
operation in a partially-completed state. They must see either the
"before" or "after" state -- though which they see may not be
predictable (at least without additional work). Operations must be
considered non-atomic unless explicitly designated as such.

S

 

[Tom]

"Atomic" comes from the Greek "atomos", which means indivisible. If you
perform an atomic operation, no outside observer (such as a signal
handler, another thread/process, etc) should be able to see the
operation in a partially-completed state. They must see either the
"before" or "after" state -- though which they see may not be
predictable (at least without additional work). Operations must be
considered non-atomic unless explicitly designated as such.

Thanks Stephen -- It's easy to understand the importance of that
characteristic when more than one program is involved.

 

[Roland Pibinger]

"Atomic" comes from the Greek "atomos", which means indivisible. If you
perform an atomic operation, no outside observer (such as a signal
handler, another thread/process, etc) should be able to see the
operation in a partially-completed state. They must see either the
"before" or "after" state -- though which they see may not be
predictable (at least without additional work). Operations must be
considered non-atomic unless explicitly designated as such.

In the context of the current discussion the real questions are: Has
the file a new name after rename() returns successfully (return value
0)? Is the change of the file name complete after successful return?
IMO, the answers are clear. I am not aware of a filesystem that allows
access to a file during rename() at the same time under the old and
the new name. But even if that were the case the proposed solution
would not be affected.

Best wishes,
Roland Pibinger

 

[Stephen Sprunk]

In the context of the current discussion the real questions are: Has
the file a new name after rename() returns successfully (return value
0)? Is the change of the file name complete after successful return?
IMO, the answers are clear. I am not aware of a filesystem that allows
access to a file during rename() at the same time under the old and
the new name. But even if that were the case the proposed solution
would not be affected.

There are OSes where if you open a file, then delete it, the opened
stream continues to be valid and you can continue reading from (or even
writing to) it until you close it, at which point it's actually removed
from disk. Other OSes will refuse to delete a file that is open. Still
others will delete the file immediately and return an error on the next
read.

In the first case, if you open a file, then rename a file over it, then
open the same file name again, you will have two streams open to the
same file name but with different contents. This is actually useful in
some scenarios (e.g. reading rotating log files), but can be confusing
the first time you encounter it.

The problem with rename potentially not being atomic isn't the above,
though. The issue is that some OSes will delete the destination name
before renaming the source file to the destination name. If one
attempts to open that file between those steps, one will get a "file not
found" error.

Worse, some network file systems won't recognize that the file contents
have changed. It's entirely possible that you can open a file, read
part of contents A, wait a while (during which someone else renames over
your file), then read part of contents B.

Of course, all of this is off-topic here since ISO C doesn't acknowledge
that multiple programs can run at the same time

S

 

[Tom]

In the context of the current discussion the real questions are: Has

There are OSes where if you open a file, then delete it, the opened
stream continues to be valid and you can continue reading from (or even
writing to) it until you close it, at which point it's actually removed
from disk. Other OSes will refuse to delete a file that is open. Still
others will delete the file immediately and return an error on the next
read.

In the first case, if you open a file, then rename a file over it, then
open the same file name again, you will have two streams open to the
same file name but with different contents. This is actually useful in
some scenarios (e.g. reading rotating log files), but can be confusing
the first time you encounter it.

The problem with rename potentially not being atomic isn't the above,
though. The issue is that some OSes will delete the destination name
before renaming the source file to the destination name. If one
attempts to open that file between those steps, one will get a "file not
found" error.

Worse, some network file systems won't recognize that the file contents
have changed. It's entirely possible that you can open a file, read
part of contents A, wait a while (during which someone else renames over
your file), then read part of contents B.

Of course, all of this is off-topic here since ISO C doesn't acknowledge
that multiple programs can run at the same time

Please bare with me... at the end is an interesting speculation that
NO renaming is necessary and an interesting UNIX question.

Amazing how a thread grows and the implications!! Logic is so
challenging at times! Once I begin to think I understand what I am
doing ... a conversation exposes things I have never considered fully
enough. Along with that comes additional growth. A very good thing.

Now to add my 2 cents of "learning observer" comment.

I very often write text files (LOG files) during a long (24+ hrs)
topological tuning run. The tuning program flushes the output to these
files after certain blocks of code are completed. During the tuning
run I often open these text files with Notepad and observe their
contents. I am careful to rename them within Notepad if I do any
editing (annotations) and want to save a particularly interesting
section of text. If I save an opened Notepad session without renaming
(once the tuning run is complete) I loose any tuning run output that
was added after the time I opened in Notepad. This is obvious! The
data from the tuning run is on the disk; however, the Notepad session
maintains the original begin file marker and the Notepad session's end
file marker is the first to be read in subsequent openings of the
file.

If I were to open in Notepad during the tuning run, delete the
contents, and then save without renaming ... I am not sure what would
happen? Yikes! The tuning program stream remains open (flushed but not
closed) and thus the file pointer within the tuning program would be
at the place of last write? But the removal by another program of the
earlier output now creates havoc?? Lets say file "NEW" is initiated
during the run and resides on the hard drive next to the open log
file. The tuning program pointer is now out of sync and could write
over the "NEW" file? At the conclusion of the tuning run and the
closing of the log file ... that log file would essentially contain
two "End-of-file" markers? One from the Notepad session and one from
the tuning program? If you then delete the file with two end-of-file
markers (speculative - not proven!) ... the begin-file-marker is
cleared and the file contents sit on the disk ready to be overwritten.
And the NEW file may be corrupted ... or may not have been overwritten
and by luck remains intact.

Thus to share data between simultaneously running programs ... all
that need be done is have Program #2 open the file in read only mode!!

No renaming necessary?

Program #2 reads the data (with internal time stamp). If the time
stamp within the data is fresh ... it acts upon it. If time stamp is
unchanged ... does nothing.

Now what a great operating system would do is NOT allow two programs
too simultaneously open a file with write privileges. I have confirmed
by observation that Win2k DOES allow both Notepad and a C program to
simultaneously open the very same file with write privilege. I am
curious ... does Unix prevent this from happening? My guess is not.
This is similar to the problem of large program development when teams
of programmers each have access too and make changes to the same code.
The latest MS studio makes a big deal about having control over this
situation and highlighting the changes such that only one set of
changes are kept. Challenging task indeed!! I wonder how MS keeps the
NEW files intact? Amazing technology!