Dynamic C to C Data Transfer

[Tom]

I need my data generating C program on computer #1 to export small
amounts of data (one - 40 byte data structure) periodically (once per
minute) to a C program on computer #2.

I am considering having computer #1 to create two duplicate data files
sequentially. (file_1.dat & file_2.dat).

Then, Computer #2 tries to open file_2.dat and upon failure it opens
file_1.dat.

Due to how small these files are ... one of these two files should
always be "available" for Computer #2.

If either file can not be opened by Computer #1 (Computer #2 is
reading it) ... then that file is simply skipped on that output cycle.
If Computer #2 sees the same time stamp on the most recently read
file_2.dat ... then it reads file_1.dat.

I only know the basics of C programming and hope this can be resolved
within C easily. Both the timing and communication issues are a
challenge for me. Perhaps using a small ram disk would prevent
hammering away on a hard drive? Most likely there is a simple solution
and I am just ignorant of the best approach. Please share how you
would solve this problem. Thanks in advance.

 

[Randy Howard]

Tom wrote

I need my data generating C program on computer #1 to export small
amounts of data (one - 40 byte data structure) periodically (once per
minute) to a C program on computer #2.

I am considering having computer #1 to create two duplicate data files
sequentially. (file_1.dat & file_2.dat).

Then, Computer #2 tries to open file_2.dat and upon failure it opens
file_1.dat.

Due to how small these files are ... one of these two files should
always be "available" for Computer #2.

It's not on topic for this newsgroup, but a sockets based (or
other network API) solution might make a lot more sense.

If either file can not be opened by Computer #1 (Computer #2 is
reading it) ... then that file is simply skipped on that output cycle.
If Computer #2 sees the same time stamp on the most recently read
file_2.dat ... then it reads file_1.dat.

You seem to be implying that you will ping pong between two
filenames so the app on the other end knows the file has been
updated. Why not simply have the consumer remove the file when
it is done with it, and the producer not write the next version
until it has been removed? Or just keep appending to the
existing file, and have the consumer do blocking reads?

I only know the basics of C programming and hope this can be resolved
within C easily. Both the timing and communication issues are a
challenge for me. Perhaps using a small ram disk would prevent
hammering away on a hard drive?

I don't see how 40 bytes once a minute will hammer on anything.
Odds are the OS fs cache will make it all invisible most of the
time anyway.

Most likely there is a simple solution
and I am just ignorant of the best approach. Please share how you
would solve this problem. Thanks in advance.

I would just open some sort of direct communication channel
between the two programs and send the data when needed and skip
the file twiddling.

 

[goose]

I need my data generating C program on computer #1 to export small
amounts of data (one - 40 byte data structure) periodically (once per
minute) to a C program on computer #2.

<snipped>

I would ask this question in a group dedicated to your platform. Try
to find a group along the lines of comp.unix.programmer or
comp.windows.programmer or similar for your platform.

goose,

 

[Roland Pibinger]

I need my data generating C program on computer #1 to export small
amounts of data (one - 40 byte data structure) periodically (once per
minute) to a C program on computer #2.
I am considering having computer #1 to create two duplicate data files
sequentially. (file_1.dat & file_2.dat).
Then, Computer #2 tries to open file_2.dat and upon failure it opens
file_1.dat.
Due to how small these files are ... one of these two files should
always be "available" for Computer #2.

Can you explain the rationale behind that proceeding?

 

[Tom]

Can you explain the rationale behind that proceeding?

The scenario is: Several networked machines. Each performing stock
analysis on individual or a small group of financial instruments and
then passing buy/sell instructions to an order placing machine.

The central machine follows/enforces portfolio rules and the results
of the individual instrument analyses must be agglomerated ... but the
individual analyses is too complex and cumbersome for one personal
computer to perform the analysis for every instrument in the
portfolio. Thus I need to use some sort of parallel processing.

The rational is: I am a novice in data transfer methods and I don't
know where to begin. I am not a proficient or professional programmer
by any stretch. HA!! (Big joke.) However, I am better than average in
technical analysis, mathematics, and process design and optimization.

My hope is there is a best method for doing this within C so that I
don't have to venture outside the structured C approach that I like.
Sort of like a common file shared by several computers and programs at
once? Several programs being able to simultaneously have open and to
have write privileges to a single file is not possible? Even with the
usage of SEEK to the file end and flushing the buffer at each write
.... two programs might be writing at the same time. Thus the single
file approach seems to this rookie as impossible. However, each
program on the server computer(s) could write to a specific file set
up for each financial instrument and the central computer could open,
read, and then close these files at a specified timed interval? The
reason to write to two files and thus duplicating the task is an
attempt to avoid timing conflicts. I am looking for functionality and
simplicity over elegance. Although I find the simplest method that is
reliable to be very elegant in my eyes. This level of data transfer
between multiple programs running simultaneously is obviously new
ground for me. As powerful as C is ... there's got to be a best method
and I'd bet the farm (if I had one) that several of the many gurus in
this user group has the knowledge.

 

[CBFalconer]

The scenario is: Several networked machines. Each performing stock
analysis on individual or a small group of financial instruments
and then passing buy/sell instructions to an order placing machine.

The central machine follows/enforces portfolio rules and the
results of the individual instrument analyses must be agglomerated
... but the individual analyses is too complex and cumbersome for
one personal computer to perform the analysis for every instrument
in the portfolio. Thus I need to use some sort of parallel
processing.

The rational is: I am a novice in data transfer methods and I
don't know where to begin. I am not a proficient or professional
programmer by any stretch. HA!! (Big joke.) However, I am better
than average in technical analysis, mathematics, and process
design and optimization.

Try getting and reading "The Little Book of Semaphores":

<http://greenteapress.com/semaphores/>

 

[Jens Thoms Toerring]

My hope is there is a best method for doing this within C so that I
don't have to venture outside the structured C approach that I like.
Sort of like a common file shared by several computers and programs at
once? Several programs being able to simultaneously have open and to
have write privileges to a single file is not possible? Even with the
usage of SEEK to the file end and flushing the buffer at each write
...

With all this you are already outside of what the C language comes
with. The C language doesn't deal with much more than a single pro-
gram running on a single machine. When you need communication be-
tween different programs already on the same machine then you must
resort to ways and means your environment allows you to use beside.
If you e.g. are on a machine where the system is POSIX compliant you
could e.g. use the socket interface to communicate between different
processes on the same or on different machines. But this is outside
of the realm of the C language and thus you have to ask these kinds
of questions in a group that deals with your environment - it's not
that people here are unwilling to help but you're asking questions
about baking bread at a carpenters shop and even though the carpen-
ter might know a lot about baking bread he usually will tell you to
go to the bakery next door where (hopefully) the real experts are;-)

Regards, Jens

 

[Tom]

With all this you are already outside of what the C language comes
with. The C language doesn't deal with much more than a single pro-
gram running on a single machine. When you need communication be-
tween different programs already on the same machine then you must
resort to ways and means your environment allows you to use beside.
If you e.g. are on a machine where the system is POSIX compliant you
could e.g. use the socket interface to communicate between different
processes on the same or on different machines. But this is outside
of the realm of the C language and thus you have to ask these kinds
of questions in a group that deals with your environment - it's not
that people here are unwilling to help but you're asking questions
about baking bread at a carpenters shop and even though the carpen-
ter might know a lot about baking bread he usually will tell you to
go to the bakery next door where (hopefully) the real experts are;-)

Regards, Jens

Thank you Jens (and others) for directing me towards an operating
system approach. I suppose I must break new ground but I was hoping
for a clever solution without having to learn about sockets and
various other complexities.

Jens' explanation and comparison between bakers and carpenters is
excellent. I understand fully. Still ... I was hoping for an easy way
out.

I guess I will put away my Milwaukee Holeshooter, Porter-Cable
circular saw, and Delta table saw and go shopping for a nice
convection air oven and a Kitchen Aid stand mixer. At least the toys
are still electrical!

I am so impressed by the knowledge of some in this group that I can't
begin to blasphemy with the claim of "real" experts being elsewhere
.... but certainly I will find other experts on my journey. I hope they
are not offended after their efforts at educating me and I then tell
them they bake a great cake? Perhaps that baking will remain our
secret. After all, real men construct things ... right? Funny how
both carpenters and bakers wear aprons!! Yeah, I have one too. )

 

[Roland Pibinger]

The scenario is: Several networked machines. Each performing stock
analysis on individual or a small group of financial instruments and
then passing buy/sell instructions to an order placing machine. ....
Sort of like a common file shared by several computers and programs at
once?

From that I suppose you machines have access to a common (LAN)
directory which they shall use to exchange data via files?

Several programs being able to simultaneously have open and to
have write privileges to a single file is not possible?

That depends on the file system (better assume nothing).

Even with the
usage of SEEK to the file end and flushing the buffer at each write
... two programs might be writing at the same time. Thus the single
file approach seems to this rookie as impossible.

Probably yes.

However, each
program on the server computer(s) could write to a specific file set
up for each financial instrument and the central computer could open,
read, and then close these files at a specified timed interval? The
reason to write to two files and thus duplicating the task is an
attempt to avoid timing conflicts.

I don't quite understand that. But let's assume we have 2 computers
(pc1, pc2) that have to exchange data via a common directory and shall
not disturb each other. I would define a name and behavior convention:
In the specified common directory pc1 opens only the file with the
name 'to_pc1.dat' and pc2 opens only the file 'to_pc2.dat'. To
exchange data between pc1 and pc2 the following steps need to be
performed:
- pc1 copies the data to the common directory with a temporary file
name
- after copying is finished pc1 renames the file to 'to_pc2.dat'
- pc2 opens (only!) the file 'to_pc2.dat' and changes it
- afterwards pc2 renames the file to 'to_pc1.dat'
- and so on
The point is that pc1 and pc2 never concurrently access (not even
read) the same file and that the rename() function is atomic. You need
to implement some sort of polling to make it work and the approach
isn't very fast. But everything can be done in Standard C (but you
should e.g. look for the sleep() function for polling).

Hope that's relevant to your problem,
Roland Pibinger

 

[Kenny McCormack]

Is the platform, by any chance, Windows?
If so, try using INI files. Works real well.

 

[Tom]

On Sat, 16 Dec 2006 23:33:27 +0000 (UTC),

Is the platform, by any chance, Windows?
If so, try using INI files. Works real well.

Thanks for the tip Kenny.

I am using WinNT on both of my computers. If this pans out ... my itsy
bitsy network will grow.

INI files as well as sockets are on my need to learn ASAP list.

Thanks again.

-- Tom

 

[Stephen Sprunk]

The scenario is: Several networked machines. Each performing stock
analysis on individual or a small group of financial instruments and
then passing buy/sell instructions to an order placing machine. ....
My hope is there is a best method for doing this within C so that I
don't have to venture outside the structured C approach that I like.
Sort of like a common file shared by several computers and programs
at once? ... Although I find the simplest method that is reliable to
be
very elegant in my eyes. This level of data transfer between multiple
programs running simultaneously is obviously new ground for me.
As powerful as C is ... there's got to be a best method and I'd bet
the farm (if I had one) that several of the many gurus in this user
group has the knowledge.

<OT>
This is a problem that just begs for you to use sockets. Polling a
variety of files, which are shared over a network, is unreliable and a
blatant hack anyways, though for some reason it always seems to be the
first thought of novice programmers developing their first client/server
app.

Sockets really aren't that hard to learn either, if you can find a
decent book or website with some examples to copy. It's new, so expect
a learning curve, but what you want to do is pretty easy, particularly
on the client side. You can whip that code quickly once you get the
basics down. The server side is a bit more complicated since you'll
have multiple sockets open, need to accept new ones, etc. while
(presumably) doing other things between reads/writes. Windows'
equivalent of POSIX's select() is your friend here, but expect to get it
wrong the first few times; it's tricky, so find some examples to study
(and copy). Once it works, though, you'll wonder why you ever thought
of sharing files...

The subtle gotcha is the data formatting. Simply writing a struct to
the socket will burn you when you port your code to a new machine (or
OS, or compiler version) due to size/padding/endianness differences.
You have to know exactly how many bits you're using for each field (e.g.
use uint32_t, not long), what order the bits are (use ntohl() et al),
and write them one by one instead of as a group (eliminate padding). Or
simply send the data as text for easier debugging and better portability
(use sprintf() and sscanf() to format and parse).

The not-so-subtle gotcha is what to do if the server receives only part
of a message -- you can't block on the read because the rest may never
show up due to a network outage or client crash -- or if the client
can't contact the server. The answers to both will vary depending on
your needs.
</OT>

S

 

[Richard Heathfield]

Tom said:

On Sat, 16 Dec 2006 23:33:27 +0000 (UTC),


Thanks for the tip Kenny.

It's a tip that doesn't do you any good, though, always assuming your
problem isn't that of opening and writing and reading and closing the
files. For that, any old format will work. But the advice is no more than
we'd expect from Mr McCormack.

The proper way to do this is via sockets, but yes, you could hack at it with
files if you wanted, if you wanted something short-term while you were busy
learning sockets.

Someone suggested starting off in the basement ("00000000.tom"), and writing
each new message in the next file up ("00000001.tom"), with the receiver
deleting as it reads. That sounds workable to me, and doesn't involve the
(easy-to-get-wrong) swapping of two filenames.

 

[Tom]

Tom said:


It's a tip that doesn't do you any good, though, always assuming your
problem isn't that of opening and writing and reading and closing the
files. For that, any old format will work. But the advice is no more than
we'd expect from Mr McCormack.

The proper way to do this is via sockets, but yes, you could hack at it with
files if you wanted, if you wanted something short-term while you were busy
learning sockets.

Someone suggested starting off in the basement ("00000000.tom"), and writing
each new message in the next file up ("00000001.tom"), with the receiver
deleting as it reads. That sounds workable to me, and doesn't involve the
(easy-to-get-wrong) swapping of two filenames.

Thanks for your incite Richard.

The basement approach to the file naming is very interesting. The
receiving computer could just attempt to open the next sequential file
on a scheduled basis and fail gently if it does not exist. Slick!
The receiver deleting prevents filling the drive. Slick again. )

Certainly sockets is more capable. In the event the receiver is not
available ... the drive space could fill. So the robustness of sockets
is clearly superior. However, there is still that benefit of rotating
between just two (or a few) file names to avoid the drive filling
scenario. My guess is a conflict would eventually surface in the
rotating approach with two machines accessing the same file name. Even
though the receiver would be restricted to read-only privileges. Just
a WAG on my part and experience is needed to go any further in that
evaluation. The sockets approach would be purely memory resident? Thus
the hard drive is not even in the loop and performance is light years
beyond file manipulation? Perhaps add another milliwatt of juice to my
cold light filament.

I appreciate everyone's suggestions. Each piece shapes my thinking and
helps start a groove in my totally smooth networking brain cell. Seems
that brain cell is teflon coated at times too. I am starting pursuit
the sockets approach (gulp). This is a daunting task for me. Simply
getting the file transfer working on my two machine network took me
days to figure out. Step-by-step instructions for the task of file
sharing on a LAN (that has been done by tens of thousands of others
and perhaps millions) are amazingly unavailable. Seems those that have
figured it out want to hoard that knowledge and turn providing the
basic instructions into a business all too often.

Classifying the file sharing approach as a short term hack is
"exactly" as it now appears. Earlier I thought it might be elegant.
(The simplest solution that works.) But now it seems to be sprouting
warts all over. Certainly the thinking process to get the hack to work
makes one appreciate the robustness of sockets.

Again, thanks! For reals.

-- Tom

 

[Tom]

<OT>
This is a problem that just begs for you to use sockets. Polling a
variety of files, which are shared over a network, is unreliable and a
blatant hack anyways, though for some reason it always seems to be the
first thought of novice programmers developing their first client/server
app.

Sockets really aren't that hard to learn either, if you can find a
decent book or website with some examples to copy. It's new, so expect
a learning curve, but what you want to do is pretty easy, particularly
on the client side. You can whip that code quickly once you get the
basics down. The server side is a bit more complicated since you'll
have multiple sockets open, need to accept new ones, etc. while
(presumably) doing other things between reads/writes. Windows'
equivalent of POSIX's select() is your friend here, but expect to get it
wrong the first few times; it's tricky, so find some examples to study
(and copy). Once it works, though, you'll wonder why you ever thought
of sharing files...

The subtle gotcha is the data formatting. Simply writing a struct to
the socket will burn you when you port your code to a new machine (or
OS, or compiler version) due to size/padding/endianness differences.
You have to know exactly how many bits you're using for each field (e.g.
use uint32_t, not long), what order the bits are (use ntohl() et al),
and write them one by one instead of as a group (eliminate padding). Or
simply send the data as text for easier debugging and better portability
(use sprintf() and sscanf() to format and parse).

The not-so-subtle gotcha is what to do if the server receives only part
of a message -- you can't block on the read because the rest may never
show up due to a network outage or client crash -- or if the client
can't contact the server. The answers to both will vary depending on
your needs.
</OT>

S

Hey Stephen -- Thank you.

I've been in situations before when the best approach was "begging" to
be used. Seems those in-the-know are the ones that see this as the
others flounder around blindfolded. Your expertise shines brightly and
I *am* convinced sockets are something I need to learn.

The tips and hints you give I need to keep fully in focus. Just one
little bit of gotcha can cause prolonged headache while in the
learning curve. Perhaps you have saved me countless hours of
frustration. )

The reason I feel sockets are so daunting is the difficulty I had
getting simple file transfer to work on my itsy bitsy network. Took me
days! And the available information for doing this simple task is hard
to find!

The trouble shooting of system's communication is also new to me. For
me the file sharing solution across a simple network was a try every
combination until it works approach. Stab in the dark and stab again.
This is because I did not know what I was doing what-so-ever and the
instructions for such task are so poorly written (or is it ... I am
the type that can turn a mole hill into a mountain of difficulty and
confusion?) I am hoping to find some first class tutorials for sockets
on Win2000. Its a good thing my nose was broken in high school
basketball ... because my guess is it will get abused a lot more
throughout this sockets learning event. My oh my how that floor sneaks
up on ya.

Thank you for awesome gotcha warnings and your direction.

-- Tom :~) ( < smiley with worked over nose!)

 

[Kenny McCormack]

Tom said:


It's a tip that doesn't do you any good, though, always assuming your
problem isn't that of opening and writing and reading and closing the
files. For that, any old format will work. But the advice is no more than
we'd expect from Mr McCormack.

You really are a brainless fool.

It looks like you don't know anything about how INI files work, and,
that just as with Jacob, any reference to things Windows drives you into
hysterics. You're one of those people who a) Don't know anything about
Windows and b) is proud of the fact.

I know. I used to be like that. Not that I am any big fan of it and
things MS in general now, but I do know a few things and have generally
learned to stop worrying about it. You will too, once you've reached
your 20th birthday or thereabouts.

All of this is totally OT, of course, this being, after all, clc.

To the OP: Please don't pay any attention to RH. He's only 17, you know.

 

[Roland Pibinger]

Someone suggested starting off in the basement ("00000000.tom"), and writing
each new message in the next file up ("00000001.tom"), with the receiver
deleting as it reads. That sounds workable to me,

Whatever naming convention you choose the gist of the solution is the
atomic rename() function. You need to copy your file with a temporary
name to the common directory and then rename it to e.g.
'00000001.tom'.

Best regards,
Roland Pibinger

 

[Roland Pibinger]

In the event the receiver is not
available ... the drive space could fill.

A feature. You may work asynchron.

However, there is still that benefit of rotating
between just two (or a few) file names to avoid the drive filling
scenario. My guess is a conflict would eventually surface in the
rotating approach with two machines accessing the same file name. Even
though the receiver would be restricted to read-only privileges.

Each 'receiver' opens only his/her file(s), similar to a real person
who opens only emails addressed to him/her. rename() is the 'send'
function in this solution. This is not highly sophisticated
interprocess communication but it works with zero administration on a
LAN (or single PC).

Best wishes,
Roland Pibinger

 

[Bill Reid]

Thanks for your incite Richard.

LOL!!! Thanks for the entertaining spelling!

Of course, all of this sounds like classic job for something like
"remote procedure calls" (RPC) just off the top of my head, so
I typed "remote procedures" into the Google(TM) search
box on my ISP home page and got about 128,000 links in
0.14 seconds.

The top link seems to contain a lot of the "C" code needed
to implement client-server RPC, so check it out and see
if this is what you want:

http://www.cs.cf.ac.uk/Dave/C/node33.html

Or if the ACTUAL problem you are trying to solve can BE
solved BEST by using RPC (note carefully the description of
RPC in the first paragraph of the link, particularly the part
about "network transport independence" in light of the
advice here about "sockets"). I'm not completely sure from
your description if that is even the case (except you seem to
require a "lock" to prevent multiple use of the same file), but if it is,
I would think that somewhere in the 128,000 or so links, you'll
have enough to proceed, or at least think about...

Now "RPC" is kind of a generalized concept that sort of came
out of other concepts such as "distributed computing", and that
it was implemented in Unix as "CORBA", and be aware
that Microsoft has their own version of it (naturally) that has gone
through several versions...I've used what they call "DCOM" in the
past, so you might want to look at that on the MSDN site (or cite!)
for stuff like that...

In any event, back the regularly-scheduled flame war!

 

[Tom]

LOL!!! Thanks for the entertaining spelling!

Of course, all of this sounds like classic job for something like
"remote procedure calls" (RPC) just off the top of my head, so
I typed "remote procedures" into the Google(TM) search
box on my ISP home page and got about 128,000 links in
0.14 seconds.

The top link seems to contain a lot of the "C" code needed
to implement client-server RPC, so check it out and see
if this is what you want:

http://www.cs.cf.ac.uk/Dave/C/node33.html

Or if the ACTUAL problem you are trying to solve can BE
solved BEST by using RPC (note carefully the description of
RPC in the first paragraph of the link, particularly the part
about "network transport independence" in light of the
advice here about "sockets"). I'm not completely sure from
your description if that is even the case (except you seem to
require a "lock" to prevent multiple use of the same file), but if it is,
I would think that somewhere in the 128,000 or so links, you'll
have enough to proceed, or at least think about...

Now "RPC" is kind of a generalized concept that sort of came
out of other concepts such as "distributed computing", and that
it was implemented in Unix as "CORBA", and be aware
that Microsoft has their own version of it (naturally) that has gone
through several versions...I've used what they call "DCOM" in the
past, so you might want to look at that on the MSDN site (or cite!)
for stuff like that...

In any event, back the regularly-scheduled flame war!

Oops. Insight, insight, insight. My phonetic guessing due to my poor
spelling ability sometimes gets me into places I don't intend to go.
That incite got past my old Agent spell checker without a hitch. It's
not misspelled ... it's just the wrong word.

I'd have googled the same if I had been smart enough to know what to
google. There are so many methods and acronyms it makes even knowing
where to start difficult for the novice.

For now I am stuck in MicroSoftville. I realize this is disgusting for
many of the guru's here. But I do not have the resources (knowledge,
time, $$, guts) to make a major switch in my computer system at this
time.

From my side of the tracks it is tough to learn from the more
knowledgable on the other side. I think the top dogs command both
sides. But this is certainly a haven for a majority of others who have
made a choice in the past that is beyond my means for now. Please
don't hate me for my skin tone, driving a Ford, or using MS products.
It's just who and where I am and I am trying to learn how to do things
beyond my current skill level.

The flame wars confuse the novices who seek help. But once you see
past them ... there is some remarkable wit thrown around in here. At
times it is outrageously funny. It makes one grow a thicker skin and I
suppose this is a good thing too. Unfortunately it is also
intimidating at times. So when in here I try my hardest to emphasize
my low skill level.

For now I consider RPC, sockets, file writing hack, and INI techniques
to all be possibilities. I'll try to learn enough about each to make
the best choice. I wonder how many other techniques will surface? In a
networking newsgroup it was suggested I consider XML within the .Net
framework and using SOAP. <- Another new acronym in the puzzle for me
that I have not even googled yet (but I will!) I think the XML is way
out of my league. That person also seems willing to point me towards
some useful socket tutorials. There are useful tutorials and sometimes
those that just add to the confusion. I've got my fingers crossed for
some good ones to come my way.

See!! All you guys encouraged me to seek help in other groups. And I
am! I do try.

It's google time for me.

Safe holidays and Merry Xmas.

-- Tom