need a good idea for catch files deleted

[elgiei]

Good morning at all,
i have to implement a server,that every n-seconds (eg. 10sec) sends to
other clients,which files and directory has been deleted or modified.

i build a n-tree, for each files on harddisk there's a node into n-
tree,
this solution is not good for large hard disk..
and i can't use inotify (it's forbidden),
and only c solutions are accepted
without third party software or external calls.
the Hard disk are both ext3 and NTFS.

anyone has a good idea for build an efficient server??

thanks a lot

elgiei

 

[Walter Roberson]

Good morning at all,
i have to implement a server,that every n-seconds (eg. 10sec) sends to
other clients,which files and directory has been deleted or modified.

i build a n-tree, for each files on harddisk there's a node into n-
tree,
this solution is not good for large hard disk..
and i can't use inotify (it's forbidden),
and only c solutions are accepted
without third party software or external calls.
the Hard disk are both ext3 and NTFS.

anyone has a good idea for build an efficient server??

No solution is possible within the constraints imposed. You are only
permitted to use C (you indicate), without third party software or
external calls, but you need external calls for two or more aspects
of the requirements:

1) C does not provide any mechanism to delay any particular amount
of time. The closest you can come is to "busy-loop", doing some
unnecessary work and checking the clock time to determine if you've
waited long enough. Busy-loops are inherently inefficient.
Most OS's provide means to delay for approximate times, but using
such an OS facility would be using an external call, not permitted
by your specifications.

2) C does not provide any mechanism to examine directories; C doesn't
know *anything* about directories. Most OS's provide means to examine
directories, but using such an OS facility would be using an external
call, not permitted by your specifications.

3) Your specifications require that information be periodically "sent"
to other clients. C does not provide any mechanims to communicate with
clients, other than whatever can be accomplished with regular files.
It is possible to come up with a reasonable information dissemination
method based upon writing out information (pre-pended with the size of
the information) to a known file, but if you want to do a traditional
client/server architecture, you would have to use OS-specific facilities
which would require external calls, not permitted by your
specifications.

i build a n-tree, for each files on harddisk there's a node into n-
tree,
this solution is not good for large hard disk..

How are you constructing your n-trees? Are you doing prefix-sharing
so that if you have figure1.eps and figure17.eps, the "figure1" portion
only gets stored once?

 

[llothar]

anyone has a good idea for build an efficient server??

Homework or real world problem ?

I'm afraid there isn't one. I used special code in my program but it
didn't work on linux reliable dnotify and on windows it always blocks
certain file operation so you can't use it either.

I came up with checking at certain times from a background thread.

 

[elgiei]

first of all thanks for your attention,
my server already works with
pthread, socket, signal for timer
and many feature offered by includes,
for external call i mean for example all the "exec" 's
family,whithin you can use any binary file on your /bin directory
(or .exe file).

my ntree has
a lot of node, the same number of files and directories.
a node is composed by n pointer one for each son

so if my hard disk is composed only by :
/home/elgiei
/home/elgiei/file2.jpg
/home/elgiei/Desktop
/home/elgiei/Desktop/file1.jpg

my ntree is :

------/home/elgiei------------------------
--------- /--------\---------------------------
file2.jpg--------/home/elgiei/Desktop
-----------------------|-----------------------
------------------file1.jpg-------------------
(remove "-" with space)

after 60secs i rebuild a new ntree and i do a diff (ntreeNEW,ntreeOLD)
and i check the differences..

but it's not a smart solution for large data.

is there a signal Handler for catch file removed ?
anyone has a better idea????

i repeat the problem
i have to implement a server,that every n-seconds (eg. 10sec) sends to
other clients,which files and directory has been deleted or
modified in the system (not by my server)
the problem is catching the deleted or modified files,whitout an
expensive ntree snapshot.

a solution:
i can read all Hard disk and i can understand which files was created
or modified in the last 60-sec,
but i don't know about the deleted files.

thanks a lot

elgiei

 

[Walter Roberson]

my server already works with
pthread, socket, signal for timer

None of those are part of standard C; those are all operating
system extensions.

and many feature offered by includes,
for external call i mean for example all the "exec" 's
family,whithin you can use any binary file on your /bin directory
(or .exe file).

This is sounding more and more like an assignment rather than a
real-life program.

my ntree has
a lot of node, the same number of files and directories.
a node is composed by n pointer one for each son
Okay.

so if my hard disk is composed only by :
/home/elgiei
/home/elgiei/file2.jpg
/home/elgiei/Desktop
/home/elgiei/Desktop/file1.jpg

my ntree is :

------/home/elgiei------------------------
--------- /--------\---------------------------
file2.jpg--------/home/elgiei/Desktop
-----------------------|-----------------------
------------------file1.jpg-------------------
(remove "-" with space)

So if there are duplicated prefixes, you store the prefix once per file?
That's less space efficient than it could be, but does make it easier
to insert or delete new nodes.

after 60secs i rebuild a new ntree and i do a diff (ntreeNEW,ntreeOLD)
and i check the differences..

but it's not a smart solution for large data.

It seems to me that you don't need to build a new ntree and do a
comparison. Provided you use a consistant ordering (e.g., depth-first,
sorted order), you should be able to start at the beginning and trace
the existing ntree through for each file: if you are processing a
file that is not part of the current n-tree then it is a new file,
and if the next file does not correspond to the next node threaded
along the leaves, then any nodes that were skipped in the process
were removed (or renamed.) It shouldn't be difficult for you to
add threading between the leaves.

is there a signal Handler for catch file removed ?

As I indicated earlier, standard C doesn't know anything about
directories -- and all it knows about files is that if you
pass a complete null-terminated string to fopen() then some
file somewhere will be opened (or the open will fail.) Standard C
doesn't know anything about what those filename strings -mean-.

Thus, of course there is no signal or any other method in standard C
of detecting that a file has been removed: the existance of such
a function would require knowing something about filesystems, which
standard C does not.

There might be a mechanism in your operating system to notice such
things, but any such facility would be OS-specific, and you would
need to inquire about it in a newsgroup that deals with your
(unnamed) operating system. (I wasn't able to deduce which OS you
are writing this for; you mentioned inotify() which appears to be
Linux-specific, and you mentioned ext3 filesystems, which appear
to be Linux-specific, but you also mentioned NTFS filesystems,
which are proprietary to Microsoft Windows.

anyone has a better idea????

What is the object of the assignment? To explore efficient use
of tree structures, or to explore how one would implement OS
facilities if those facilities were not already provided?

 

[elgiei]

None of those are part of standard C; those are all operating
system extensions.

ok i can use operative extensions infact in my code i used
#ifdef WIN32
and the else for linux..

So if there are duplicated prefixes, you store the prefix once per file?
That's less space efficient than it could be, but does make it easier
to insert or delete new nodes.

i don't know if i really understand the prefixes
in each file i already stores path/file
i don't write in that picture for space's problem.

/home/elgiei/file2.jpg--------/home/elgiei/Desktop
---------------------------------------------|-------------------
-----------------------------/home/elgiei/file1.jpg--------

and really each node has a md5 (message digest) value
it's the sum of the md5 of his son
so if nothing is changed i don't need to compare both tree, so i
compare only the modified branch
(if someone don't understand this part can asks but it it's not useful
for the solution of the problem)
That's less space efficient (really less space efficient) but it's
faster (really faster)

It seems to me that you don't need to build a new ntree and do a
comparison. Provided you use a consistant ordering (e.g., depth-first,
sorted order), you should be able to start at the beginning and trace
the existing ntree through for each file: if you are processing a
file that is not part of the current n-tree then it is a new file,
and if the next file does not correspond to the next node threaded
along the leaves, then any nodes that were skipped in the process
were removed (or renamed.) It shouldn't be difficult for you to
add threading between the leaves.

yes it's a good solution,
but i build my new ntree becouse comparison is faster,
and after 60-sec it becomes the old ntree i have to build a new ntree
for comparison.

As I indicated earlier, standard C doesn't know anything about
directories -- and all it knows about files is that if you
pass a complete null-terminated string to fopen() then some
file somewhere will be opened (or the open will fail.) Standard C
doesn't know anything about what those filename strings -mean-.

Thus, of course there is no signal or any other method in standard C
of detecting that a file has been removed: the existance of such
a function would require knowing something about filesystems, which
standard C does not.
There might be a mechanism in your operating system to notice such
things, but any such facility would be OS-specific, and you would
need to inquire about it in a newsgroup that deals with your
(unnamed) operating system. (I wasn't able to deduce which OS you
are writing this for; you mentioned inotify() which appears to be
Linux-specific, and you mentioned ext3 filesystems, which appear
to be Linux-specific, but you also mentioned NTFS filesystems,
which are proprietary to Microsoft Windows.

Yes my server must play on linux and win32
D'oh might be..
Anyone knows?
It's a great idea i'll search for win32 newsgroup and linux too
(suggested groups?)

What is the object of the assignment?
To explore efficient use
of tree structures, or to explore how one would implement OS
facilities if those facilities were not already provided?

is do a server for win and linux..
but the really problem is check in the better way,files deleted and
modified in filesystem ext3 Ntfs

 

[elgiei]

my final solution is :

save my bigs ntree in memory (i'm studing)
"Memory Mapped Files"
i hope it will be useful for the community.

 

[CBFalconer]

elgiei wrote: *** and top-posted - fixed ***

first of all thanks for your attention, my server already works
with pthread, socket, signal for timer and many feature offered by
includes, for external call i mean for example all the "exec" 's
family,whithin you can use any binary file on your /bin directory
(or .exe file).

my ntree has a lot of node, the same number of files and
directories. a node is composed by n pointer one for each son

so if my hard disk is composed only by :
/home/elgiei
/home/elgiei/file2.jpg
/home/elgiei/Desktop
/home/elgiei/Desktop/file1.jpg

my ntree is :

------/home/elgiei------------------------
--------- /--------\---------------------------
file2.jpg--------/home/elgiei/Desktop
-----------------------|-----------------------
------------------file1.jpg-------------------
(remove "-" with space)

after 60secs i rebuild a new ntree and i do a diff
(ntreeNEW,ntreeOLD) and i check the differences..
but it's not a smart solution for large data.

is there a signal Handler for catch file removed ?
anyone has a better idea????

i repeat the problem
i have to implement a server,that every n-seconds (eg. 10sec)
sends to other clients,which files and directory has been
deleted or modified in the system (not by my server)
the problem is catching the deleted or modified files,whitout an
expensive ntree snapshot.

a solution:
i can read all Hard disk and i can understand which files was
created or modified in the last 60-sec,
but i don't know about the deleted files.

I have preserved your entire post, less sigs, while fixing the
top-post.

Please do not top-post. Your answer belongs after (or intermixed
with) the quoted material to which you reply, after snipping all
irrelevant material. See the following links:

--
<http://www.catb.org/~esr/faqs/smart-questions.html>
<http://www.caliburn.nl/topposting.html>
<http://www.netmeister.org/news/learn2quote.html>
<http://cfaj.freeshell.org/google/> (taming google)
<http://members.fortunecity.com/nnqweb/> (newusers)