Help in file locking and synchronization

[ramesh.thangamani]

I am working on a script which queries LDAP for a group of people and
stores the details in XML preserving the hierarchy. I have used a perl
module to create the XML file, the same module can parse it when
needed. I will also have a perl script to create the XML file daily so
that the details in the XML file are in sync with the LDAP details.

Now the problem here is there can be synchronization problem when the
XML is getting generated
by more than one user running the script at the same time. What could
be the best approach for this problem. Any suggestions ?

- I have seen file locking stuff but not sure what needs to be done
to avoid any issues .
- Will be better to have some timeout and wait to write the file ?.
- Or should I skip if the file has already been created like that etc.

 

[Skye Shaw!@#$]

I am working on a script which queries LDAP for a group of people and
stores the details in XML preserving the hierarchy.

Now the problem here is there can be synchronization problem when the
XML is getting generated
by more than one user running the script at the same time. What could
be the best approach for this problem. Any suggestions ?

perldoc -q lock

- Will be better to have some timeout and wait to write the file ?.

Well flock() will block if another process holds a lock on the given
file. Although the blocking can be indefinite.

- Or should I skip if the file has already been created like that etc.

You say that the XML file should be /in sync/ with LDAP. Process A
generates the XML, the LDAP structure is changed, then process B
starts. B will pull the most accurate data, so skipping due to the
file created by A would not be in line with your requirements.

You can also avoid any file locking and just have your program attempt
to open a lock file via sysopen and the O_EXCL and O_CREAT flags.

 

[xhoster]

I am working on a script which queries LDAP for a group of people and
stores the details in XML preserving the hierarchy. I have used a perl
module to create the XML file, the same module can parse it when
needed. I will also have a perl script to create the XML file daily so
that the details in the XML file are in sync with the LDAP details.

Now the problem here is there can be synchronization problem when the
XML is getting generated
by more than one user running the script at the same time. What could
be the best approach for this problem. Any suggestions ?

- I have seen file locking stuff but not sure what needs to be done
to avoid any issues .

It depends on exactly what issues you want to avoid. You could have
your programs, when they decide to build a new file, do so in a local
uniquely named temp file, then rename that to be the standard location.
(I don't know how this will work on Windows, renaming an open file or
renaming a file to overwrite the entry for an open file). This should be
OK in the sense that the file should not be corrupted by multiple writers,
but it won't prevent 20 programs from all slamming your LDAP server at the
same time, each trying to build the new file.

- Will be better to have some timeout and wait to write the file ?.

It is hard to say what this even means. What operation is being timed
out? How do you detect it?

- Or should I skip if the file has already been created like that etc.

Again, I don't know what this means. Could you show the proposed code that
would implement it?

Xho

 

[ramesh.thangamani]

How big is the XML file? If not too big then you can lock LDAP table (I
suppose MySQL), generate XML on the fly and unlock LDAP table.

--

Petr Vileta, Czech republic
(My server rejects all messages from Yahoo and Hotmail. Send me your mail
from another non-spammer site please.)

The XML file is just 65 KB.

 

[ramesh.thangamani]

It depends on exactly what issues you want to avoid. You could have
your programs, when they decide to build a new file, do so in a local
uniquely named temp file, then rename that to be the standard location.
(I don't know how this will work on Windows, renaming an open file or
renaming a file to overwrite the entry for an open file). This should be
OK in the sense that the file should not be corrupted by multiple writers,
but it won't prevent 20 programs from all slamming your LDAP server at the
same time, each trying to build the new file.

I am working on Linux platform so no need to worry about Windows
issues. In addition to the LDAP details i query database to add some
more details, so my aim is to avoid LDAP and DB calls so that i can
query just the XML file for details. Creating a temp file is nice. May
be if user asks for XML file generation i should create a temporary
file and then try to make it the original XML file if no one is trying
to write, but will this be a better idea ?.

 

[xhoster]

I am working on Linux platform so no need to worry about Windows
issues. In addition to the LDAP details i query database to add some
more details, so my aim is to avoid LDAP and DB calls so that i can
query just the XML file for details. Creating a temp file is nice. May
be if user asks for XML file generation i should create a temporary
file and then try to make it the original XML file if no one is trying
to write, but will this be a better idea ?.

You probably don't need to worry about other people trying to write.
The main file is never written to, it is only ever read, or replaced
with a new (already written, as a temporary file) file by a "rename". So
at no time is the main file written to. You will want the temp file to be
on the same device as the main file it is to replace, so that rename
is atomic.

Xho