Looking for general advice on complex program

J

Josh English

Maybe not to the gurus here, but for me, this is a complex problem and I want to make sure I understand the real problem.

All of this is in Python 2.7 and wxPython

I have several XML files on a shared drive.
I have applications on other machines that need to access this XML file.
These applications need to read and write to this file.
These applications need to a) be alerted to a file change, or b) monitor the file for changes and regular intervals.

In my code, I have XManager classes (using a Singleton pattern) that reads each XML file into a tree (using ElementTree). The XManager class can read the file, make changes to the tree, and write the file as needed.

Now I'm expanding to the multiple application level, and I think I understand what I need to do, and I'm unsure about the exact processes.

I've been trying to have the XManagers check periodically if the XML file they monitor has changed. Since I don't want to mess up the GUI with constant hanging, I think I can use the thread or threading modules to create a recurring timed check, and then I need a separate check to see if the file isin use before reading or writing.

I also need, I think, to have a way to check if the GUI is editing a node before the XManager reads the file, so the thread needs to be interrupted, or paused, because I don't know if threads would stop if a wxDialog is beingshow modally or not.

So, am I on the right track here?

josh
 
B

Billy Mays

I remember reading that file locking doesn't work on network mounted
drives (specifically nfs mounts), but you might be able to simply create
a 'lock' (mydoc.xml.lock or the like) file for the XML doc in question.
If that file exists you could either hang or silently give up. Not
sure if that helps.
 
C

Cameron Simpson

| I remember reading that file locking doesn't work on network mounted
| drives (specifically nfs mounts), but you might be able to simply
| create a 'lock' (mydoc.xml.lock or the like) file for the XML doc in
| question. If that file exists you could either hang or silently
| give up. Not sure if that helps.

There are two approaches to this. Plain old make-a-file won't work - it
is racy (and as mentioned, you can't rely on the various lock
facilities).

You can create a file while your umask is 0777; it will be non-writable
immediately (no chmod required), preventing another attempt to make it.

My personal habit is to make a directory for the lock; mkdir
also can't happen twice to the same name, you don't need to fiddle you
umask (racy and annoying, and problematic if you're using multiple
threads), _and_ you can put meta info inside it, like pid files etc.

Cheers,
--
Cameron Simpson <[email protected]> DoD#743
http://www.cskk.ezoshosting.com/cs/

I had a wierd dream with Ken Thompson in it once.
- George Politis <[email protected]>
 
C

Chris Angelico

There are two approaches to this.
You can create a file while your umask is 0777... [or]
My personal habit is to make a directory for the lock

Both viable options; I'd be inclined toward the second. Or, here's a
third option. Instead of writing to a shared network drive, submit
requests on a TCP socket direct to the monitor program. Spin off a
thread that does this:

* Wait for incoming socket connection
* Set overall cutoff timer; if (say) 2 seconds pass, kill the connection
* Authenticate the client (if applicable)
* Accept the update data, sanitize if necessary
* Write the file to disk
* Notify the XManager
* Loop.

Do all this on *one thread* and then you eliminate all race
conditions. Good use of a TCP listen queue and the cutoff timer will
mean that applications aren't actually kept waiting, but they're still
rigidly locked into a queue - depending on how frequent your updates
are, this could be a problem. If you need simultaneous updates, spin
off a new thread for each socket connection, and then use something
simple like a mapping of file name to semaphore to ensure no two try
to update the same file at once.

By moving the actual file read/writes to a single computer, you
simplify the task of notifying the parent. In fact, if there's only
one process that needs to be made aware of the change, the job's even
easier - all you need to do is change a variable or call a method or
whatever it be, right there in the socket handler.

Chris Angelico
 
G

geremy condra

Maybe not to the gurus here, but for me, this is a complex problem and I want to make sure I understand the real problem.

All of this is in Python 2.7 and wxPython

I have several XML files on a shared drive.
I have applications on other machines that need to access this XML file.
These applications need to read and write to this file.
These applications need to a) be alerted to a file change, or b) monitor the file for changes and regular intervals.

In my code, I have XManager classes (using a Singleton pattern) that reads each XML file into a tree (using ElementTree). The XManager class can read the file, make changes to the tree, and write the file as needed.

Now I'm expanding to the multiple application level, and I think I understand what I need to do, and I'm unsure about the exact processes.

I've been trying to have the XManagers check periodically if the XML filethey monitor has changed. Since I don't want to mess up the GUI with constant hanging, I think I can use the thread or threading modules to create a recurring timed check, and then I need a separate check to see if the file is in use before reading or writing.

I also need, I think, to have a way to check if the GUI is editing a nodebefore the XManager reads the file, so the thread needs to be interrupted,or paused, because I don't know if threads would stop if a wxDialog is being show modally or not.

So, am I on the right track here?

I'd try watchdog[0] before I went to the trouble of rolling my own.

Geremy Condra

[0]: http://packages.python.org/watchdog/
 
M

Michael Hrivnak

Multiple clients reading from and writing to a central collection of
related data is a problem that has been largely solved. Use a
database, and have the clients act on it with transactions. There's
no reason to re-invent the wheel.

You could have the clients connect to the database directly over the
network. Usually not ideal, but still much better than trying to
share XML files.

Or you could write a very simple HTTP-based API, maybe even following
a basic REST pattern. Heck, a tiny Django app with this REST plugin
might do nearly all the work for you!

http://code.google.com/p/django-rest-interface/

If you aren't a fan of database design, it's ok to keep it simple.
Almost anything you do, even pickling entire XML nodes and sticking
them in a text field in the database, would be better than trying to
share XML files and deal with file locking over the network.

This approach is also likely to involve MUCH less network traffic and
consume far less resources on the clients.

Michael
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

No members online now.

Forum statistics

Threads
473,744
Messages
2,569,482
Members
44,901
Latest member
Noble71S45

Latest Threads

Top