Design of XML log

[denzel]

Hello,

i have a big design problem. Until today, we had a text log system, wich
worked pretty good. But we decided to use XML for several reasons. This
change is a lot more complicated than we thought. The thing is that in
XML, you can't juste append some text, you need to repect a DTD... And i
can't design the new log system because of this! Here are my constraints:

1/ An application crash must be detected (this was done by the log file
still present until now)
2/ If several instance of the application are started, nothing must be
erased/lost (new constraint)
3/ The log file must keep the "extrem" events which are "throwed"
between main entry and UI appearance, and between UI disappearing and
application exiting, which make user questioning pretty hard!

So for example, let's say i'm starting the app, the UI is not yet
showned. In the log directory, there are 3 log files. I don't know
- If they belong to another application instance
- If they belong to a crashed session, in which case, i must send them,
whith user approval of course.
- If they belong to the previous, cleanly closed session, parse them to
see if they contains an exception. If they do, ask user approval to send
them.
- All of this must not take too long (i think i must use a background
thread)

It would be better to ask the user at the end of the session, but i need
to close the XML stream to be able to email it, but i also need to log
if the email sending fails...

Some others non technical constraints:
- my boss said "no sockets" for checking other instances
- my boss said "one day there won't be any bugs left in our software so
the log is no priority"
- i must finish the log system for tomorrow evening...

I'm pretty lost...

Have you ever done something like that? Have you some advices? I will
not use log4j, or similar, because i don't want to lost my work, and i'm
pretty sure we will have some specific features to add for our software.

Thanks for any help, really.

 

[Thomas Weidenfeller]

i have a big design problem. Until today, we had a text log system, wich
worked pretty good. But we decided to use XML for several reasons.

Which would be? The fact that you are now in deep trouble is apparently
because of the choice of using XML for logging. If you can rationally
argue with the decision makers try to get that decision changed ASAP.

This
change is a lot more complicated than we thought. The thing is that in
XML, you can't juste append some text, you need to repect a DTD... And i
can't design the new log system because of this!

So apparently the decision makers didn't know the implications of the
decision. If you can rationally argue with them present the facts and
try to get the decision changed.

1/ An application crash must be detected (this was done by the log file
still present until now)

A rater good indication is if the XML root element is not properly
closed (missing end tag). However, to correctly identify this condition
you would need to parse the log file.

An alternative trick is to use a different name for the log file as long
as it is incomplete. And only rename it to the final name, once you
closed the XML and are done with it.

2/ If several instance of the application are started, nothing must be
erased/lost (new constraint)

So either each instance needs its own log file, or you have to properly
merge the log data of one or more applications into one file.

For the first one you need a unique file per application instance.
File.createTempFile() can give you such files.

The later seems only be possible if you utilize some central server, a
central lock, file lock or something similar which polices the writing
to the log file. However, using a file lock can have very bad
implications on the application (instances must wait for each other).

3/ The log file must keep the "extrem" events which are "throwed"
between main entry and UI appearance, and between UI disappearing and
application exiting, which make user questioning pretty hard!

I don't know what this should mean.

Some others non technical constraints:
- my boss said "no sockets" for checking other instances

What does you boss want?

1) The guarantee that only a single instance can run
1a) on a single machine?
1b) in the local network?

2) The guarantee that multiple instanced don't mess up things?
2a) on a single machine?
2b) in the local network?

1a) Consider file locking (see the java.nio package)
1b) Sockets
2a) Use different files (see File.createTempFile())
2b) Usually the same as 2a)

- my boss said "one day there won't be any bugs left in our software so
the log is no priority"

No comment.

- i must finish the log system for tomorrow evening...

If you can do it. If not, talk to him.

Have you ever done something like that?

Yes. I have worked with XML logs. There are telecom standards where XML
log file formats are required. It ain't pretty.

Have you some advices?

Don't use XML. Talk to the decision makers - assuming they are rational.

I will
not use log4j, or similar, because i don't want to lost my work,

Huch? Sounds like the decision makers are not rational. If you
permanently live in fear it is probably time to update your CV. If this
is a reason to get fired at your company there are probably hundreds of
other irrelevant things which can get you fired, too. And sooner or
later it will happen. It sounds that it might happen happen sooner.

/Thomas

 

[denzel]

Which would be? The fact that you are now in deep trouble is apparently
because of the choice of using XML for logging. If you can rationally
argue with the decision makers try to get that decision changed ASAP.

Th emain reason is be able to use XSLT on it, for easy (future)
reporting/transformations.

The 2 feature we must have very fast is:
- coloring log entries based on gravity
- making html reports
- de-obfuscating exception stack traces

So apparently the decision makers didn't know the implications of the
decision. If you can rationally argue with them present the facts and
try to get the decision changed.

I can't do that.

A rater good indication is if the XML root element is not properly
closed (missing end tag). However, to correctly identify this condition
you would need to parse the log file.

An alternative trick is to use a different name for the log file as long
as it is incomplete. And only rename it to the final name, once you
closed the XML and are done with it.

Of course! thanks for the tip!!!

So either each instance needs its own log file, or you have to properly
merge the log data of one or more applications into one file.

For the first one you need a unique file per application instance.
File.createTempFile() can give you such files.

The later seems only be possible if you utilize some central server, a
central lock, file lock or something similar which polices the writing
to the log file. However, using a file lock can have very bad
implications on the application (instances must wait for each other).
ok

I don't know what this should mean.

well, the entry point is something like:

public static void main(String[] args) {
// 1/ init the log file
// 2/ if a log file is found, ask the user if he accept to send it

// 3/ init the UI
}

I can't ask the user before UI initialisation. Which mean i must read
all the logs, cleanup the complete logs, and ask the user if i can send
them.

Same thing when the application is exiting.

What does you boss want?

1) The guarantee that only a single instance can run
1a) on a single machine?
1b) in the local network?
nope

2) The guarantee that multiple instanced don't mess up things?
2a) on a single machine?
2b) in the local network?

actually, this is a desktop software, so local network is not an issue.
so 2a)

1a) Consider file locking (see the java.nio package)
1b) Sockets
2a) Use different files (see File.createTempFile())
2b) Usually the same as 2a)

let's go for multiple files

>
No comment.

i can't agree more

Yes. I have worked with XML logs. There are telecom standards where XML
log file formats are required. It ain't pretty.

ok so i'm not the only one facing these issues. Always good to know.

Don't use XML. Talk to the decision makers - assuming they are rational.

Well, even if XML is hard to make works, when it will work, there a lot
of thing which can be done in an easier way.

Huch? Sounds like the decision makers are not rational. If you
permanently live in fear it is probably time to update your CV. If this
is a reason to get fired at your company there are probably hundreds of
other irrelevant things which can get you fired, too. And sooner or
later it will happen. It sounds that it might happen happen sooner.

sorry. I badly choose my words. What i meant is "i don't want to lose
what i've already done". I can't go back to text log. I must find a way,
even if not perfect.


Thx a lot for your help.

 

[Andy Dingley]

i have a big design problem. Until today, we had a text log system, wich
worked pretty good. But we decided to use XML for several reasons.

Dubious decision, IMHO. Append operations suck in XML, not because the
DTD enforces validity, but because the XML syntax itself enforces
well-formedness and so you always need to keep the root elements
closing tag sitting at the back end of the file. You can't just open a
simple text file and keep appending to the tail of it, like your
Grandpa used to.

Use log4j (the panacea of logging). If you want to juggle XSLT over it
later, then take a simple closed logfile and turn it into XML (a
couple of lines of code tops - do it in Python or something
lightweight), then run XSLT over that.

 

[Daniel Pitts]

Hello,

i have a big design problem. Until today, we had a text log system, wich ....
Some others non technical constraints:
- my boss said "no sockets" for checking other instances
- my boss said "one day there won't be any bugs left in our software so
the log is no priority"
- i must finish the log system for tomorrow evening...

I'm pretty lost...

Look into log4j.

Having to finish the system by "tomorrow evening", it sounds like you
weren't given enough time for the number of problems you need to solve.

I hate to say it, but my real suggestion is start looking for a new
job. Any "boss" who says "one day there won't be any bugs left in our
software" is a terrible and unrealistic manager. Only the most trivial
software can ever be proven to be bug free.

I feel for you, I really do.

Good luck,
Daniel.

 

[Mark Rafn]

i have a big design problem. Until today, we had a text log system, wich
worked pretty good. But we decided to use XML for several reasons. This
change is a lot more complicated than we thought. The thing is that in
XML, you can't juste append some text, you need to repect a DTD.

What are the reasons you decided to use XML? I suspect they can be met by a
saner implementation than trying to write your logs as XML documents.

I'd advise staying with text logs, and converting to an XML version every
hour/day/whenever.

So for example, let's say i'm starting the app, the UI is not yet
showned. In the log directory, there are 3 log files. I don't know
- If they belong to another application instance

If there's a file that this instance didn't create, it clearly belongs to
another instance, either running or ended.

- If they belong to a crashed session, in which case, i must send them,
whith user approval of course.

If your log has heartbeats, you can just find logs that have had no activity
for N seconds, which haven't already been "sent", whatever that means.

- If they belong to the previous, cleanly closed session, parse them to
see if they contains an exception. If they do, ask user approval to send
them.

See above.

- All of this must not take too long (i think i must use a background
thread)
Fine.

It would be better to ask the user at the end of the session, but i need
to close the XML stream to be able to email it, but i also need to log
if the email sending fails...

Yeah, write a text log, but convert to XML before sending. The end of the log
will have status of the send, so future instances can easily tell whether it
needs re-sending. Or have a different directory for the archive of sent logs,
or rename the log when it gets sent, or whatever.

Some others non technical constraints:
- my boss said "no sockets" for checking other instances

Ok. Data in the logs themselves, or location/name of the logfiles then must be
sufficient to determine what to do.

- my boss said "one day there won't be any bugs left in our software so
the log is no priority"

OK. Don't do it then, and work on something that DOES have priority.
I'm serious - if your boss says this, you have no option but to (gently, if
possible) correct him. Or just ignore him and do whatever you want, but
you'll have to endure him yelling occasionally and someday find a new job.

- i must finish the log system for tomorrow evening...

Why? You said it was not a priority.

Have you ever done something like that?

Sure. Logging is a common activity. Crash detection and recovery are also
done by a lot of apps. Sending reports to a server happens all the time.

I will not use log4j, or similar, because i don't want to lost my work,

I don't understand. log4j makes a lot of things much easier, and doesn't lost
any work that I know of, whatever that means.

and i'm pretty sure we will have some specific features to add for our
software.

Sure. Which is one reason to use log4j - you can then concentrate on your
specific features rather than reinventing yet another wheel.

 

[EJP]

The XMLFormatter in java.util.logging seems to always close the file
correctly, even on a CTRL/C, at least in 1.5. I seem to remember it
didn't in 1.4. java,util.logging has its limitations but if you must
have XML logging and don't need to define the format yourself it could
be the answer.

 

[denzel]

Thursday "morning", and everything is commited

Daniel: tanks

Andy : turning the text log into XML is not that easy. An entry contains
(mainly) a title, a message, and an exception with its causes. Assuring
no loss of data would be harder than doing XML log i think (i don't know
python enough to do it).

Others: thanks for your help

Here is the final "design"/algorithm

1/ At the begining of the software, scan the log dir for log files
1a/ log files with "_complete_" in the name are put in the list
"completeLogs"
1b/ log files with "_incomplete_" in the name are put in the list
"incompleteLogs"
1c/ completeLogs are scanned, if they contains "exception", "warning"
or similar, put them in "interestingCompleteLogs"

2/ install the log in a new log file "_incomplete_"

2/ when the UI appears:
2a/ if there are incomplete logs, ask the user if he's using several
simultaneous instances
2a1/ if yes, ignore the incomplete log files
2a2/ if no, the incomplete logs are "marked" as to be sent

2b/ if "interestingCompleteLogs" is not empty, or they are incomplete
logs to send, ask the user if it's ok with him to send them

2b1/ if yes, send an email with the log files as attachements

2c/ if no exception occurs, whether or not the files have been sent
(depending of the user choice) erase them

3/ when the application is quitting, rename the current log file by
replacing "_incomplete_" with "_complete_"

The last problem which was not for yesterday evening is to handle
incomplete XML logs. I hope it can be done with Xerces SAX.

NB: one idea i didn't have time to test is to use a pseudo XML, without
headers and root tag.

 

[Andy Dingley]

NB: one idea i didn't have time to test is to use a pseudo XML, without
headers and root tag.

Search comp.text.xml There's an easy and fairly common hack to do that
"properly", by wrapping your XML file up within another and treating it
as an entity.