Design of XML log

Discussion in 'Java' started by denzel, Nov 7, 2006.

  1. denzel

    denzel Guest

    Hello,

    i have a big design problem. Until today, we had a text log system, wich
    worked pretty good. But we decided to use XML for several reasons. This
    change is a lot more complicated than we thought. The thing is that in
    XML, you can't juste append some text, you need to repect a DTD... And i
    can't design the new log system because of this! Here are my constraints:

    1/ An application crash must be detected (this was done by the log file
    still present until now)
    2/ If several instance of the application are started, nothing must be
    erased/lost (new constraint)
    3/ The log file must keep the "extrem" events which are "throwed"
    between main entry and UI appearance, and between UI disappearing and
    application exiting, which make user questioning pretty hard!

    So for example, let's say i'm starting the app, the UI is not yet
    showned. In the log directory, there are 3 log files. I don't know
    - If they belong to another application instance
    - If they belong to a crashed session, in which case, i must send them,
    whith user approval of course.
    - If they belong to the previous, cleanly closed session, parse them to
    see if they contains an exception. If they do, ask user approval to send
    them.
    - All of this must not take too long (i think i must use a background
    thread)

    It would be better to ask the user at the end of the session, but i need
    to close the XML stream to be able to email it, but i also need to log
    if the email sending fails...

    Some others non technical constraints:
    - my boss said "no sockets" for checking other instances
    - my boss said "one day there won't be any bugs left in our software so
    the log is no priority"
    - i must finish the log system for tomorrow evening...

    I'm pretty lost...

    Have you ever done something like that? Have you some advices? I will
    not use log4j, or similar, because i don't want to lost my work, and i'm
    pretty sure we will have some specific features to add for our software.

    Thanks for any help, really.
    denzel, Nov 7, 2006
    #1
    1. Advertising

  2. denzel wrote:
    > i have a big design problem. Until today, we had a text log system, wich
    > worked pretty good. But we decided to use XML for several reasons.


    Which would be? The fact that you are now in deep trouble is apparently
    because of the choice of using XML for logging. If you can rationally
    argue with the decision makers try to get that decision changed ASAP.


    > This
    > change is a lot more complicated than we thought. The thing is that in
    > XML, you can't juste append some text, you need to repect a DTD... And i
    > can't design the new log system because of this!


    So apparently the decision makers didn't know the implications of the
    decision. If you can rationally argue with them present the facts and
    try to get the decision changed.

    > 1/ An application crash must be detected (this was done by the log file
    > still present until now)


    A rater good indication is if the XML root element is not properly
    closed (missing end tag). However, to correctly identify this condition
    you would need to parse the log file.

    An alternative trick is to use a different name for the log file as long
    as it is incomplete. And only rename it to the final name, once you
    closed the XML and are done with it.

    > 2/ If several instance of the application are started, nothing must be
    > erased/lost (new constraint)


    So either each instance needs its own log file, or you have to properly
    merge the log data of one or more applications into one file.

    For the first one you need a unique file per application instance.
    File.createTempFile() can give you such files.

    The later seems only be possible if you utilize some central server, a
    central lock, file lock or something similar which polices the writing
    to the log file. However, using a file lock can have very bad
    implications on the application (instances must wait for each other).


    > 3/ The log file must keep the "extrem" events which are "throwed"
    > between main entry and UI appearance, and between UI disappearing and
    > application exiting, which make user questioning pretty hard!


    I don't know what this should mean.

    > Some others non technical constraints:
    > - my boss said "no sockets" for checking other instances


    What does you boss want?

    1) The guarantee that only a single instance can run
    1a) on a single machine?
    1b) in the local network?

    2) The guarantee that multiple instanced don't mess up things?
    2a) on a single machine?
    2b) in the local network?

    1a) Consider file locking (see the java.nio package)
    1b) Sockets
    2a) Use different files (see File.createTempFile())
    2b) Usually the same as 2a)

    > - my boss said "one day there won't be any bugs left in our software so
    > the log is no priority"


    No comment.

    > - i must finish the log system for tomorrow evening...


    If you can do it. If not, talk to him.

    > Have you ever done something like that?


    Yes. I have worked with XML logs. There are telecom standards where XML
    log file formats are required. It ain't pretty.

    > Have you some advices?


    Don't use XML. Talk to the decision makers - assuming they are rational.

    > I will
    > not use log4j, or similar, because i don't want to lost my work,


    Huch? Sounds like the decision makers are not rational. If you
    permanently live in fear it is probably time to update your CV. If this
    is a reason to get fired at your company there are probably hundreds of
    other irrelevant things which can get you fired, too. And sooner or
    later it will happen. It sounds that it might happen happen sooner.

    /Thomas
    --
    The comp.lang.java.gui FAQ:
    http://gd.tuwien.ac.at/faqs/faqs-hierarchy/comp/comp.lang.java.gui/
    ftp://ftp.cs.uu.nl/pub/NEWS.ANSWERS/computer-lang/java/gui/faq
    Thomas Weidenfeller, Nov 8, 2006
    #2
    1. Advertising

  3. denzel

    denzel Guest

    Thomas Weidenfeller wrote:

    > Which would be? The fact that you are now in deep trouble is apparently
    > because of the choice of using XML for logging. If you can rationally
    > argue with the decision makers try to get that decision changed ASAP.


    Th emain reason is be able to use XSLT on it, for easy (future)
    reporting/transformations.

    The 2 feature we must have very fast is:
    - coloring log entries based on gravity
    - making html reports
    - de-obfuscating exception stack traces

    > So apparently the decision makers didn't know the implications of the
    > decision. If you can rationally argue with them present the facts and
    > try to get the decision changed.


    I can't do that.

    > A rater good indication is if the XML root element is not properly
    > closed (missing end tag). However, to correctly identify this condition
    > you would need to parse the log file.
    >
    > An alternative trick is to use a different name for the log file as long
    > as it is incomplete. And only rename it to the final name, once you
    > closed the XML and are done with it.


    Of course! thanks for the tip!!!

    > So either each instance needs its own log file, or you have to properly
    > merge the log data of one or more applications into one file.
    >
    > For the first one you need a unique file per application instance.
    > File.createTempFile() can give you such files.
    >
    > The later seems only be possible if you utilize some central server, a
    > central lock, file lock or something similar which polices the writing
    > to the log file. However, using a file lock can have very bad
    > implications on the application (instances must wait for each other).


    ok

    > I don't know what this should mean.


    well, the entry point is something like:

    public static void main(String[] args) {
    // 1/ init the log file
    // 2/ if a log file is found, ask the user if he accept to send it

    // 3/ init the UI
    }

    I can't ask the user before UI initialisation. Which mean i must read
    all the logs, cleanup the complete logs, and ask the user if i can send
    them.

    Same thing when the application is exiting.


    > What does you boss want?
    >
    > 1) The guarantee that only a single instance can run
    > 1a) on a single machine?
    > 1b) in the local network?


    nope

    > 2) The guarantee that multiple instanced don't mess up things?
    > 2a) on a single machine?
    > 2b) in the local network?


    actually, this is a desktop software, so local network is not an issue.
    so 2a)


    > 1a) Consider file locking (see the java.nio package)
    > 1b) Sockets
    > 2a) Use different files (see File.createTempFile())
    > 2b) Usually the same as 2a)


    let's go for multiple files

    >> - my boss said "one day there won't be any bugs left in our software
    >> so the log is no priority"

    >
    > No comment.


    i can't agree more


    > Yes. I have worked with XML logs. There are telecom standards where XML
    > log file formats are required. It ain't pretty.


    ok so i'm not the only one facing these issues. Always good to know.

    > Don't use XML. Talk to the decision makers - assuming they are rational.


    Well, even if XML is hard to make works, when it will work, there a lot
    of thing which can be done in an easier way.


    > Huch? Sounds like the decision makers are not rational. If you
    > permanently live in fear it is probably time to update your CV. If this
    > is a reason to get fired at your company there are probably hundreds of
    > other irrelevant things which can get you fired, too. And sooner or
    > later it will happen. It sounds that it might happen happen sooner.


    sorry. I badly choose my words. What i meant is "i don't want to lose
    what i've already done". I can't go back to text log. I must find a way,
    even if not perfect.


    Thx a lot for your help.
    denzel, Nov 8, 2006
    #3
  4. denzel

    Andy Dingley Guest

    denzel wrote:

    > i have a big design problem. Until today, we had a text log system, wich
    > worked pretty good. But we decided to use XML for several reasons.


    Dubious decision, IMHO. Append operations suck in XML, not because the
    DTD enforces validity, but because the XML syntax itself enforces
    well-formedness and so you always need to keep the root elements
    closing tag sitting at the back end of the file. You can't just open a
    simple text file and keep appending to the tail of it, like your
    Grandpa used to.

    Use log4j (the panacea of logging). If you want to juggle XSLT over it
    later, then take a simple closed logfile and turn it into XML (a
    couple of lines of code tops - do it in Python or something
    lightweight), then run XSLT over that.
    Andy Dingley, Nov 8, 2006
    #4
  5. denzel

    Daniel Pitts Guest

    denzel wrote:
    > Hello,
    >
    > i have a big design problem. Until today, we had a text log system, wich

    ....
    > Some others non technical constraints:
    > - my boss said "no sockets" for checking other instances
    > - my boss said "one day there won't be any bugs left in our software so
    > the log is no priority"
    > - i must finish the log system for tomorrow evening...
    >
    > I'm pretty lost...


    Look into log4j.

    Having to finish the system by "tomorrow evening", it sounds like you
    weren't given enough time for the number of problems you need to solve.

    I hate to say it, but my real suggestion is start looking for a new
    job. Any "boss" who says "one day there won't be any bugs left in our
    software" is a terrible and unrealistic manager. Only the most trivial
    software can ever be proven to be bug free.

    I feel for you, I really do.

    Good luck,
    Daniel.
    Daniel Pitts, Nov 8, 2006
    #5
  6. denzel

    Mark Rafn Guest

    denzel <> wrote:
    >i have a big design problem. Until today, we had a text log system, wich
    >worked pretty good. But we decided to use XML for several reasons. This
    >change is a lot more complicated than we thought. The thing is that in
    >XML, you can't juste append some text, you need to repect a DTD.


    What are the reasons you decided to use XML? I suspect they can be met by a
    saner implementation than trying to write your logs as XML documents.

    I'd advise staying with text logs, and converting to an XML version every
    hour/day/whenever.

    >So for example, let's say i'm starting the app, the UI is not yet
    >showned. In the log directory, there are 3 log files. I don't know
    >- If they belong to another application instance


    If there's a file that this instance didn't create, it clearly belongs to
    another instance, either running or ended.

    >- If they belong to a crashed session, in which case, i must send them,
    >whith user approval of course.


    If your log has heartbeats, you can just find logs that have had no activity
    for N seconds, which haven't already been "sent", whatever that means.

    >- If they belong to the previous, cleanly closed session, parse them to
    >see if they contains an exception. If they do, ask user approval to send
    >them.


    See above.

    >- All of this must not take too long (i think i must use a background
    >thread)


    Fine.

    >It would be better to ask the user at the end of the session, but i need
    >to close the XML stream to be able to email it, but i also need to log
    >if the email sending fails...


    Yeah, write a text log, but convert to XML before sending. The end of the log
    will have status of the send, so future instances can easily tell whether it
    needs re-sending. Or have a different directory for the archive of sent logs,
    or rename the log when it gets sent, or whatever.

    >Some others non technical constraints:
    >- my boss said "no sockets" for checking other instances


    Ok. Data in the logs themselves, or location/name of the logfiles then must be
    sufficient to determine what to do.

    >- my boss said "one day there won't be any bugs left in our software so
    >the log is no priority"


    OK. Don't do it then, and work on something that DOES have priority.
    I'm serious - if your boss says this, you have no option but to (gently, if
    possible) correct him. Or just ignore him and do whatever you want, but
    you'll have to endure him yelling occasionally and someday find a new job.

    >- i must finish the log system for tomorrow evening...


    Why? You said it was not a priority.

    >Have you ever done something like that?


    Sure. Logging is a common activity. Crash detection and recovery are also
    done by a lot of apps. Sending reports to a server happens all the time.

    >I will not use log4j, or similar, because i don't want to lost my work,


    I don't understand. log4j makes a lot of things much easier, and doesn't lost
    any work that I know of, whatever that means.

    >and i'm pretty sure we will have some specific features to add for our
    >software.


    Sure. Which is one reason to use log4j - you can then concentrate on your
    specific features rather than reinventing yet another wheel.
    --
    Mark Rafn <http://www.dagon.net/>
    Mark Rafn, Nov 8, 2006
    #6
  7. denzel

    EJP Guest

    The XMLFormatter in java.util.logging seems to always close the file
    correctly, even on a CTRL/C, at least in 1.5. I seem to remember it
    didn't in 1.4. java,util.logging has its limitations but if you must
    have XML logging and don't need to define the format yourself it could
    be the answer.
    EJP, Nov 9, 2006
    #7
  8. denzel

    denzel Guest

    Thursday "morning", and everything is commited :)

    Daniel: tanks

    Andy : turning the text log into XML is not that easy. An entry contains
    (mainly) a title, a message, and an exception with its causes. Assuring
    no loss of data would be harder than doing XML log i think (i don't know
    python enough to do it).

    Others: thanks for your help

    Here is the final "design"/algorithm

    1/ At the begining of the software, scan the log dir for log files
    1a/ log files with "_complete_" in the name are put in the list
    "completeLogs"
    1b/ log files with "_incomplete_" in the name are put in the list
    "incompleteLogs"
    1c/ completeLogs are scanned, if they contains "exception", "warning"
    or similar, put them in "interestingCompleteLogs"

    2/ install the log in a new log file "_incomplete_"

    2/ when the UI appears:
    2a/ if there are incomplete logs, ask the user if he's using several
    simultaneous instances
    2a1/ if yes, ignore the incomplete log files
    2a2/ if no, the incomplete logs are "marked" as to be sent

    2b/ if "interestingCompleteLogs" is not empty, or they are incomplete
    logs to send, ask the user if it's ok with him to send them

    2b1/ if yes, send an email with the log files as attachements

    2c/ if no exception occurs, whether or not the files have been sent
    (depending of the user choice) erase them

    3/ when the application is quitting, rename the current log file by
    replacing "_incomplete_" with "_complete_"

    The last problem which was not for yesterday evening is to handle
    incomplete XML logs. I hope it can be done with Xerces SAX.

    NB: one idea i didn't have time to test is to use a pseudo XML, without
    headers and root tag.

    :D
    denzel, Nov 9, 2006
    #8
  9. denzel

    Andy Dingley Guest

    denzel wrote:

    > NB: one idea i didn't have time to test is to use a pseudo XML, without
    > headers and root tag.


    Search comp.text.xml There's an easy and fairly common hack to do that
    "properly", by wrapping your XML file up within another and treating it
    as an entity.
    Andy Dingley, Nov 9, 2006
    #9
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Henrik_the_boss
    Replies:
    0
    Views:
    2,643
    Henrik_the_boss
    Nov 5, 2003
  2. Amratash
    Replies:
    0
    Views:
    517
    Amratash
    Apr 13, 2004
  3. =?Utf-8?B?VG9tIFdpbmdlcnQ=?=

    My.Log.Writeexception not writing to Application Event Log.

    =?Utf-8?B?VG9tIFdpbmdlcnQ=?=, Jan 20, 2006, in forum: ASP .Net
    Replies:
    0
    Views:
    2,369
    =?Utf-8?B?VG9tIFdpbmdlcnQ=?=
    Jan 20, 2006
  4. unomystEz
    Replies:
    0
    Views:
    542
    unomystEz
    Nov 19, 2006
  5. vj
    Replies:
    0
    Views:
    677
Loading...

Share This Page