VFAT + Linux/Knoppix bug?

Discussion in 'Java' started by lbrtchx@hotmail.com, May 7, 2007.

  1. Guest

    Hi,
    ~
    I just noticed a bug the while coding, that made me spend quite a big
    of time eyeballing, cleaning up and mentally remapping my code and I
    believe to have reduced to a bug caused by some nasty VFAT + Linux/
    Knoppix impedance
    ~
    1._ I am using Linux/Knoppix 5.1.1 based on the 2.6.19 Linux kernel
    ~
    sh-3.1# su -
    root@Knoppix:~# uname -r
    2.6.19
    ~
    2._ which I booted with the cheatcode
    ~
    "knoppix acpi=off noapic testcd init 2"
    ~
    because the Linux/knoppix version 5.1.1 based on the 2.6.19 fills the
    dmesg output with errors:
    CPU0: 01(01)
    APIC error on CPU0: 01(01) (<- lots of these)
    spurious APIC interrupt on CPU#0, should never happen. (also, but not
    as many as the previous one)
    ~
    3._ I dumped the output of dmesg
    ~
    root!tty1:/# dmesg > 00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt
    ~
    4._ I tried mounting the vfat fs /dev/hdb6 using
    ~
    root!tty1:/# mount /media/hdb6
    ~
    but then while moving/copying files to directories in this fs I would
    get
    ~
    root!tty1:/# mv *.txt /media/hdb6/bugs
    root!tty1:/# mv failed to preserve ownership for '/media/hdb6/bugs/
    00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt' Operation not permitted
    ~
    mounting the fs using the stanza
    ~
    root!tty1:/# mount -t vfat /dev/hdb6 /media/hdb6
    ~
    appears to rid you of these ownership transfer losses
    ~
    5._ I then started X via
    ~
    root!tty1:/# startx
    ~
    6._ opened a shell went to a folder in the vfat fs and created files
    with the same name but with the difference of a lower and upper case
    extension
    ~
    root@Knoppix:/media/hdb6/prjx/bugs# echo "test lower case extension" >
    test.txt
    root@Knoppix:/media/hdb6/prjx/bugs# echo "test upper case extension" >
    test.TXT
    ~
    7._ but even if the file with the upper case version is there (which
    you only see by invoking it with its full name)
    ~
    root@Knoppix:/media/hdb6/prjx/bugs# ls -l test.TXT
    -rwxr-xr-x 1 root root 26 May 7 06:40 test.TXT
    ~
    Linux is not 'seeing it' if you wild card expressions or just list
    all files
    ~
    root@Knoppix:/media/hdb6/prjx/bugs# ls -l *.TXT
    ls: *.TXT: No such file or directory
    root@Knoppix:/media/hdb6/prjx/bugs# ls -l
    total 176
    -rwxr-xr-x 1 root root 12889 May 7 05:59
    00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt
    -rwxr-xr-x 1 root root 12889 May 7 05:59 02_dmesg_mount_-
    t_vfat_dev_hdb6_media_hdb6.txt
    -rwxr-xr-x 1 root root 29501 May 7 05:05
    1st_trial_00_dmesg_after_boot.txt
    -rwxr-xr-x 1 root root 650 May 7 05:12
    1st_trial_04_fdisk_dev_hdb.txt
    -rwxr-xr-x 1 root root 1542 May 7 06:39 LinuxVFatBug00.txt
    -rwxr-xr-x 1 root root 1342 May 7 06:35 LinuxVFatBug00.txt~
    -rwxr-xr-x 1 root root 3687 May 7 05:59 cp--help.txt
    -rwxr-xr-x 1 root root 208 May 7 06:02 md5sum00.txt
    -rwxr-xr-x 1 root root 526 May 7 06:02 md5sum02.txt
    -rwxr-xr-x 1 root root 26 May 7 06:40 test.txt
    ~
    root@Knoppix:/media/hdb6/prjx/bugs# ls -l *.txt
    -rwxr-xr-x 1 root root 12889 May 7 05:59
    00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt
    -rwxr-xr-x 1 root root 12889 May 7 05:59 02_dmesg_mount_-
    t_vfat_dev_hdb6_media_hdb6.txt
    -rwxr-xr-x 1 root root 29501 May 7 05:05
    1st_trial_00_dmesg_after_boot.txt
    -rwxr-xr-x 1 root root 650 May 7 05:12
    1st_trial_04_fdisk_dev_hdb.txt
    -rwxr-xr-x 1 root root 1542 May 7 06:39 LinuxVFatBug00.txt
    -rwxr-xr-x 1 root root 3687 May 7 05:59 cp--help.txt
    -rwxr-xr-x 1 root root 208 May 7 06:02 md5sum00.txt
    -rwxr-xr-x 1 root root 526 May 7 06:02 md5sum02.txt
    -rwxr-xr-x 1 root root 26 May 7 06:40 test.txt
    ~
    root@Knoppix:/media/hdb6/prjx/bugs# md5sum *.txt
    4d0a8122383095005be55c8daed3aac9
    00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt
    4d0a8122383095005be55c8daed3aac9 02_dmesg_mount_-
    t_vfat_dev_hdb6_media_hdb6.txt
    0291b2e145614198136599426f012e42 1st_trial_00_dmesg_after_boot.txt
    3120b0f6172325eb4fad5c502b390b7e 1st_trial_04_fdisk_dev_hdb.txt
    8380bad18b88152267686976bcbf5213 LinuxVFatBug00.txt
    f7dd8b7cc5d6115be6b09186106862d1 cp--help.txt
    7b9be97a510ba89b8d06cdff28e02ed8 md5sum00.txt
    ec0a023c6a8c31c47232640a16f27ea4 md5sum02.txt
    76744305c32618225193b5556c8b091e test.txt
    ~
    root@Knoppix:/media/hdb6/prjx/bugs# md5sum *.TXT
    md5sum: *.TXT: No such file or directory
    ~
    Obviously this impedance regarding file names/extensions would
    transpire to the JVM, since it works on top of the OS services +
    utilities and the OS + VFAT FS
    ~
    However what I don't quite get is why a class implementing
    http://java.sun.com/javase/6/docs/api/index.html such as:
    ~
    // __
    class DirFilter implements FilenameFilter{
    public boolean accept(File FlDir, String aFl){
    boolean IsAccept = (FlDir.isDirectory());
    if(IsAccept){
    File Fl = new File(FlDir, aFl);
    IsAccept = Fl.exists();
    if(IsAccept){
    aFl = Fl.getAbsolutePath();
    if(!IsAll){
    IsAccept = false;
    for(int i = 0; (i < aExtAr.length) && !IsAccept; ++i){
    IsAccept = aFl.endsWith(aExtAr);
    }
    }// (!IsAll)
    }// (Fl.exists())
    }// (FlDir.isDirectory())
    // __
    return(IsAccept);
    }
    }
    ~
    would not pick all files with both extensions
    ~
    when you go
    ~
    DirFilter FlsFltr = new DirFilter();
    File[] XFls = FlDir.listFiles(FlsFltr);
    ~
    even if it detected in its accept implemented method both extensions
    just fine!!!
    ~
    I degugged and eyeballed it carefully, but you should check it out
    yourself
    ~
    I think there is impedance among the three parts:
    ~
    * the OS Linux honoring this MS FAT32 requirements file system
    filenames in which you get filenames stored in a case-sensitive
    manner, but
    filenames being treated as if they were case-insensitive. So this
    might be creating this file name "ghosts" problem
    * the JVM not knowing what to do with this conflict
    ~
    I code mostly Java and I need to use FAT32 to keep the files in my
    external drive easily accessible when I am working in a Linux,
    Solaris, WIndows or MacOS box, but this filename thing is creating
    problems
    ~
    How do people keep their data portable regarding file system names?
    Just using all lower or uppercase?
    ~
    Thanks
    lbrtchx
    , May 7, 2007
    #1
    1. Advertising

  2. [f'up-to ignored. post here, read here]

    (solution to problem further down in this post)

    <> wrote:
    > root!tty1:/# mv failed to preserve ownership for '/media/hdb6/bugs/
    > 00_dmesg_acpi_eq_off_noapic_testcd_init_2.txt' Operation not permitted


    This is irrelevant. (this is not critizism, but information)
    It means, that vfat doesn't support changing of file-owner
    as attemted by the "mv" command.

    > root!tty1:/# mount -t vfat /dev/hdb6 /media/hdb6
    > appears to rid you of these ownership transfer losses


    No, it doesn't. If you now no longer get these warnings,
    this must have a different reason.

    > root@Knoppix:/media/hdb6/prjx/bugs# echo "test lower case extension" >
    > test.txt
    > root@Knoppix:/media/hdb6/prjx/bugs# echo "test upper case extension" >
    > test.TXT


    Both were targetted to the *same* file!

    vfat is not case-sensitive, but case-preserving.
    the first echo created a file "test.txt", the second
    recognized test.TXT as being already there, and overwrote
    its contents. test.txt now contains "... upper ...".
    Since it was already there, it preserved the filename
    under which it was created (not last accessed)

    If you then access it with a differently cased name, the
    system will actually find and access it, but if you want
    to apply wildcards, then unix-case-sensitivity creeps in,
    and it won't find files that don't match the pattern in
    a case-sensitive way.

    > root@Knoppix:/media/hdb6/prjx/bugs# ls -l test.TXT
    > -rwxr-xr-x 1 root root 26 May 7 06:40 test.TXT


    > root@Knoppix:/media/hdb6/prjx/bugs# ls -l *.TXT
    > ls: *.TXT: No such file or directory


    ....$ ls -l test.*
    -rwxr-xr-x 1 root root 26 May 7 06:40 test.txt

    > Obviously this impedance regarding file names/extensions would
    > transpire to the JVM, since it works on top of the OS services +
    > utilities and the OS + VFAT FS


    sure it does. but the rules are easy:
    you can access a file on a case-insensitive filesystem with
    any case you like.
    If you use wildcards, the wildcard-matching is case-sensitive.

    > IsAccept = aFl.endsWith(aExtAr);

    This is the culprit: here you do the case-sensitive filtering yourself!

    If you need to deal with case-insensitive filesystems, you had better
    do it so:
    IsAccept = aFl.toLowerCase().endsWith(aExtAr.toLowerCase());
    (or likewise with toUpperCase() in both places.)

    > How do people keep their data portable regarding file system names?
    > Just using all lower or uppercase?


    If you need to select by extension, and the extension of the files
    could be any case, you need extra care for filtering (making the
    filtering itself case-insensitive).

    I don't think there is a way to do this automatically for
    files on case-insensitive filesystems only.

    PS: I've seen that one cannot rename a file on a case-INsensitive
    filesystem from one case to another. Not even under Windows.
    you have to rename it to a dummy name first, then rename it back
    to proper case.
    Andreas Leitgeb, May 7, 2007
    #2
    1. Advertising

  3. Andreas Leitgeb wrote:
    >
    >> IsAccept = aFl.endsWith(aExtAr);

    > This is the culprit: here you do the case-sensitive filtering yourself!
    >
    > If you need to deal with case-insensitive filesystems, you had better
    > do it so:
    > IsAccept = aFl.toLowerCase().endsWith(aExtAr.toLowerCase());
    > (or likewise with toUpperCase() in both places.)
    >
    >> How do people keep their data portable regarding file system names?
    >> Just using all lower or uppercase?

    >
    > If you need to select by extension, and the extension of the files
    > could be any case, you need extra care for filtering (making the
    > filtering itself case-insensitive).
    >


    This is harder to do correctly than it might seem. The Windows
    comparison of filenames is not the same as a Java case insensitive
    comparison. Examples where it differs include various Turkish
    dotted/dotless letter I. The best you can do is to convert the file
    names to UPPER case and compare that. Do NOT convert to lower case, the
    result is different for those Turkish characters and some others.

    If your operating system is Mac OSX, which by default also has a case
    insensitive file system, then the file name comparison is probably
    different.

    Mark Thornton
    Mark Thornton, May 7, 2007
    #3
  4. Mark Thornton <> wrote:
    > Andreas Leitgeb wrote:
    >>> IsAccept = aFl.endsWith(aExtAr);

    >> This is the culprit: here you do the case-sensitive filtering yourself!
    >> If you need to deal with case-insensitive filesystems, you had better
    >> do it so:
    >> IsAccept = aFl.toLowerCase().endsWith(aExtAr.toLowerCase());
    >> (or likewise with toUpperCase() in both places.)

    >
    > This is harder to do correctly than it might seem. The Windows
    > comparison of filenames is not the same as a Java case insensitive
    > comparison. Examples where it differs include various Turkish
    > dotted/dotless letter I. The best you can do is to convert the file
    > names to UPPER case and compare that.


    If you go to international territory, you really can't win, anyway ;-/

    I didn't know about these OS(*)-peculiarities. I really thought, that
    unicode was clear about case-mapping and character-equivalence, and
    that windows just like java would adhere to unicode standard (with
    all its immanent faults).

    Anyway, thanks for the hint of transforming to uppercase. It seems
    to be at least better than lowercase-folding, when we target
    international scripts.

    Another example: In greek there are also two versions of sigma which
    then translate to same capital. I don't know whether windows
    considers the two lowercase versions equivalent or not.
    Greek also becomes fun with accented (tonos) vowels - are these
    equivalent to their base-versions in Windows?
    Einai ola ellhnika gia mena...

    (*) actually they are human-language-peculiarities, which some pieces of
    software account for better than others, or interpret differently.
    Andreas Leitgeb, May 8, 2007
    #4
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Fred LaBar

    mosix knoppix and JBoss

    Fred LaBar, Jul 14, 2004, in forum: Java
    Replies:
    0
    Views:
    446
    Fred LaBar
    Jul 14, 2004
  2. Replies:
    0
    Views:
    497
  3. Jon Monteleone

    Installing Tkinter on knoppix

    Jon Monteleone, Nov 9, 2005, in forum: Python
    Replies:
    2
    Views:
    277
    Michele Simionato
    Nov 9, 2005
  4. Thierry Lam
    Replies:
    1
    Views:
    326
    Jonathan N. Little
    Aug 9, 2007
  5. RoboHobby

    How to update Java on KNOPPIX.

    RoboHobby, Nov 16, 2007, in forum: Java
    Replies:
    0
    Views:
    286
    RoboHobby
    Nov 16, 2007
Loading...

Share This Page