return to the begin of InputStream

T

Tom Anderson

(Marginally topical) [...]

Also, it's a *very* bad idea to purge /tmp blindly, even if you're
careful only to purge files that haven't been modified in a while. I
recall working with a server application that put files in /tmp and
mmap'ed them to share memory between its multiple processes. Since
simple paging I/O to and from a file opened a week ago doesn't change
the files' modification date, along came the customer's /tmp-purging
cron job and BLOOEY went the server ...

Hang on, the files were open, right? So how could they be deleted? Or is
the point that the directory entries were deleted, so when new processes
were spawned, they couldn't open the file? And since when did writing to
a file via an mmap not change its modification time, anyway?

(The topicality margin gets even thinner)

You're right: An open file can't be deleted. However, its directory
entry is removed. Then, when the application spawns a new process and
that new process tries to share memory with the others by opening and
mmap'ing the now-unfindable file, BLOOEY. (When the customer first
reported trouble, I immediately asked whether there was a cron job or
some such that periodically purged old files from /tmp. Customer
asserted -- vehemently and a bit angrily -- that OF COURSE there wasn't.
So we cobbled together some DTrace to monitor file deletions in /tmp,
and caught the non-existent cron job red-handed ...)

It's at moments like this that i start whistling the Dragnet theme tune.
As for file modification times, I confess an incomplete grasp of
exactly which operations do and do not update them. However, just
poking a new value into a page that's mmap'ed from a file is not enough
to update the time stamp. Can you imagine the overhead if every memory
write trapped to the kernel to update the time?

Right, but the kernel could update the mtime when it flushed pages to
disk. The timestamp would reflect the time of the flush, not the write to
the page, but that's how it is with buffered stream IO too. Or am i
misunderstanding again, and there was no writing to disk happening?

I take it you didn't have SysV shmget/shmat here?
It wasn't my choice. It wasn't even my company's choice.
The third party who wrote the application chose to do things that
way, and even went so far as to include "do_not_delete" as part
of the files' names.

!

My company has a recent and hopefully short-lived tendency to put some
fairly non-temporary things in /tmp - various files created during the
lifetime of an app installation, like search indices, runtime
configuration, etc. It really ought to go in /var or somewhere like that,
but /tmp was the first place that sprang to mind, and since this is so far
just on development machines with fairly short-lived instances, it hasn't
yet gone terribly wrong.

tom
 
M

Martin Gregorie

Works OK for Linux and most *nixen, don't know about others.
Just checked with a text editor and Fedora 10: the change timestamp is
updated when the file is overwritten, not when the file is closed.

I don't have anything to hand that could try this with a mmapped file or
for random writes.
 
E

Eric Sosman

On Sat, 5 Dec 2009, Eric Sosman wrote:

(Marginally topical) [...]
(The topicality margin gets even thinner) [...]
[...]
As for file modification times, I confess an incomplete grasp of
exactly which operations do and do not update them. However, just
poking a new value into a page that's mmap'ed from a file is not
enough to update the time stamp. Can you imagine the overhead if every
memory write trapped to the kernel to update the time?

Right, but the kernel could update the mtime when it flushed pages to
disk. The timestamp would reflect the time of the flush, not the write
to the page, but that's how it is with buffered stream IO too. Or am i
misunderstanding again, and there was no writing to disk happening?

(Topicality margin now thin enough to use in a microtome)

No disk writes. The tmpfs file system on Solaris tries to
stay memory-resident as much as possible, and only goes to the
swap area(s) if RAM becomes awfully scarce.
I take it you didn't have SysV shmget/shmat here?

Had 'em, sure, but the folks who wrote the application
chose not to use them. (Not by default, anyhow: There was a
way to use shm instead of /tmp files, but it involved some
serious headaches for the administrator of the application
and was seldom done.)

Anyhow, this has really strayed pretty far from Java. If
for some reason you're interested in pursuing it further, I
suggest you E-mail me and we'll take it off-line.
 
T

Tom Anderson

On 12/9/2009 10:49 AM, Tom Anderson wrote:
On Sat, 5 Dec 2009, Eric Sosman wrote:

(Marginally topical) [...]
(The topicality margin gets even thinner) [...]
[...]
As for file modification times, I confess an incomplete grasp of
exactly which operations do and do not update them. However, just
poking a new value into a page that's mmap'ed from a file is not
enough to update the time stamp. Can you imagine the overhead if every
memory write trapped to the kernel to update the time?

Right, but the kernel could update the mtime when it flushed pages to
disk. The timestamp would reflect the time of the flush, not the write
to the page, but that's how it is with buffered stream IO too. Or am i
misunderstanding again, and there was no writing to disk happening?

(Topicality margin now thin enough to use in a microtome)

I disagree! This group is about being a java programmer. Java programmers
do things like work with mmapped working files used by long-lived
processes, and as such, this is relevant to them, by whom i mean us.
No disk writes. The tmpfs file system on Solaris tries to
stay memory-resident as much as possible, and only goes to the
swap area(s) if RAM becomes awfully scarce.

Ah, yes, that sort of changes the semantics. On a disk-resident FS,
changing the contents of an mmapped file doesn't change the bytes on disk
immediately; they will usually be written some time after the instructions
doing the write are finished. Is the point that in Solaris tmpfs, the same
pages are used for the mmap buffer and the definitive contents of the
file? In that case, there is no flushing to disk, and so no chance to
update a timestamp.

Now, i have been assuming that writing to a mapped file which *does* live
on disk affects the timestamp. Here is a test:

http://urchin.earth.li/~twic/Code/MMapTest.java

And here are the results on a machine which identifies itself as "Linux
localhost 2.6.18-92.1.6.el5PAE #1 SMP Wed Jun 25 14:21:46 EDT 2008 i686
i686 i386 GNU/Linux":

t0 = 1260468714000
t1 = 1260468714000
t2 = 1260468714000
t3 = 1260468714000
file = Mapd file

In other words, writing to a mmapped file doesn't change the mtime. Yikes!

tom
 
M

Martin Gregorie

Now, i have been assuming that writing to a mapped file which *does*
live on disk affects the timestamp. Here is a test:
.......
In other words, writing to a mmapped file doesn't change the mtime.
Yikes!
Something about this niggled me - the effect of using coarse
(millisecond) time resolution and a fast (1.6 GHz) CPU when checking
the effect of single statements, so I rewrote your program to add 1000 mS
pauses between interesting groups of statements.

This shows that, on a Linux system anyway, the modification timestamp is
updated when a RandomAccessFile is written or a MappedByteBuffer is
changed. No other operations on an existing file including flushing its
buffers (FileDescriptor.sync() and MappedByteBuffer.force()) affect the
amended timestamp.

Here are the labelled timestamps:

$ java MMapTest
1 Create,open raf : = 1260562842000
2 Write : = 1260562843000
3 Sync : = 1260562843000
4 Mapped,put : = 1260562845000
5 Force : = 1260562845000
6 Close : = 1260562845000
7 Reopen raf : = 1260562845000
File content : = Mapd file
8 Read : = 1260562845000
9 Close : = 1260562845000

and here's the modified code:

===============MMapTest.java===================

import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.FileChannel.MapMode;

public class MMapTest
{
public static void main(String[] args)
throws IOException, InterruptedException
{
File f = File.createTempFile("test", ".mmap");
RandomAccessFile raf = new RandomAccessFile(f, "rw");
showModTime(1, "Create,open raf ", f);
raf.write("Test file\n".getBytes("UTF-8"));
showModTime(2, "Write ", f);
raf.getFD().sync();
showModTime(3, "Sync ", f);
FileChannel channel = raf.getChannel();
MappedByteBuffer buf =
channel.map(MapMode.READ_WRITE, 0, f.length());
buf.put("Mapd".getBytes("UTF-8"));
showModTime(4, "Mapped,put ", f);
buf.force();
showModTime(5, "Force ", f);
channel.close();
showModTime(6, "Close ", f);
raf = new RandomAccessFile(f, "r");
showModTime(7, "Reopen raf ", f);
System.err.println(" File content : = " + raf.readLine());
showModTime(8, "Read ", f);
raf.close();
showModTime(9, "Close ", f);
}

private static void showModTime(int n, String label, File f)
throws InterruptedException
{
long t = f.lastModified();
System.err.println(n + " " + label + ": = " + t);
Thread.sleep(1000);
}
}

===============MMapTest.java===================
 
T

Tom Anderson

Something about this niggled me - the effect of using coarse
(millisecond) time resolution and a fast (1.6 GHz) CPU when checking the
effect of single statements, so I rewrote your program to add 1000 mS
pauses between interesting groups of statements.

My reasoning was that were were interested in at what stage the timestamp
changed, but not what it changed to. You're right that if one of the
operations had a delayed effect, in my implementation, it would show up
under the timestamp for another operation, and checking for that is
prudent.
This shows that, on a Linux system anyway, the modification timestamp is
updated when a RandomAccessFile is written or a MappedByteBuffer is
changed.

So it does. What OS are you on?

Results from a java claiming to be Java(TM) 2 Runtime Environment,
Standard Edition (build 1.5.0_19-b02-306) Java HotSpot(TM) Client VM
(build 1.5.0_19-138, mixed mode, sharing) on a machine calling itself
Darwin localhost 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00
PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh powerpc:

1 Create,open raf : = 1260569419000
2 Write : = 1260569420000
3 Sync : = 1260569420000
4 Mapped,put : = 1260569420000
5 Force : = 1260569420000
6 Close : = 1260569420000
7 Reopen raf : = 1260569420000
File content : = Mapd file
8 Read : = 1260569420000
9 Close : = 1260569420000

So no change from writing to the file via an mmap.

From java version "1.5.0" gij (GNU libgcj) version 4.3.2 on what purports
to be Linux urchin 2.6.26-2+urchin.1-686-bigmem #1 SMP Tue Nov 24 16:50:32
GMT 2009 i686 GNU/Linux:

1 Create,open raf : = 1260569830000
2 Write : = 1260569831000
3 Sync : = 1260569831000
4 Mapped,put : = 1260569833000
5 Force : = 1260569833000
6 Close : = 1260569833000
7 Reopen raf : = 1260569833000
File content : = Mapd file
8 Read : = 1260569833000
9 Close : = 1260569833000

Change! Obamatastic!

From java java version "1.5.0_14" Java(TM) 2 Runtime Environment, Standard
Edition (build 1.5.0_14-b03) J on Linux localhost 2.6.18-92.1.10.el5 #1
SMP Tue Aug 5 07:41:53 EDT 2008 i686 athlon i386 GNU/Linux:

1 Create,open raf : = 1260570108000
2 Write : = 1260570109000
3 Sync : = 1260570109000
4 Mapped,put : = 1260570109000
5 Force : = 1260570109000
6 Close : = 1260570109000
7 Reopen raf : = 1260570109000
File content : = Mapd file
8 Read : = 1260570109000
9 Close : = 1260570109000

No change.

So now i'm completely baffled.
No other operations on an existing file including flushing its
buffers (FileDescriptor.sync() and MappedByteBuffer.force()) affect the
amended timestamp.

Here are the labelled timestamps:

$ java MMapTest
1 Create,open raf : = 1260562842000
2 Write : = 1260562843000
3 Sync : = 1260562843000
4 Mapped,put : = 1260562845000
5 Force : = 1260562845000
6 Close : = 1260562845000
7 Reopen raf : = 1260562845000
File content : = Mapd file
8 Read : = 1260562845000
9 Close : = 1260562845000

tom
 
M

Martin Gregorie

My reasoning was that were were interested in at what stage the
timestamp changed, but not what it changed to. You're right that if one
of the operations had a delayed effect, in my implementation, it would
show up under the timestamp for another operation, and checking for that
is prudent.
Yes, that's clear. After posting, I modified my code so I could vary the
wait time and found that the timestamp change lost its correlation with
the program action if I set the wait to less than 450 mS.

At first I thought this showed that timestamp changes were not
synchronised with the write actions but then I realised that the
timestamp is only 1 second accuracy, so probably the timestamp only
detectably updates with the first write after the least significant
second digit has changed.

The lack of correlation with flush operations isn't surprising for a
*nix: the inode would be updated in memory and would show immediately in
mtime queries. Some time later, the affected blocks would be flushed to
disk but this is merely an internal filing system housekeeping operation,
unlike updating mtime in the inode, so you wouldn't expect the fs sync to
affect the mtime value.
So it does. What OS are you on?
Fedora 10, kernel version 2.6.27.38-170.2.113.fc10.i686
java version "1.6.0_07"
Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing)

Running on an Intel(R) Pentium(R) Dual CPU T2330 @ 1.60GHz
Results from a java claiming to be Java(TM) 2 Runtime Environment,
Standard Edition (build 1.5.0_19-b02-306) Java HotSpot(TM) Client VM
(build 1.5.0_19-138, mixed mode, sharing) on a machine calling itself
Darwin localhost 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10
18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh
powerpc:

1 Create,open raf : = 1260569419000
2 Write : = 1260569420000
3 Sync : = 1260569420000
4 Mapped,put : = 1260569420000
5 Force : = 1260569420000
6 Close : = 1260569420000
7 Reopen raf : = 1260569420000
File content : = Mapd file
8 Read : = 1260569420000
9 Close : = 1260569420000

So no change from writing to the file via an mmap.
I know very little about PPC Macs so can't comment about this. However,
I'm a little surprised that your second Linux run does the same: its
still kernel 2.6 so, barring bugs, I wouldn't expect a difference.
Similarly, I'd be quite surprised if the JVM has any effect on this
because mtime updating is, or should be, an OS function.

It may be worth increasing the wait time to 1500 - 2000 mS and running
your three tests again to see if that changes anything.
 
A

Arne Vajhøj

Good idea. Another in the same vein, if the stream is a network stream
that can't be re-opened (easily), then make a temp file and copy it to
the temp file. Then you can re-open the temp file as random access.

If it is a disk file, then I would expect it to be one of
the safest ways of doing it.

Arne
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Members online

Forum statistics

Threads
474,261
Messages
2,571,040
Members
48,769
Latest member
Clifft

Latest Threads

Top