WHY doesn't C know anything about directories?

[Kenneth Brody]

A. Every computer I work with (locally numbering into the hundreds)
has a directory structure. It would be nice if the language supported
these machines.

[...]

Just because every computer that _you_ work with has a directory
structure, doesn't mean that all computers do. Plus, those that
do have directory structures don't all have the same kind of
structure, as noted elsethread. (For the record, every computer
that I currently program in C on has a directory structure, and
they all have a similar way of representing directory paths.
However, I have coded in C on a system with a very different
method of representing directories. We don't support that
platform anymore, though there is still conditional code in some
of the modules to handle this difference.)

Also, just because the C language standard doesn't support such a
construct doesn't mean that other standards which do support them
can't be used. Most of the systems I work on have support for a
set of calls with names like opendir(), readdir(), closedir(), and
so on, which seem to be pretty standardized, though I don't see
any reference to a formal standard.

Perhaps POSIX has such support?

--
+-------------------------+--------------------+-----------------------+
| Kenneth J. Brody | www.hvcomputer.com | #include |
| kenbrody/at\spamcop.net | www.fptech.com | <std_disclaimer.h> |
+-------------------------+--------------------+-----------------------+
Don't e-mail me at: <mailto:[email protected]>

 

[Keith Thompson]

A. Every computer I work with (locally numbering into the hundreds)
has a directory structure. It would be nice if the language supported
these machines.

[...]

Yes, it would be nice, but it's not very practical. There are enough
differences between how directories are implemented on different
systems that providing a portable interface is very difficult. Note
that, in the grand scheme of things, Unix and Windows are actually
very similar; if they were the only operating systems in existence, a
"portable" directory interface wouldn't be difficult.

Every operating system that supports directories provides a
system-specific interface to work with directories. A hypothetical
portable interface would inevitably leave out some of the
functionality of the system-specific interface, and a lot of
programmers would just use the system-specific interface even if a
portable interface existed.

Programs that deal with directories tend to be system-specific anyway.
This is to some extent a chicken-and-egg problem, but solving it would
be difficult.

I'm not *necessarily* saying that a standard directory interface is
impossible, but there are a lot of obstacles that many programmers
might not be aware of. David Tribble has proposed such an interface;
see <http://david.tribble.com/text/c0xdir.html>. (I haven't taken a
very close look at it myself.)

 

[David Mathog]

And every computer in the world is a VAX^WSun^WWintel^Wx64.

Therefore, if a VAX^WSun^WWintel^Wx64 can do it, it should be in every
programming langauge in the world.

You left off DOS, Mac, and BeOS ;-).

In short, yes, I think that it's daft that C supports fopen() without
also providing a portable way to navigate to the file other
than by creating a path string, which we all know not to be in the least
bit portable. And as I've said a several times already, it's trivial to
provide a test function so that a program can determine at run time
if directory operations are supported. Which I think is still pretty
much a red herring since the overlap between embedded controller
software and the sorts of programs which would benefit from portable
directory navigation is pretty close to the null set.

On the flip side, because there is no way to do this, useful software
that does end up being ported often needs a lot of rewriting to work
around the platform specific directory name and file name conventions,
and also the directory navigation and listing methods, which often are
found scattered around in various parts of the code.

Regards,

David Mathog

 

[Mark McIntyre]

A. Every computer I work with (locally numbering into the hundreds)
has a directory structure.

And a lot of the ones I work with don't.

It would be nice if the language supported
these machines.

No need, since platform-specific extensions do it very well.

C. How many keys do these proposed database filing systems require?
The proposed navigation calls already support UP, ACROSS, and DOWN
navigation directions. That should be sufficient to navigate within
a database so long as a list of valid keys can be retrieved and
the number of keys needed to index a file location is finite

I don't think you understand how a database filesystem works. Consider
an ordinary RDBMS. A row isn't 'down' from a table, and a row in a
different table isn't 'across'. if you select * from table where
filetype=bla and filedescription like gibble you don't get some sort
of hierarchical tree returned.
--
Mark McIntyre

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it."
--Brian Kernighan

 

[Eric Sosman]

David Mathog wrote On 06/07/07 17:01,:

You left off DOS, Mac, and BeOS ;-).

In short, yes, I think that it's daft that C supports fopen() without
also providing a portable way to navigate to the file other
than by creating a path string, which we all know not to be in the least
bit portable. And as I've said a several times already, it's trivial to
provide a test function so that a program can determine at run time
if directory operations are supported. Which I think is still pretty
much a red herring since the overlap between embedded controller
software and the sorts of programs which would benefit from portable
directory navigation is pretty close to the null set.

On the flip side, because there is no way to do this, useful software
that does end up being ported often needs a lot of rewriting to work
around the platform specific directory name and file name conventions,
and also the directory navigation and listing methods, which often are
found scattered around in various parts of the code.

In my own experience, "a lot" of rewriting has not
been necessary. If your experience is otherwise, that
might be taken as evidence that the notion of directory
has too many system-specific quirks to be suitable for
standardization ...

 

[David Mathog]

On Thu, 07 Jun 2007 08:30:41 -0700, in comp.lang.c , David Mathog


No need, since platform-specific extensions do it very well.

There is a need because it's a PITA to port those platform specific
extensions to another set of platform specific extensions.

I don't think you understand how a database filesystem works. Consider
an ordinary RDBMS. A row isn't 'down' from a table, and a row in a
different table isn't 'across'. if you select * from table where
filetype=bla and filedescription like gibble you don't get some sort
of hierarchical tree returned.

UP ACROSS and DOWN define navigation on a grid on a 2D surface, which
may or may not have a tree based file system mapped onto it. If a
database can be mapped so that every file it contains lies on
that 2D surface then UP ACROSS and DOWN should be sufficient to navigate
to any of those files. (Not necessarily trivially to any SET of those
files, as could be done with SQL select statements.) Also the
navigation path to each cell, and especially from cell to cell, may not
be unique, but that's true now on Unix directory trees with links
between directories in place.

Another way to think of it with respect to a database
method would be "Less Restrictive", "As Restrictive", "More
Restrictive". Let's ignore the middle one for a minute and work the
example you cited. Admittedly it is a bit counter
intuitive if you're thinking in terms of SQL select statements.

In your example we need to get to all files "in":
some table, filetype=bla, filedescription=gibble.

Start at the database "TOP" (nothing selected)

Navigate DOWN: Lists all tables, from list select a table.
Navigate DOWN: List all columns in table, from list select "fileteype".
Navigate DOWN: List all values in column, from list select "bla".

(if we directory list at this point we get all files for which the
database has filetype=bla.)

The next step is where it becomes counter intuitive. To further
restrict the specification ("DOWN") after one column has been
set requires selecting another column and a key value in it, like this:

Navigate DOWN: List all Columns in table ([OTHER than the one(s)
already selected] ie, further restrict list),
from list select "filedescription".
Navigate DOWN: List all values in column, from list select a
value like "gibble".

List directory: all files for filetype=bla, filedescription=gibble.

UP is even less familiar since it is multivalued.

Navigate UP: lists all less restricted paths, which in this case are:
1. column=filetype(value=bla),column=filedescription
2. column=filetype,column=filedescription(value=gibble)

Admittedly this is all AND logic, but it should be sufficient to
find any file stored in the database system, or to place a new
file in the system. It cannot do SETS of files with as much
flexibility as a SQL select statement, but then, none of the native
directory tools I'm aware of are designed for operating on sets of
directories.

The biggest problem I see is that if files in a column were ONLY
organized by numeric indices (for instance a collection number) the list
that came back would be both humongous and essentially meaningless, so
that selecting the next value would be next to impossible. This is
exactly analogous though to creating a zillion subdirectories
with equally meaningless names. Unfortunately an organization
I've encountered a few times.

My point still being, that with C modified this way the program
just sees standard function calls for navigating UP and DOWN and
lists of results, and C never exposes the underlying database or
directory organization. This would allow the same program to
run unchanged on an OS with a tree file system or a database
file system. That would, I think, be a good thing.

Regards,

David Mathog

 

[Chris Torek]

... should be sufficient to
find any file stored in the database system, or to place a new
file in the system. It cannot do SETS of files with as much
flexibility as a SQL select statement, but then, none of the native
directory tools I'm aware of are designed for operating on sets of
directories.

[more snippage]

My point still being, that with C modified this way the program
just sees standard function calls for navigating UP and DOWN and
lists of results, and C never exposes the underlying database or
directory organization. This would allow the same program to
run unchanged on an OS with a tree file system or a database
file system. That would, I think, be a good thing.

It might. I suspect, however, that people would end up not using
it, in part because (as you note above) it may be sufficient for
some purposes, but it will not capture the "true nature" of the
underlying OS.

In general, it seems that when people want to operate on directories,
they want to do many (or most or even all) of the things that the
underlying OS does with them. A "subset" interface is, by definition,
not good enough; and a complete one is, by definition, not portable
enough. (They almost always *start* with something that can be
done with a simple "enumerate all files that are members of given
directory" type interface; it just turns out that, once that works,
they suddenly care about whether the file is the latest version or
not, on VMS; or whether it is marked "has system attribute", on
DOS; and so on.)

There do seem to be some exceptions, and those may be common
enough to warrant some sort of "directory access library". But
that is really a topic for comp.std.c, rather than comp.lang.c.

 

[Walter Roberson]

UP ACROSS and DOWN define navigation on a grid on a 2D surface, which
may or may not have a tree based file system mapped onto it.

Another way to think of it with respect to a database
method would be "Less Restrictive", "As Restrictive", "More
Restrictive".

Since the invention of symbolic links, unix filesystems have not
been tree-based, just approximately tree based. It sounds to me
as if there would be difficulty in translating the unix semantics
that the unix .. ("UP", "Less Restrictive") does not refer to
a fixed location.

NFS mounts also sound tricky to handle, especially cross-mounts
and automounts . Once you have those semantics built in, you also
have to worry about SANs (Storage Area Networks) (which system is
the owner of this shared file?) and about DMF (Data Migration
Facility), which might have multiple levels of storage with files
(and even file -names-) brought back from "near-line" storage at
need. It seems to me that the "reselect and do a fileset AND" semantics
you suggest require that all the names be present at the same time.

There a Microsoft Windows NTFS facility whose name I do not recall at
the moment, partly related to Alternate Data Streams (ADS), which
allowed a directory entry to be a program which interprets the
rest of the path, sort of an API for device driver- like objects.
I'm not sure how that would fit in with your scheme.


Is the facility you propose practical for removable media? Floppies
might be pretty moribund now, but there are USB Keychains, CDs and DVDs,
camera storage, iPods -- pretty high volumes of removable storage,
and which people have come to expect will be trivial to plug in and
have "just work". Would the Database type approach be efficient enough
at rebuilding the file information -quickly- when activating the
removable media, and at removing the information -quickly- when
deactivating the removable media?

 

[Ian Collins]

There have been a series of questions about directory operations, all of
which have been answered with "there is no portable way to do this".

More of an aside, but you might be interested in the design decisions
behind the proposed C++ filesystem library:

http://www.boost.org/libs/filesystem/doc/tr2_proposal.html#Design

 

[Richard Bos]

A. Every computer I work with (locally numbering into the hundreds)
has a directory structure. It would be nice if the language supported
these machines.

Every computer I work with has a graphics screen. It would be nice if
the language supported these machines. But it doesn't, and there are
very good reasons for that.

B. If there is no way to navigate the database (for instance, no way
to recover a list of the keys it accepts) then

directory_support()

would return false, as it would on an embedded controller.

The question is not of _whether_ the file system is navigable, the
question is _how_. This is not as easy as it sounds at first. Oh, it's
all very well if you only assume Windows and Unix and blithely disregard
such matters as softlinks and devices-as-files, but such a simplified
approach would never work for an ISO Standard.

Files themselves could be included in the Standard because one can
whittle down working with them to a few very portable operations. Open
and close, create and remove, read and write, get, restore and seek
position. And even those have been curtailed to an extent that
invariably trips up all-the-world's-a-desktop-toy programmers: not only
is there no file size function, but you can't even reliably go to the
end of a file and ask for a position in bytes. And this is not, as some
would think, for no better reason than to pester Winblows programmers;
it is because that simply cannot be done on some systems, but those
systems have files that are powerful enough to make giving them the rest
of the file functions worthwhile.
With directories, however, matters are a little more complicated. Even
on systems where you have directories and you can navigate them, there
is no simple way to specify _how_ to navigate them completely and
reliably that works on all systems. If you think that "go to top of
directory tree; list all files; find first sub-directory; recurse; find
next subdirectory; re-recurse" suffices and will do the job on all
systems out there, you simply haven't had much experience of many OSes.
Hell, you probably haven't even considered how you handle Windows drive
letters...

Richard

 

[Flash Gordon]

Richard Bos wrote, On 08/06/07 08:31:

With directories, however, matters are a little more complicated. Even
on systems where you have directories and you can navigate them, there
is no simple way to specify _how_ to navigate them completely and
reliably that works on all systems. If you think that "go to top of
directory tree; list all files; find first sub-directory; recurse; find
next subdirectory; re-recurse" suffices and will do the job on all
systems out there, you simply haven't had much experience of many OSes.
Hell, you probably haven't even considered how you handle Windows drive
letters...

A psuedo-directory one level up that has the drives as simple letters...

I admit I don't know enough about things like database file systems, but
could one not do something very simple like:

int setdir(const char *dir);
Where interpretation of the string is implementation defined, so for a
database filesystem it would be a database query and a unix/dos/windows
filesystem it would be a patch as you would use for chdir.

Then add:
int resetdirpos(void);
To initialise your "directory pointer" to the first file in the "directory"
int readdir(char *name);
To read a name from the directory moving the "directory pointer" on to
the next directory.

Oh, and a function:
int currdir(char *dir);
Where as with setdir the string put in to dir is implementation defined.

It would be enough for me (I could do my own"file globbing" as I read
through the directory. It might not be enough for others (a directory
name of ".." might mean go up on Unix, but not on VMS), but I *think*
that by leaving the strings up to the implementation to define (taking a
page out of how fopen was defined) it allows it to be of *some* use on
any system with multiple files and a way of selecting a subset of all
the files on the system.

 

[Chris Hills]

Every computer I work with has a graphics screen. It would be nice if
the language supported these machines. But it doesn't, and there are
very good reasons for that.

Every computer I work with has 30mm automatic cannon attached to it.....
.... And a screen but no file system

there are many thousands of different systems that use C. the core
language covers them. We do not need every variation added to the core.

 

[Walter Roberson]

Also, just because the C language standard doesn't support such a
construct doesn't mean that other standards which do support them
can't be used. Most of the systems I work on have support for a
set of calls with names like opendir(), readdir(), closedir(), and
so on, which seem to be pretty standardized, though I don't see
any reference to a formal standard.

Perhaps POSIX has such support?

[OT]
Yes, ISO/IEC 9945-1 (IEEE Std 1003.1, aka POSIX.1 from 1990).

It leaves as implementation defined whether a leading // has
special meaning (basically, it's allowed as an escape into
other kinds of filesystems or network filesystems). It also
does not define much beyond the standard user/group/other permissions;
the attempt later to standardize access control lists was dropped
(and the authors of that draft now consider standardized ACLs a dead topic.)

 

[David Mathog]

More of an aside, but you might be interested in the design decisions
behind the proposed C++ filesystem library:

http://www.boost.org/libs/filesystem/doc/tr2_proposal.html#Design

That was interesting. Especially the part about OpenMVS being an HP
trademark ;-).

Regards,

David Mathog

 

[David Mathog]

Richard Bos wrote, On 08/06/07 08:31:



A psuedo-directory one level up that has the drives as simple letters...

Ditto for VMS.

Regards,

David Mathog

 

[David Mathog]

there are many thousands of different systems that use C. the core
language covers them. We do not need every variation added to the core.

Actually I agree, but I diverge on the way it is "not covered". The
current method seems to be to leave out any functionality which
is not universal, whereas I think it would be better for the language to
include highly common, but not universal, functionality, but supply test
functions more or less like this:

#include <stddir.h>
#include <stdgraphics.h>

/* compile time checking */
#ifdef STDDIR_SUPPORT
#ifdef STDGRAPHICS_SUPPORT
#endif
#endif

/* or run time checking */

if(directory_support(){
if(graphics_support()){
}
}
else {
}

How much work are we talking about for compilers on systems that don't
offer this support? Seems like practically none. Here's part of
stddir.h for such a system:

/* sttdir.h - disable all directory options portably on this system */
#ifndef _STDDIR_H
#define _STDDIR_H
#define STDDIR_SUPPORT 0
typedef struct is_dirstruct DIRSTRUCT;
struct is_dirstruct{ int notsupported; };
/* prototypes */
int directory_support(void);
DISTRUCT *directory_create_direstruct(void);int
directory_otherfunction(DIRSTRUCT *arg1, char *arg2);
/* etc. */

/* actual functions */
int directory_support(void){ return 0; }
DISTRUCT *directory_create_direstruct(return NULL);
int directory_otherfunction(DIRSTRUCT *arg1, char *arg2){ return 0; }
/* etc. */

#endif /* stddir.h */

The language standard itself could easily define this include file for
compilers

Should graphics also be part of the language? That's in some ways a
more interesting question than for directory operations, because
there are so many different ways it could be done. I'm thinking no
though, because right now one can already choose a cross platform
graphics library and work with that. Also graphics is vastly more
complex than directory or database models of file organization.

Regards,

David Mathog

 

[¬a\\/b]

Directory structures are antiquated loads of crap.

We should put everything into a database (including making the file
system be a database).

For file systems, a network model or hierarchy is probably better than
a relational model.

At any rate, the C language itself is the wrong place to reform
things.

IMO-YMMV.

the operations "read or write" to directory or files are all the same

all that is not there are standard functions that donig above
operations

 

[Szabolcs Nagy]

Should graphics also be part of the language? That's in some ways a
more interesting question than for directory operations, because
there are so many different ways it could be done. I'm thinking no

now that's where you completely wrong sir

there are so many different ways directories could be done (different
concepts, different performance charasteristics for different kinds of
access, etc)
and --as it was mentioned earlier-- system specific solutions will
always beat the extremely narrow special use case of the standard

i don't understand how can you still insist on this with so many
helpful answers on the topic

it's like you would want to standardize mouse input events and if
someone has a different input device then he'll just suffer a little
performance loss or return ENOT_IMPLEMENTED

 

[Chris Hills]

Actually I agree, but I diverge on the way it is "not covered". The
current method seems to be to leave out any functionality which
is not universal, whereas I think it would be better for the language
to include highly common, but not universal, functionality,

File systems are not "highly common". They may be in your field but
not in a large number of others. Also there are many different types of
file system. The most common one is not on a PC... smart cards have a
very different file system... Most people have more smart cards than
PC's

but supply test functions more or less like this:

#include <stddir.h>
#include <stdgraphics.h>

/* compile time checking */
#ifdef STDDIR_SUPPORT
#ifdef STDGRAPHICS_SUPPORT
#endif
#endif

/* or run time checking */

if(directory_support(){
if(graphics_support()){
}
}
else {
}

How much work are we talking about for compilers on systems that don't
offer this support? Seems like practically none.

Most compilers will support ifdef BUT

We have been round this loop before for graphics. (Several times)

There are many very good graphics systems out there for all manner of
things. From instrument panel type libraries to various GUIs other than
the ones with Windows, Solaris, 3(?) for Linux, MAC, etc etc

So will your new "standard" graphics incorporate these or be different?
Likewise for file systems (NOT "directory structures" )

At the moment all bar about 5 compilers do not have full C99 compliance
and are never likely to. Their markets do not require or ask for it. If
you add Graphics, file systems etc the net result is 95% of the most
common compilers on the market will ignore it. .

Should graphics also be part of the language? That's in some ways a
more interesting question than for directory operations, because
there are so many different ways it could be done. I'm thinking no
though, because right now one can already choose a cross platform
graphics library and work with that. Also graphics is vastly more
complex than directory or database models of file organization.

There are cross platform graphics libraries out there now. Your
standard will have to incorporate al of them or it will be ignored. If
it incorporates al of them it is pointless

 

[Mark McIntyre]

There is a need because it's a PITA to port those platform specific
extensions to another set of platform specific extensions.

I'm one of the few people I know who has experienced the fun of
porting from (64-bit) VMS to (32-bit) Unix and (16-bit) DOS. Believe
me, porting directory support was the /least/ of my issues.

A well designed app will be immune to such vagaries anyway - for
instance paths to data, config files, etc etc should be externalised.

UP ACROSS and DOWN define navigation on a grid on a 2D surface,

Databases don't have 2d surfaces.

database can be mapped so that every file it contains lies on
that 2D surface

Yes, but they can't. The whole point is that its a /relational/
database, with multiple keys cutting any-which-way over your data.

In your example we need to get to all files "in":
some table, filetype=bla, filedescription=gibble.

Start at the database "TOP" (nothing selected)

Navigate DOWN: Lists all tables, from list select a table.
Navigate DOWN: List all columns in table, from list select "fileteype".
Navigate DOWN: List all values in column, from list select "bla".

And this is efficient how exactly ? Compared to a single,
well-optimised select ?

This is the crux: /some/ OSen offer simple hierarchical directory
trees, not really searchable except on crude criteria. Even so, these
differ significantly in how data is addressed. Consider that unix
stores one version of each file, while VMS stores as many as you like.
Some OSen offer significantly more complex multidimensional storage,
which are fully searchable. Basic operations like "list all files in
this directory" are meaningless in these OSen.

My point still being, that with C modified this way the program
just sees standard function calls for navigating UP and DOWN

And you end up either with hideously inefficient functions, which are
be very inflexible, or with hideously complicated functions with
zillions of parameters and bitfields filled with flags, all required
in order to get it to be efficient and/or operate with your particular
OS. Witness the cack that goes into the Windows APIs, all in order to
provide compatiblity with a half-dozen slightly different versions of
the same OS.

--
Mark McIntyre

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it."
--Brian Kernighan