Constellations

[Roedy Green]

I often run into the problem of having to specify a set of files to be
processed by an application. I wrote a crude command line tool to let
you specify a list of files, and a list of directories, with a switch
to allow including subdirectories. I thought I should write something
a little more powerful that let you specify nested wildcards of
alternating include exclude overriding rules based on XML. You would
compose your constellations, then just specify them on the command
line or alternatively a simple list of files and dirs.

The problem has percolated to top of my stack. Some time ago I wrote
some ideas at http://mindprod.com/jgloss/constellation.html on how it
might work.

This might be of more general use. It will be a free package with open
source. If you have some thoughts, please speak now while it is easier
to incorporate them.

--
Roedy Green Canadian Mind Products
http://mindprod.com
PM Steven Harper is fixated on the costs of implementing Kyoto, estimated as high as 1% of GDP.
However, he refuses to consider the costs of not implementing Kyoto which the
famous economist Nicholas Stern estimated at 5 to 20% of GDP

 

[Arne Vajhøj]

I often run into the problem of having to specify a set of files to be
processed by an application. I wrote a crude command line tool to let
you specify a list of files, and a list of directories, with a switch
to allow including subdirectories. I thought I should write something
a little more powerful that let you specify nested wildcards of
alternating include exclude overriding rules based on XML. You would
compose your constellations, then just specify them on the command
line or alternatively a simple list of files and dirs.

The problem has percolated to top of my stack. Some time ago I wrote
some ideas at http://mindprod.com/jgloss/constellation.html on how it
might work.

This might be of more general use. It will be a free package with open
source. If you have some thoughts, please speak now while it is easier
to incorporate them.

Why not make your code an ant task and reuse ant's ability to
specify directories and files.

Reuse !

Arne

 

[Tom Anderson]

Why not make your code an ant task and reuse ant's ability to
specify directories and files.

Reuse !

Why not use find and xargs? If you're talking about reuse, the unix shell
is the most successful component architecture in history.

tom

 

[Arne Vajhøj]

Why not use find and xargs? If you're talking about reuse, the unix
shell is the most successful component architecture in history.

None Unix people would not have a clue about how to use those.

Java people should know how to use ant.

Arne

 

[Roedy Green]

Why not use find and xargs?

xargs runs the utility multiple times, with an arbitrary sublist of
the files each time. It is a tool to get around too short a command
line. I am after something so that the app can process all the files
at once as a lump, possibly consolidating them, sorting them etc.

I would see constellations as ways of describing files to back up,
to upload to ta website, to tidy, to search etc.

You want a set of dirs, with exceptions, and exceptions to the
exceptions, and exceptions based on regexes, extensions and wildcards.
--
Roedy Green Canadian Mind Products
http://mindprod.com
PM Steven Harper is fixated on the costs of implementing Kyoto, estimated as high as 1% of GDP.
However, he refuses to consider the costs of not implementing Kyoto which the
famous economist Nicholas Stern estimated at 5 to 20% of GDP

 

[David Segall]

When I have that sort of problem, I just make the program read a list of
files from standard input. The list of files can be built using any
tools the user likes - I use a mix of find, grep, and sed, but that is a
matter of taste and UNIX experience.

As I understand it Roedy wants to provide that list of files using
only XML. If he succeeds he can provide a DTD and, consequently, an
easy implementation in most modern IDEs. I think that is likely to be
simpler than your solution for those that don't know UNIX or a
comparable scripting language. Are you arguing that the list cannot be
expressed using XML or that expressing the list using a combination of
find, grep and sed would be significantly simpler than using XML?

 

[Stefan Ram]

As I understand it Roedy wants to provide that list of files
using only XML.

XML is a language not a tool. Therefore, I'd write »provide a
list of files in XML«, not »provide a list of files using only
XML«.

If he succeeds he can provide a DTD and, consequently, an easy
implementation in most modern IDEs.

If this is easier to program in Java, it will be easier in any
environment, whether a text editor or an IDE.

I would not use »in« here, but »using«. I implement something
/in/ a language (like »Java«) /using/ a program (like an IDE).

Can you show an example code here to explain how it is easier
than reading just lines from a text file?

final java.util.Scanner in = new java.util.Scanner( java.lang.System.in );
while( in.hasNextLine() ){ final java.lang.String line = in.nextLine(); ... }

The above code was written within a minute without using
an IDE. (But I have not tested it, so it still might contain
errors. However, it might provide the general idea.)

I think that is likely to be simpler than your solution for
those that don't know UNIX or a comparable scripting language.
Are you arguing that the list cannot be expressed using XML or
that expressing the list using a combination of find, grep and
sed would be significantly simpler than using XML?

»find«, »grep« and »sed« are part of a programmer's toolbox.
They are available for every operating system. A programmer
usually knows them or, otherwise, will benefit from learning
how to use them.

They are not interactive, but can be used to automate tasks.
A programmer will prefer to use /non-interactive/ tools to
automate repeated tasks. An IDE is an interactive tool.

Again, you are comparing tools (»find«, »grep«, and »sed«)
with a language (»XML«). Therefore, it is difficult to
understand what you are talking about.

You could compare a program, like »find«, with features of
another program, like »eclipse«.

Or, you could compare languages, like »XML«, with features
of other languages, like »file of text lines« (plain text).

 

[Tom Anderson]

None Unix people would not have a clue about how to use those.

It's never too late to learn!

Java people should know how to use ant.

Really? I've never used it. Well, i think i've invoked it once or twice,
but that's it.

tom

 

[Tom Anderson]

xargs runs the utility multiple times, with an arbitrary sublist of the
files each time. It is a tool to get around too short a command line.

That's one of its uses. But it's also useful for bundling up multiple
arguments to a command (as long as they're less than 4 kBish in total),
which is what we're doing here.

I am after something so that the app can process all the files at once
as a lump, possibly consolidating them, sorting them etc.

Yup, like that.

I would see constellations as ways of describing files to back up, to
upload to ta website, to tidy, to search etc.

You want a set of dirs, with exceptions, and exceptions to the
exceptions, and exceptions based on regexes, extensions and wildcards.

All doable with a little bit of shell script using find and xargs!

I'm not saying your idea of doing it with XML and a custom tool or library
is a bad one, merely that there is already a way to do this.

tom

 

[Lew]

Really? I've never used it. Well, i think i've invoked it once or twice,
but that's it.

Which does not alter the validity of Arne's assertion a jot.

 

[Arne Vajhøj]

As I understand it Roedy wants to provide that list of files using
only XML. If he succeeds he can provide a DTD and, consequently, an
easy implementation in most modern IDEs. I think that is likely to be
simpler than your solution for those that don't know UNIX or a
comparable scripting language. Are you arguing that the list cannot be
expressed using XML or that expressing the list using a combination of
find, grep and sed would be significantly simpler than using XML?

There are excellent XML tools in library, but it is still not
easier to parse XML than a list with one filename per line.

The find/grep/sed stuff used to generate that that list would
still need to be done to generate the XML file. And that would
most definitely be more work than just creating the list.

Arne

 

[Arne Vajhøj]

It's never too late to learn!
>

Really? I've never used it. Well, i think i've invoked it once or twice,
but that's it.

Among the users of a Java library the probability of them knowing
ant is simply bigger than of knowing xargs.

If the target was another group, then it could be different (if it were
as an example Linux sysadms, then I am sure it would be different).

Arne

 

[Roedy Green]

When I have that sort of problem, I just make the program read a list of
files from standard input. The list of files can be built using any
tools the user likes - I use a mix of find, grep, and sed, but that is a
matter of taste and UNIX experience.

Over the last 24 hours I have come round to that point over view too.
The advantage is the tools for creating constellations can then be
used with software that is constellation unaware. Further, the app
needs no RAM overhead to analyse/evaluate the constellation. Further
the user can look at the list of files no verify it, or hand tune it
before feeding it to a app.

For example, winzip already accepts a such a linear list of files. All
you need do is put a @on the front of the list file name.
--
Roedy Green Canadian Mind Products
http://mindprod.com
PM Steven Harper is fixated on the costs of implementing Kyoto, estimated as high as 1% of GDP.
However, he refuses to consider the costs of not implementing Kyoto which the
famous economist Nicholas Stern estimated at 5 to 20% of GDP

 

[Roedy Green]

»find«, »grep« and »sed« are part of a programmer's toolbox.
They are available for every operating system. A programmer

But they lack the WORA of a something in Java. Everybody needs to do
backups, not just programmers.
--
Roedy Green Canadian Mind Products
http://mindprod.com
PM Steven Harper is fixated on the costs of implementing Kyoto, estimated as high as 1% of GDP.
However, he refuses to consider the costs of not implementing Kyoto which the
famous economist Nicholas Stern estimated at 5 to 20% of GDP

 

[Stefan Ram]

For example, winzip already accepts a such a linear list of files. All
you need do is put a @on the front of the list file name.

This example was fetched too far:

»For a large number of source files, list the file names
in a file, separated by blanks or line breaks. Then use
the list file name on the javac command line, preceded by
an @ character«

http://java.sun.com/j2se/1.4.2/docs/tooldocs/windows/javac.html

 

[Stefan Ram]

But they lack the WORA of a something in Java.

You do not have to write them, because they are already written.

And they run everywhere, since they are available for free for
every major operating system.

Everybody needs to do backups, not just programmers.

What has this to do with the topic of UNIX tools?

 

[Tom Anderson]

But they lack the WORA of a something in Java. Everybody needs to do
backups, not just programmers.

I don't see how writing XML is more user-friendly than writing find
commands (neither are very friendly to the non-technical). If you're
proposing to provide a friendlier interface via software, then that could
be backended onto find - from an admittedly cursory inspection, you
constellation grammar looks largely isomorphic to find's grammar.

tom

 

[David Segall]

XML is a language not a tool. Therefore, I'd write »provide a
list of files in XML«, not »provide a list of files using only
XML«.

You might, but I don't think you would convey what the OP is trying to
do. I believe he wants to use an XML based language as the source
language to a program that will output a list of files.

If this is easier to program in Java, it will be easier in any
environment, whether a text editor or an IDE.

I would not use »in« here, but »using«. I implement something
/in/ a language (like »Java«) /using/ a program (like an IDE).

Can you show an example code here to explain how it is easier
than reading just lines from a text file?

final java.util.Scanner in = new java.util.Scanner( java.lang.System.in );
while( in.hasNextLine() ){ final java.lang.String line = in.nextLine(); ... }

The above code was written within a minute without using
an IDE. (But I have not tested it, so it still might contain
errors. However, it might provide the general idea.)

I can see why my reference to IDEs was confusing and I should have
either explained it or, more sensibly, omitted it. NetBeans _already
has_ the facility to create an XML document using a specified DTD. It
will check that the document conforms to the DTD and provide code
hints and code completion based on the DTD. I believe that most other
IDEs either have a similar facility or it is available as a plug in.

»find«, »grep« and »sed« are part of a programmer's toolbox.
They are available for every operating system. A programmer
usually knows them or, otherwise, will benefit from learning
how to use them.

They are not interactive, but can be used to automate tasks.
A programmer will prefer to use /non-interactive/ tools to
automate repeated tasks. An IDE is an interactive tool.

Again, you are comparing tools (»find«, »grep«, and »sed«)
with a language (»XML«). Therefore, it is difficult to
understand what you are talking about.

You could compare a program, like »find«, with features of
another program, like »eclipse«.

Or, you could compare languages, like »XML«, with features
of other languages, like »file of text lines« (plain text).

We have a language problem If XML was a language I could compare it
to other languages like Java and COBOL. I would argue that the
parameters required by find, grep or sed are also languages. I don't
think introducing ill-defined categories like "tools" or excessively
broad categories like "programs" is useful. Sed and a payroll program
are both programs and, depending on the user, tools. Comparing them on
either basis is not fruitful.

 

[Stefan Ram]

We have a language problem If XML was a language I could
compare it to other languages like Java and COBOL.

What do you think, the »L« in »XML« stands for?

See also:

http://www.w3.org/TR/2000/REC-xml-20001006.html

 

[Roedy Green]

I don't see how writing XML is more user-friendly than writing find
commands

I don't think find commands have much hope of being used by
non-programmers. They have been around a long time, and you still
don't see them used that way. XML is not much better. What is
plausible is a specialised GUI editor for describing constellations
that edits XML. These XML constellation files could then be
imported/expanded/modified by programs or programmers.

What I am hoping to do is provide stable rules for selecting files
that handle files appearing and disappearing, that can then feed the
entire set to various utilities, without each utility needing to be
clever. I want a standard way of specifying constellations that works
across many utilities, including ones I did not write.

--
Roedy Green Canadian Mind Products
http://mindprod.com
PM Steven Harper is fixated on the costs of implementing Kyoto, estimated as high as 1% of GDP.
However, he refuses to consider the costs of not implementing Kyoto which the
famous economist Nicholas Stern estimated at 5 to 20% of GDP