Interesting utilities

[jacob navia]

I am developing in the Mac OS X (10.7) system, and it has some nice
features that I hadn't seen in any windows/unix system.

One of them is the "mdfind" utility that is incredibly powerful:

mdfind some-identifier

will look in the whole machine for "some-identifier" and write you in
standard output the name of all files that have a match. It is like

grep -R "some-identifier" /*.*

but it comes back in seconds instead of hours like grep.

It is very useful, and I never found something even remotely equivalent
under linux/sun os

Another nifty thing is the splint code checker. (This one is probably
present under linux also but I never saw it under windows)

The problem with it is that it is incredibly verbose, swamping the
few interesting diagonstics under a mountain of rubbish. After a long
session with it I have trimmed it to give me reasonable short output.

I put all options in a shell script:

/bin/sh
splint -formatconst -dependenttrans -globstate -compdef -mayaliasunique
-paramuse +boolint -immediatetrans -retvalother -noeffect
-unqualifiedtrans -compdestroy -temptrans -branchstate -usereleased
-kepttrans -compmempass -nullstate -nullpass -nullret -boolops
-mustfreeonly -incondefs +charint -retvalint -predboolint -realcompare $1

 

[Roberto Waltman]

I am developing in the Mac OS X (10.7) system, and it has some nice
features that I hadn't seen in any windows/unix system.

mdfind some-identifier ...
but it comes back in seconds instead of hours like grep.

That probably means that it looks into a keyword database, built by
indexing the files in the background, over many hours instead of
seconds.
And yes, the same functionality is available under Linux and Windows.

Another nifty thing is the splint code checker. (This one is probably
present under linux also but I never saw it under windows)

A windows installer is available at the home page:
http://www.splint.org/

 

[jacob navia]

Le 15/10/11 06:35, Roberto Waltman a écrit :

That probably means that it looks into a keyword database, built by
indexing the files in the background, over many hours instead of
seconds.
And yes, the same functionality is available under Linux and Windows.

Sorry but I have never even heard about something like that under
windows/linux.
Maybe you have an example?

A windows installer is available at the home page:
http://www.splint.org/

Thanks, I overlooked that one.

 

[Geoff]

Sorry but I have never even heard about something like that under
windows/linux.
Maybe you have an example?

It's called Spotlight on OS/X and Windows Search since the release of
Windows XP. I can't remember what it was called in Windows 2000.
Settings are available in the Windows Control panel applet named
"Indexing Options" on Windows 7.

 

[Roberto Waltman]

jacob navia wrote:

[ re: File indexing / search applications ]

... something like that under
windows/linux.
Maybe you have an example?

http://en.wikipedia.org/wiki/Windows_Search
http://www.microsoft.com/download/en/details.aspx?id=23
http://www.vandenoever.info/software/strigi/akademy2006.pdf
http://www.wikinfo.org/index.php/Comparison_of_desktop_search_software
http://en.wikipedia.org/wiki/Beagle_(software)

The last page says the Beagle project, (only one I used under Linux,)
is not being developed anymore. I was not aware of that.

 

[Kaz Kylheku]

I am developing in the Mac OS X (10.7) system, and it has some nice
features that I hadn't seen in any windows/unix system.

One of them is the "mdfind" utility that is incredibly powerful:

mdfind some-identifier

will look in the whole machine for "some-identifier" and write you in
standard output the name of all files that have a match. It is like

grep -R "some-identifier" /*.*

but it comes back in seconds instead of hours like grep.

So it is not more powerful than grep, since it computes the same thing. The
word you are looking for is "faster".

Can it find any regular expression in seconds?

It is very useful, and I never found something even remotely equivalent
under linux/sun os

How hard do you look?

I use id-utils and ctags for doing this over programming projects.
In id-utils there is a "mkid" utility which indexes the identifiers found
in a directory tree. This is gathered into a big file called ID.
The ID file is queried with "lid".

I just set up an Ubuntu system where "apt-get install id-utils"
got me mkid in seconds.

Another nifty thing is the splint code checker. (This one is probably
present under linux also but I never saw it under windows)

There is a direct pointer to a Windows download page on the Splint
home page! This kind of thing is not going to come preinstalled on a Wintel PC
from Best Buy.

The problem with it is that it is incredibly verbose, swamping the
few interesting diagonstics under a mountain of rubbish. After a long
session with it I have trimmed it to give me reasonable short output.

This is by design. To get value out of splint you have to read the
documentation and start using the annotations.

Splint can statically check things like that malloced objects are freed
somewhere, at least for some programs (maybe not in general).

 

[Kaz Kylheku]

Le 15/10/11 06:35, Roberto Waltman a écrit :

Sorry but I have never even heard about something like that under
windows/linux.
Maybe you have an example?

http://freshmeat.net/search?q=file+indexing&submit=Search

45 hits, some of them relevant-looking.

http://freshmeat.net/search?q=file+search&submit=Search

251 hits.

Man, you really are still a goof, sorry to say.

Windows has dynamic file indexing since Vista, if not before.

 

[Ian Collins]

So it is not more powerful than grep, since it computes the same thing. The
word you are looking for is "faster".

Can it find any regular expression in seconds?


How hard do you look?

I use id-utils and ctags for doing this over programming projects.
In id-utils there is a "mkid" utility which indexes the identifiers found
in a directory tree. This is gathered into a big file called ID.
The ID file is queried with "lid".

I just set up an Ubuntu system where "apt-get install id-utils"
got me mkid in seconds.

These utilities are all well and good on a relatively static filesystem,
but the fun begins when you want to keep the database as up to date as
possible.

I know all about the fun involved, I have to maintain an index over a
couple of multi-TB filesystems with tens of millions of files in each!

 

[Kaz Kylheku]

What about `locate' from GNU Findutils?

Locate searches only file names, not their contents.

That's been a staple of Linux distros as far as I can remember (which could
potentially takes us back as far as 1993).

Long before operating systems "for the rest of them" started getting
any kind of indexing.

 

[jacob navia]

Le 16/10/11 03:24, Kaz Kylheku a écrit :

http://freshmeat.net/search?q=file+indexing&submit=Search

45 hits, some of them relevant-looking.

http://freshmeat.net/search?q=file+search&submit=Search

251 hits.

Man, you really are still a goof, sorry to say.

Windows has dynamic file indexing since Vista, if not before.

Sure, but it doesn't index everything, as I have seen lately.
Only some selected directories. SUre you can change that but still,
it doesn't match the easy of use of not doing anything


I used the find utility in windows XP, and the terrible interface
with the dog just convinced me to avoid it completely. In any case it
wasn't a background process as it is in the mac


Under linux there *could* be an indexing utility around of course,
but it is not automatic like mdfind.

What is interesting in the mac is the number of automatic utilities
like mdfind, for instance the indredibly useful backup utility, that
work out of the box without any need for big configurations, etc.

 

[jacob navia]

Le 16/10/11 03:24, Kaz Kylheku a écrit :

http://freshmeat.net/search?q=file+indexing&submit=Search

45 hits, some of them relevant-looking.

http://freshmeat.net/search?q=file+search&submit=Search

251 hits.

Man, you really are still a goof, sorry to say.

Windows has dynamic file indexing since Vista, if not before.

Sure sure....

In Wikipedia the program "tracker" is listed as a search utility for
linux.

Downloaded the package from
http://projects.gnome.org/tracker
Then:

bzip2 -d tracker-0.1.12.bz2
untarred result
changed to the direzctory and typed ./configure
../configure fails with:
configure: error: glib-compile-schemas not found.

Searched google for that.

After much searching I found out that actually I have that package
installed in /usr/bin.

Apparently I am not the only one havinng that problem, there are many
hits about that error message. But there are no solutions. Some of
the solutions point to upgrading to a newer version of the OS...
but I am running already the latest Ubuntu (I try to keep current since
under linux if you aren't upgrading every 2-3 months the OS nothing
will work any more)

The problem is that I have the package but the
configure script doesn't fdind it (it is in /usr/bin... I really do not
understand why that directory is not searched)

I edit the configure script and see that there is an environment
variable that should contain the path to those "compile-schemas"
I set the variable, rerun, nothing, same error.

Now I discover that I am not really a shell/ configure / language
expert. I can't figure out why the thing isn't found I give up.

Wasted time: Approx 1 hour.

mdfind came with the mac already, nothing to do for me.

But of course I am a "goofer"...

Look Mr Kaz, I am well aware that an indexing utility could
exist for linux even one that after several hours of work
could even work better than mdfind.

But... really, I do not feel like wasting more time in it.
Yes, it is free, I did not pay for my Ubuntu version. But I
have learned the hard way that unless you want to specialize in
linux administration you will be wasting so much time with it that it
is just not worth trying to do things like this for somebody
that is not a linux admin...


I wonder how would I explain the usage of the linux utility to
my wife...

"Look Annie, just download that source package"

Then, there is no install program, you have to type
../configure

etc...
`
Good luck with that

:-(

jacob

 

[jacob navia]

Well, there was another way, sorry.

The ubuntu distribution has a package for tracker binaries.
Excellent!

I download the package and apparently installs without problems.

Obviously there isn't any documentation at all. After some searching
I discover that it installed in "Accessories". I click in the tracker
line and a window appears with a text field called "Search"

I type "iList" and press return. Since I downloaded the source code
of the containers library in a subdirectory under /home/jacob
I know that this word appears.

Nothing, nothing changes, everything remains empty. Mmmmmm...
Maybe I haven't configured something?

Open a shell "man tracker" gives no answer, no documentation.

I type

tracker-<TAB>

and see a list of tracker programs. I call the configuration
program and... yes, my home directory is listed under the
directories to be searched recursively.

I see another utility: tracker-search. I write

$ tracker-search iList
Results: 0

Ahhh that is why everything is empty. It doesn't find anything.
Why?

Mistery. The configuration program says that it has written the
configuration and I see /home/jacob in the recursive directory
list.

I go to /home/jacob/ccl and write:

$ grep iList *.c
containers.h: extern ListInterface iList;

Yes, I am not dreaming...

Well, I do not know what is going on. There must be some configuration
option I did not find out. Probably after some hours of work I could
get this stuff running, who knows?

 

[Kaz Kylheku]

Well, there was another way, sorry.

The ubuntu distribution has a package for tracker binaries.
Excellent!

Yes, I just found it in three seconds while looking at your
other post:

Is this it?

http://packages.ubuntu.com/lucid/tracker

I would not try to compile from scratch packages that have any
complex dependencies. (Not as a Ubuntu user, that is.)

I download the package and apparently installs without problems.
Obviously there isn't any documentation at all. After some searching

I searched the web instead to find the home of this thing.

http://projects.gnome.org/tracker/documentation.html

"All the documentation is on the live wiki"

http://live.gnome.org/Tracker/Documentation/GettingStarted

Much of the documentation is developer-oriented. This looks like an
experimental new project of some sorts.

Well, I do not know what is going on. There must be some configuration
option I did not find out. Probably after some hours of work I could
get this stuff running, who knows?

The wiki page says there is is a:

"tracker-miner-fs | The file system crawler to mine data from local files"

I doubt that you will find anything if this has not run and done its job.