I Want To Index Every File On My LAN.

[tinjaw]

I know there are various threads about desktop search applications here on DC, but I want to not be tied down to just one computer. I want to find an open source application that will spider my entire LAN (mostly via SMB, but also HTTP and FTP if possible) and index every file. I want it to grab meta data and use plugins to index the contents of every file. I want thumbnails of image files. I want it to go into compressed archives. I want want to search it all via a web frontend. I want to be able to click on the search results and have either my preferred web browser, FTP client, or (Windows) file manager open to the file.

The first place I am going to look is to see if Google has something like this that is free/opensource. I have found many of the bits and pieces for such an application at the Apache Software Foundation, but it would all need to be glued together.

[update] I just found IBM OmniFind Yahoo! Edition. It looks like it just might do what I want. I am reading through the website right now.

[update 2] Microsoft has countered this with Search Server 2008 Express.

[J-Mac]

I have no clue as to what kind of indexing it offers, but have you looked into Amazon S3 at all?  S3 stands for "simple storage service" and a lot of small businesses, startups, etc. are using it.  I am not certain at all just how the index works within that system, but since places like Smugmug online photo service are using it for massive amounts of storage, it must have a pretty good retrieval system - which I think would  require a capable index system.

I don't know if this is even a consideration for you, but it's surely worth a look!

Jim

[housetier]

I don't want you in my LAN, tinjaw! Stay off my harddisk

[tinjaw]

I have no clue as to what kind of indexing it offers, but have you looked into Amazon S3 at all?

S3 is a very useful tool, but not suitable for this purpose. It is predominately a storage mechanism. Think of it as a remote disk drive where you pay for only what you use.

If S3 were to be used for something like this 1) you'd have to mirror every file on S3 2) you'd have to upload those terabytes of files over you internet connection and 3) you'd still need to write the software to do everything else.

[tinjaw]

I don't want you in my LAN, tinjaw! Stay off my harddisk

Don't worry. I won't take up much space. I'll just relax in a small ~ file in your temp directory.