Most important aspect of Info Keeping

[CodeTRUCKER]

One can use up the better part of a week reviewing all the threads available on DonationCoder, but I have found it odd there is a lack of focus on the importance of INSTANT RETRIEVAL. 

<rant>
Perhaps it's just me and everyone else knows this natively, but retrieval has to be the kingpin of any record keeping of every sort, yet far too often I find myself scratching my head wondering where is the emphasis on this fundamental ability?   

Why keep any data if you can't get it back easily?   Why use a software to collect data and then have to wait or not even find it (TopicScape, are you listening)?  How come every note taking, note keeping, web snippet gathering, mind mapping, etc. software out there does not present/contain a simple and powerful search capability for all text data it contains?  Every developer of this type of app should make the primary feature of their product INSTANT RETRIEVAL then build additional capability of their software while not breaking this primary function.

"INSTANT RETRIEVAL"

That's it short and sweet.
</rant>

Disclaimer: IMHO.

[Renegade]

FINALLY~! Someone else sees this!!!!

AGREED 1,000%~!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Did I mention that I agree?

Well... Here's goes a quick explanation...

Developers are fundamentally lazy.

Most data is stored in relational databases, and getting it out requires writing more than 1 SQL statement. Also, if you want to actually be able to filter results and get some specific results, you need to design the database properly, which is beyond the capabilities of most developers. No. Really. I mean that.

You can't just start JOINing tables all willy-nilly all over the place. You need to design the database according to how you will pull data out of it. Further, you need to put indexes on things to speed up data retrieval. You also need to actually THINK about what the data is and type it accordingly instead of just throwing it into an INTEGER or VARCHAR(50) field. You need to know the kinds of ways that you'll pull data out, and plan for things like multiple SELECTs instead of 1 massive JOIN. This brings up memory issues and caching issues. etc. etc. etc. The larger the database, the more problematic this is. Also, the more dynamic the database is (you are inserting records and deleting records often), the more problematic this is.

All of that requires a LOT of work for larger or more complex databases.

SUMMARY: Most developers are too lazy to do that, lack the skills to do it, or are under massive pressure to "get it done", and do not have the time to do it properly.



I think this goes double for web development. There seems to be far less competence when I consistently see places that they are not doing any kind of sensible error checking. (e.g. How difficult is it to remove spaces from a phone number or credit card number instead of making the user do it?) Also, the attitude towards web pages is that they are a commodity, and are undervalued. So, you end up with good developers that have mildly retarded managers and customers demanding insane schedules that border on irresponsible. How can you possibly deliver a truly great product when you view it as a commodity? A potato is a potato, so who cares?

I believe that this is partial fallout from the business mindset that we have today, which is basically, "bleed every *&%^% penny you possibly can out of everywhere and *&^% everything else as long as we make $#!+-loads of money", or something like that.

Good. Fast. Cheap. Pick 2. Web development is almost always the latter 2. After all, the web is just a "commodity".

[barney]

Renegade, you are a cynic .
CodeTRUCKER, while most of us share your attitude(s), many of us have given up expecting the results your wish would mandate.

Give ya a ferinstance. 

InfoSelect (IS) is advertised as a PIM, but my usage is almost exclusively data storage - to the extent of something like 4-5 gigabytes of data (in 20 MB files .)  (I've been using InfoSelect since before it was a Windows app.)  After three (3) years, the 2007 version has been updated to a 2010 version, currently in public beta stage.  One of the things I've loved about the previous version was that you could search on text of a specific colour.  (Incidentally, search within IS is very fast compared to any other such app I've used or even tried.) That enabled me to implement a tagging system for all notes, or search for specific files color coded by type, e.g., Linux, MS, Script, and the like.

In their upgraded system, colored text search is no longer available.  Why?  It was there, but they broke it (along with several other items, but that's another thread).

They simplified the search capabilities because, I suspect, it was too much work to maintain it in the upgrade .  Or, perhaps, they simply overlooked it ?!?

The bottom line, for most developers that have wandered in and out of my experience, is that search is just too damned hard to implement.  Creative or really fast search is well nigh an effor equal to or greater than the effort to create the original program.

It's doubtful you'll ever find the application you'd like to see , although I really hope you do.  Should that happen, please announce it here  .

[Stoic Joker]

Having built (from scratch) several of the data systems our company uses, I can attest to the fact that it's not as easy as it sounds. I too made some of the very mistakes Renagade ranted about above - Due not surprisingly to "time constraints" - But I am working on getting these fixed.

Craftsman it seems are a breed that died out years ago.

[Renegade]

Many years ago I started writing a web app but had to abandon it. I did stress tests and it failed badly. I simply didn't know what I know now. It was a typical CRUD app, like most web apps.

But building from scratch? WOW! I'd never attempt that in a million years. It's simply far too difficult.

Search is a very difficult thing to do well. RDBMSs make it easy if you're using one. But still, you need to work to get it done right. Search from scratch? That's BALLS! BIG BALLS! Like AC/DC Big Balls! The biggest balls of all!