In search of ... universal download tracker

[barney]

Folk,

Way back in the Win98/Win98SE days, we had a TSR that did naught but record every bit of software, specified in a list, that was downloaded to a particular machine.  Basically, I could go to any given machine in my group and see just what had been downloaded in a given time period.  (The track record was stored on a network drive, so I didn't have to physically visit the machine in question.)

I would like to set something like that working on my home system.  I'd like to be able to pull up a [specified] catalog of everything that was downloaded, organizable by date/time or alphabetically.  This would pretty much be driven by a list of file extensions to record, e.g., .swf, .exe, .jpg, .dll, and the like.  Then the software would record date, time, file name, file extension, size, download origin, perhaps what triggered the download.

My searches have turned up a number of scripts to record downloads from a Website or vendor, as well as a few that were ISP specific for monitoring overall bandwidth usage - basically from recorders.  However, I cannot seem to find anything that will record downloads to a machine other than a few download managers.  (Thought of using a sniffer, something like Snort, but don't think that would work - too much irrelevant traffic.)

Example:  Adobe, in particular, likes to pull stuff down, then tell me it's available.  'Nother example:  Win pulls down updates, then tells me they're available.

There are a few other applications that do much the same thing.  I want to be able to go to the historical record to see when these downloads were accomplished.  It would help immensely with some troubleshooting.

Anyone have any references for something of this nature?

[40hz]

Secunia Personal Software Inspector (PSI) comes pretty close to what you're describing. It;s free for personal use.

It's primarily designed to be a version checker/vulnerability assessment tool. But it also keeps track of changes, versions, and updates as part of that process so it should provide you with much of the information you're looking for.

In search of ... universal download tracker

In search of ... universal download tracker

You'd need to enable the logging option if you wanted to keep full change details.

Heavy-duty info on what's already on your machine can be obtained by using Belarc Advisor. It's a PC auditing tool which is also free for personal use. It won't give you installation dates, but it will provide software version information in excruciating detail along with a very useful security vulnerability score.

(partial output:)

In search of ... universal download tracker


Belarc is something many techs run to get the "lay of the land" whenever they get a machine in for service. One of my favorite diagnostic tools.

Luck!

[barney]

Heavy-duty info on what's already on your machine can be obtained by using Belarc Advisor. It's a PC auditing tool which is also free for personal use. It won't give you installation dates, but it will provide software version information in excruciating detail along with a very useful security vulnerability score.

-40hz (November 28, 2011, 07:09 AM)

Roger excruciating detail  !

Didn't think about PSI.  Run it as a matter of course, but forgot about the logging feature.  I'll set logging on, see what it gives me.  Thanks  .

[cyberdiva]

Heavy-duty info on what's already on your machine can be obtained by using Belarc Advisor. It's a PC auditing tool which is also free for personal use. It won't give you installation dates, but it will provide software version information in excruciating detail along with a very useful security vulnerability score.

-40hz (November 28, 2011, 07:09 AM)

Hmmm....I run Belarc from time to time, and I agree that it can provide some useful information about a computer, but I disagree strongly about the usefulness of its security vulnerability score.  Mine has always been abysmal, but I've never really figured out why or what to do about it.  There are well over 400 items in its Security Benchmark check list, and even in sections where I've had a mix of green check marks and red x's, the score for the section has been 0.00.  It lists 110 items (all with red x's) under "Internet Explorer 8," which I don't even have on my computer.  Finally, off to the side, it says the following:

"How can you reduce your security vulnerability?  The local group policy editor (accessed by running the gpedit.msc command) can be used to configure security settings for your computer.  Windows home editions don't include that editor, but most security settings can also be made with registry entries instead.  Warning: Applying these security settings may cause some applications to stop working correctly.  Back up your system prior to applying these security templates or apply the templates on a test system first."

Sounds like I'd be crazy to let Belarc "fix" the security settings on my computer (which has never had a security problem).

[40hz]

@cyberdiva - Advisor's security assessment is geared towards enterprise users on corporate networks rather than home users. I (like you) don't have any personal machines that garner high scores either but I'm (also like you) more than comfortable with the level of security they do have. My corporate client's users, on the other hand, come in as close to a perfect score as I can get - and my clients will put up with.  

For home users, I'd view the security report as more educational than anything else. If it raises your awareness of just how much is involved securing Windows it will have served its purpose. Besides, it also covers a dozen things that are easy to do that sill will make a huge difference in how secure your PC is. Most people aren't aware of half of them.

If it causes major angst feel free to ignore it.  

[MilesAhead]

Local group policy isn't all that great for preventing apps from running anyway. For awhile I used it in Windows Seven to prevent IE from running at all.  Then one day I installed some software that liked to run IE. Up it came!!!  Turns out the installer has higher priority than my policy settings.

I came to the conclusion that if I wanted to be certain IE never starts the only way is uninstall it.  Instead I just never initialized any of the settings. I set CCleaner to remove anything on the off chance I might run it for a couple of minutes. I believe it comes up by default in privacy mode(or whatever buzz word they use to mean the same thing.) Likely it deletes everything on close.  But if it doesn't CCleaner gets it.

Plus I have a script for removing all index.dat files on the partition.

[40hz]

@MilesAhead - pretty rude installer!

Give some software admin privileges and it takes advantage of you apparently.  

Hope you complained to their support forum about it.

[MilesAhead]

@MilesAhead - pretty rude installer!

Give some software admin privileges and it takes advantage of you apparently.  

Hope you complained to their support forum about it.

-40hz (November 28, 2011, 05:29 PM)

I don't even remember what it was. Probably a codec pack. I don't install those kitchen sink packs anymore.

[MilesAhead]

@barney that TSR monitor sounds familiar. Do you recall if it was one of those PC Magazine utilities?  I remember back at the end of the 80s start of the 90s just about every month was yet another TSR in assembly language. Some fun stuff esp. if you had a lowly XT clone like I did.

edit: could have been a PCMag TSR or something similar.  Win98 was still Dos then the Gui loading, with some DPMI stuff. But I believe you could still use Dos type TSRs with it.

[barney]

Oh, yeah!

I munched on PCMag's monthly little dividend.  Funny thing is, a lot of those TSRs still worked up until Vista, even though they weren't NT technology  .

Prolly was a PCMag thing.  Might have been one (1) of Karen Kenworthy's VB apps - she did a bunch of 'em, and I still use some of 'em from time to time.  Just do not recall the name - too many years [and beers, likely] gone by  - only the functionality.  I winnowed through my archive of her apps, but nothing jumped out at me.  If it was one (1) of the assembly bits, I'd be hard-pressed to locate it, assuming I still have it.

[IainB]

Not sure if this is relevant, but I think MS Security Essentials is perpetually monitoring all downloads to disk, so it probably maintains a log of downloads somewhere.

[barney]

It would definitely be relevant if I could just figure out where to find such a log  .  Nothing I see in the configs makes note of such a thing.  Actually, I'd be kinda surprised if it was a text file that I could read .  But if I do happen to find such, I'll give a yell -surely there are others who could make use of such a thing .

[cyberdiva]

@cyberdiva - Advisor's security assessment is geared towards enterprise users on corporate networks rather than home users.

-40hz (November 28, 2011, 04:40 PM)

Ah, thanks 40hz.  I had no idea.

For home users, I'd view the security report as more educational than anything else. If it raises your awareness of just how much is involved securing Windows it will have served its purpose. . . .
If it causes major angst feel free to ignore it.  

-40hz (November 28, 2011, 04:40 PM)

Well, with well over 400 items on its list, most of which I don't understand, it's not what I'd call educational.  The only thing it raises is not my awareness but my blood pressure.   It doesn't cause major angst, but I'll nonetheless feel free to ignore it.

[MilesAhead]

This probably doesn't have all the info you want. But you may be surprised what it shows:

http://www.softpedia...dex-dat-Viewer.shtml

[MilesAhead]

It might be worth experimenting a bit with index.dat files. If they really do log every download, at least via http, then you may be able to get some program to monitor the files for changes.

If you can detect index.dat file changes then it should be possible to get the newest information. You could feed the filenames into some app that gets stuff like size, modification date etc.. from the OS

I would look for some open source index.dat reader as that may give a clue to formatting the file contents into something easy to handle.

Note:

I just looked in my index.dat viewer and it showed some files I downloaded without using the browser. Who knows? Anything downloaded on a network may be logged in one of the index.dat files. It may at least provide a trigger for an info gathering app.

[barney]

Index.dat Viewer does show a great deal of info, but no way to determine date/time - save perhaps, for the time stamp on the file itself?  Don't think that will work.  Several of the index.dat files show a modified date earlier than the created date, at least in FileLocator Pro.

However, it's a start, might prove to be useful in a slightly different context.
 

[MilesAhead]

Right. What I'm saying is you'd have to monitor the index.dat file for changes.  When change, get new entries. If they are files, you have the path so get the info.  Some other program would do the monitoring.

Only thing is watching programs that use Win API to monitor stuff it seems they keep the drive very busy.  I'm not sure if you can monitor individual files for changes or only folders.

[4wd]

Just looking at index.dat kind of restricts you to IE only doesn't it?

I wonder if you could do it using the SDK of Everything ?

Since it already monitors all file changes across the whole computer, (except removables and non-NTFS), if you set a filter for the extensions you want and have your program just log new file creations via Everything' SDK.  The only thing you miss out on would be originating website.

Filtering, originating website - doesn't URL Snooper already do most of this ?
Since it uses WinPcap it's not restricted to a browser.

I'm sure mouser would be amenable to enhancing URL Snooper to include missing functions given the right incentive   or  

[MilesAhead]

Just looking at index.dat kind of restricts you to IE only doesn't it?

-4wd (November 30, 2011, 06:59 PM)

That's what I thought.  Nope. I never use IE.  But stuff I downloaded in other browsers and not even using http or a browser app is showing up. I think it keys on network access.  Also it made some notes that I looked at my services list using the services applet.  One would probably have to study them for some weeks to figure out all the stuff they log.

Seems Windows API gives a folder change notification.  But from what I've been reading it's not so great.  Since there are only a few index.dat that matter it's probably simpler to periodically use FindFirstFile() and get FileLastWriteTime.  Compare it to the write time returned at the last check etc.. if newer, read the file and find what's new.

The hazard doing it that way is index.dat locations probably vary with every Windows flavor.

I don't know of any canned code that just monitors one file.  But I haven't searched that hard yet.

[4wd]

That's what I thought.  Nope. I never use IE.  But stuff I downloaded in other browsers and not even using http is showing up.

-MilesAhead (November 30, 2011, 07:22 PM)

Not here, only shows what IE and those programs that use IE Core services have done - shows nothing done by Pale Moon, (optimised Firefox).  It has a total of three lines in it.

I don't know of any canned code that just monitors one file.  But I haven't searched that hard yet.

FileNotify and FileNotify2

[MilesAhead]

Not here, only shows what IE and those programs that use IE Core services have done - shows nothing done by Pale Moon, (optimised Firefox).  It has a total of three lines in it.

-4wd (November 30, 2011, 07:35 PM)

What flavor of windows are you running and what are you using to read the index.dat files?

I'm running Windows Seven and it logs everything I download.

FileNotify and FileNotify2

I still think it uses directory change notification API. I don't see one for individual files.

http://msdn.microsof...364417(v=vs.85).aspx

[4wd]

Not here, only shows what IE and those programs that use IE Core services have done - shows nothing done by Pale Moon, (optimised Firefox).  It has a total of three lines in it.

-4wd (November 30, 2011, 07:35 PM)

What flavor of windows are you running and what are you using to read the index.dat files?

I'm running Windows Seven and it logs everything I download.

-MilesAhead (November 30, 2011, 08:19 PM)

Win7HP x64 and that index.dat viewer you linked above.  I've tried opening every index.dat on the system, (total of 18), and it shows the contents as either:
a) empty,
b) a lot of non-URL stuff, or
c) the three lines I mentioned above, which are URL related but which I have never used in Pale Moon.



Everything shows the last change to an index.dat being over an hour ago and I've downloaded files and visited other websites since then.  If, on the other hand, I load up IE and visit a website, index.dat is updated immediately.

FileNotify and FileNotify2

I still think it uses directory change notification API. I don't see one for individual files.

I'm not sure if it uses the API for picking up directory changes since it installs its own service but it probably does.  You can specify a filter for a particular file in FileNotify2, (eg. index.dat), within a directory but I haven't used this particular function myself, (only used FileNotify - non .NET version).

[MilesAhead]

Don't know what to tell you.  Maybe x64 is different. I'm on 32 bit. Believe me, there's a lot of stuff tracked on my machine. Stuff downloaded not even using http never mind a browser.

edit: yeah, I think it's because the viewer is 32 bit. I just fired up my Vista64 system and it shows less than a dozen entries.  Folder redirection is keeping it from seeing all the index.dat files is my guess.


As for individual file change notification you specify the directory and a filter what changes in the directory will trigger the notification.  The fact that you specify a file only means the code gets the change notification and makes the call ReadDirectoryChangesW() to find out what changed. If it's the file you selected, it fires your action.

If you have 4000 files under that folder and are monitoring all sub-folders I think the disk is going to be busy. It's cheaper just to call FindFirstFile() on a timer.

[4wd]

Don't know what to tell you.  Maybe x64 is different. I'm on 32 bit. Believe me, there's a lot of stuff tracked on my machine. Stuff downloaded not even using http never mind a browser.

-MilesAhead (November 30, 2011, 08:54 PM)

Just tried it on my XP Pro 32bit machine - same thing.

Nothing shown in index.dat until I start using IE.

This leads me to believe that just monitoring index.dat would be next to useless unless you can enforce a policy of IE only on the monitored machine and even that would be no guarantee.

I still think URL Snooper, (or similar), is the only way you're going to catch everything.  You either have to monitor the network connection or monitor for every file change/creation and filter - either way will cause a hit of some kind.

edit: yeah, I think it's because the viewer is 32 bit. I just fired up my Vista64 system and it shows less than a dozen entries.  Folder redirection is keeping it from seeing all the index.dat files is my guess.

-MilesAhead (November 30, 2011, 08:54 PM)

The viewer seems to work fine here, if it shows everything that IE, (and IE x64), has done then it's looking at the right index.dat AFAICT - it just doesn't show what any other non-MS API calling program has done.

Also:

I've tried opening every index.dat on the system, (total of 18), and it shows the contents as either:

-4wd (November 30, 2011, 08:44 PM)

As a matter of interest, what browser are you using that is causing entries to appear?

[MilesAhead]

Evidently the behavior is too inconsistent to be generally useful. Except maybe to MS.

Btw, I just booted XP 32 bit for chuckles. I don't use IE or any browser that uses IE engine. I brought up the viewer. It was empty.  Closed it. Opened Firefox. Did a couple downloads.  Brought up viewer.  Downloads logged. Still running FF 4 on the XP side as I don't boot XP very often.

But that's neither here nor there.