Author Topic: NANY 2019 - Windows App - Process Watcher (Read 36802 times)

mouser · « **on:** November 06, 2018, 01:34 PM »

Process Watcher - a bare bones minimal utility for watching and logging new and changed processes found running on your pc.

PORTABLE version: https://www.donation...sWatcherPortable.zip
INSTALLER version: https://www.donation...cessWatcherSetup.exe

Process Watcher is a tool that lives down in your system tray and logs when it sees a new or changed process running on your system.

This is a very early release of what will hopefully be a more elaborate tool, but as of now it has only one purpose: To log each time it sees a new running process, or sees a new version of a previously seen running process.
When it sees a changed process, it will log what has changed (file timestamps, size, exe properties/comments).

So this utility may be useful for those who are interested in seeing what processes run on their computer in the background, or logging when a program is silently updated, etc.

You can easily view the log from the system tray menu, or simply open it in any text editor.

NOTE: It is currently for 64-bit versions of windows only, though a 32-bit version could be built if desired.

NOTE 2: This version will by default set itself to autostart with windows when you run it. You can change this by editing the ProcessWatcher.ini file.

wraith808 · « **Reply #1 on:** November 06, 2018, 10:33 PM »

Sounds interesting just from the name. Sometimes I just need to watch processes to see if they spike in CPU usage. So I'll be watching this entry with interest!

mouser · « **Reply #2 on:** November 07, 2018, 04:12 AM »

Right now I'm mostly refactoring code, and thinking what I could do with it..

The current version I'm working on simply watches all running processes -- and while I have code written to monitor cpu and memory use (code I use in my Process Tamer and Process Piglet apps) -- currently for the new I am working on a different focus. The new app is just trying to keep a record of all processes it sees, and log each time it sees a new or changed process (application) running.

That is, the first time you run the app, every running process it sees will be "new" to it, and it will write out to it's log every file it has found running, and the information it can collect about it (process name, company, version, filedate, etc.).
So the first time you run it your log will show info on 100-200 processes.

But then it remembers having seen these processes running.. So over the course of the following months/years, it will only be adding to its log when it sees a new process run for the first time, OR an update of one of the existing processes (via changed file properties, or if you wish, a full file contents hash).

When it finds a new or changed process, it will write a report to the log file saying so and listing what has changed.

In this way, you are meant to leave this app running at all times on your pc, or a pc you are trying to troubleshoot or maintain. And you can check the log at any time to see exactly when some process was updated or run for the first time.

It could be useful for people in identifying when a malware hit them, or what new program someone may have installed on a pc that they shouldn't have, etc.

Whether other people think this might be a useful tool, I'm not sure...

Later I may consider adding more functionality to the app for not just logging the first time it sees a new/changed process, but other things like you mention -- reports when apps use certain amounts of memory or cpu. Or even a more elaborate process use tracking feature, which keeps track of how much time you have spent running each process (or how much cumulative cpu use processes have used over long times).

I'm focusing on keeping it as lean, fast, and resource friendly as possible, compiled in visual studio c++ with a 64bit build and the resident app having no gui, so it should not use noticable resources.

Ath · « **Reply #3 on:** November 07, 2018, 06:37 AM »

Ha, sounds interesting.
Most difficult part would be to organize the logging in such way that it actually makes sense when looking at it, without being overwhelmed with 'all that data'

Maybe the memory/cpu use-log should be separated, possibly a logfile per exe (path+filename, possibly reversed, and : and \ replaced by another character?) and include the pid inside the logfile to be able to filter on. (Hint: the LogStash family of products, no ties)

mouser · « **Reply #4 on:** November 09, 2018, 01:35 AM »

If anyone wants to try a really early alpha:

http://donationcoder...her_x64_portable.zip

What to know:

It's portable
It's 64-bit build only
There is no GUI, just run it and you'll see any icon in the tray which you can use to exit.
2 files will be created in the directory when you run it (ProcessWatcher_HistoryData.dat, and ProcessWatcher_HistoryLog.txt); the former is the data file it will read and write on each run; the latter is the human-readable log file that is the bread-and-butter of the app. This is the file you will want to look at to see if the program is useful to you.
You are meant to just leave this app running always. Restart it on reboot. The log file will show you every time it sees a new process for the first time, or any time a process has changed (was updated, etc.).

Remember the real value of this app is in logging the first time it sees a new or changed process. This can be useful if programs launch in the background, and never inform you.

By occasionally checking the ProcessWatcher_HistoryLog.txt you may discover some applications that are occasionally running, performing update checks and similar things in the background.

anandcoral · « **Reply #5 on:** November 09, 2018, 03:32 AM »

Ahhhh... Looks like after marriage you can read minds too.

I was planning a process lister which will show the process names as they are loaded by the os in a small non focus window. So that I can see which process are loading in background without my knowledge.

Now you just read my mind and made it. There goes my NANY 2019.
Well I will have to re-start again.

Anyway, thanks a lot for the app. It is definitely useful to me

Regards,

Anand

mouser · « **Reply #6 on:** November 09, 2018, 07:05 PM »

One thing I should emphasize in case it's not clear -- this program is designed to remember which applications it has seen running before, and only log when it sees a new program running (or when it detects a change to a process that it has seen before). As such, it's most useful as a tool to have running always in the background, so you can see a nice list of every application that has been run on your computer, while still keeping the list manageable because it's not adding log entries every time it sees the same program.

mouser · « **Reply #7 on:** November 10, 2018, 01:07 AM »

So here are some possibilities for where this program could go:

Have it store data at regular interval on cpu and memory use of every running app. So that you could look back at the last day/week/month and examine your computer activity in, say, 5 minute intervals. I'm not sure how useful this would be. Maybe add some additional tracking of what app had the foreground. Add the ability to show some basic statistics about what how much you are using each application..Maybe useful for people interested in tracking how much time they use different applications.
Do #1 above but also incorporate some automatic screenshotting.. Basically merge in functionality of my Automatic Screenshotter application.
Add-in alert functionality, to trigger alerts when certain applications use more cpu or memory than some threshhold, much like my Process Piglet application.
Add in priority manipulation and affinity control, much like my Process Tamer application.

wraith808 · « **Reply #8 on:** November 10, 2018, 10:40 AM »

I think making those options would be a good thing, i.e. where to store, how often to take snapshots, and max size of file. Also, I think that using a standard type log for this would make it useful, i.e. csv or tab-separated so that the data could be processed easily. I like where it's going though.

cranioscopical · « **Reply #9 on:** November 10, 2018, 02:27 PM »

I notice that HistoryLog.txt starts at the earliest entry and newer entries are appended. Would it be feasible to (have an option to) reverse that, so that latest entries appear first?

mouser · « **Reply #10 on:** November 10, 2018, 06:28 PM »

Would it be feasible to (have an option to) reverse that, so that latest entries appear first?

No, because appending to the file is fast and reliable and the opposite is not true.

But I think the real solution here is eventually a nice (standalone) gui tool for more nicely viewing the contents of the file in a proper grid or something. That will wait until I figure out the final kind of data to be saved. But yeah, the solution to more pleasant viewing of the data gathered is a gui tool for it.

mouser · « **Reply #11 on:** November 10, 2018, 09:24 PM »

Also, I think that using a standard type log for this would make it useful, i.e. csv or tab-separated so that the data could be processed easily.

You are right, I'm open to suggestions.. If you look at the format I'm using now, it's basically JSON (though not 100% valid json since I don't properly escape the strings). I do think I would like to save in a format that would make it easier for someone else write a GUI tool to browse history.

But first I have to figure out what kind of information I will be logging. Right now both of these data files are just logging the occasional process entry so it's not that important. But if I start to log performance data constantly for every running process then it will be important to use a data format that is compact.

Ath · « **Reply #12 on:** November 11, 2018, 04:24 AM »

Well, about that file-format: I was going to ask you to use real JSON, as the current content isn't quite readable using standard tools/libraries, but also seeing your remark about compact/heaps of data, I'd suggest to switch to CSV, as that's usually the most compact and usable for non-complex data like this (unless you want to put the performance/usage data inside the JSON object ofcourse...).

Continuing on the subject of including performance/usage data: Using (structured) JSON storage would require you to rewrite the entire file when updating a single performance item, so the policy of 'only appending' isn't going to work.
I've suggested (reply #3) to create a file-per-exe for the performance/usage data, and when using CSV that would be a feasible and fast solution, including append-only. It will generate a lot of files though, so a configurable log directory would be nice to have.

I might even be tempted into creating a GUI for this data, but you'd have to have the storage format finalized first.

mouser · « **Reply #13 on:** November 11, 2018, 04:47 AM »

There are 2 kinds of files that this program will be writing -- those where speed and space is important and those where it is not.

These files are not speed/space sensitive because they only get written out occasionally.. So JSON is a reasonable choice. I'll bit the bullet and make them fully json compliant.

Then there may be some files that are sensitive to speed and size, like cpu and memory use files which are updated very frequently. For those I'll have to decide on some other format, or even use a proper database.

Ath · « **Reply #14 on:** November 11, 2018, 05:26 AM »

That sounds like a very solid path

anandcoral · « **Reply #15 on:** November 12, 2018, 04:26 AM »

Looks like the path do not have the option I was dreaming of.

Should I go ahead and make one as I envisioned for NANY 2019 ?

Regards,

Anand

mouser · « **Reply #16 on:** November 12, 2018, 04:28 AM »

Can you explain what feature you were interested in?

anandcoral · « **Reply #17 on:** November 12, 2018, 06:31 AM »

Can you explain what feature you were interested in?
-mouser (November 12, 2018, 04:28 AM)

I was planning a process lister,
which will show the process names, time etc.,
as they are loaded by the os,
in a small non-focus window.

So that I can see, which process are loading in background without my knowledge.

Regards,

Anand

Ath · « **Reply #18 on:** November 12, 2018, 06:47 AM »

I was planning a process lister,
which will show the process names, time etc.,
as they are loaded by the os,
in a small non-focus window.
-anandcoral (November 12, 2018, 06:31 AM)

I think this should be one of the tasks of a GUI on Process Watcher logs.
Maybe you could create that kind of a GUI? (I might start it, depending on available time and motivation)

Another GUI tool/task would be for filtering/searching the log-data, and that could even be the same tool, AFAICS.

wraith808 · « **Reply #19 on:** November 12, 2018, 08:38 AM »

Another GUI tool/task would be for filtering/searching the log-data, and that could even be the same tool, AFAICS.
-Ath (November 12, 2018, 06:47 AM)

This is what I use for that on my logs: https://www.logfusion.ca/

Ath · « **Reply #20 on:** November 12, 2018, 02:01 PM »

Hm, though LogFusion looks like a fine log viewer, I'm afraid it doesn't handle the json-like format that is currently output by ProcessWatcher. We'll have to wait for mouser to choose another format before LogFusion is useful in this context.

KodeZwerg · « **Reply #21 on:** November 12, 2018, 03:34 PM »

Hm, though LogFusion looks like a fine log viewer, I'm afraid it doesn't handle the json-like format that is currently output by ProcessWatcher. We'll have to wait for mouser to choose another format before LogFusion is useful in this context.
-Ath (November 12, 2018, 02:01 PM)

Or put up a request to convert file. If you append me some small examples (before/after) i write one for you.
Until now i did not have time to test this application at all. LogFusion is also unknown to me.

mouser · « **Reply #22 on:** November 12, 2018, 03:37 PM »

The file format I will use is not decided yet, so I advise no one spend any time coding anything yet..
Even more importantly, the log file produced currently is just the log of processes seen, and is not the main log file that will be of any interest.. That is still to come.

wraith808 · « **Reply #23 on:** November 12, 2018, 04:02 PM »

Hm, though LogFusion looks like a fine log viewer, I'm afraid it doesn't handle the json-like format that is currently output by ProcessWatcher. We'll have to wait for mouser to choose another format before LogFusion is useful in this context.
-Ath (November 12, 2018, 02:01 PM)

Haven't tried it with JSON, but as what's there is only json-like, was waiting until he decided what that format might be in order to test. Just showing as a suggestion for what I use for log files in general.

mouser · « **Reply #24 on:** November 12, 2018, 11:05 PM »

It HAS been interesting checking out the log of processes that are seen by the application. Unlike a real-time log, you actually notice some surprises in this one -- of processes that run only occasionally in the background. Support tools, update checkers, etc. for other programs, etc.