DonationCoder.com Forum
DonationCoder.com Software => Older DC Contests and Challenges => N.A.N.Y. 2019 => Topic started by: mouser on November 06, 2018, 01:34 PM
-
Process Watcher - a bare bones minimal utility for watching and logging new and changed processes found running on your pc.
- PORTABLE version: https://www.donationcoder.com/Software/Mouser/ProcessWatcher/downloads/ProcessWatcherPortable.zip
- INSTALLER version: https://www.donationcoder.com/Software/Mouser/ProcessWatcher/downloads/ProcessWatcherSetup.exe
Process Watcher is a tool that lives down in your system tray and logs when it sees a new or changed process running on your system.
This is a very early release of what will hopefully be a more elaborate tool, but as of now it has only one purpose: To log each time it sees a new running process, or sees a new version of a previously seen running process.
When it sees a changed process, it will log what has changed (file timestamps, size, exe properties/comments).
So this utility may be useful for those who are interested in seeing what processes run on their computer in the background, or logging when a program is silently updated, etc.
You can easily view the log from the system tray menu, or simply open it in any text editor.
NOTE: It is currently for 64-bit versions of windows only, though a 32-bit version could be built if desired.
NOTE 2: This version will by default set itself to autostart with windows when you run it. You can change this by editing the ProcessWatcher.ini file.
[ You are not allowed to view attachments ]
-
Sounds interesting just from the name. Sometimes I just need to watch processes to see if they spike in CPU usage. So I'll be watching this entry with interest!
-
Right now I'm mostly refactoring code, and thinking what I could do with it..
The current version I'm working on simply watches all running processes -- and while I have code written to monitor cpu and memory use (code I use in my Process Tamer and Process Piglet apps) -- currently for the new I am working on a different focus. The new app is just trying to keep a record of all processes it sees, and log each time it sees a new or changed process (application) running.
That is, the first time you run the app, every running process it sees will be "new" to it, and it will write out to it's log every file it has found running, and the information it can collect about it (process name, company, version, filedate, etc.).
So the first time you run it your log will show info on 100-200 processes.
But then it remembers having seen these processes running.. So over the course of the following months/years, it will only be adding to its log when it sees a new process run for the first time, OR an update of one of the existing processes (via changed file properties, or if you wish, a full file contents hash).
When it finds a new or changed process, it will write a report to the log file saying so and listing what has changed.
In this way, you are meant to leave this app running at all times on your pc, or a pc you are trying to troubleshoot or maintain. And you can check the log at any time to see exactly when some process was updated or run for the first time.
It could be useful for people in identifying when a malware hit them, or what new program someone may have installed on a pc that they shouldn't have, etc.
Whether other people think this might be a useful tool, I'm not sure...
Later I may consider adding more functionality to the app for not just logging the first time it sees a new/changed process, but other things like you mention -- reports when apps use certain amounts of memory or cpu. Or even a more elaborate process use tracking feature, which keeps track of how much time you have spent running each process (or how much cumulative cpu use processes have used over long times).
I'm focusing on keeping it as lean, fast, and resource friendly as possible, compiled in visual studio c++ with a 64bit build and the resident app having no gui, so it should not use noticable resources.
-
Ha, sounds interesting.
Most difficult part would be to organize the logging in such way that it actually makes sense when looking at it, without being overwhelmed with 'all that data' :o
Maybe the memory/cpu use-log should be separated, possibly a logfile per exe (path+filename, possibly reversed, and : and \ replaced by another character?) and include the pid inside the logfile to be able to filter on. (Hint: the LogStash family of products (https://www.elastic.co/solutions/logging), no ties)
-
If anyone wants to try a really early alpha:
- http://donationcoder.com/Software/Mouser/ProcessWatcher/downloads/ProcessWatcher_x64_portable.zip
What to know:
- It's portable
- It's 64-bit build only
- There is no GUI, just run it and you'll see any icon in the tray which you can use to exit.
- 2 files will be created in the directory when you run it (ProcessWatcher_HistoryData.dat, and ProcessWatcher_HistoryLog.txt); the former is the data file it will read and write on each run; the latter is the human-readable log file that is the bread-and-butter of the app. This is the file you will want to look at to see if the program is useful to you.
- You are meant to just leave this app running always. Restart it on reboot. The log file will show you every time it sees a new process for the first time, or any time a process has changed (was updated, etc.).
Remember the real value of this app is in logging the first time it sees a new or changed process. This can be useful if programs launch in the background, and never inform you.
By occasionally checking the ProcessWatcher_HistoryLog.txt you may discover some applications that are occasionally running, performing update checks and similar things in the background.
-
Ahhhh... Looks like after marriage you can read minds too.
I was planning a process lister which will show the process names as they are loaded by the os in a small non focus window. So that I can see which process are loading in background without my knowledge.
Now you just read my mind and made it. There goes my NANY 2019.
Well I will have to re-start again.
Anyway, thanks a lot for the app. It is definitely useful to me :)
Regards,
Anand
-
One thing I should emphasize in case it's not clear -- this program is designed to remember which applications it has seen running before, and only log when it sees a new program running (or when it detects a change to a process that it has seen before). As such, it's most useful as a tool to have running always in the background, so you can see a nice list of every application that has been run on your computer, while still keeping the list manageable because it's not adding log entries every time it sees the same program.
-
So here are some possibilities for where this program could go:
- Have it store data at regular interval on cpu and memory use of every running app. So that you could look back at the last day/week/month and examine your computer activity in, say, 5 minute intervals. I'm not sure how useful this would be. Maybe add some additional tracking of what app had the foreground. Add the ability to show some basic statistics about what how much you are using each application..Maybe useful for people interested in tracking how much time they use different applications.
- Do #1 above but also incorporate some automatic screenshotting.. Basically merge in functionality of my Automatic Screenshotter (https://www.donationcoder.com/software/mouser/popular-apps/automatic-screenshotter) application.
- Add-in alert functionality, to trigger alerts when certain applications use more cpu or memory than some threshhold, much like my Process Piglet (https://www.donationcoder.com/software/mouser/other-windows-apps/process-piglet) application.
- Add in priority manipulation and affinity control, much like my Process Tamer (https://www.donationcoder.com/software/mouser/popular-apps/process-tamer) application.
-
I think making those options would be a good thing, i.e. where to store, how often to take snapshots, and max size of file. Also, I think that using a standard type log for this would make it useful, i.e. csv or tab-separated so that the data could be processed easily. I like where it's going though.
-
I notice that HistoryLog.txt starts at the earliest entry and newer entries are appended. Would it be feasible to (have an option to) reverse that, so that latest entries appear first?
-
Would it be feasible to (have an option to) reverse that, so that latest entries appear first?
No, because appending to the file is fast and reliable and the opposite is not true.
But I think the real solution here is eventually a nice (standalone) gui tool for more nicely viewing the contents of the file in a proper grid or something. That will wait until I figure out the final kind of data to be saved. But yeah, the solution to more pleasant viewing of the data gathered is a gui tool for it.
-
Also, I think that using a standard type log for this would make it useful, i.e. csv or tab-separated so that the data could be processed easily.
You are right, I'm open to suggestions.. If you look at the format I'm using now, it's basically JSON (though not 100% valid json since I don't properly escape the strings). I do think I would like to save in a format that would make it easier for someone else write a GUI tool to browse history.
But first I have to figure out what kind of information I will be logging. Right now both of these data files are just logging the occasional process entry so it's not that important. But if I start to log performance data constantly for every running process then it will be important to use a data format that is compact.
-
Well, about that file-format: I was going to ask you to use real JSON, as the current content isn't quite readable using standard tools/libraries, but also seeing your remark about compact/heaps of data, I'd suggest to switch to CSV, as that's usually the most compact and usable for non-complex data like this (unless you want to put the performance/usage data inside the JSON object ofcourse...).
Continuing on the subject of including performance/usage data: Using (structured) JSON storage would require you to rewrite the entire file when updating a single performance item, so the policy of 'only appending' isn't going to work.
I've suggested (reply #3) to create a file-per-exe for the performance/usage data, and when using CSV that would be a feasible and fast solution, including append-only. It will generate a lot of files though, so a configurable log directory would be nice to have.
I might even be tempted into creating a GUI for this data, but you'd have to have the storage format finalized first.
-
There are 2 kinds of files that this program will be writing -- those where speed and space is important and those where it is not.
These files are not speed/space sensitive because they only get written out occasionally.. So JSON is a reasonable choice. I'll bit the bullet and make them fully json compliant.
Then there may be some files that are sensitive to speed and size, like cpu and memory use files which are updated very frequently. For those I'll have to decide on some other format, or even use a proper database.
-
That sounds like a very solid path :Thmbsup:
-
Looks like the path do not have the option I was dreaming of.
Should I go ahead and make one as I envisioned for NANY 2019 ?
Regards,
Anand
-
Can you explain what feature you were interested in?
-
Can you explain what feature you were interested in?
-mouser
I was planning a process lister,
which will show the process names, time etc.,
as they are loaded by the os,
in a small non-focus window.
So that I can see, which process are loading in background without my knowledge.
Regards,
Anand
-
I was planning a process lister,
which will show the process names, time etc.,
as they are loaded by the os,
in a small non-focus window.
-anandcoral
I think this should be one of the tasks of a GUI on Process Watcher logs.
Maybe you could create that kind of a GUI? (I might start it, depending on available time and motivation)
Another GUI tool/task would be for filtering/searching the log-data, and that could even be the same tool, AFAICS.
-
Another GUI tool/task would be for filtering/searching the log-data, and that could even be the same tool, AFAICS.
-Ath
This is what I use for that on my logs: https://www.logfusion.ca/
-
Hm, though LogFusion looks like a fine log viewer, I'm afraid it doesn't handle the json-like format that is currently output by ProcessWatcher. We'll have to wait for mouser to choose another format before LogFusion is useful in this context.
-
Hm, though LogFusion looks like a fine log viewer, I'm afraid it doesn't handle the json-like format that is currently output by ProcessWatcher. We'll have to wait for mouser to choose another format before LogFusion is useful in this context.
-Ath
Or put up a request to convert file. If you append me some small examples (before/after) i write one for you.
Until now i did not have time to test this application at all. LogFusion is also unknown to me.
-
The file format I will use is not decided yet, so I advise no one spend any time coding anything yet..
Even more importantly, the log file produced currently is just the log of processes seen, and is not the main log file that will be of any interest.. That is still to come.
-
Hm, though LogFusion looks like a fine log viewer, I'm afraid it doesn't handle the json-like format that is currently output by ProcessWatcher. We'll have to wait for mouser to choose another format before LogFusion is useful in this context.
-Ath
Haven't tried it with JSON, but as what's there is only json-like, was waiting until he decided what that format might be in order to test. Just showing as a suggestion for what I use for log files in general.
-
It HAS been interesting checking out the log of processes that are seen by the application. Unlike a real-time log, you actually notice some surprises in this one -- of processes that run only occasionally in the background. Support tools, update checkers, etc. for other programs, etc.
-
The file format I will use is not decided yet, so I advise no one spend any time coding anything yet..
-mouser
was waiting until he decided what that format might be in order to test.
-wraith808
And that is why I'm waiting too... :) :up:
Or put up a request to convert file.
-KodeZwerg
The only reason to convert a logfile is just before putting it into something like LogStash, syslog, or another log agregator. Converting just for display is a last resort only, so no, I shouldn't need (to write) me a converter ;)
-
Would it be possible to have a second version that does not include the Microsoft listed files?
Or maybe allow us to create a white list of files to ignore in writing out the log entry. This way if we are trying to track down a specific file we can eliminate ones we already know about.
-
I may use Protocol Buffers as the data file format. See here: https://www.donationcoder.com/forum/index.php?topic=46840
-
I may use Protocol Buffers as the data file format. See here: https://www.donationcoder.com/forum/index.php?topic=46840
-mouser
I'd shy away from that for this. You want them to be human readable, right? That's been the deciding factor in a lot of cases...
-
If you don't need or care about data being human readable, try: EDIFACT :P
-
New version uploaded with proper installer (and portable version), and fully functioning right-click tray menu to quickly bring up the log file.
-
I was planning to add a lot more functionality, but a kitchen renovation went horribly wrong and i've spent the last month cursing my life. Today what should have been a simple kitchen cabinet repainting job shit the bed and we're back to square -5.
-
Another inconvenience.. Because I wrote this is plain old visual studio for maximum efficiency, there is no normal fancy GUI for options like I normally write.. In fact in the current incarnation there is only one option, whether to start with windows or not.. It's on by default but you can change that setting by editing the ProcessWatcher.ini file and changing the StartWithWindows setting to 0.
-
I was planning to add a lot more functionality, but a kitchen renovation went horribly wrong and i've spent the last month cursing my life. Today what should have been a simple kitchen cabinet repainting job shit the bed and we're back to square -5.
-mouser
Welcome to Husbands Club.
We all go through this patch in the beginning years then we get habituated :)
Regards,
Anand
-
Avast seems to terminate the process and won't let the installer run, insisting I submit it to them for analysis. I have not done so yet, but just a heads up.
[ You are not allowed to view attachments ]
I have never, EVER, seen such behavior from it before...
EDIT: It seems to have sent something without my permission, but it does indeed give me an opportunity! I have installed the software, disabling the shields, and reactivated the software, seeing what, if anything, avast does to it, and what, if anything, avast updates with to let them know the file's signature is safe or not.
-
How strange!
Can you try the portable version and see if it complains about that one?
ps.
If anyone wants me to build a 32-bit build, just let me know.
-
[ You are not allowed to view attachments ]
I am forever referring to programmers as "Rare File Architects" henceforth... okay, maybe not...
Avast informed me the installer version was just fine though!
-
It would seem useful to have a pop up bubble or something when new and/or update processes happen and entries were added to the log. Otherwise are people supposed to look at this after they discover something going awry? Or after they add/update something to see what it did. ie. proactive vs reactive
[update]
Perhaps something like the options ... alarms/alerts section in Process Piglet that I just started trying.
[/update]
-
It would seem useful to have a pop up bubble or something when new and/or update processes happen and entries were added to the log.
agreed, will be added soon.
-
Hello,
Is there any way to read the log file in a table form like in excel?
the log file is showing
"NEW PROCESS" : {
"File" : "C:\Windows\System32\svchost.exe",
"ID" : "1",
"Size" : "47.55kb",
"Discovered" : "Nov 27, 2019 - 12:02 PM"
what im thinking is
|Status | File | ID | Size | Discovered |
|NEW PROCESS|C:\Windows\System32\svchost.exe| 1 |47.55kb| Nov 27, 2019 - 12:02 PM|
Thank you!