Welcome Guest.   Make a donation to an author on the site July 23, 2014, 05:02:09 AM  *

Please login or register.
Or did you miss your validation email?


Login with username and password (forgot your password?)
Why not become a lifetime supporting member of the site with a one-time donation of any amount? Your donation entitles you to a ton of additional benefits, including access to exclusive discounts and downloads, the ability to enter monthly free software drawings, and a single non-expiring license key for all of our programs.


You must sign up here before you can post and access some areas of the site. Registration is totally free and confidential.
 
Your Support Funds this Site: View the Supporter Yearbook.
   
   Forum Home   Thread Marks Chat! Downloads Search Login Register  
Pages: [1]   Go Down
  Reply  |  New Topic  |  Print  
Author Topic: DONE: Search a folder of text files for an expression, copy the line of text  (Read 3931 times)
ohfine
Charter Member
***
Posts: 5

View Profile Give some DonationCredits to this forum member
« on: June 13, 2011, 01:53:33 PM »

I am monitoring my internet connection (cable TV, phone, and internet) due to previous issues with the service frequently dropping off line.

I am using "EasyNetMonitor" which sends out a ping to Google.com every minute and then makes a one line log entry of “online” or “lost connection”. Each day has it's own text log file in the "Log" folder. This gives me an idea of how many times the service drops out and something to show the cable provider.

I have been copying each text log file, importing it into a spreadsheet, sorting for "lost connection", copy the results to another tab, delete the old tab, and then format the cell backgrounds based on date using alternating colors. The end result is monthly columns showing each lost connection with the days set off by alternating colors.

It works, but I was hoping somebody could help me develop a better way to get the job done. I am not a coder and only know enough about spreadsheets, macros, and batch files to get into trouble.

Any assistance would be appreciated.
Logged
skwire
Moderator
*****
Posts: 4,015



Another Coding Snack request? Om nom nom...

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #1 on: June 13, 2011, 02:53:26 PM »

Can you provide some samples of your log files and the resulting spreadsheets you create?
Logged

Ath
Supporting Member
**
Posts: 2,179



see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #2 on: June 14, 2011, 01:25:44 AM »

You could search google for 'logfile analyzer' (as I did), most top 10 results are free/open source tools with lots of features and options. I'd give one of them a go.
Logged

AndyM
Charter Member
***
Posts: 615


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #3 on: June 14, 2011, 10:44:15 AM »

from a command prompt, the Find command will do what you ask

find "expression" Filename

It will output each line that includes 'expression'.  Standard output is to the monitor, but you can redirect the output to a file:

find "lost connection" LogfileName > OutputFilename

The Output text file will be a list of lines, each one including 'lost connection'.  You can then load this file into a spreadsheet and add your colors (conditional formatting perhaps, odd dates one color, even dates another?)
« Last Edit: June 14, 2011, 10:48:29 AM by AndyM » Logged
MilesAhead
Member
**
Posts: 4,681



View Profile WWW Give some DonationCredits to this forum member
« Reply #4 on: June 14, 2011, 01:47:01 PM »

Also I'd look at some of the Linux utilities ported to Windows. Usually there are many more built in features than similar Windows command line tools. Grep and Find are good examples.  They may take some set up if they need  a bash shell environment.  But as example, find can search for a pattern and execute a command on every match.

Plus they tend to be stable rather than changing behavior with every minor OS version update.

Logged

ohfine
Charter Member
***
Posts: 5

View Profile Give some DonationCredits to this forum member
« Reply #5 on: June 15, 2011, 07:18:46 PM »

There is a text file for each day. So after a month there are 30 text files in the log folder.

Text file:
3/1/2011 19:22:16      www.google.com      Online!      26 msec.  IP:
3/1/2011 19:23:16      www.google.com      Online!      24 msec.  IP:
3/1/2011 19:24:21      www.google.com      Lost connection!      
3/1/2011 19:25:16      www.google.com      Online!      21 msec.  IP:
3/1/2011 19:26:16      www.google.com      Online!      28 msec.  IP:
3/1/2011 19:27:16      www.google.com      Online!      40 msec.  IP:
3/1/2011 19:28:16      www.google.com      Online!      38 msec.  IP:
3/1/2011 19:29:16      www.google.com      Online!      33 msec.  IP:
3/1/2011 19:30:21      www.google.com      Lost connection!      
3/1/2011 19:31:16      www.google.com      Online!      37 msec.  IP:
3/1/2011 19:32:16      www.google.com      Online!      49 msec.  IP:
3/1/2011 19:33:16      www.google.com      Online!      23 msec.  IP:


The end product in the spreadsheet is two columns. Each line is then colored grouping the dates together: all 03-01-11 would be yellow, all 03-02-11 would be blue, all 03-03-11 would be yellow, etc

Spreadsheet:
3/1/2011 21:33:23      Lost connection!
3/1/2011 21:41:23      Lost connection!
3/1/2011 23:14:24      Lost connection!
3/3/2011 16:55:02      Lost connection!
3/3/2011 17:16:02      Lost connection!

My biggest problem is processing each text file one by one. I would like to process multiple text files at the same time. If I can get all the "lost connection" lines into an output file, inserting them into a spreadsheet would not be a problem for me.

'logfile analyzers'
I checked and they appear to be over my head and overkill for what I am trying to do.

Using the Find command at the command prompt sounds like it might work if it would process all the text files in the folder at the same time.

conditional formatting perhaps, odd dates one color, even dates another?
Great idea! I never thought of that. I'll have to learn how to do that.

Thanks to all for taking time to help out.

Could someone point me to the help file on posting screenshots for this forum?


Logged
MilesAhead
Member
**
Posts: 4,681



View Profile WWW Give some DonationCredits to this forum member
« Reply #6 on: June 15, 2011, 09:00:29 PM »

to search all .txt files in a folder would be trivial with Grep.  The only thing you have to watch for with Windows versions of Grep is often they want to make a gui out of it. It's really more effective as a command line utility.

I believe this one is a stand-alone windows binary. I'd take a look at it first:

http://gnuwin32.sourceforge.net/packages/grep.htm

Logged

AndyM
Charter Member
***
Posts: 615


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #7 on: June 15, 2011, 09:30:04 PM »

The Find command will work on multiple files.

At a command prompt, make your Log folder the current directory.

Then type:

     find "Lost connection!" *.* > lc.txt

The lc.txt file would have all the dates from all the files in your Log folder.  Then load the lc.txt file into Excel, which will then have each Lost Connection line in it's own row.  Is it important to delete the "www.google.com" ?

The conditional formatting will be a little tricky because of the way your dates are displayed - not a consistant mm/dd/yyyy, but single and double digits for the month and day.

Edit:  it just occured to me that you might have to put the lc.txt file in a different directory so it isn't processed by the Find command along with the rest of the files (*.*)
« Last Edit: June 15, 2011, 09:32:32 PM by AndyM » Logged
Target
Honorary Member
**
Posts: 1,382



see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #8 on: June 15, 2011, 10:07:32 PM »

this little AHK script should do what you want

[copy or print]
#singleinstance
setbatchlines, -1

path = 20d04fe0-3aea-1069-a2d8-08002b30309d

iniread, PATH, CTN.INI, LOGS,Folder

fileselectfolder, PATH, *%path%

iniwrite, %path%, CTN.INI, logs, folder

msgbox, 4,, Delete the Log Files?

loop, %path%/*.txt
{
    loop, read, %a_loopfilefullpath%
    {
        if a_loopreadline contains Lost connection
        {
            stringleft,f1, a_loopreadline,20
            F1 = %F1%
            stringright, F2, a_loopreadline,22
            f2 = %f2%
            fileappend,%f1%`,%f2%`n, %path%/LogOuts_%a_yyyy%-%a_MM%.csv
        }
    }

    ifmsgbox yes
        filedelete, %a_loopfilefullpath%
}


assumes your log files are TXT files (you can edit this, though you'll need to recompile if you want to run it as an executable), outputs the relevant lines to a CSV file.

prompts for the path the the logs (path is stored in an ini file for future runs)

Option to delete the log files as you go...


* connect_the_nots.ahk (0.65 KB - downloaded 89 times.)
* connect_the_nots.exe (412.03 KB - downloaded 83 times.)
Logged

"Look wise, say nothing, and grunt. Speech was given to conceal thought" - Sir William Osler
cranioscopical
Friend of the Site
Supporting Member
**
Posts: 4,164



see users location on a map View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #9 on: June 16, 2011, 12:45:58 PM »

Quote from: Target
connect_the_nots.ahk (0.65 KB - downloaded 1 times.)

Good name, quite a few good points and never a cross word.
Logged

Chris
Target
Honorary Member
**
Posts: 1,382



see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #10 on: June 16, 2011, 05:48:17 PM »

Quote from: Target
connect_the_nots.ahk (0.65 KB - downloaded 1 times.)

Good name, quite a few good points and never a cross word.

better than a kick in the colon Thmbsup
Logged

"Look wise, say nothing, and grunt. Speech was given to conceal thought" - Sir William Osler
ohfine
Charter Member
***
Posts: 5

View Profile Give some DonationCredits to this forum member
« Reply #11 on: June 16, 2011, 08:17:14 PM »

MilesAhead,

I installed Grep and took a look at it. It is a bit confusing to me, but I am going to keep it and try to learn more about using it.

AndyM,

I followed your example for the command line, output to a different folder, and it worked. I didn't mess it up. It even separated the output by dates!

I don't think I need to put it into a spreadsheet unless I want to try to colorize it. If I do, it would be easy enough for me to do it manually.

Target,

I love the name you came up with! I ran the .exe and the .ini file worked, but I can't find the output .csv file. What did I do wrong?


Be proud people! This is one of the few places where one does not have to be afraid of asking questions or asking for help.

To all of you, Thank You for your time and patience.
Logged
Target
Honorary Member
**
Posts: 1,382



see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #12 on: June 16, 2011, 08:32:41 PM »

I love the name you came up with! I ran the .exe and the .ini file worked, but I can't find the output .csv file. What did I do wrong?

sorry, my bad, I should have mentioned that in my original post - output is to the log file directory.  File naming format is LogOuts_YYYY-MM.CSV

note also that if you run it more than once a month it will just append to any pre-existing output file for that month (unless you've renamed said file...)
Logged

"Look wise, say nothing, and grunt. Speech was given to conceal thought" - Sir William Osler
kyrathaba
N.A.N.Y. Organizer
Honorary Member
**
Posts: 3,008



while(! dead_horse){beat}

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #13 on: July 28, 2011, 07:30:58 AM »

Looks like this thread could be marked "Finished".  Nice work, Target  thumbs up
Logged

Win 7 Home Premium 64bit-SP1 AMD Athlon II X2 220 Socket AM3 (938) @ 2.1GHz 6GB RAM Firefox 26.0
_________________________________________________________________________________________

I'm fighting against patent trolls. Join me and tell your representative to support the #SHIELDAct: https://eff.org/r.b6JJ /via @EFF

My DC page: http://kyrathaba.dcmembers.com | My blog: http://williambryanmiller.com/ | Proofreading Service: http://bit.ly/1fQSqQP

Pages: [1]   Go Up
  Reply  |  New Topic  |  Print  
 
Jump to:  
   Forum Home   Thread Marks Chat! Downloads Search Login Register  

DonationCoder.com | About Us
DonationCoder.com Forum | Powered by SMF
[ Page time: 0.041s | Server load: 0.01 ]