delete all the lines that do NOT start with a specific string in multiple files

[kalos]

hello

I have an amount of text files and I want to delete all the lines that do NOT start with a specific string

how can I do this?

thanks

EDIT: the program must support unicode characters

[housetier]

on linux a combination of grep and the shell's for loop could accomplish this:

[Select]

export BAN="whatever should not be kept"
mkdir after
for textfile in *.txt; do grep $textfile -v ^$BAN > after/$textfile; done

This should work if the files all have a .txt extension.

"grep -v" prints lines that do NOT match; "^" means beginning-of-line; "$BAN" will be replaced by whatever you put between the quotation marks in the export statement above. The output from grep is then put into a new file with the same filename, but in a different directory.

If you have cygwin, you can do this under windows as well.

[bgd77]

kalos, could you specify if this is for Windows or Linux?

[TucknDar]

edit:
completely misunderstood... sorry

[kalos]

it's for winxp
thanks

[Ehtyar]

Well, I now despise Windows for having such a totally sucky console, but I managed to get this working.

As an exercise for myself I compiled GNU grep myself. It required a couple of patched but all-in-all relatively simple. If you're not capable of this yourself, you can get an only slightly outdated binary from UnxUtils (I've included my build in the attached zip). Converting the shell script to batch was a huuuge pain in the ass because (again) the Windows console sucks, and doesn't honor piping symbols in a for loop in the same way it would as a standalone command (you have to enclose your command in parens prior to the >, < or | symbol). God help you trying to find that in Google (I ended up randomly guessing it). Finally, grep on Windows converts CRLF line endings to plain LF (for internal operational reasons), so we need the todos utility to convert them back again.

So here you are, give it a try and let me know how it goes.

Ehtyar.

[edit]
Er...should this not be moved to the coding snack new request forum?
[/edit]

[edit2]

If you have cygwin, you can do this under windows as well.

-housetier (March 14, 2009, 12:30 PM)

MSYS will manage it also, minus the enormity of Cygwin.
[/edit2]

[bgd77]

I think that the only way to do it is how Ehtyar suggested. i'm no expert in Windows Command Prompt, but I do not think that it is powerful enough to make changes in files. If someone knows better, please correct me!

One alternative would be to install ActivePerl or Python and to write a small program to do this, if you think that all the trouble is worthwhile.

[Ehtyar]

*sigh*, life would indeed be easier if Windows came with a posix-compatible shell or Strawberry Perl, but alas, here I am passing around utilities that others would've had on almost any other operating system. I think, however, that this is a debate for another day.

kalos, would you let us know how it goes please?

Ehtyar.

[rjbull]

Converting the shell script to batch was a huuuge pain in the ass because (again) the Windows console sucks, and doesn't honor piping symbols in a for loop in the same way it would as a standalone command

-Ehtyar (March 14, 2009, 05:44 PM)

Ehtyar,

Take a look at Horst Schaeffer's List MODifier LMOD, which is a great workaround for making batch files to get round these kind of problems.

[Ehtyar]

I do like the sound of that utility rjbull, thanks for recommending it. However, the problem is no longer a problem, as I found the solution. I was complaining about the lack of documentation regarding the Windows console. Perhaps my searching skills failed me, but if you have to resort to guessing the syntax of any language, the documentation is in pretty poor shape.

Ehtyar.

[David1904]

TextHarvest is useful for keeping or deleting lines from text files. It is very simple to use.
You can find it on Tucows or other download sites.

[TheQwerty]

I believe since Windows NT there's been a command, findstr, which acts similar to grep, but I'm not sure about the support for Unicode.

You should be able to do something like this:

[Select]

findstr /V "^needle" input.txt>output.txt

The /V tells it to print the lines that do not match the regular expression "^needle" (any line beginning with "needle").

[bgd77]

I've tried it and searched some forums. findstr does not support unicode. 

So, find must be used, and it seems to be really complicated to use it for this purpose.

[bgd77]

I think that this can be easily done with Windows Power Shell. Does anyone know more things about it?

[app103]

Could probably do it with WSH and javascript too.

[Krishean]

i could do this in javascript/wsh, i just need to know:

1. how large will the files be
2. how much memory are you willing to let the script use (more memory = faster, less = slower)