topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • September 23, 2018, 06:49 AM
  • Proudly celebrating 13 years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: Big Data tools  (Read 239 times)

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,671
    • View Profile
    • Donate to Member
Big Data tools
« on: September 14, 2018, 09:47 AM »
Hello!
Is there a 'free for commercial use' software that opens and processes with regex large text files of 20GB?
Thanks!

Ath

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 3,251
    • View Profile
    • Donate to Member
Re: Big Data tools
« Reply #1 on: September 14, 2018, 10:44 AM »
Is there a 'free for commercial use' software that opens and processes with regex large text files of 20GB?

TL;DR; Yes.

Suggestions:
  • Any Linux distro with a non-commercial license (most will fit)
  • Linux-tools for Windows (if it's supposed to run on Windows, you didn't say it should)
In any of these environments use tools like grep, sed, awk, perl, python etc. for text processing.

  • 'Plain' Windows (7 and newer)[/i]
Use powershell, like shown & explained in your other thread.

Shades

  • Member
  • Joined in 2006
  • **
  • Posts: 2,372
    • View Profile
    • Donate to Member
Re: Big Data tools
« Reply #2 on: September 14, 2018, 11:02 AM »
RJ TextEd

Is freeware, multi-lingual and the website says it is allowed to be used in commercial environments too. Donations are accepted. This text editor has support for regex and I tested (previous versions of) the software with 2GByte log files and search/replace actions in those files were fast on a 7 year old PC (16GB RAM, i5 CPU (3th generation, 3.5GHz)).

Caveats:
- Regex I haven't played with, so how well/functional the support is I can't tell.
- Haven't tested with log files bigger than 2GByte, so if 20GByte log files can be processed, I can't tell.

Ath

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 3,251
    • View Profile
    • Donate to Member
Re: Big Data tools
« Reply #3 on: September 14, 2018, 12:02 PM »
Haven't tested with log files bigger than 2GByte, so if 20GByte log files can be processed, I can't tell.
Well, I did and it wasn't nice,
Created a large file using this method
Installed RJ TextEd and tried to open it, with this result:

Screenshot - 14-09-2018 , 19_00_32.png

kalos

  • Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 1,671
    • View Profile
    • Donate to Member
Re: Big Data tools
« Reply #4 on: September 14, 2018, 12:36 PM »
Is there a 'free for commercial use' software that opens and processes with regex large text files of 20GB?

TL;DR; Yes.

Suggestions:
  • Any Linux distro with a non-commercial license (most will fit)
  • Linux-tools for Windows (if it's supposed to run on Windows, you didn't say it should)
In any of these environments use tools like grep, sed, awk, perl, python etc. for text processing.

  • 'Plain' Windows (7 and newer)[/i]
Use powershell, like shown & explained in your other thread.


Thanks but I want to also be able to view the content, like EmEditor

Ath

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 3,251
    • View Profile
    • Donate to Member
Re: Big Data tools
« Reply #5 on: September 14, 2018, 12:59 PM »
I want to also be able to view the content,

Well, you could just buy EmEditor, it's not that expensive, or EditPad Pro, that does also handle really big files with ease.

For a freebee, check out the really old, and a bit harsh to download, MadEdit, it opened a > 20GB file without a hitch, and supports regex search and replace.



Like I suggested above, and me and others before, it would really get you more helpful answers if you specify what you want, need and desire complete instead of letting us pull those details with pliers, in the dark >:(

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 9,711
    • View Profile
    • Donate to Member
Re: Big Data tools
« Reply #6 on: September 14, 2018, 01:31 PM »
https://stackoverflo...uge-large-text-files

You can try those and see if any satisfy your need.

From those, my personal recommendation is EmEditor, 010, or LogExpert.

https://github.com/zarunbal/LogExpert

I don't know what processes means, however in that context.  And on this Ath is correct.  You want better recommendations, give better requirements.